CURSOR-BASED ADAPTIVE QUANTIZATION FOR DEEP NEURAL NETWORK
Authors: Bapu Li(*)
Yanwen Fan(*)
Zhiyu Cheng
Yingze Bao (* means equal contribution)
Venue:  (2020)

Abstract: Deep neural network (DNN) has rapidly found many applications in different scenarios.
However, its large computational cost and memory consumption are barriers
to computing restrained applications. DNN model quantization is a widely
used method to reduce the DNN storage and computation burden by decreasing
the bit width. In this paper, we propose a novel cursor based adaptive quantization
method using differentiable architecture search (DAS). The multiple bits’
quantization mechanism is formulated as a DAS process with a continuous cursor
that represents the possible quantization bit. The cursor-based DAS adaptively
searches for the desired quantization bit for each layer. The DAS process can
be solved via an alternative approximate optimization process, which is designed
for mixed quantization scheme of a DNN model. We further devise a new loss
function in the search process to simultaneously optimize accuracy and parameter
size of the model. In the quantization step, based on a new strategy, the closest
two integers to the cursor are adopted as the bits to quantize the DNN together to
reduce the quantization noise and avoid the local convergence problem. Comprehensive
experiments on benchmark datasets show that our cursor based adaptive
quantization approach achieves the new state-of-the-art for multiple bits’ quantization
and can efficiently obtain lower size model with comparable or even better
classification accuracy.

Under review as a conference paper at ICLR 2020
CURSOR-BASED ADAPTIVE QUANTIZATION FOR DEEP NEURAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural network (DNN) has rapidly found many applications in different sce- narios. However, its large computational cost and memory consumption are bar- riers to computing restrained applications. DNN model quantization widely used method to reduce the DNN storage and computation burden by decreasing the bit width. In this paper, we propose a novel cursor based adaptive quanti- zation method using differentiable architecture search (DAS). The multiple bits' quantization mechanism is formulated as a DAS process with a continuous cursor that represents the possible quantization bit. The cursor-based DAS adaptively searches for the desired quantization bit for each layer. The DAS process can be solved via alternative approximate optimization process, which is designed for mixed quantization scheme of a DNN model. We further devise a new loss function in the search process to simultaneously optimize accuracy and parameter size of the model. In the quantization step, based on a new strategy, the closest two integers to the cursor are adopted as the bits quantize the DNN together reduce the quantization noise and avoid the local convergence problem. Compre- hensive experiments on benchmark datasets show that our cursor based adaptive quantization approach can efficiently obtain lower size model with comparable or even better classification accuracy.
INTRODUCTION
Deep learning (DL.) has achieved great successes in varied fields such as gaming, natural language processing, speech recognition, computer vision and on. However, its huge computational burden and large memory consumption still intimidate many potential applications, especially for mobile devices and embedded systems.
number of efforts have been devoted to compress the DL model size and accelerate its training and test speed. These efforts can be roughly categorized into four major classes: network pruning
(Han et al. (2015); Anwar et al. (2015); 5 Peng et al. (201)) /huang et al.) (2018). low rank approxi- mation (Tai et al. (2015) Wang al. (2018) Hayashi al. (2019)). knowledge distillation (Hinton ct al. (2015); Zagoruyko & Komodakis (201))) and network quantization (Courbariaux & Bengio (2016) Lin et al. (2015): Wu et al. 2015 : Polino al. 2018): Zhang et (2018)). Among them, network quantization methods, joinly optimizing the whole network weights, activations or gradi- ents with low bit (such as 8 bits or even bit), show great potential in compressing model size and accelerating inference time. In addition, quantization based approaches are preferable for mobile devices and embedded systems since these devices are gradually equipped by specifically designed low bit computing hardware.
Although existing quantization based approaches, which mainly use one fixed bit to represent the whole DNN model, yields encouraging compression ratio while keeping the model's performance, we argue that simply using only fixed bit for quantization is not the optimal choice for the trade-off' between model size and its performance. For example, run model on chips with strict memory limitations, bit or 2 bits' quantization suffers from severe accuracy loss (Rastegari et al. (2016))
while 16 bits' or 8 bits' quantization cannot significantly reduce the model size.
To address the above problem, we propose a cursor based adaptive quantization method to derive a different number of bits in different layers for DNN model compression, i.e., search for the best

---

Under review as a conference paper at ICLR 2020
configuration of different bit quantization for different layers in neural network model. Distine- tive from most algorithms aforementioned, our approach is motivated by recent neural architecture search (NAS) that aims to find better performance neural architecture with less calculations less size automatically. The key in our algorithm is using a continuous cursor that represents the bit quantization scheme for each layer. For different layers, many cursors will be adaptively searched during the NAS process. Since the cursor itself is continuous and the whole search procedure can be considered as a differentiable architecture search (DAS) process, which can be effectively solved based on an alternative optimization strategy. A novel cost function that considers the model com- pression and prediction accuracy is also proposed in the DAS process. In the searching process of the cursors, quantization process is applied to compress the model size at the same time. To re- duce the possible quantization noise and local convergence problem, we make use the closest two integer bits to the cursor to quantize the weights for each layer in a DNN model. We validate our proposed method with image classification tasks on CIFAR IO,,,,,II R.O,CIFARION 100 and ImageNet. Compre- hensive experiments MobileNetV2 show that the proposed cursor based quantization method achieves remarkably better performance of compression ratio with ignorable accuracy drop even better accuracy.
some backbone DNN models such as ResNet20, ResNet18, ResNet56 and
In summary, the contributions of this work are four-fold:
We cast the adaptive quantization of neural network as problem of neural architecture search. A continuous cursor is proposed to represent the possible quantization bit, leading to a more efficient search space.
A novel regularization function is proposed to optimize model compression in the search process. Thus the search for the cursor position and weights can be efficiently solved in an alternative optimization manner.
Two nearest neighbor integers to the cursor are adopted with a carefully designed strategy to implement the quantization of the network to reduce the quantization noise and avoid possible local convergence.
We comprchensively evaluate the proposed adaptive quantization method on some bench- mark datasets and achieve new state-of-the performance for different number of bits quantization of neural network.
2 RELATED WORK
Recently, a lot of new quantization approaches have been proposed, enabling the quantized com- pressed model to compete with their full precision counterparts. Clustering method is applied for the weight codebook representation (Han et al. (2015)). and then the network is retrained to get better quantized centroids. In (Zhang et al. (2018). 8)), the authors jointly trained DNN and its asso- ciated quantizes to reduce the noticeable predication accuracy gap between the quantized model and its full precision one. A direct differentiable quantization method was introduced in (Louizos et al. (2018) with promising test accuracy. A new activation quantization method that takes an activation clipping parameter was proposed in (Choi al. (2018) to ensure the suitable quantization scale.
Some efforts have been taken on quantization of the neural network with a different number of bits used for different layers. In (Lin al.) (2015)). signallto-qqaanizzaionn-ooiiei
ratio (SQNR) is applied on layer weight to evaluate the effects of quantization error. Based on SQNR, different bits were used for quantization of each layer, yielding about 20% model size reduction without accuracy loss in their tests. The authors of (Wang et al. (2018)) proposed an automated mixed precision quantization scheme based on reinforeement learning (RL) to achieve better latency on different hardware platforms such as edge devices and cloud data center. They also claimed that their actor- critic model produced efficient actions that result in better lateney and less energy consumption with negligible loss of accuracy.
In the past few years, a new trend has been witnessed for network design, i.e., neural architecture search (NAS). RL based approaches are first utilized to generate network with high accuracy (Zoph)
& Le (006)). ), and they also build a strong basis for the following recent works such as (Gao
let al. 22019; ; Guo et al. (2018)). Then, evolution based approach (Liang et al. (2018) is further
applicd to obtain the possible optimal solution in the large search space. Both of these two category
approaches tend to yield large amount of computational burden because NAS is treated blackbox
2

---

Under review as a conference paper at ICLR 2020
optimization problem in a discrete domain, yielding large number of architecture evaluations, and thus run very slow even on the most advanced GPU machine. To alleviate this problem, in 2018, the authors (Liu et al. (2018) proposed differentiable approach to accelerate the search of good neural network by relaxation of the possible operation on the cell level structure. Wu et al. recently proposed new approach find the mixed bits for different layers by applying differentiable NAS (DNAS) method based on a model of super net (Wu et al. (2018), which is kind of directed acyelic graph. They considered the quantization as a problem of sampling on a stochastic super net, where a Gumbel softmax function is applied to make the sampling process differentiable.
We cast the different bit quantization for DNN as a cursor based adaptive architecture search prob- lem, and it is different from the traditional direct quantization works and the learning based mixed bits' quantization approaches. Moreover, is also distinctive from DART' and DNAS in the method- ology itself.
3
BURSOR-BASEL ADAPTIVE QUANTIZATION
3.1 NEURAL ARCHITECTURE SEARCH
is well known that DNN model needs much time to design its structure. Neural architecture search (NAS) recently emerged as a new methodology to overcome this challenge. It designs the optimal architecture of neural network by considering all possible factors such number of layers, width of each layer, different operators in each layer and so on. Two key concepts are related to a NAS process, i.e., search space and search strategy. All the possible combinations of the major factors that determine a network structure constitute the search space. Generally speaking, the search space of DNN is very large, leading huge computational task. As such, the previous NAS works first devise normal and reduction cell (Pham et al. )018). Such kind of motif is then repeated to build the final neural network. Another definition is about search strategy, that is, how to transverse in such large search space. With each searched network structure, the performance of it will be evaluated. A typical search method is random search, however, its computational efficiency is not ideal. Therefore, most recent works (Cai et al. (2018); Liu et al.] (2018) have been proposed along this big direction to improve the search efficiency as much as possible.
3.2 SEARCH SPACE FOR QUANTIZATION PROBLEM
Quantization has also been a very hot research topic in the past few years. Rounding function, vector quantization or stochastic function are typically applied to implement quantization to compact the model size while maintaining equivalent performance acceptable loss. Some other approaches also use stochastic or probabilistic methods to quantize the neural network. Most previous methods simply apply one kind of bit quantization to the whole network due to the simplicity of implemen- tation. A few recent works begin to utilize different bit quantization scheme to further improve the compression ratio and prediction accuracy. If we consider quantization choice as a part of the neural architecture, we can estimate its corresponding search space. Let us take Resent20 as an example and if we decide to quantize the neural network with the possible bit width of 1, 2, 4, 8, 16, 32, then all the possible quantization choices for ResNet20 will be 620. In the context of NAS, this is very large number for the search space. Hence, evaluation of so many designs one by one seems infeasible right now.
3.3 DIFFERENTIABLE CURSOR SEARCH FOR ADAPTIVE QUANTIZATION
The discrete search space of the above quantization scheme so large. If we further consider the possible bit choice for each layer as a virtual continuous cursor in the range of [2, the cursors then become significant parts of the architecture for a neural network model. Here, we define the cursor as a position that is related to the quantization choice for each layer. Its value is a floating- point number within [2]. If we assume a DNN has N layers, and each layer has a different cursor, denoted by C2, ..., CN together with their weights of Wc. our goal is to find a good combination of C12 .... CN to achieve better prediction accuracy and compression ratio. As such, for the whole neural network it be described as an optimization problem that minimizes the loss on the validation data after training through the minimization of the loss on the training data as follows (Liu et al. (2018))
3

---

Under review as a conference paper at ICLR 2020
min Mim, ((
s.t. We. = =rgminEmm y)) (Lossv (C, Wc))
(I)
where C' represents the cursor vector, Wo- is the weights corresponding to the optimal C, Lossr(C, Wc) and Lossv(C. Wc) is the respective training and validation loss function based on the cursors and weights with condition of C. Dr and Dy represents the training and valida- tion dataset respectively, (I, y) and y²) means data from the training and validation dataset. It should be noted that using training and validation data is a tradition to derive the weight parameters and architecture parameters in the field of NAS, which is a little bit different from the other prob- lems in deep learning. For simplicity and efficiency, in this paper, we let LossT Wc) equals to Lossv(C (C, Wc) and assume they share the same form of Loss(C, Wc). To consider both the prediction accuracy and model size, we design the loss function as a combination of cross entropy and model size as follows:
Loss(C, (C) = CrossEEntropyCC, We) + x Lossc
(2)
where CrossEntropy(C. Wc) is the widely used cross entropy function, encoding the prediction accuracy of the model. The reason why we add a regularization item to the loss function is first because it may determines the model's performance compromise hetween the accuracy and quan- tization, which directly determines the model size, and À is a regularization coefficient that adjusts the trade-off of accuracy and compression. In addition, it may prevent overfitting to some extent. Concerning the loss related to Lossc, we focus on the model size change with quantization. So we design it in the form of Eq. B). which will be introduced in details in the next subsection.
The above process is a bi-level optimization problem, which requires to deduce higher order deriva- tives and is hard to obtain an exact solution. An approximated iterative solution can be applied instead, so we alternatively take the optimization strategy in weight and cursor space to update We based on the training losses from Dr and renew C²' based on the validation losses from Dv. By solv- ing this bi-level optimization problem using an alternative approximation approach, the cursors can be efficiently searched by gradient based optimization approach such as Adam. Our later experimen- tal results also show that this alternative optimization method may yield a good solution with high compression ratio and accuracy. Compared to the original discrete search space, this search method is more efficient because the design of continuous cursor and the direct gradient based optimization approach. The whole differentiable cursor search for adaptive quantization based on the alternative optimization of Wc and C' is described in Algorithm| I With the training and validation set, initial- ized cursor value and pretrained model as inputs, our algorithm first quantizes the weights in each layer of the network using the two most close integers to the cursor and calculates the loss based on the training data for forward process, and then it updates the original 32 bit weight by gradient descent algorithm. For the subsequent validation, it also first quantizes the network with the two most close integers to the cursor, and use them to obtain the validation error, then our algorithm utilizes this error to update the cursors with gradient descend algorithm. It should be emphasized that we utilize the original 32 bit weight in the above training and validation step, and use it for sharing when implementing quantization with the cursor's two neighbor integers. The key step in the algorithm about quantization with the nearest two integers to the cursor will be elaborated in the subsequent section. The outputs of the whole algorithm are rounded cursor values for each layer and its quantized network model.
It should be noted that our proposed cursor based differentiable search is different from DARTs (Liu et al. (018)) in the following three aspects. First, DARTs method considers the possible operation each layer as a mixture of primitive operations. Here, we directly make use of cursor to represent the quantization bit for each layer, no similar mixture operation exists in the whole search algorithm. Second, in DARTs, each primitive operation is assigned with a probability through a softmax function. Cursor based search is optimized directly without probability. Third, DARTs approach concentrates on the cell structure, but we apply the DAS directly on the whole network. Compared to DNAS (Wu et al. (2018)), our approach is also distinctive. For DNAS, the authors build a stochastic super net first to describe all the possible quantization choices, then a sampling approach based on Gumbel sofumax function that enables the discrete distribution to be continuous and differentiable is applied in each layer of the super net. Our cursor based differentiable search has no super net or sampling process in the pipeline. Hence, the subsequent solutions to the optimization problem is also completely different. In short, the proposed method requires no relaxation anymore as in both DARTs and DNAS approach.
4

---

Under review as a conference paper at ICLR 2020
input :
'The training set Dr, validation set Dv, initialized cursors C, pretrained 32-bit weight W, and the batch size
while not reaching the target epochs or not converge do
Sample data from training data Dr;
Quantize the network using two integers that are closest to the cursor, calculate the loss T.
training data with 2):
Update the weight W by gradient descent W = W Vw L(C, W);
Sample data from validation data Dv:
Quantize the network using two integers that are closest to the cursor, calculate the loss L on validation data with Eq. 2
Update the cursor C' by gradient descent C" = - V L(C,W) end
output: Rounded cursor values for each layer and quantized network
Algorithm Differentiable Cursor Search for Adaptive Quantization
3.4 TRAINING FOR NETWORK QUANTIZATION
Aiming for DNN quantization, we attempt to apply the cursor that represents the bit to quantize the weight layers. Unfortunately, the cursor obtained during the search is fractional number, which cannot be directly used for quantization. One choice is to round the cursor to some integers, but it may cause rather large quantization error if choose the rather distant bits. On the other hand, if we directly round the cursor to its nearest integer, it may not efficiently represent the variation of cursor. For example, if cursorl and cursor2 for different epochs in the same layer are 2.6 and 2.8 respectively, they will be rounded to the same integer 3, yielding no change in the model size for this layer when implementing quantization. In addition, in the whole search process, such one integer choice may result in local convergence because the iteration process of one integer quantization may get stuck in a local minimum region for the cursor search. To alleviate the above two problems, we propose instead to make use of the nearest lower and upper integer bound at the same time in the search training process. Compared to directly using the nearest one neighbor to quantize, the lower and upper integer bounds may produce more variations in the loss function that describes the quantization effects, yielding effective gradient changes to update the cursors more efficiently. Subsequent experiments also demonstrate that this design obtain better quantization performance compared to simply applying rounding function on the searched cursor. As such, the loss function for model size part in Eq. ( is designed follows:
Lossc (Cost(C)))
where is postive coefficient, Cost(C) is quantization related continuous cost function with cursor as its variable. In this work and for the convenience of implementation, we further design it as:
N S. S.X. Lossc = 2is
(4)
EN S.
where S, is defined as the size of the 1th layer in bits when each of parameters is represented by bit. In fact, for a trained model, the size of layer in bits, i.e., S: is constant for lay i. Since c; is a continuous cursor, we may consider the above equation differentiable with respect G--
When implementing the quantization for each layer, we utilize the DoReFa-Net (Zhou et al. (016) quantization:
W/k 2Qu( tanh(w)
0.5) -
(5)
2max([tanh(x))
where wo represents the full precision weight of a model and Q+(.) is the k-bit quantization function that transforms continuous value E [0, 1] to k-bit output y E [0, 1] as below:
round((2* - 1) x x)
(6)
where round function is the typical rounding operation used in quantization. In other words, in the process of quantization, after searching the possible quantization bit of e in each layer, its
5

---

Under review as a conference paper at ICLR 2020
corresponding two nearest neighbor integers will be applied to 6q. and Eq.(5) quantize each layer of the network.
In the forward process of the proposed quantization scheme, output of the 8th layer, assuming O can be described with the following equation:
M O, Σ ReLu(1 la --11) x s.t. Σ MeLu(111cs 911) -gl))
where ReLu(x) max(0, x) is the Rectified Linear Unit that zeros out the negative values, is the cursor in one layer, 9 denotes the jth quantization bit width within M total quantization choices, and Oj is the corresponding output of one convolution layer of the model when quantizing with 45- Due to the introduction of Ru((r) in the forward process, the whole loss function may not the point of zero. In such a cornrr case, fortunately. the gradient may be still also estimated through the method of straight-trouugh-ee tbroooogg-sssimaarrSSSE (Guo (2018). whose details are beyond the scope of this paper. In this work, we mainly consider q₁ E 3, 6, 7,8), M = 8 and C E [1, 8), because it has been observed that with greater than 8 bits, the neural network's performance almost has no degradation (Elthakeb et al. (2018)). We also concentrate on such a design in all our experiments. In fact, there is also new trend that investigates the possible quantization with bit that not power of 2 (Elthakeb al. (2018); Park et al. (2018b)a) a): Wang et al. )018)) In addition, some hardware such as FPGA also gradually supports efficient quantization using such bits (Wang et al. (2018); Wei et al. (2019)).
be differetiable
Based on Eq. C) it can be found that there is only two closest integers that take effects in each layer's 's output. As such, our proposed search algorithm can much reduce the coupling of different quantization operations, leading more effective search of good cursor. Assuming the cursor's lower and upper bound integer in the tth layer is 0;1 and 0,2, we can define two coefficients d, and d,2 below:
d, 1 -aal: d22 = 1 la:2 cs]
(8)
where e; represents a cursor searched in the 7th layer, d, and di represents how close the bounds are to cursor. The closer it is, the larger the coefficient is. A continuous move of a cursor then adjusts the weights continuously. Then, based on (7), given the input of one convolution layer, output of it (we only consider the quantization of the convolution layers in this work in the forward process of quantization can be simplified and described with the following equation:
O du x (Cone(X; (X, Wn) + di2 Conu(X, W22)
where Wa and W,2 are the temporary weights in one layer after quantization using 0a and 0,2 based on its corresponding 32 bit weight, Conu is the convolution operation. The 32 bit weight will be updated as the whole algorithm iterates, while Wn and W12 will be recalculated in the forward process based on the new a; and 0,2 In other words, we activate the two closest bits to the cursor, and sum the convolution results of these two quantization bit choices based on the coefficients of the L1 distance. As such, the whole process is differentiable. This can be also intuitively explained by that the outcome of the desired quantization scheme for each layer in the forward process might be represented by weighted sum of the two different quantization schemes using the approximated closest bit choices. Hence, the proposed scheme may find the best quantization scheme for the whole network in the cursor searching process based on the alternative optimization solution.
After the approximate alternative optimization approach converges or reaches the target epoch num- ber, the final quantization bit in each layer can be obtained by applying rounding operation on each cursor for inference. The final quantized model may also need to be finetuned based on the quanti- zation bit.
4 EXPERIMENTS
Currently, we only apply quantization on the weights and use full precision activations. In addition, also follow the traditions in the domain of DNN quantization avoid the quantization of the first and last layer in a model. In all the experiments, we take ResNet18, ResNet20, ResNet56 (He et al.) (015)) or MobileNetV2 (Sandler et al. (2018) as the backbone models. Please be noted that these

---

Under review as a conference paper at ICLR 2020
135-
quamn ioten srrgers
quani Ail two stegers
034-
232
1280000 MB.
0.3
236
(a) Training Loss
(b) Validation Loss
Figure I: Model size loss change for different quantization scheme.
models should be pretrained to obtain the floating point models first. For the initialization of cursors in each layer, all of them are set with bits for the convenience of iteration. When the cursors are obtained by our method, the model may be further fine tuned get its final accuracy, which is practical tradition in the fields of NAS and quantization.
As for the parameter À in Eq (2) and in the loss of model size in (3. rather optimal set of them is chosen as (0.25, 0.3) after trials. Based on our experiments, we also generalize trial rule for A. We control the balance of the two loss parts in Eq. to ensure the ratio of the first and second part between [0.5, 2.0]. We also study the influence of À in the experiments to show that in most cases, the cursor based adaptive quantization scheme is not senstive to its change if is at a larger interval of À > 0.1. Concerning the learning rate schedule of weight and cursor, we apply cosine annealing method to adjust them. The minimum learning rate for them is 0.001 and 0.0001 respectively.
4.1 COMPARISON OF MODEL SIZE LOSS
To show the validity of quantization approach using two integer bounds nearest to the cursor, we first implement the search process by comparing to using only one nearest integer of the cursor. We analyze their model size losses, i.e., (3. to show the great distinction in the training process.
Here we apply ResNet20 on CIFARIO dataset to demonstrate the optimization process. For illus- trative purpose, we only draw the training and validation loss change of the model size, i.e., the second term of Eq. the first 20 epochs. As shown in Figure D the red curve represents the training and validation loss of model size using one nearest integer to implement quantization, while the blue one denotes the training and validation loss of size obtained by using two neighbor integers nearest to the curser searched by the proposed scheme. The major differences in these two tests lic at the quantization choices. In fact, we also tried some other parameters and random initialization for one integer quantization scheme, and similar curves can be found. Obviously, the blue one looks more smooth and natural for convergence process. The red loss may lead to a strong possibility that the cursors are stuck in a local minimum region instead. From this Figure, we can clearly no- tice that the training and validation loss of the model size using only one integer quantization will keep constant immediately in the first epoch, which is NOT desired for cursor search because it will cause no change in the model size anymore. In fact, the cursor values obtained by the one neighbor scheme tend to be bit for all layers. The reason why the one integer quantization scheme fails may be because, in most cases, the weights in one layer span rather small range, one lower integer quantization may lead to the same quantization results on the weights the training process. Such same quantization results further yield almost no change in the backward gradient process, which is not beneficial for the optimal cursor search. The designed two integers' quantization process, on the other hand, can map the cursor to two different integer values, leading to efficient change in the model size even for the weights in rather a small value range. Figure I also shows that Eq. is continuous and differntiable.
7

---

Under review as a conference paper at ICLR 2020
4.2 SEARCH PROCESS ANALYSIS
To get some insights of adaptive cursor search algorithm, investigate its iteration process in this subsection. For illustration only, we take MobileNetV2 on CIFAR1O 10 as an example. Its search process depicted in Figure 2 with the quantization bits igmored due to space limitation. Here the abscissa and vertical coordinate respectively represents the compression ratio and prediction accuracy. should be noted that here our proposed algorithm runs 10 epochs only to clearly show the variation of performance. In addition. because of the cosine annealing scheduler, such an iteration process may also be representative. From Figure 2 search scheme, it first begins at the lower left region (lower accuracy and compression) and then gradually assembles to the upper right region (higher accuracy and compression). Meanwhile, there some small vibrations in the whole process, for example, from epoch 8 to epoch 9, there is some increase in accuracy as well as compression ratio, but from epoch 9 to epoch 10, there slight reduction in both measures. It can also be noticed that the search process is rather stable and gathers to the final upper right region with better accuracy and compression ratio. We also observed similar pattern for ResNet20 and ResNet56 on CIFARIO, but we ignored the picture of them because of space limitation. The reason why the search process of our method can reach to region with high prediction accuracy and compression ratio may be due to the alternative optimization approach to solve this bi-level problem with two goals. In addition, the regularization item may also play positive role this process.
we observe that for the proposed adaptive cursor
90.0
87.5
prediction accuracy
85.0
72.5
12
14
18
20
compression ratio
Adaptive search process for MobileNetV2 on CIFAR10, the number beside the points represent the epoch.
Figure 2:
4.3 IMPACT OF REGULARIZATION COEFFICIENT
The regularization coefficient in Eq. 2 determines the balance between the model precision and size. In this part, we carry out some experiments to analyze the influence of it on the whole perfor- mance. We choose À = 0.9, 0.7, 0.5, 0.25, 0.1. 0.05, 0.01, and we test its effects model. For the purpose of illustration, we test ResNet20 on CIFAR10. To directly show the effects of our cursor based differentiable search, we do NOT implement finetune step for all these results af- ter finishing the cursor search. The results of the quantized ResNet20 on CIFAR10 is demonstrated in Table and all the results are obtained by implementing the search with 200 epochs.
the quantized
8

---

Under review as a conference paper at ICLR 2020
Table Performance of ResNet20 on CIFARIO with different
0.9
0.7
0.5
0.25
0.1
0.05
0.01
Accuracy(%)
90.10
90.14
90.18
91.79
91.03
91.16
91.58
Compression ratio
30.37
30.00
30.09
28.33
28.43
19.46
13.95
From T we observe that forλ >= 0.1, the whole performance of the proposed quantization method is rather steady, that is, the accuracy and compression ratio of the quantized model maintain concentrated region with the accuracy about 90% while the compression ratio about 29.00. When À < 0.1, the cursor based adaptive quantization approach may still have good performance of prediction but gradually loses its effects on model compression. This can be explained that when the regularization becomes gradually weak, it does NOT exert its compression effects so well as when the coefficient is large. This further validates the effectiveness of the regularization function proposed in this paper.
4.4 CIFARIO RESULTS
We demonstrate our cursor hased adaptive quantization algorithm on CIFAR10 benchmark dataset with ResNer20, ResNet56 MobileNetV2. For ResNet20,we compare the accuracy and compres- sion ratio of the proposed approach to some related or similar works such as DNAS (Wu et al. (2018). TTQ (Zhu et al. ()166). PACT (Choi al. (2018)) and LQE (Zhang et al. (2018) with Resnet20 on CIFAR-10, 10, and the details of accuracy and compression ratio are shown in Table It can noticed that, compared the other related works, our method achieves much better com- pression ratio while achieving comparable or better classification accuracy on CIFAR10 dataset. The reason why the proposed approach better than the quantization methods such as LQF,, TTQ and PACT may be due to the adaptive cursor based search mechanism. By considering both the model accuracy and compression ratio, the cursor based approach can effectively search different quantization bit for each layer whole, leading to better compression ratio with better accuracy. Compared to DNAS, the reason for our better performance terms of CR is partially due to that the two closest integers' quantization scheme produces less quantization error in each layer. In addition, it may be also because of our multiple lower bits' design in the search process.
Table 2: Performance comparison with other works using ResNet20 on CIFARIO
Accuracy(%%
Compression Ratio
Ours
92.27
28.3
DNAS(most efficient)
92.00
16.6
DNAS(most accurate)
92.72
11.6
LQE(2 bit)
91,80
16.0
TTQ(2 bit)
91.13
16.0
PACT(2 bit)
89,70
16.0
Baseline(32 bit)
92.06
1.0
We also apply the proposed approach ResNet56 and compare its performance with DNAS (Wul eet al. (2018), ) and the results are recorded in Table 5 We further test the proposed approach on MobilenetV2, 2, with the results shown in Table Because of space limitation, we put the detailed Tables and descriptions in Appendix.
4.5 CIFAR 100 RESULTS
To further show the effectiveness of the proposed scheme, we test our method on CIFAR 100 dataset using ResNet20, ResNet56 and MobileNerV2. We illustrate compressed ResNer20's performance compared to the original one on CIFAR100 in Table 3, it should be pointed out that here we also do not fine tune the original model, its accuracy may not be the best one in the literature. For ResNet20, our approach achieves a good compression ratio of 11.6 while maintaining a comparable accuracy of

---

Under review as a conference paper at ICLR 2020
Table 3:
Performance of ResNet 20 CIFAR100
Accuracy
Compression Ratio
Ours
68.18
11.6
Original(32 bit)
68.30
1.0
The performances of the quantized network of ResNet56 and MobileNerV2 on CIFARI00 datasets are presented Table 7 and in Table [ respectively, which are also placed in Appendix due to limited space. We notice that both quantized models show little better accuracy with impressive compression ratios of 17.2 and 12.9 for ResNet56 and MobileNetV2 respectively.
4.6 IMAGENET RESULTS
In this subsection, we apply ResNet18 and MobileNetV2 to ImageNet dataset, which is a much larger dataset compared to CIFARIO and CIFARIO0 Here, as in (Han et al. (2015); Wang et al. (2188), we present two sets of our results, i.e., the most efficient result and most accurate one to compare conveniently.
We record the performance of the proposed method with ResNet18 on ImageNet in Table as well comparisons LQE (Zhang et al. (20T8)), TTQ (Zhu al. (2016)), PACT (Choi et al. (2018)) methods. From Table 4 can be noticed that, compared to the original 32 bit model, the most accurate result of our algorithm achieves promising compression rate of 13.9 with slight accuracy drop of 0.15%, and for the most efficient one, our algorithm achieves an accuracy of 68.80% and an impressive compression ratio of 18.1. The most accurate result of our algorithm shows much better accuracy over LQE, TTQ and PACT methods although the compression ratio is a little bit smaller. As for the most efficient one, both the accuracy and compression ratio are better than those of LQE, TTQ and PACT, validating the effectiveness of the proposed scheme. The results of MobileNetV2 on ImageNet are illustrated Appendix, please refer to Table for details.
Table 4: Performance comparison with other works using ResNet18 on ImageNet
Accuracy(%)
Compression Ratio
Baseline(32 bit)
69.75
I.0
Ours(most accurate)
69.60
13.9
Ours(most efficient)
68.80
18.1
LQE(2 bit)
68.00
16.0
TTQ(2 bit)
66.60
16.0
PACT(2 bit)
64.40
16.0
5 CONCLUSIONS
In this paper, we have proposed a novel cursor based DAS algorithm for obtaining the mixed pre- cision DNN model. Different from most of the traditional approaches, which choose quantization configuration using heuristics or learning based rules, we adaptively choose the quantization bit for each layer in the DNN model from the perspective of NAS. cursor based search algorithm with alternative manner is applied for efficient optimization. The nearest two neighbor integers to the cursor are used to implement the quantization in the training process to reduce the quantization noise and avoid local convergence. The result of our algorithm is the adaptive bit width choice for different layers as whole. Extensive experiments with some typical models demonstrate that the proposed approach provides dramatic compression capability with accuracy par with better than the ofte--f-thhe-r of methods on benchmark datasets. In the near future, we will apply the proposed scheme to object detection tasks further show its possible wider application. We may also utilize some hardware platforms to test some performance such as inference time and resource consumption.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning convolittonn neural networks. CoRR, abs/1512.08571,, 2015. (tl.htttpp//arttu..oooo/a/a/////
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformatann for efficient architecture search. CoRR, htss/880602239)) 2018. URL http::/aaxii..oot////
abs/1806. 02639
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. PACT: htpp activation for quantized neural networks. CoRR,
2018. URL.htttp:/arx..v.oom abs 06085
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. URL http://arxiv.
org/ har//bb//tt / 602. 02830
Ahmed T. Elthakeh, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Es- maeilzadeh. Releq: reinforcement learning approach for deep quantization of neural networks. CoRR, abs/1811.01704, 2018.
01704
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graphnas: Graph neural architecture search with reinforcement learning. CoRR, 0bs/1904.00981, 2019. URL http://arxiv.
org/abs/1904 09981
Minghao Guo, Zhao Zhong, Wei Wu, Dahua Lin, and Junjie Yan. IRLAS: inverse reinforcement
learning for architecture search. CoRR, abs/1812.05285, 2018. URL http::/arxiv.om
bbss18122 05285
Yunhui Guo. A survey methods and theories of quantized neural networks. CORR, 0ass/808.00752.. 2018. /.KL/ttpp//arxll.orrg/////
04752
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.001444 2015.
Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, and Shin ichi Maeda. Einconv: Exploring unex- plored tensor decompositions for convolutional neural networks. ArXiv, abs/1908. 908.04471, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770- 778. 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv e-prints, art. arXiv:1503.02531 Mar 2015.
Jason Zhi Liang, Elliot Meyerson, and Risto Miikkulainen. Evolutionary architecture search for deep multitask networks. CoRR, abs/1803.03745, 2018. URL httt::://rxxiv.oo///2...
18C3.03745
Darryl Dexu Lin, Sachin Talathi, and V. Sreekanth Annapureddy. Fixed point pointiotittttooootioonnn deep convolutional networks. CoRR, 0as/1511.0699) 2015. URL. httt:://axxv.orrg//eee
1511. 06393
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, 2018 URL htth: /tttt:///txxii/t org/ abs/1806. 09055
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling.
Relaxed quantization for discretized neural networks. CoRR, abs/1810.01875. 2018. URL
http:// attp/riiiii..///...//.aaa/.aaatv org. abogg/abs/eee 18100 01875
Eunhyeok Park, Dongyoung Kim, Sungjoo Yoo, and Peter Vajda. Precision highway for ultra low- precision guantization. CoRR, abs/1812.00981, 2018a. URL http: /ttt:://arxiin org/ abs/
T812 2. 09818
11

---

Under review as a conference paper at ICLR 2020
Eunhyeok Park, Sungjoo Yoo, and Peter Vajda. Value-aware quantization for training and inference of neural networks. CoRR, Abs//800.07880, 2018b. URL http:ttttt//iii/////// wrw. 1804.
07802
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference
Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5113-5122, Long Beach, California, USA, 09-15 Jun 2019, PMLR. URL http: 1/procediings ngs. ml www..ppress..m www.srressvvi. wttpss//v///p/m/ .ttttmll
Hicu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. CoRR, abs/1802.03268, 2018. URL /tt/tt////rxxlv.rr///a/s
18C2. 03268
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. CoRR, abs/1802.05666 2018.
org. /a////b//// aaaaaaas..11 1802. 05668
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. CoRR, abs//660.00579, 2016. URL
http://arxii/
05279
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and seg- mentation. CoRR, abs/ /B//1801.04881, 2018. URL/attp://arxiia httppp///aaxii..oo/aab//////a/// wwtp:///ggaaaaaaaaaeaaeaeee 04381
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, and Weinan Convolutional neural networks with low-rank regularization. arXiv e-prints, art. 1r1 1.66677 Nov 2015.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware-aware automated quan- tization. CoRR, abs/181 2018. URL http:: /ttp:///axvv..v http:///aaiiii..craaa//1/1111111 08886
Xin Wei, Wenchao Liu, Lei Chen, Long Ma, He Chen, and Yin Zhuang. Fpga-based hype im- plementation of quantized neural networks for remote sensing applications. Sensors, 19(4), 2019. ISSN 1424-8220. doi: 10.3390/s1904099 URL (tthss/::/// www mdpi 19/4/924
wtwpi.ccm//aaaa wwm. 1424 82207
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. CoRR, abs/1812.0000 2018
abs 1812. 00090
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng- Quantized convolutional neural networks for mobile devices. CoRR, abs/1512.006733, 2015. URL http::/arxiv:
brg/abs/1512. 06473
Sergey Zagonuyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. CoRR, abs/1612.039988, 2016.
URL http: / http://xxvvvv. org/ abs /aabss//11...c 03928
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. CoRR, abs/1807.10029, 2018 URL
org. abs / 1807 10029
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, 2016. URL htttt:
/1666.6666660 06160
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. CoRR, abs/1612.01064, 2016. URL httt: http:://aaxvvvv.o org/ abs 2616661616 01 064
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jin-Hui Zhu. Diserimination-awware channel pruning for deep neural networks. CoRR, Abs//810.11 1809, 2018. UtL/http://arxivv
wwwp:/iwww.roraeeoooooooo abs hwtp://////ppaa.aatota/pe
11809
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. CoRR, abs/ Abs//161..0178. 2016. URL htth:t/taxxx...oo httt:://axxv..oooo/a////1111
abs 01578
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Due to space limitations, we put number of experimental results here.
A.1 RESNET56 ON CIFARIO
We apply the proposed approach to ResNet56 and compare its performance with DNAS (Wu et al. (0188), and the results are recorded in T In this Table, we notice that ResNet56 compressed using our quantization approach shows higher compression ratio of 20.6 compared to the most efficient DNAS's result 18.9, while its accuracy almost the same. As for the most accurate one of DNAS, our quantized model is still comparable, while the compression ratio is much higher compared to 14.6. It should also be pointed out that due to different environments and other imple- mentation differences in the whole process, the accuracy of ResNet56 baseline is not good as the one in DNAS. It should also be noted that DNAS takes cutout operation in its experiments, also test such an operation to show the comparison. may be noticed that our test with cutout achieves a accuracy of 94.48% with an impressive compression ratio of 23.3.
Table 5: Performance comparison with other works using ResNet56 on CIFARIO
Accuracy(%) (%)
Compression Ratio
Ours
93,45
20.6
Ours cutout
94.48
23.3
Baseline of our implementation(32 bit)
93.39
1.0
DNAS + eutout most efficient)
94.12
18.9
DNAS + cutout most accurate)
94.57
14.6
Baseline in DNAS(32 bit)
94.42
1.0
A.2 MOBILENETV2 ON CIFARIO
We further apply the proposed approach to MobilenetV2, which is a typical DL model for mobile devices and embedded systems, with the results shown in Table It can be noticed that our adaptive cursor based quantization shows better classification accuracy of 93.28% as well promising compression ratio of 12.4.
Table 6: Performance of MobileNetV2 on CIFARIO
Accuracy
Compression Ratio
Ours
93.28
12.4
Original(32 bit)
92.39
1.0
A.3 RESNET56 ON CIFAR100
We also test ResNet56 on CIFAR 100 and present its corresponding results in Table 7 We can see that the proposed algorithm yields little bit better accuracy(1.00) (1.05%) together with an impressive compression ratio of 17.2.
Table 7: Performance of ResNet56 on CIFARION 100
Accuracy
Compression Ratio
Ours Original
71.84
17.2
70.79
1.0
13

---

Under review as a conference paper at ICLR 2020
A.4 MOBILENETV2 ON CIFARI00
The quantized MobilenetV2 on CIFARI00 and its corresponding results are demonstrated in Table As for MobileNetV2, the proposed algorithm yields little bit better accuracy together with impressive compression ratio of 12.9.
Table 8: Performance of MobileNetV2 on CIFAR100
Accuracy
Compression Ratio
Ours
68.04
12.9
Original(32 bit)
67.96
1.0
A.5 MOBILENETV2 ON IMAGENET
The performance of MobileNetV2 on ImageNet illustrated Table together with comparisons to some related works such HAQ (Wang et al. (2018) and deep compression (Han et al. (2015). In Table 9 we notice that, for the most accurate result, the quantized MobileNerV2 model using approach shows slight accuracy loss (71.65%
72.19% of the original 32 bit model) while achieves an encouraging compression ratio of 9.1. It may also be observed that the accuracy of our most accurate one little bit higher than the corresponding most accurate results of HAQ and deep compression together with a better compression ratio. While for the most efficient one, our algorithm shows a compression ratio of 14.3, which better than that of HAQ, but smaller than that of deep compression. However, our approach demonstrates a dramatically better accuracy of 70.59% compared to the corresponding 66. 66.75% of HAQ and 58.07% of deep compression, which again validated the advantage of our algorithm.
Table 9: Performance comparison with other works using MobileNetV2 on ImageNet
Accuracy(%5) %))
Compression Ratio
Baseline(32 bit)
72.
1.0
Ours(most accurate) Ours (most efficient)
71.65
9.1
70.59
14.3
HAQ(most accurate)
71.47
7.5
HAQ(most efficient)
66.75
13.9
Deep compression(mos (most accurate)
71.24
8.0
Deep compression (most efficient)
58.07
16.0
14


Reviews:
[{"review_id": "Syx6-A1ZqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel way to determine automatically the quantization level at each layer of \ndeep neural networks while training. The quantization can result in a more compact network requiring\nless memory for storing the parameters (weights), with minimal drop in accuracy. \nIf I understand correctly, the main idea in the paper is utilizing a continuous representation of the quantization using a continuous cursor, and a new differential approach from Neural Architecture Search (NAS), to create a differential learning algorithm for the problem of finding the optimal quantization. \nOn the positive side, the authors present very good experimental results on popular networks like ResNet, \nand improve on other method's compression accuracy, with little to no reduction in accuracy. \nHowerver, the paper is not clearly and carefully enough written to convey the author's results which makes it a borderline paper in my opinion. \n\n\nIn my understanding, compressed (quantized) networks can have different advantages besides a more compact representation of the network: \nFor example, this can also reduce training time because all operations are performed with words with fewer bits - but the authors focus only on the compression ratio and do not discuss this issue. In fact, it seems that their algorithm actually requires more training time than standard training of neural networks, because you start with training w with full 32-bit representation, and then train with different quantizations simultaneously. \nIt would be good if the authors describe clearly how do different methods (theirs and the others mentioned) compare to each other in terms of savings of different resources (training time, compression, inference/prediction time for the trained networks etc.)  and which of these is more important (e.g. the latter in mobile devices).\n\nAnother practical issue is that using a different number of bits for each layer may complicate the design and software/hardware implementation of a network, compared to say allocating 4-bits or 8-bits to each weight. It is correct that you can reduce the overall number of bits, but you may reduce it even further if you allow a different number of bits for each individual weight. I assume that there is a price in power/memory/speed etc. to this non-homogenous property, and it may be better to get a slightly lower compression ratio provided that there is more uniformity in the network weights.  This is not a critique of this particular paper, but of the entire framework of flexible quantization. \n\nThe English of the manuscript could be greatly improved. For example: \nPage 2: \"..multiple bits for different layers..\" - I assume the authors mean \"..a different number of bits used for different layers ..\"\nPage 2, bottom: \"The authors (Wang et al. 2018) presented ...\" - the sentence is long and unclear. \nMany sentences starting with 'And' - for example: \"And a Gumbel softmax function ...\" (page 3) \n\nThere are a few vague and non-informative statements which do not contribute to the understanding of the field and the authors' contribution. For example: \n- Page 3: \"A typical search method is random search, however, its efficiency is not ideal.\" (does efficiency refer to computational efficiency here?)\n- Page 4: \"The reason why we add regularization item to the loss function is because the regularization can prevent overfitting to some extent\" (I thought that here the main purpose of regularization is to get a trade-off between accuracy and compression). \n- Page 6: Regarding parameters choice: 'a rather optimal set of them is chosen ..' - this seems quite arbitrary and it is not clear how to choose parameters for the authors' method for future architectures and datasets. \n\n\nThere are also terms that are known to expert but could use a short explanation/reference - for example: \n- Adam optimizer \n- Cosine annealing\n- Gumble Softmax\n\nThe mathematical equations are definitions are not clear enough and contain errors: \n- Eq. (1): there is (x',y') twice in the equation, but in the sentence afterward there is also (x,y). \nThere are D_T, D_V with and without tilde. \nThe space of maximization of w for a given quantization is not defined (I'm assuming all weights vectors w\nwhich can be represented using the number of bits in the quantization). \n- Eq. (3) - the loss is loosely defined - what is 'parameter size'? the number of bits for all the weights in each layer? \n- Eq. (4) - is w or w_k the full precision weights? why are x and y in the interval [0,1]?\n- Eq. (6) - c_i represnts the i-th layer, but d_1,d_2, a_1,a_2 are fixed. Are they also different for different layers?   \nAlso, a_1 and a_2 are the boundaries of the cursor - are they set to [0,1]? or [0,32]? \n- The authors defined Conv - the convolution operation - but what are W_1 and X? vectors? how is the convolution\ndefined precisely? it is also not common in math papers to use '*' for  product\n\nIt is not entirely clear to me how exactly the authors define a differentiable loss and a gradient for the quantization part Loss_q.  When a cursor parameter changes (e.g. c_i from 2.5 to 2.6) then the loss is defined by quantization to the two consecutive integers (in that case 2,3 ) and there is a continuous mixture parameter between them which can be changed smoothly, which is nice. But at some point, the parameter will reach 3, and then the quantization will use the two integers 3 and 4 - the fact that 3 has weight 1 in the loss (and 2,4 have weight zero) makes the loss continuous, but is the loss differentiable at this point?  some explanation is needed here on if this is a problem and how is it avoided/solved. \n\nAlgorithm 1: The description in Figure 1 can be made more precise. The algorithm seems to alternate between \noptimizing the cursor c, and optimizing the weights w for a given value of c. \nThere are also missing details and parameters like gradient step size.  What does Grad-C * L_C mean? is the gradient multiplied by the loss?? or only applied to it? \nIn the sentence \"Quantize the network ... to update the loss\" - which loss is updated and how? L_V, L_T? something else? \nA detailed description of the algorithm and parameters or the actual code used by the authors would improve the understanding of their method. \n\nSection 4.1: Figure 2 compares the loss of one-integer vs. two-integers quantization scheme. The authors argue that their two integers scheme is better because it is smoother. But the loss for one integer is actually lower - so why wouldn't it be better to use this one?  \nwill the two-integer method eventually reach a lower loss? \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e4UrfvqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors developed a novel quantization technique that yields layer-wise different mixed-precision quantization. To do so, they alternatively update the pre-trained weights and the quantizer, which they call cursor. The following two features distinguish this paper: using two precision values around the cursor's value (instead of the closest one) and regularizing the parameter size using a new loss function. Because the whole process is differentiable, the appropriate precision to each layer can be found fast. Thanks to these efforts, this method balances the compression rate and accuracy well on CIFAR-10 and ImageNet.\n\nUnfortunately, the details of these two features are hardly supported in conceptual, theoretical, and experimental manners. \n(1) The reviewer guesses that \"parameter size after quantization in one layer\" in equation 3 is related to \"layer_size\" in equation 8. However, the relationship is unclear and cannot convince the reviewer that the newly proposed loss (equation 3) is differentiable. Also, equation 7 ($f=d_1*(Conv(X, W_1)+d_2*Conv(X*W_2))$) is difficult to make the reviewer understand why and how this method works.\n(2) The preliminary experiment in section 4.1 seems to claim that a single cursor leads a network to local minima, by only showing training loss. The reviewer thinks that the authors need to show validation loss as well to claim the failure of the single cursor.\n\n\nThe followings are minor comments.\n* To the best of the reviewer's knowledge, the term \"cursor\" is the authors' original one. Therefore, the authors need to write its definition.\n * The mathematical notations in this paper are confusing. (1) Each face has different meanings. For example, $max$ means $m \\times a \\times x$, rather than the max operator $\\max$. (2) operator * is used in arbitral as both unitary and binary without any comments or definitions.\n* The reference is required to be format and cited correctly. For example, [He et al. 2015] is accepted to CVPR 2016 but is not mentioned. \n* The authors claim that \"comprehensive experiments\" are done. However, the authors' proposal is experimented only on CIFAR-10 and ImageNet with MobileNet V2 and ResNet-18, while DNAS, for example, is verified on more variety of data and networks, plus object detection tasks. The reviewer thinks that the method proposed in this paper requires more comprehensive experiments.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlKxH2N5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about using quantization to compress the DNN models. The main idea is to use NAS to obtain the mixed precision model. More specifically, it adaptively chooses the number of quantization bit for each layer using NAS by minimizing the cross-entropy loss and the total number of bits (or model size) used to compress the model. The experiment is on CIFAR and ImageNet, and compared with other quantization methods showing better accuracy.\n\nI have somes questions on this paper:\n\n1) Is Eq(1) standard for quantization optimization? Any reference? Normally, we aim to minimize the loss on the training set, not the validation set,  or sometimes generalization loss on data from the data distribution. \n\n2) For experiment, it is interesting to see how the compression ratio changes over the accuracy--- that is a curve with x-axis on compression ratio, and y axis  accuracy so that we can have a sense of how the accuracy and compression rate trade-offed.\n\n3) What is the training time for NAS for finding the 'optimal' bits for each layer? Although we might not care much about the training time for compression task, I just want to have a sense of how training works. Also what is the reason for not compressing the first and last layers? Do these two layers taken into account in the final compression ratio computation?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to compress DNNs by quantization. The core idea is to use NAS techniques to adaptively set quantization bits at each layer. The proposed method is shown to achieved good results on the standard benchmarks. 
Through our final discussion, one reviewer agreed to raise the score from ‘Reject’ to ‘Weak Reject’,  but still on negative side. Another reviewer was not satisfied with the author’s rebuttal, particularly regarding the appropriateness of training strategy and evaluation. Moreover, as reviewers pointed out, there were so many unclear writings and explanations in the original manuscript. Although we admit that authors made great effort to address the comments, the revision seems too major and need to go through another complete peer reviewing. As there was no strong opinion to push this paper, I’d like to recommend rejection. 
Decision: Reject