Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors developed a novel quantization technique that yields layer-wise different mixed-precision quantization. To do so, they alternatively update the pre-trained weights and the quantizer, which they call cursor. The following two features distinguish this paper: using two precision values around the cursor's value (instead of the closest one) and regularizing the parameter size using a new loss function. Because the whole process is differentiable, the appropriate precision to each layer can be found fast. Thanks to these efforts, this method balances the compression rate and accuracy well on CIFAR-10 and ImageNet.

Unfortunately, the details of these two features are hardly supported in conceptual, theoretical, and experimental manners. 
(1) The reviewer guesses that "parameter size after quantization in one layer" in equation 3 is related to "layer_size" in equation 8. However, the relationship is unclear and cannot convince the reviewer that the newly proposed loss (equation 3) is differentiable. Also, equation 7 ($f=d_1*(Conv(X, W_1)+d_2*Conv(X*W_2))$) is difficult to make the reviewer understand why and how this method works.
(2) The preliminary experiment in section 4.1 seems to claim that a single cursor leads a network to local minima, by only showing training loss. The reviewer thinks that the authors need to show validation loss as well to claim the failure of the single cursor.


The followings are minor comments.
* To the best of the reviewer's knowledge, the term "cursor" is the authors' original one. Therefore, the authors need to write its definition.
 * The mathematical notations in this paper are confusing. (1) Each face has different meanings. For example, $max$ means $m \times a \times x$, rather than the max operator $\max$. (2) operator * is used in arbitral as both unitary and binary without any comments or definitions.
* The reference is required to be format and cited correctly. For example, [He et al. 2015] is accepted to CVPR 2016 but is not mentioned. 
* The authors claim that "comprehensive experiments" are done. However, the authors' proposal is experimented only on CIFAR-10 and ImageNet with MobileNet V2 and ResNet-18, while DNAS, for example, is verified on more variety of data and networks, plus object detection tasks. The reviewer thinks that the method proposed in this paper requires more comprehensive experiments.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper is about using quantization to compress the DNN models. The main idea is to use NAS to obtain the mixed precision model. More specifically, it adaptively chooses the number of quantization bit for each layer using NAS by minimizing the cross-entropy loss and the total number of bits (or model size) used to compress the model. The experiment is on CIFAR and ImageNet, and compared with other quantization methods showing better accuracy.

I have somes questions on this paper:

1) Is Eq(1) standard for quantization optimization? Any reference? Normally, we aim to minimize the loss on the training set, not the validation set,  or sometimes generalization loss on data from the data distribution. 

2) For experiment, it is interesting to see how the compression ratio changes over the accuracy--- that is a curve with x-axis on compression ratio, and y axis  accuracy so that we can have a sense of how the accuracy and compression rate trade-offed.

3) What is the training time for NAS for finding the 'optimal' bits for each layer? Although we might not care much about the training time for compression task, I just want to have a sense of how training works. Also what is the reason for not compressing the first and last layers? Do these two layers taken into account in the final compression ratio computation?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a novel way to determine automatically the quantization level at each layer of 
deep neural networks while training. The quantization can result in a more compact network requiring
less memory for storing the parameters (weights), with minimal drop in accuracy. 
If I understand correctly, the main idea in the paper is utilizing a continuous representation of the quantization using a continuous cursor, and a new differential approach from Neural Architecture Search (NAS), to create a differential learning algorithm for the problem of finding the optimal quantization. 
On the positive side, the authors present very good experimental results on popular networks like ResNet, 
and improve on other method's compression accuracy, with little to no reduction in accuracy. 
Howerver, the paper is not clearly and carefully enough written to convey the author's results which makes it a borderline paper in my opinion. 


In my understanding, compressed (quantized) networks can have different advantages besides a more compact representation of the network: 
For example, this can also reduce training time because all operations are performed with words with fewer bits - but the authors focus only on the compression ratio and do not discuss this issue. In fact, it seems that their algorithm actually requires more training time than standard training of neural networks, because you start with training w with full 32-bit representation, and then train with different quantizations simultaneously. 
It would be good if the authors describe clearly how do different methods (theirs and the others mentioned) compare to each other in terms of savings of different resources (training time, compression, inference/prediction time for the trained networks etc.)  and which of these is more important (e.g. the latter in mobile devices).

Another practical issue is that using a different number of bits for each layer may complicate the design and software/hardware implementation of a network, compared to say allocating 4-bits or 8-bits to each weight. It is correct that you can reduce the overall number of bits, but you may reduce it even further if you allow a different number of bits for each individual weight. I assume that there is a price in power/memory/speed etc. to this non-homogenous property, and it may be better to get a slightly lower compression ratio provided that there is more uniformity in the network weights.  This is not a critique of this particular paper, but of the entire framework of flexible quantization. 

The English of the manuscript could be greatly improved. For example: 
Page 2: "..multiple bits for different layers.." - I assume the authors mean "..a different number of bits used for different layers .."
Page 2, bottom: "The authors (Wang et al. 2018) presented ..." - the sentence is long and unclear. 
Many sentences starting with 'And' - for example: "And a Gumbel softmax function ..." (page 3) 

There are a few vague and non-informative statements which do not contribute to the understanding of the field and the authors' contribution. For example: 
- Page 3: "A typical search method is random search, however, its efficiency is not ideal." (does efficiency refer to computational efficiency here?)
- Page 4: "The reason why we add regularization item to the loss function is because the regularization can prevent overfitting to some extent" (I thought that here the main purpose of regularization is to get a trade-off between accuracy and compression). 
- Page 6: Regarding parameters choice: 'a rather optimal set of them is chosen ..' - this seems quite arbitrary and it is not clear how to choose parameters for the authors' method for future architectures and datasets. 


There are also terms that are known to expert but could use a short explanation/reference - for example: 
- Adam optimizer 
- Cosine annealing
- Gumble Softmax

The mathematical equations are definitions are not clear enough and contain errors: 
- Eq. (1): there is (x',y') twice in the equation, but in the sentence afterward there is also (x,y). 
There are D_T, D_V with and without tilde. 
The space of maximization of w for a given quantization is not defined (I'm assuming all weights vectors w
which can be represented using the number of bits in the quantization). 
- Eq. (3) - the loss is loosely defined - what is 'parameter size'? the number of bits for all the weights in each layer? 
- Eq. (4) - is w or w_k the full precision weights? why are x and y in the interval [0,1]?
- Eq. (6) - c_i represnts the i-th layer, but d_1,d_2, a_1,a_2 are fixed. Are they also different for different layers?   
Also, a_1 and a_2 are the boundaries of the cursor - are they set to [0,1]? or [0,32]? 
- The authors defined Conv - the convolution operation - but what are W_1 and X? vectors? how is the convolution
defined precisely? it is also not common in math papers to use '*' for  product

It is not entirely clear to me how exactly the authors define a differentiable loss and a gradient for the quantization part Loss_q.  When a cursor parameter changes (e.g. c_i from 2.5 to 2.6) then the loss is defined by quantization to the two consecutive integers (in that case 2,3 ) and there is a continuous mixture parameter between them which can be changed smoothly, which is nice. But at some point, the parameter will reach 3, and then the quantization will use the two integers 3 and 4 - the fact that 3 has weight 1 in the loss (and 2,4 have weight zero) makes the loss continuous, but is the loss differentiable at this point?  some explanation is needed here on if this is a problem and how is it avoided/solved. 

Algorithm 1: The description in Figure 1 can be made more precise. The algorithm seems to alternate between 
optimizing the cursor c, and optimizing the weights w for a given value of c. 
There are also missing details and parameters like gradient step size.  What does Grad-C * L_C mean? is the gradient multiplied by the loss?? or only applied to it? 
In the sentence "Quantize the network ... to update the loss" - which loss is updated and how? L_V, L_T? something else? 
A detailed description of the algorithm and parameters or the actual code used by the authors would improve the understanding of their method. 

Section 4.1: Figure 2 compares the loss of one-integer vs. two-integers quantization scheme. The authors argue that their two integers scheme is better because it is smoother. But the loss for one integer is actually lower - so why wouldn't it be better to use this one?  
will the two-integer method eventually reach a lower loss? 


review_title: Official Blind Review #3

Meta-review: This paper presents a method to compress DNNs by quantization. The core idea is to use NAS techniques to adaptively set quantization bits at each layer. The proposed method is shown to achieved good results on the standard benchmarks. 
Through our final discussion, one reviewer agreed to raise the score from ‘Reject’ to ‘Weak Reject’,  but still on negative side. Another reviewer was not satisfied with the author’s rebuttal, particularly regarding the appropriateness of training strategy and evaluation. Moreover, as reviewers pointed out, there were so many unclear writings and explanations in the original manuscript. Although we admit that authors made great effort to address the comments, the revision seems too major and need to go through another complete peer reviewing. As there was no strong opinion to push this paper, I’d like to recommend rejection. 
Decision: Reject