Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work attempts to improve DRL based mixed-precision network compression. Compared to the standard action-critic method, he proposed method inserts an additional candidate evaluation step on k proposed actions as shown in Algorithm 1. As the paper title suggests, the proposed augmentation is “simple” yet practically brings non-trivial performance improvement. 

I am not a real expert on RL. In general, I appreciate the idea in this work. It generates higher-quality action proposals by heuristically evaluating a few of them, either using a quick calculation of inference accuracy or direct quantization loss. The proposed idea is intuitive and well validated by experiments. 

I have several suggestions for improving this work: the distance-based indicator is proved to be inferior in terms of performance, mentioned by authors. It is still suggested to report its accuracies in Tables 1 and 2, for a better view of two variants.  A few issues in experiments need clarification: is the reported time the total before convergence, or the per-episode time? The proposed method takes much fewer episodes to search but an additional q-value indicator brings more computations. More analysis regarding the complexity is required. There are a few ad hoc treatments in the method, such as memorizing the accuracy of specific bit value at a layer. Does it consider the evolution of the entire network (since the change of other layers have impact)? The theoretic analysis is based on assumptions that can be only empirically verified (yet not convincingly verified in the experimental sections). It is not clear to me the true value of section 3.2.

review_title: It modifies a step in DRL based method that quantizes each networks with different bit numbers. A good engineering trick.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ========
Summary

This paper studies the DNN quantization using deep reinforcement learning. The paper proposes an augmented DRL which introduces a Q-value indicator to refine action selection. The proposed approach has been applied to several image classification baselines and has compared with several recent DRL based quantization approach, achieving a similar compression rate without accuracy decrease.  In addition, compared to previous methods, the learning speed has been improved by 4.5-64x. 

=========
Pros

1. The paper is well written and easy to follow.  It is very clear even for an audience who may not be an expert on DNN quantization. 
2. The idea is simple and reasonable to introduce an additional Q-value indicator to refine action selection.  Given the improvement in performance,  despite the simplicity, the method does provide great performance. 


==========
Concerns/Comments

1.  Is there any result about the final quantization configuration of the compressed model? Is there any takeaway about the quantization pattern? 

2.  What is the action space used in the experiments? How many bits were the model used? 

3. In Table 1, could you provide the original accuracy of the models rather than the accuracy delta?  In addition, is that possible to quantize deeper models like ResNet-101 or ResNet 152? Is the method extendable to the latest model architectures such as ResNeXt, DenseNet, visual transformers?
















review_title: Simple but effective method with strong performance improvement

final_score: rating: 7: Good paper, accept
review_content: This paper describes an improved way to determine weight quantization bit lengths using reinforcement learning, by injecting model evaluation directly into action selection.  Building upon a DRL setup where the action at each timestep corresponds to selecting a bit value for each layer, the method adds a "Q-value indicator" function Q~ that selects among a set of candidate actions, and filters based on the model performance quantizing the layer to each level.  This seems to form a hybrid between DRL and greedy search, using a greedy criteria Q~ to filter proposals made by the DRL agent.  Experiments show very good performance, with similar or better quantization levels and accuracy as other DRL-based methods and much faster runtime.

The method is well described overall, though I would have liked more details on the mu and Q networks including their initializations and how well the mu proposals span the action space initially (see questions below).

Likewise, I wonder if the mechanism of improvement is mostly through the initial guess provided by greedy search with Q~ at the beginning of training.  An alternative may be to edit the action network mu using a guess made by greedy profiling (see question 3 below) -- was anything like this explored?

Although I wonder whether the method may be able to be further simplified, I still find this a good paper overall, offering an effective way to reduce time to produce quantized models with no accuracy hit.  A few additional baselines of even simpler systems, including greedy search and greedy policy initialization, could help provide more context and assess the use of Q~ as a filter.



Additional questions:

1.  The descretized action space only has 8 values per layer.  What happens if one evaluates all 8 with the Q~ indicator and remove the candidate generation network?  Is this equivalent to a layerwise greedy search?  What is the final accuracy and quantization speed of this baseline?

2.  How is the last layer of the expanded actor function mu^ initialized?  Randomly, or in such a way that the initial outputs correspond to different discrete bit sizes?  How many candidate actions are there?  How well do the candidate values span the action space to allow for guidance by Q~, particularly in the beginning episodes?

3.  Instead of having multiple candidates, is it possible to introduce an output bias per layer in the actor function mu, initialized to the bit sizes determined by greedy profiler search with Q~ ?  That is, to use a function mu_l(s) = mu(s) + b(l), where b(l) = argmax_a Q~(a) for each layer l, then use regular DRL (not ADRL) with this mu_l that has an initial guess using greedy search from the profiler?

4.  I don't see where the system is encouraged to use fewer bits.  If more bits generally leads to higher accuracy, what makes the system learn to output smaller bits in its actions?  Both Q~ augmentation and the reward R = acc_quant - acc_orig depend only on accuracy.

5.  How different are the proposal actions from one another (do they span the space of possible bit lengths), and how do they change over the course of model selection?  Is the initial max according to Q~ used most of the time or are there times when the proposal index selected by Q~ changes from one episode to the next?
review_title: effective approach, interested in further details on action proposals

Meta-review: This paper proposes a simple yet effective approach for determining weight quantization bit lengths using RL. All the reviewers agree that the simplicity and performance improvements are a strong plus point. There are some concerns on applicability which have been sufficiently handled by rebuttal. AC recommends accepting the paper.
Decision: Accept (Poster)