Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an entropy-driven mixed-precision quantization strategy. The basic idea is to formulate the neural network as an information system, and then define a quantization entropy score to capture the expressiveness of the system. Finally, the Quantization Bits Refinement algorithm quantizes the activation and weights to meet the peak memory and model size constraint.
strengths_and_weaknesses: Strength:
1. The concept of the quantization entropy score looks interesting.
2. The proposed solution outperforms SOTA works.

Weakness:
1. The proposed solution does not work for GPU.



questions: 1. The QBR strategy looks heuristic, is it possible to fall into an oscillation process?
2. Why only 8 bit and 2 bit in Figure 7?
limitations: The proposed QE-scores can only use CPU resources instead of GPU.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents an entropy-driven mixed-precision quantization strategy to design efficient CNNs for resource constrained IoT devices. The paper proposes a new metric, Quantization Entropy Score (QE-Score), to evaluate the CNN model capacity without the need of training the model. QE-Score is applicable when different layers are quantized to different bit widths, and thus can significantly speedup the design space exploration for efficient models. Extensive experiments have been performed to show that the proposed quantization strategy discovers tiny models with higher accuracy under strict resource constraints. 
strengths_and_weaknesses: ### Originality

While the connections between deep learning models and information systems have been studied and the entropy has been related with accuracy in the literature, it appears to be an original idea to evaluate CNN model capacity via a single QE-Score metric under the quantization scenario. 

The discussion on the relation among quantization budget, neural architecture, initialization and entropy (accuracy) is original and useful. 


### Quality

**Strengths**

The proposed QE-Score seems to be a simple yet effective metric to estimate the model capacity. This shows potential for practical IoT deployment. 

The evaluation of the proposed strategy is extensive and covers various aspects. 

Necessity of mixed-precision quantization is clearly illustrated by the MobileNet layer breakdown in Fig 2. 


**Weaknesses**

Various technical descriptions in Section 3 are unclear, and I am challenged to interpret some of the plots. For example:
* Equation 6: what variable is the summation applied on? Q based on its different value ranges?
* Fig 6: it appears to me most subplots can show positive correlation between QE-Score and accuracy. What is so special about the plot of $\sigma_A=5$ and $\sigma_W=4$ plot? Why do you say that "QE-Scores are gradually positive(ly) correlated with accuracy until $\sigma_A=\sigma_W=4$ Then, we adjust $\sigma_A=5$ and $\sigma_W=4$ to rank the diversity of activations and weights on accuracy"? What do you mean by ranking the diversity of activations and weights?
* The Quantization Bits Reinforcement (QBR, Fig 7) strategy is very unclear. Where do you explain this figure? Why do you need this refinement when you already have QE-Score to guide the allocation of bit-widths? You should at least briefly introduce this algorithm in the main text. 

The fundamental assumption on the system is unclear: To execute the CNN on IoT devices, do you need the full model to fit in its memory? Or do you only need a single layer (the currently executed layer) to fit in the memory? From Fig 2, it seems that you are imposing a layer-wise constraint. From other parts of the paper, it seems that the constraints are on the full CNN. 

From the description, it seems that the main purpose of QE-Score is to simplify neural architecture search (e.g., channel widths, bit widths). Then why does initialization of X and W matter? Specifically, since you will train the model after fixing the architecture, the effect of different initialization scales will be automatically addressed by batch-norm or layer-norm. 

While the empirical results seem to justify QE-Score, its derivation seems problematic. Activation has a major impact on the distribution. For example, with ReLU, half of values will be cropped to 0 if the input follows 0-mean Gaussian. With a deep CNN, the final layer distribution will be far from Gaussian. Thus the derivation of QE-Score does not follow a realistic assumption. 


### Clarity

The initial portion of the paper is clear to motivate mixed-precision quantization. However, the technical description is overall unclear (see details above). 

Tables and figures in experiments clearly show the advantages of the proposed strategy over the baseline w.r.t. various different metrics. 


### Significance

Judging from the empirical gains, the proposed method makes a significant contribution to enable efficient IoT inference with compressed models. 

However, evaluations are mostly performed on a single backbone model (MobileNet). It is unclear if the proposed method can generalize to other CNN or non-CNN models. 

The technical significance seems to be limited. First of all, the QE-Score and the proposed algorithm are both mostly heuristic based. Secondly, due to the questions on the technical sections (listed above), it is unclear if the proposed method brings in-depth insights to the community. 

questions: Please clarify the questions regarding the technical descriptions (see comments on “Quality”). 

Please provide more details on the main algorithm steps (e.g., Quantization bits refinement, Evolution algorithm). 

limitations: The paper does not discuss potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Previous works use an inefficient two-stage method that quantizes after designing a small network. However, the proposed method uses a one-stage method that can jointly optimize network design and mixed-precision quantization. Therefore, it is not only efficient in terms of time-cost, but also achieved high performance in various tasks.
strengths_and_weaknesses: Strengths:
1) The ranking strategy from the perspective of entropy and Quantization Bits Refinement in the evolutionary algorithm considering the HW budget are interesting and novel.
2) This paper clearly defined the problems and demonstrated solutions through solid mathematical analysis. 
3) This paper is very clear and easy to read. I think that it would help both researchers related to this field and general readers to give a little bit more background on TinyML.
4) The experiment showed the superiority of the proposed method on various datasets (ImageNet, VWW, WIDER FACE).

Weaknesses:
1) For all experiments, MobileNet-V2 was used as the base model. In particular, the analysis of peak memory and model size presented in Section 2 is also limited to MobileNet-V2. However, when a different model is used, the peak memory for each layer will also be different. Can we assume that the results of MobileNet-V2 are representative? The authors should show that the proposed method is compatible with other models.
2) Since the key contribution asserted in this paper is the 1-stage approach, it is necessary to thoroughly compare the proposed method with the 2-stage methods. In other words, comparative experiments with more combinations of SOTA network design studies + SOTA mixed-precision studies should be presented.
3) Experimental results in terms of power (energy) consumption should be presented. As the authors know, the most important factor in an IoT device is energy consumption.
4) It is necessary to explain whether the proposed method is compatible with more complex networks (e.g., SSD, instance segmentation, Transformer). (If possible, the experimental results need to be presented together.) Also, it is difficult to understand which network was used for the current Tiny Object Detection experiment.
5) [1] was used as a quantization-aware training (QAT) method. However, model performance may be improved by [1] in this paper. So, the current results cannot clearly show the effectiveness of the proposed method. Further experiments excluding the impact of QAT should be presented.
6) In this paper, only SRAM and Flash are used as constraints, but it seems that DRAM should be considered.
7) Why didn't the authors use the commonly used bit-precision configuration {2, 4, 8, 16} in the mixed-precision setting? Additional explanations related to this should be provided.
8) In Table 3, the appropriate reference should be added on which tool was used to measure CO2 emission.

[1] Learned step size quantization. ICLR’20.
questions: 1) Can you provide applicability and experimental results for networks other than MobileNet-V2?
2) Can you show additional experimental results related to Weaknesses 2), 3) and 5)?
3) Can you provide additional experimental results and explanations regarding the compatibility of the proposed method?
4) Can you provide additional explanations related to Weaknesses 6) and 7)?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new mixed-precision quantization in a new perspective of information entropy. Since the proposed Quantization Entropy Score can approximately measure the expressiveness of neural networks and is easy to compute, the authors can do the search in a short time.
strengths_and_weaknesses: Strengths
1. The paper is well developed and organized. The tables and figures are great.
2. The authors extend the previous work of viewing neural networks from the perspective of information entropy. Since quantization can be treated as an operator, it is natural to apply this perspective to the quantization problem.
3. The experiments are convincing. The authors cover three benchmarks.

Weaknesses
1. I am wondering if there is any work on using the entropy-based scores to conduct neural architecture search (NAS). If yes, the related work should be discussed and compared. If no, it should be a contribution of this paper. The authors should also show that the score can be used to search for a full precision model efficiently.
2. How can the proposed method extend to other machine learning models, such as Transformers?
questions: 1. Since the search cost is very low, why do the authors expand the search space? We may expect to find better results in a larger solution space. For instance, could you relax the constraints mentioned in Line 186 (three layers in each block share the same precision value)?
2. In Table 5, why does the QBR achieve better results? From my understanding, the QBR uses prior knowledge to shrink the search space. Without QBR, we should explore a larger space and achieve better results than that with QBR.
3. Could the authors discuss other hardware-related metrics, such as latency, throughput, and energy?

limitations: Limitations and social impacts are well discussed in Appendix E and F.
review_title: null

Meta-review: metareview: this paper proposes a joint architecture design and quantization method by casting it as an Entropy Maximization process. reviewers give consensus acceptance. 
confidence: Certain
Decision: Accept