Entropy-Driven Mixed-Precision Quantization for Deep Network Design
Authors: Zhenhong Sun
Ce Ge
Junyan Wang
Ming Lin
Hesen Chen
Hao Li
Xiuyu Sun
Venue: NeurIPS 2022 Accept (2022)

Abstract: Deploying deep convolutional neural networks on Internet-of-Things (IoT) devices is challenging due to the limited computational resources, such as limited SRAM memory and Flash storage. Previous works re-design a small network for IoT devices, and then compress the network size by mixed-precision quantization. This two-stage procedure cannot optimize the architecture and the corresponding quantization jointly, leading to sub-optimal tiny deep models. In this work, we propose a one-stage solution that optimizes both jointly and automatically. The key idea of our approach is to cast the joint architecture design and quantization as an Entropy Maximization process. Particularly, our algorithm automatically designs a tiny deep model such that: 1) Its representation capacity measured by entropy is maximized under the given computational budget; 2) Each layer is assigned with a proper quantization precision; 3) The overall design loop can be done on CPU, and no GPU is required. More impressively, our method can directly search high-expressiveness architecture for IoT devices within less than half a CPU hour. Extensive experiments on three widely adopted benchmarks, ImageNet, VWW and WIDER FACE, demonstrate that our method can achieve the state-of-the-art performance in the tiny deep model regime. Code and pre-trained models are available at https://github.com/alibaba/lightweight-neural-architecture-search.

Entropy-Driven Mixed-Precision Quantization for Deep Network Design
Zhenhong Sun'
Ce Ge!
Junyan Wang 12,+
Ming Lin
1,3+ 1,3
Hesen Chen
Hao Li"
Xiuyu Sun
Alibaba Group University of New South Wales
Ammzon.com Inc
Abstract
Deploying deep convolutional neural networks Internet-of- Things (IoT) devices is challenging due to the limited computational resources, such as limited SRAM memory and Flash storage. Previous works re-design small network for loT devices, and then compress the network size by mixed-precision quantization. This two-stage procedure cannot optimize the architecture and the corresponding quantization jointly, leading to sub-optimal tiny deep models. In this work, we propose one-stage solution that optimizes both jointly and automatically. The key idea of our approach is to cast the joint architecture design and quantization as an Entropy Maximization process. Particularly, our algorithm automatically designs a tiny deep model such that: 1) Its representation capacity measured by entropy is maximized under the given computational budget; Each layer is assigned with proper quantization precision; 3) The overall design loop can be done on CPU, and no GPU required. More impressively, our method can directly search high-expressiveness architecture for IoT devices within less than half CPU hour. Extensive experiments on three widely adopted benchmarks, ImageNet, VWW and WIDER FACE, demonstrate that our method can achieve the state-of-the-at performance in the tiny deep model regime. Code and pre-trained models are available http:://ghhhbb.commmaaabb///ggge hew.-eeeeeeaaaccccttteeeeeeeeeee
Introduction
With the attention of many intelligent IoT applications that have grown recently, the demand for low- cost and low-energy IoT devices is significantly increasing, such as tiny NPUs and microcontroller units (MCUs). These devices have hight-resource like hundreds KB on-chip memory (SRAM), MB-level storage (Flash) and low speed computing ability. Thus, how to design and deploy efficient tiny deep models for IoT devices becoming main topic Tiny Machine Learning (TinyML) [
As most IoT devices have very limited on-chip memory, one of the most critical challenges in TinyML control the peak memory during inference. Some works re-design the network architecture maintain the peak memory occupation 27
33 [6] by network transformed techniques, in which accessible and efficient technique is to slice input images into multiple patches that the inference can be split into multiple independent processes to meet the peak memory constraint for calculation. In addition, Neural Architecture Search (NAS) is a popular tool re-design tiny with an elaborate search space. After re- re-designing the
39 TO 17
deep models automatically [10
17
33
16
Work done in Alibaba Group.
t CCorespondence to: Xiuyu Sun <xiuyu.sx@@alibe www.:ussxeelllbbbbbbi...ooomm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Quantization Entropy
Gaussian Initialization
Resource Maximization
x (i
wi - o
ow
QE-Score
ACC
4 Callbration
QBR
w - (u
Evolutionary Algorithi
821 b (Mb)el lorie))
Score
Figure 1: Illustration of our proposed strategy for deep network design on IoT devices, which consists of quantization entropy score, Gaussian initialization calibration, and resource maximization.
neural architecture, the promotable stage is to reduce the memory occupation via mixed-precision Nevertheless, the incoherence of such a two-stage design procedure leads to the inadequate utilization of resources, therefore producing sub-optimal models within tight resource requirements for IoT devices.
quantization [34 36 I
35
10
26
38 4
To compensate for the sub-optimality of the two-stage design, this work studys how to optimize quantization in more efficient manner. Re-
the network architecture jointly with cently, traiiing-free based approaches [22 6 32 31 have emerged for neural architecture search, alternative proxy to rank candidate networks instead a training-basee based accuracy indicator, they achieve commendable effec- tiveness for the full-precision model design. Hence, considering the progress of the proxy mechanism, training-free technology could provide hands-down feasibility for integrated architecture searching, but there is still lack of key techniques for cooperating mixed-precisioe quantization. Thus, explore advancing the training- free technology to optimize both architecture and quantization in one-stage lightweight way.
accelerating the progress of the model design. Owing to constructing
Inspired from [II, deep neural network can be regarded as an information system, and the information of the system can be proportional to the entropy of the last feature map, which reflects expressiveness of the network. Further, we reformulate the informationalized network with mixed-precision quantization and measure its entropy the proxy for the performance of the quantization network. Concretely, we first comprehend the quantization effect on the entropy of the information system and propose Quantization Entropy Score (QF-Score) to calculate the entropy. Then, according to the study of how Gaussian initialization on variables affects the value QE-Score, design grid calibration process to determine the Gaussian initial values. Finally, Quantization Bits Refinement (QBR) strategy embedded the evolutionary algorithm is introduced to maintain weights and lower-precision precision activations in the front few-layer, while the opposite is in the latter few layers. The entire process illustrated in Figure
The key contributions of our work are summarized as follows:
To the best of our knowledge, we first present the ranking strategy of mixed- -rexis-ireeiioo quantization networks in the entropy view. And Quantization Entropy Score is proposed with calibrated initialization to measure expressiveness of the network.
Quantization Bits Refinement is proposed adjust mixed quantization bits, which can ensure that each layer is assigned with proper quantization precision, and maximize the utilization of memory and storage resources on the IoT devices.
Benefitting from the QE-Score, half a 64-core CPU hour. Extensive experiments demonstrate that our searched model can achieve ffate-of-the-as art performance on IoT devices for both classification and detection tasks.
approach can achieve architecture searching within less than
2 Related Work
Tiny Deep Models. Recent attempts [27 39 10, 17 33, 16
have been made to design efficient neural networks by reducing the peak memory of the front layers intuitively. Some works 22 39 [6]
aim to reduce the resolution of the front layers by slicing the image to several patches. For instance, RaScaNet [39] learns the representation of the whole image using a recurrent neural network through raster-scanning image reading pipeline. Others works [10 17
33 [6] focus investigating Neural

---

Architecture Search (NAS) to design architecture under the peak memory budget of IoT devices. For example, MCUNet series [6] address that automatically optimizes the search space to adapt the tiny resource constraints. Then, NAS search is performed for handling the tiny and diverse memory constraints on various microcontrollers. These works re-design the network architecture to maintain the peak memory occupation, however the mixed-precision quantization strategy is not considered to further reduce resource utilization.
Mixed-precision Quantization. Mixed-precision quantization search
10 26
38 4 can reduce the peak memory, flash storage and computation cost of neural networks. Some methods [4 36 26 4 focus on mixed-precision quantization on specific models (i.e., ResNet [
and MobileNetV2 (28 without jointly combining with the architecture design. For example,
DNAS [6]] formulates quantizing different layers with different bit-widths as neural architecture search problem, and proposes a novel differentiable neural architecture search framework to efficiently explore it with gradient-based optimization. Other works [3 I0, 38 attempt to jointly optimize the network architecture on cloud and mobile models and quantization by training each candidate network or big supernet on various candidate architectures. For instance, APQ [35] conducts a joint search between neural architecture design, pruning policy, and quantization policy, and proposes a rredictor-transte method to tackle the high cost of quantization-aware accuracy training. However, the architecture design in these methods pays less attention to the constraints of the IoT devices. Therefore, how to jointly design the tiny deep models and do mixed- seeecddprecison quantization is still worth exploring.
Training-free NAS methods. Some attempts 22
proposed training-free strategies for architecture searching, which construct an alternative proxy to rank the initialized networks without training. For example, SynFlow 32 preserves the total flow of synaptic strengths through the network at initialization, subject to a sparsity constraint as the proxy. Specially, Zen-NAS [[8] uses the gradient norm of the input image as ranking score, and MAE-DET (3] estimates the differential entropy of the last feature map to represent the expressiveness of a network, based on the maximum entropy theory [24). However, these methods construct the proxy full-precision models, and can't 't directly indicate the correlation between mixed-precision quantization and accuracy. To address the above issues, work aims to optimize the network architecture jointly with mixed-precio quantization in one-stage, considering the lightweight neural architecture search approach.
32
18
31
3 Problem Statement
3.1 Constraints of IoT Devices
Although significant research has been done to investigate architecture searching for IoT devices, designing models under limited resources remains a challenging issue. Limited on-chip SRAM memory and Flash storage are the major constraints on 1oT devices, especially for MCUs. For example, of-te-of-the-ar ARM Cortex-M7 MCU merely has 512kB SRAM and 2MB Flash, which is impossible to run the off-the-shelf ResNer-50 [3] or MobileNetV2 28 As typical tiny model, full-precision MobileNetV2 needs 5.6M peak memory and 13.5M storage, while the int8-precision version needs 1.4M peak memory and 3.4M storage [17], which far exceeds the limitation of mainstream IoT devices. The distribution of memory and model size of MobileNetV2 is illustrated as shown in Figure 2 On the other hand, the limited resource budget might be satisfied by lower precision quantization, but the performance of the models will drop more.
12 16 for deploying deep learning models
3.2 Low-precision Quantization
To analyze the performance drop, we conduct an overall low-precision quantization [9] on Mo- bileNetV2 except for the first and last layers, which follow the common practice of using 8-bit for high accuracy [25 42 2 19. As shown in Table T and Figure we observe that lower bit precision lower accuracy. Only "A3W2" model fits both 512KB memory limit and 2MB storage limit with 23.17% ACC decline, which uses 3-bit for all activations and 2-bit for all weights. However, in "A3W2" Model, the activations in latter stages and the weights in front stages can utilize larger bits quantization, which change less memory and storage according to the imbalanced resource utilization in Figure Thus, the resource utilization of fixed- pred-siecision quantization is not high some stages, which means we need mixed- precision quantization to improve accuracy. Because mixed-precision
3

---

Memory Storaaee (KBB
1280
Lower Lower
MBV2-INTS Memory
1024
MHV2-ATW2 Menory
MBVV-INIS Modal Size
768
MBV2-A3W2 Model Size
512KB Memory. Constraint
512
Lowerj Higher
256
10
11
13
18
Stage3
Block Index
Staged
Figure 2: Imbalanced memory and model size distribution on MobileNerV2 (28
Top five blocks
determine the memory bottleneck of the entire network due to high resolution, and last five blocks occupy 80% model size due to higher output channels. "A3W2" model could satisfy both 512KB memory limit and 2MB storage limit, which uses 3-bit for activations and 2-bit for weights.
quantization uses lower bit on tight-resource position and higher bit on rich-resource position to ensure precision stability. Therefore, we should apply neural architecture search on mixed-precision quantization for selected IoT devices.
Table I: TOP-1 ACC of fixed-precision MobileNetV2 models ImageNet with 120 training epochs. Bold values meet the 512KB SRAM limit, and underline values meet the 2MB Flash limit.
1280
1024
512KB Constraint
Memory
512
Activation Bit
Weight Bit
A2
A3
A4
A5
A6
2
3
4
5
6
8
Model Size (MB)
2MB Constraint
3
47.43
59.38
62.78
63,59
64.06
64.28
4
55.54
64.74
67.78
68.50
68.59
68,94
5
56.66
66.31
69.23
69.75
69.99
70.25
6
57.73
66.62
69.11
70.00
70.07
70.48
Figure 3: Peak memory and model size of
8
57.89
66.69
69.25
70.02
70.17
70.60
fixed deed-precision MobileNetV2 models.
4
Quantization Entropy
In this section, we propose a Quantization Entropy Score for measuring the expressiveness of the mixed-precision quantization model.
4.1 Maximum Entropy for Full-precision Models
According to the Maximum Entropy Principle [5] and its successful deep learning applications 29
5, 40, 31.. a deep neural network can be regarded as an information system, and the differential entropy of the last output feature map represents the expressiveness of the system. Given by the following theorem:
Theorem For any continuous distribution P(x) of mean μ and variance o², its differential entropy is maximized when Pzx) is a Gaussian distribution ((,, d²).
The differential entropy of a distribution is upper bounded by Gaussian distribution with the same mean and variance. Suppose x is sampled from Gaussian distribution N(4, o²), the differential entropy (24] H of is then:
+.00
1(x)
-log(p(z))p(z)dd da log(o²))
where p(x) represents the probability density function r. Moreover, the entropy H represents the expressiveness of a deep system, which correlated with the performance of a deep neural network [3I]. Specifically, given convolutional network with L layers of weights W ....... W which are initialized from standard Gaussian distribution, the forward inference is given by:
r' = * -1) for L,
(

---

where z' denotes the 1th layer feature map and o(-) represents the activation function. For simplicity. the bias of the convolutional layer is set to zero and the activation function omitted. As we initialize the input x: from Gaussian distribution, the upper bound entropy H(F) of the network FF proportional to lxz() where o² (x*) represents the variance of the x² and is computed by the forward inference. To this end, we can measure the expressiveness of full-precision networks without training and inference. The detailed description can be found [3].
Further, according to the product law of expectation (33 and Bienaymé's 's identity in probability theory 20], we can obtain the expectation and variance element of x layer feature map in our system by:
K4 Σ h=
E B(x:) = 0, x²(x)))
0 (achhww)) x o² ( dhwe)
(3)
where {K), K.) represents the kernel size of the first layer in the CNN, and C denotes its input channels size. Further extend to the 1-layer convolution, we obtain corresponding layer's expectation and variance:
Kj Σ h=1
K
Cl-1
B(x!) 0, (x
0 (1'cthe) o²
(W chnc)
(4)
The detailed derivation process is present in Appendix A. Next, we will explore how to introduce quantization into the calculation process.
4.2 Quantization Entropy for Mixed-Precision Models
Mixed-precisiee quantization to the network F requires inserting low-precision conversion behind Gaussian initialized input and weights. Learning from the practices of entropy coding [30 24 with fixed quantization step size of 1, given the variable R to quantize symmetrically, the conversion of N-bit quantization representation [26] can be defined as:
Q round - 2-1 2N 2 1)),
where clamp returns R with values below -2 to 2~-1--1 According to the work of [31] the input of each layer is zero-mean distribution when deriving the entropy, so that the upper bound of Q quantization value Q can be given by:
-2 and values above 2 1 set
2N-1, Then, the variance of the
a o²(NN) = Σ [P(Q) x - 12) P(Q) = 1 Rr
P(R),
(6)
JR.
where Q E [-2NN1 2N11 and 82 denotes the truncated variance of Q, P(Q) is the probability of Q relied on the distribution of R. Therefore, P(Q) equals to: (1) [1) - (R), when Q -2N-11 (2) f P(R), when-2N-1 N- Q N-1, (3) 1 P(R), when Q = 2N-
As shown in Figure variance of the variable. Given a Gaussian distribution with a and N bit precision, the decrease is fixed, so that we can pre-compute the decrease, which will speed up the computation of Eq. 6 The pre-computed quantization standard deviation ((N) is demonstrated in Table 2
figure out that the quantization of Gaussian variable will decrease the
Then, apply the quantization to Eq. 4 the o²(xz) of l layer become larger during the depth increase. As we set the quantization step as in Eq. 5 the distribution in Figure 4 will be much smoother, and the probability will close to O. So that the different bits of quantization can not be distinguished in the next layer. If we use dynamic quantization step in each layer to avoid this phenomenon, the computation will be more complex. Therefore, we adopt a scaling parameter o3 (3 to normalize the Tj to σ instead of adjusting the quantization step. Following the weights obey (O ow)
18 31,, then use N!-1 and No, bit to quantize the l-layer's input and weight respectively, x
Eq. can be revised as
K| kf_
0² da(x))
82 (2( x (N x (o)
(3(
(a (7)
5

---

0.08
231
431
Table Look up table of ((N) according to and N bit. Low-precision leads to small variance.
Probabiity
112
-81
125
0.04
0.00
N Bit Precision
Values
1.00
1.04
1.04
1.04
1.04
1.04
1.04
Figure 4: N = {2, 4, 8} bit quantization on Gaus- sian variable. The upper and lower bounds repre- sent truncation. Shaded areas represent quantiza- tion.
1.47
1.94
2.02
2.02
2.02
2.02
2.02
1.73
2.89
3.85
4.01
4.01
4.01
4.01
1.82
3.26
5.04
5.96
6.00
6.00
6.00
Subsequently, the variance x. can be obtained by propagating the variances from previous layers:
o² = (2(
III
(8)
1=1
where x is initialized by 0... Refer to the Eq.
I the upper bound entropy H (F) is proportional to:
H(F) & Σ log C" - (002 (NWW 3/o + +lg((d2);
(9)
where (N(N and 6² (Nw) are calculated by Eq. with initial standard deviation σл and ow. We name the value of Eq. as Quantization Entropy Score (QE-Score), and detailed derivation is intro- duced Appendix A. As the QE-Score only de- pends the structural parameters of the network, quantization function, and initial standard deviation a and dw. to rank candidate networks during search, which greatly reduces the dependence hardware. In the next section, we will show how to determine the initial standard deviation σ and ow.
70
TOP-1 ACC (59
ATM S794
55
ASW; 271-551
234-588
compute QE-Score CPUs
10
14
16
18
BitOps
Figure 5: ACC vs. BitOps. "AXWY" represents X-bit for all activations and Y-bit for all weights. A4W3 model has better accuracy than A3W4 model.
4.3 Gaussian Initialization Calibration
TOP-1 ACC
A3986
AGW3
60
60
60
A383
W2AR
50
Al
50
50
1000
Scures U6 OA and Owy
Scores @ 0A-2 0y-
Scores @ Ox and mw
TOP-1 ACC (%)
70
70
60-
50
50
800
1000
1200
1400
900
1000
1100
1200
1300
800
900
1000
1100
1200
1300
@ d and Ggy
Scores
@ C Ow
Figure 6: Gaussian initialization calibration between the accuracy and QE-Score on fixed- precision MobileNetV2 models.
We apply the commonly used MobileNerV2 architecture with fixed-precision quantization for cali- bration, and these models have been summarized Table According to the relationship between accuracy and computation in Figure 5 and the pre-computed quantization standard deviation & (NV in

---

2 we can figure out that: 1) Increasing the initial standard deviation o will help us distinguish the disparity entropy between different bits; 2) Quantization on activations and weights has differ- ent effects on accuracy. Therefore, we set different values of σ and ow to show differences between activations and weights. In summary, we use a grid calibration process to seek for the appropriate initial A and dw. and demonstrate the calibration process in Figure [
According Figure we observe that when gradually increasing the values of σ and ow, QE- Scores are gradually positive correlated with accuracy until σ ow = 4. Then, we adjust aA and dw rank the diversity of activations and weights on accuracy. Benefit from the Gaussian initialization calibration, QE-Score can help us rank various architectures without training and evaluating during search, and design eigeessxressivenesss architecture for IoTT devices.
4.4 Resource Maximization for IoT Devices
We apply Evolutionary Algorithm (EA) along with fow-precision quantization to obtain the optimal architecture by the proposed QE-Score for IoT devices. Based the observation in Figure 2 the front few layers dominate the peak memory of the entire model due to high resolution, and the distribution of model weights generally shows upward trend with the increase of feature map channels. Considering the tight SRAM memory and Flash storage constraints of loT devices, weights and lower-precision activations in the front few layers is an eligible strategy, while it is the opposite in the latter few layers. Therefore, we propose a quantization bits refinement strategy (Figure 7 embedded in the EA to realize it, to assess the bit distribution and make appropriate adjustments to maximize resources utilization. The detailed algorithm is introduced in Appendix B.
maintaining higher-preciste
8bit
Greater
Smaller
8bit
6bit
Peak Memory vs. Budget
6bit
2bit
2bit
Higher Peak Memory
Lower Peak Memory
FC
Input
Lower Model Size
Higher Model Size
Block
8bit
6bit
Model Size vs. Budget
6bit
2bit
Smaller
Greater
2bit
Figure 7: Quantization Bits Refinement (QBR) strategy for a candidate architecture redistribute the mixeedprecisions. For activations, we scale the mixed-precision to make the peak memory meet the budget. Accordingly, we also increase or decrease the mixeedprecision smallest blocks or the largest block separately to guarantee the model size approaches the budget. Orange curve arrows in the four corners mean the adjustment scale of the precision value from bit to bit or 8 bit to bit.
the weights from the
5 Experiments
5.1 Implementation Details
Searching Settings. The evolutionary algorithm and super-block definition follow the work of [I] The evolutionary population N is set as 512 with total 500000 iterations. To compare with common mix- mix-qqantization search (34 36 II 35) and IoT device applications [6 27 39), build our network with MobileNetV2-basee blocks, using the inverted bottleneck block with expan- sion ratio in [1.5, 6j. Specifically, low-precision values are random selected from (2, 3, 4, 5, 6, 8) in our search, and three layers in each block share the same precision value.
Datasets and Training Settings. We use two standard benchmarks in this work: ImageNet g and Visual Wake Words (VWW) 77. For ImageNet- IK dataset, our models are trained for 240 epochs without special indication. All models are optimized by SGD with batch size of 512 and Nesterov momentum factor of 0.9. Initial learning rate set to 0.4 with cosine learning rate scheduling 21), and the weight decay is set to 4c-6. Besides, to validate the expansibility of our method, we further
7

---

evaluate it on the WIDER FACE [3 object detection dataset. When training with low-precision do the quantization. For VWW and WIDER FACE, we follow the training settings in 16 More implementation details are introduced in Appendix C.
values, we follow the implementation
5.2 Mixed- Pixcis-recision Comparison
To compare with other mixed-precision search methods 34 36
we conduct our searching on the MobileNerV2-lene under the computation budget (BitsOps), and the BitsOps budget is 19.2G to compare with MobileNerV2-Sbii, and 7.0G for MobileNerV2-44ii..
1I
TOP-1 ACC (%6)
70-
MosileNeV2--bts
70
MchileNerV2 his
Random Sumplin
68
Random Sampliss
66-
66
10P-1
64-
64-
62
62
60
1150
1200
1250
1300
1350
60
Scores
BitOps
Figure 8: Correlations between TOP-1 accuracy and QE-Score with 100 random sampled networks. Green circle points represent 1OP-10 score models in all random samples.
Random Correlation Study. We randomly selected 100 models without QE-Score under the same searching space to verify the correlation between our QF-Score and accuracy on ImageNet, before comparing with SOTA methods. Results are shown in Figure X and we can observe that: 1) The proposed QE-Score is closer to the linear growth relationship with TOP-1 accuracy than BitOps; 2) The searched mixed-7.06 model outperforms the best random model with TOP-1 accuracy nearly 2%. According to these observations, our QE-Score is valid to rank various architectures instead of training and evaluating.
Comparison with SOTA Models. Table presents the results of fixed- and mixed- precision searching. Obviously, our models outperform state-of-the-arr mixed- peeeeiiin methods. In particular, our mixed-19.20 model has 2.9% accuracy boost than MobileNetV2-8D5 baseline, and our mixed- 7.0G model has better accuracy than MobileNerV2-4hii baseline (from 68.9% to 70.8%). Note that, as we apply formula calculation rather than inference, our QE-Score can use CPU instead of GPU. This will significantly reduce the dependence on hardware. More impressively, the marginal cost of CO2 emission of our method is two orders of magnitudes smaller than other work, benefiting from computational efficiency.
Table 3: Comparison with state-of of-the-art efficient models with mixed-precision quantization. MBV2- 4bit use 4-bit for the overall layers except for the first and last layer. t: 64 cores of Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz.
Search
Design Cost
Model Size (MB)
BitOps
ImageNet
CO2e
Model
Quant.
Devices
(hours)
(G)
TOP-1
(marginal)
MBV2 28
8-bit
3.4
19.2
71.9%
MBV2 128
4-bit
2.3
7.0
68 98.9%
MBV2+HAQ
mixed
GPUs
96N
71.9%
27. 27.23N
DNAS 36
mixed
GPUs
300N
57.3
74.0%
11.34N
SPOS
mixed
GPUs
288+24N
51.9
74.6%
82+6.811 81N
APQ
35
mixed
GPUs
2400+0.5N
16.5
74.1%
672+0.144 14N
APQ
35
mixed
GPUs
2400+0.5N
23.6
75.1%
67240.144
Ours-19 19.2-
mixed
CPUst
0.5N
3.2
18.8
74.8 %
0.19N
Ours-7.0G
mixed
CPUs+
0.5N
2.2
6.9
70.8 %
0.19N

---

5.3 Tiny Image Classification
Large-scale Classification on ImageNet. We searched tiny models under three hardware specifi- cations, and comparative results with recent state-of-the-ar ML solutions are shown in Table 4 We can figure out that our model outperforms others on all three SRAM/Flash budgets. Under the tight constraints of 256kB SRAM and IMB Flash, our model significantly improves the TOP-1 accuracy by 6.2% and 4.5% over 8-bit and 4-bit quantized MCUNet, respectively. Under 512kB SRAM and 1MB Flash, our model achieves a new record of 72.8% TOP-1 accuracy. We think these remarkable achievements benefit from improved SRAM/Flash utilization, which means more accurate mixeddprreisste quantization can specialize higher-capacioo structures on resource-constraing IoT devices. Note that MCUNetV2 employed patch-based inference scheduling to reduce memory usage, which is an orthogonal technique to mixed-prec precision quantization.
Table 4: Comparison of ImageNet classification accuracy on IoT devices
256kB SRAM, 1MB Flash
320kB SRAM,
1MB Flash
512kB SRAM, Flash
Model
Quant.
Mem
Size
Acc.
Mem
Size
Acc.
Mem
Size
Acc.
MBVI 14
26
mixed
<256kB
<IMB
60.2%
<512BB
<2MB
68.0%
MBV2 28
8-bit
308KB
0.72MB
49.0%
Proxyless
8-bit
292kB
0.72MB
56.2%
MCUNct-int8 17
8-bit
238kB
0.70MB
60.3%
293kB
0.70MB
61.8%
452kB
L.65MB
68.5%
MCUNct-int4
4-bit
233kB
0.67MB
62.0%
282kB
0.67MB
63.5%
498XB
L55GBB
70.7%
MCUNetV2
1T6
8-bit
196kB
0.79MB
64.9%
465kB
L667BB
7188%
Ours
mixed
253kB
0.73MB
66.5%
308kB
0.7IMB
68.2%
507kB
1.67BB
72.8%
Low-energy Application on Visual Wake Words. The Visual Wake Words dataset (VWW) can represent a realistic IoT use-case of identifying the presence persons. Experimental results on VWW dataset are shown in Figure We can observe that model is superior to MCUNet in both accuracy and memory utilization. The accuracy is boosted to 93% with slightly fewer memory requirements. To be comparable with MCUNetV2, we used the same patch-based inference scheduling to further reduce the memory footprint. Without accuracy loss, the runtime peak memory can be decreased by about 3.5x.
94
$21B
Table 5: Strategy studies for ImageNet classification on loT devices.
VWW Anc (%)
92
Mctive
30-
SRAM Budget, Flash Budget
Model
256kB, 1MB
320kB, IMB
512kB, 2MB
84
60
120
150
240
300
360
Ours-int8
62.7%
63.9%
70.4%
Pesk R.M
Ours-int4
64.4%
65.5%
71.0%
Figure 9:
Comparison of different MCU
Ours-mixed
65.6%
66.6%
71.7%
models on Visual Wake Word (VWW) ac- curacy within 256KB peak memory.
w/o QBR
Ours-mixed
66.5 %
68.2%
72.8%
Resource Maximization. Table 5 our final model with fixed-bit counterparts and shows the effectiveness of QBR. Although 4-bit quantization better than 8-bit quantization, there is still a gap between fixed- and mixed- precision quantization. Note that even without QBR, our mixed-bit bit network superior to fixed-bit MCUNet and MCUNetV2 in Table Fig. 10 visualizes peak memory, bits distribution, and model size of our searched model under 320KB SRAM and 1MB Flash. Comparison of models with or without the QBR displayed in Figure IT Obviously, our method maintains higher-precision weights and power-pre activations in the front few layers, while opposite in the latter few layers. This demonstrates that QBR can strengthen resource utilization by explicitly embedding prior design knowledge.

---

Medel tot
Menoryy or Storage (K33)
64
RAM solo QH12
8 8
Stregg
Weights
11
13
15
16
17
18
Indox
Bluck Indox
Figure 10: Peak memory, bits distribution and models size of mixed-precision model.
Figure 11: Model size and Peak Memory of models with or without the QBR.
5.4 Tiny Object Detection on WIDER FACE
Table Comparison of face detection on WIDER FACE, The hard subset the most authoritative benchmark since it contains the faces in easy and medium subsets [19].
mAP
Model
Peak RAM
MACs
Easy
Medium
Hard
EagleEye
1.7MBB
0.08G
0.74
0.70
0.44
RNNPool P7
1.17MB
0.10G
0.77
0.75
0.53
MCUNetV2 [6
762kB
0.11G
0.85
0.81
0.55
Ours-Face
650kB
0.04G
0.82
0.81
0.77
The proposed QF-Score method can be adopted to various visual tasks, and demonstrate its gener- alization ability on object detection task. Following the pipeline of [19], We conducted experiments on WIDER FACE [37] and results are shown in Table We follow the quantization strategy in [27 that memory usage is analyzed for detector backbone and reported in FP32 to build our detection model "Ours-Face". According
Table we observe that our model can achieve competitive mAP performance at all three subsets with 1.8x low peak memory and 2x less computation than EagleEye 41 and RNNPool (27), which verifies the generalization ability of our method.
Conclusion
In this work, we propose an entropy driven mixed-precisiin quantization strategy to address chal- lenges of deep network design on loT devices. In particular, we first present the ranking strategy of mixed-precision quantization networks in the entropy view by formulating the neural network information system, and propose QE-Score with calibrated initialization to measure the expressiveness of the system. Then we propose the Quantization Bits Refinement within evolution al- gorithm adjust mixed-precioe precision quantization. Finally, the obtained network achieve state-of-the-an performance on loT devices for both classification and detection tasks within less than half a 64-core CPU hour searching.
References
[1] Colby R Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, Robert Hurtado, David Kanter, Anton Lokhmotov, et al. Benchmarking tinyml systems: Challenges and direction. arXiv preprint arXiv:2003.0482 2020.
[2] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems, 32, 2019.
[3] Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations. 2019.
[4] Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed sxexed-precsson neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2349-
2358, 2020.
10

---

[5] Kwan Ho Ryan Chan, Yaodong Yu, Chong You. Haozhi Qi, John Wright, and Yi Ma. ReduNet: A white-box deep network from the principle of maximizing rate reduction. arXiv preprint arXiv:210.. 10446. 2021.
[6] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search imagenet in four gpu hours: A theoretically inspired perspective. In Proceedings of the International Conference Learning Representations, 2021.
[7] Aakanksha Chowdhery, Pete Warden, Jonathon Shlens, Andrew Howard, and Rocky Rhodes. Visual wake words datasct. arXiv preprint arXiv: (et::00065552 2019.
[8] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 lece, 2009.
[9] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. 2019.
[10] Igor Fedorov, Ryan P Adams, Matthew Mattina, and Paul Whatmough. Sparse: Sparse architecture search for cans resource-constrained microcontrollers. Advances in Neural Information Processing Systems.
32, 2019.
[1]] Zichao Guo, Xiangyu 7hang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path onc-shot neural architecture scarch with uniform sampling. In European Conference Computer Vision, pages 544-560. Springer, 2020.
[12] Albert Gural and Boris Murmann. Memory-optinat direct convolutions for maximizing classification accuracy in embedded applications. In International Conference on Machine Learning, pages 2515-2524.
PMLR, 2019.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint wwXiv:1704.0444 04861, 2017.
[15] Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
[16] Ji Lin, Wei-Ming Chen, Han Cai. Chuang Gan, and Song Han. Mcunetv2: deemory-efficiee patch- based inference for tiny deep learning. In Anmual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[17] Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al. Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems,
11722, 2020.
[18] Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas:
A zero-shot nas for high-performance deep image recognition. In 2021 IEEE/CVF International Conference on Computer Vision, 2021.
[19] Yang Liu, Fei Wang. Jiankang Deng. Zhipeng Zhou, Baigui Sun, and Hao Li. Mogface: Towards deeper appreciation on face detection. arXiv preprint arXiv:2103. 1139, 2021.
[20] Michel Loeve. Probability theory. Courier Dover Publications, 2017.
[21] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[22] Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowlcy. Neural architecture search without training.
In International Conference on Machine Learning, pages 7588-7598. PMLR, 2021.
[23] Alexander McFarlane Mood. Introduction to the theory of statistics. 1950.
[24] Kenneth H Norwich. Information, sensation, and perception. Academic Press San Diego, pp. 81-82, 1993.
[25] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifi- cation using binary convolutional neural networks. In European conference on computer vision, pages 525-542. Springer, 2016.
[26] Manuele Rusci, Alessandro Capotondi, and Luca Benini. Memory mixed low precision quantization for enabling deep network inference on microcontrolleers. Proceedings of Machine Learning and Systems. 2:326-333 335, 2020.
[27] Oindrila Saha, Aditya Kusupati, Harsha Vardhan Simhadri, Manik Varma, and Prateek Jain. Rnnpool: efficient non- pooling for ram constrained inference. Advances in Neural Information Processing Systems, 33:20473-20484, 2020.
[28] Mark Sandler, Andrew Howard, Menglong 7hu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenety2: Inverted residuals and linear bottlenecks. In Proceedings of the IEF.F conference computer vision and pattern recognition, pages 4510-4520, 2018
[29] Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advami, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information boutlnnck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019.
[30] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
11

---

[31] Zhenhong Sun. Ming Lin, Xiuyu Sun, Zhiyu Tan, Hao Li, and Rong Jin. Mae-det: Revisiting maximum entropy principle zero-shot nas for efficient object detection. In International Conference on Machine Learning. PMLR, 2022.
[32] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data iteratively conserving synaptic flow. In Proceedings the Interaatinnal Conference Neural Information Processing Systems, 2020.
[33] Hasan Unlu. Efficient neural network deployment for microcontroller. arXiv preprint arXiv:2007.01345
2020.
[34] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8612 8612-8620, 2019.
[35] Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang. Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2078-2087 2087, 2020.
[36] Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed preci- sion quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv: 1812.00090 2018.
[37] Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[38] Zhaohui Yang. Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. Searching for low-bit weights in quantized neural networks. Advances in neural information processing systems,
33:4091-4102, 2020.
[39] Jachyoung Yoo, Dongwook Lee, Changyong Son, Sangil Jung, ByungIn Yoo, Changkyu Choi, Jac-Joon Han, and Bohyung Han. Rascanet: Learning tiny models by raster-scanning images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13673-13682, 2021.
[40] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434, 2020.
[41] Xu Zhao, Xiaoqing Liang, Chaoyang Zhao, Ming Tang, and Jinqiao Wang. Real-time multi-scale face detector on embedded devices. Sensors, 19(9)2158, 2019.
[42] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606.06116 2016.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Present abstract and Section [
(b) Did you describe the limitations of your work? [Yes] Present in Appendix D.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Present in Appendix E.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read it and ensure that our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Present in Section and Section 4
(b) Did you include complete proofs of all theoretical results? [Yes] Present Appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] The code released described
at the end of abstract. The data is public. The instructions is present in Section 4 Section
(b) Did you specify all the training details (c.g., data splits. hyperparameters, how they were
chosen)? [Yes] Present in Section 4 Section and and Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] Error bars are not reported because would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Search uses cores of Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz and training conducted on 8 NVIDIA V100 GPUs.
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Present in Section 5.1 and Section 5.4
(b) Did you mention the license of the assets? [No] Assets in the paper are widely used, and the license is already mentioned in reference papers.
(c) Did you include any new assets either in the supplemental material as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] The data we used are all public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicaall??
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Q1Q-Oeg-VKM", "reviewer": "Reviewer_TPBM", "summary": "The paper proposes a new mixed-precision quantization in a new perspective of information entropy. Since the proposed Quantization Entropy Score can approximately measure the expressiveness of neural networks and is easy to compute, the authors can do the search in a short time.", "questions": "1. Since the search cost is very low, why do the authors expand the search space? We may expect to find better results in a larger solution space. For instance, could you relax the constraints mentioned in Line 186 (three layers in each block share the same precision value)?\n2. In Table 5, why does the QBR achieve better results? From my understanding, the QBR uses prior knowledge to shrink the search space. Without QBR, we should explore a larger space and achieve better results than that with QBR.\n3. Could the authors discuss other hardware-related metrics, such as latency, throughput, and energy?\n", "limitations": "Limitations and social impacts are well discussed in Appendix E and F.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is well developed and organized. The tables and figures are great.\n2. The authors extend the previous work of viewing neural networks from the perspective of information entropy. Since quantization can be treated as an operator, it is natural to apply this perspective to the quantization problem.\n3. The experiments are convincing. The authors cover three benchmarks.\n\nWeaknesses\n1. I am wondering if there is any work on using the entropy-based scores to conduct neural architecture search (NAS). If yes, the related work should be discussed and compared. If no, it should be a contribution of this paper. The authors should also show that the score can be used to search for a full precision model efficiently.\n2. How can the proposed method extend to other machine learning models, such as Transformers?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LsEb7vRccOI", "reviewer": "Reviewer_gBWZ", "summary": "This paper proposes an entropy-driven mixed-precision quantization strategy. The basic idea is to formulate the neural network as an information system, and then define a quantization entropy score to capture the expressiveness of the system. Finally, the Quantization Bits Refinement algorithm quantizes the activation and weights to meet the peak memory and model size constraint.", "questions": "1. The QBR strategy looks heuristic, is it possible to fall into an oscillation process?\n2. Why only 8 bit and 2 bit in Figure 7?", "limitations": "The proposed QE-scores can only use CPU resources instead of GPU.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The concept of the quantization entropy score looks interesting.\n2. The proposed solution outperforms SOTA works.\n\nWeakness:\n1. The proposed solution does not work for GPU.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J0PfAuR_Z6g", "reviewer": "Reviewer_Aoyu", "summary": "Previous works use an inefficient two-stage method that quantizes after designing a small network. However, the proposed method uses a one-stage method that can jointly optimize network design and mixed-precision quantization. Therefore, it is not only efficient in terms of time-cost, but also achieved high performance in various tasks.", "questions": "1) Can you provide applicability and experimental results for networks other than MobileNet-V2?\n2) Can you show additional experimental results related to Weaknesses 2), 3) and 5)?\n3) Can you provide additional experimental results and explanations regarding the compatibility of the proposed method?\n4) Can you provide additional explanations related to Weaknesses 6) and 7)?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The ranking strategy from the perspective of entropy and Quantization Bits Refinement in the evolutionary algorithm considering the HW budget are interesting and novel.\n2) This paper clearly defined the problems and demonstrated solutions through solid mathematical analysis. \n3) This paper is very clear and easy to read. I think that it would help both researchers related to this field and general readers to give a little bit more background on TinyML.\n4) The experiment showed the superiority of the proposed method on various datasets (ImageNet, VWW, WIDER FACE).\n\nWeaknesses:\n1) For all experiments, MobileNet-V2 was used as the base model. In particular, the analysis of peak memory and model size presented in Section 2 is also limited to MobileNet-V2. However, when a different model is used, the peak memory for each layer will also be different. Can we assume that the results of MobileNet-V2 are representative? The authors should show that the proposed method is compatible with other models.\n2) Since the key contribution asserted in this paper is the 1-stage approach, it is necessary to thoroughly compare the proposed method with the 2-stage methods. In other words, comparative experiments with more combinations of SOTA network design studies + SOTA mixed-precision studies should be presented.\n3) Experimental results in terms of power (energy) consumption should be presented. As the authors know, the most important factor in an IoT device is energy consumption.\n4) It is necessary to explain whether the proposed method is compatible with more complex networks (e.g., SSD, instance segmentation, Transformer). (If possible, the experimental results need to be presented together.) Also, it is difficult to understand which network was used for the current Tiny Object Detection experiment.\n5) [1] was used as a quantization-aware training (QAT) method. However, model performance may be improved by [1] in this paper. So, the current results cannot clearly show the effectiveness of the proposed method. Further experiments excluding the impact of QAT should be presented.\n6) In this paper, only SRAM and Flash are used as constraints, but it seems that DRAM should be considered.\n7) Why didn't the authors use the commonly used bit-precision configuration {2, 4, 8, 16} in the mixed-precision setting? Additional explanations related to this should be provided.\n8) In Table 3, the appropriate reference should be added on which tool was used to measure CO2 emission.\n\n[1] Learned step size quantization. ICLR’20.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DNsmRaM7D5v", "reviewer": "Reviewer_xZvK", "summary": "This paper presents an entropy-driven mixed-precision quantization strategy to design efficient CNNs for resource constrained IoT devices. The paper proposes a new metric, Quantization Entropy Score (QE-Score), to evaluate the CNN model capacity without the need of training the model. QE-Score is applicable when different layers are quantized to different bit widths, and thus can significantly speedup the design space exploration for efficient models. Extensive experiments have been performed to show that the proposed quantization strategy discovers tiny models with higher accuracy under strict resource constraints. ", "questions": "Please clarify the questions regarding the technical descriptions (see comments on “Quality”). \n\nPlease provide more details on the main algorithm steps (e.g., Quantization bits refinement, Evolution algorithm). \n", "limitations": "The paper does not discuss potential negative societal impact. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nWhile the connections between deep learning models and information systems have been studied and the entropy has been related with accuracy in the literature, it appears to be an original idea to evaluate CNN model capacity via a single QE-Score metric under the quantization scenario. \n\nThe discussion on the relation among quantization budget, neural architecture, initialization and entropy (accuracy) is original and useful. \n\n\n### Quality\n\n**Strengths**\n\nThe proposed QE-Score seems to be a simple yet effective metric to estimate the model capacity. This shows potential for practical IoT deployment. \n\nThe evaluation of the proposed strategy is extensive and covers various aspects. \n\nNecessity of mixed-precision quantization is clearly illustrated by the MobileNet layer breakdown in Fig 2. \n\n\n**Weaknesses**\n\nVarious technical descriptions in Section 3 are unclear, and I am challenged to interpret some of the plots. For example:\n* Equation 6: what variable is the summation applied on? Q based on its different value ranges?\n* Fig 6: it appears to me most subplots can show positive correlation between QE-Score and accuracy. What is so special about the plot of $\\sigma_A=5$ and $\\sigma_W=4$ plot? Why do you say that \"QE-Scores are gradually positive(ly) correlated with accuracy until $\\sigma_A=\\sigma_W=4$ Then, we adjust $\\sigma_A=5$ and $\\sigma_W=4$ to rank the diversity of activations and weights on accuracy\"? What do you mean by ranking the diversity of activations and weights?\n* The Quantization Bits Reinforcement (QBR, Fig 7) strategy is very unclear. Where do you explain this figure? Why do you need this refinement when you already have QE-Score to guide the allocation of bit-widths? You should at least briefly introduce this algorithm in the main text. \n\nThe fundamental assumption on the system is unclear: To execute the CNN on IoT devices, do you need the full model to fit in its memory? Or do you only need a single layer (the currently executed layer) to fit in the memory? From Fig 2, it seems that you are imposing a layer-wise constraint. From other parts of the paper, it seems that the constraints are on the full CNN. \n\nFrom the description, it seems that the main purpose of QE-Score is to simplify neural architecture search (e.g., channel widths, bit widths). Then why does initialization of X and W matter? Specifically, since you will train the model after fixing the architecture, the effect of different initialization scales will be automatically addressed by batch-norm or layer-norm. \n\nWhile the empirical results seem to justify QE-Score, its derivation seems problematic. Activation has a major impact on the distribution. For example, with ReLU, half of values will be cropped to 0 if the input follows 0-mean Gaussian. With a deep CNN, the final layer distribution will be far from Gaussian. Thus the derivation of QE-Score does not follow a realistic assumption. \n\n\n### Clarity\n\nThe initial portion of the paper is clear to motivate mixed-precision quantization. However, the technical description is overall unclear (see details above). \n\nTables and figures in experiments clearly show the advantages of the proposed strategy over the baseline w.r.t. various different metrics. \n\n\n### Significance\n\nJudging from the empirical gains, the proposed method makes a significant contribution to enable efficient IoT inference with compressed models. \n\nHowever, evaluations are mostly performed on a single backbone model (MobileNet). It is unclear if the proposed method can generalize to other CNN or non-CNN models. \n\nThe technical significance seems to be limited. First of all, the QE-Score and the proposed algorithm are both mostly heuristic based. Secondly, due to the questions on the technical sections (listed above), it is unclear if the proposed method brings in-depth insights to the community. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept