AutoQ: Automated Kernel-Wise Neural Network Quantization 
Authors: Qian Lou
Feng Guo
Minje Kim
Lantao Liu
Lei Jiang.
Venue:  (2020)
Keywords: AutoML
Kernel-Wise Neural Networks Quantization
Hierarchical Deep Reinforcement Learning

Abstract: Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

Published as a conference paper at ICLR 2020
AUTOQ: AUTOMATED BERNEL-WIST NEURAL NETWORK QUANTIZATION"
*
Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang
(louqian, fengguo, minje, lantao, jiang60feiu.ed. Indiana University Bloomington
ABSTRACT
Network quantization is one the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) low-power mobile devices. Recent network quantization techniques quantize each weight kernel a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts deep reinforcement
obtain only sub-optimal results. It is difficult for learning (DRL) Deep Deterministic Policy Gradient (DDPG)- based agents to find kernel-wise QBN configuration that can achieve reasonable inference accuracy. kernel-wise network quanti-
In this paper, we propose a hierarchical-DRI-basse zation technique, AutoQ, automatically search QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quan- tized by the ofate-of-the-art DRL-based schemes, the same models quantized by AutoQ reduce the inference latency by 54.06% and decrease the inference energy consumption by 50.69% averagely, while achieving the same inference accuracy.
INTRODUCTION
Although convolutional neural networks (CNNs) have been the dominant approach (Sandler al., 2018) to solving a wide variety of problems such as computer vision and recommendation sys- tems, it is challenging deploy CNNs to mobile devices having only limited hardware resources and tight power budgets, due to their huge essential computing overhead, e.g., an inference of Mo- bileNetV2 (Sandler al. 2018) involves 6.9.M weights and 585M floating point operations.
Several approaches such as pruning (He et al. 2018) and low-rank approximation (Denton et al., 2014) are proposed to reduce the inference computing overhead of CNNs. Network quantiza- tion /Wang et al., 2019) Lin et al., (017)) becomes one of the most hardware friendly CNN ac- celeration techniques by approximating real-valued weights and activations to QBN fixed-point representations, and performing inferences using cheaper fixed-point multiple-accumulation (MAC) operations, where QBN is the quantization bit number.
Instead of using one QBN for the whole CNN, the layer-wise network quantization (Wang et al. 2019 Elthakeb et al. 2018) assigns a QBN to the weights of each convolutional layer, and searches another QBN for the activations of the same layer to decrease the inference computing overhead. But the inference cost of the layer-wise quantized CNNs is still prohibitive for low-power mobile de- vices powered by batteries. Recent works (Zeng al. 2019 Choukroun et al. 2019b Zhang al.)
2018
et al. 2019 Krishnamoorthi 2018 Sasaki et al. 2019) find that various weight kernels of
"This work supported in part by NSF CCF-1908992 and CCF-1909509.

---

Published as a conference paper at ICLR 2020
Count
250
meean-8.012 atd=0.127
Count
250
means-4.05 0.005
Top-1 Accuracy
71%
std=0,
198
69%
outliers
0
-0.5
0.5
67%
Count
250
meanm-0.025 atd+0.112 128
Count
250
mean= 2000
65%
network-wist quantization
layer-wise quantization
outliers
63%
kernel-wise quantization
-0.25
0.25
0
0.0
0.5
50
100
150 Latency (ms)
200
250
Figure 1: The weight distribution of kernels.
Figure 2: Inference accuracy and latency.
convolutional layer (ResNet-18) exhibit different variances shown in Figure and hence have differ- amounts of redundancy. Therefore, they quantize each weight kernel independently for higher accuracy by calculating a Q B Neleement scaling factor vector for each kernel, rather than globally quantize all the kernels of layer as a whole. To reduce different amounts of redundancy among different weight kernels, these kernel-wise network quantization techniques should have searched a QBN for each kernel of each layer in a CNN. However, the search space of choosing a QBN for each weight kernel is too large, so prior kernel-wise network quantization (Zeng et al.) [2199] Choukroun jet al., 2019b, Zhang et al. 2018 Li et al. 2019 Krishnamoorthi 2018 (asaki et al. 2019) still uses the same QBN for the entire CNN. As Figure shows, compared to the layer-wise quantized model, on the same FPGA accelerator (Umuroglu et al. (2019)), the kernel-wisg quantized model (assigning QBN each weight kernel and choosing QBN for each activation layer) improves the inference accuracy by ~ 2% (ImageNet) with the same computing overhead (inference latency).
and
How to decide QBN for each weight kernel is the most important task of the kernel-wise network quantization, since the QBNs have a large impact on the inference accuracy, latency and hardware overhead. Determining QBN for each weight kernel via hand-crafted heuristics is sophisticated that even machine learning experts can obtain only sub-optimal results. Recent works (Wang et al.) 2019 Elthakeb al., 2018) automatically select a QBN for each layer of CNN through deep reinforcement learning (DRL) agent without human intervention. However, it is still difficult for low-power mobile devices such as drones and smart glasses adopt the layer-wise quantized CNN models. These mobile devices are very sensitive to the bit-width of fixed-poin point MAC operations and memory access during inferences due to their limited battery lifetime and hardware resources. Kernel-wise network quantization assigning QBN to each weight kernel and searching QBN for each activation layer of a CNN becomes a must to enable the efficient deployment of deep CNNs on mobile devices by reducing the inference computing overhead. Although it is straightforward to perform kernel-wise quantization via DRL, it takes ultra-long time for DRL agent to find a proper QBN for each weight kernel of CNN. As CNN architectures are becoming deeper, it is infeasible to employ rule-based domain expertise or conventional DRI--bseed techniques to explore the exponentially enlarging search space of kernel-wise network quantization.
In this paper, we propose a Sherarchical-DRL-baee search QBN for each weight kernel and choose QBN for each activation layer of a CNN for accurate kernel-wise network quantization. AutoQ comprises a high-level controller (HLC) and a low-level controller (LLC). The HLC chooses QBN for each activation layer and generates goal, the average QBN for all weight kernels of convolutional layer, for each layer. Based on the goal, the LLC produces an action, QBN, to quantize each weight kernel of the layer. The HLC and LLC simultaneously learn by trials and errors, i.e., penalizing inference accuracy loss while rewarding smaller QBN. We also build state space, a goal and an action space, an intrinsic reward and an extrinsic reward for AutoQ. Instead of proxy signals including FLOPs, number of memory access and model sizes, we design the extrinsic reward to take the inference latency, energy consumption and hardware cost into consideration.
agent, AutoQ, to automatically and rapidly
2 BACKGROUND AND RELATED WORK
Quantization. Recent works (Lin et al. 2016 Zhou et al. 2017 Jacob et al., 2018 McKinstry et al.) 2018 Zhang et al.] 2018)) quantize the real-valued weights and activations to fixed-point representa- tions, so that the model size is reduced and inferences can use low-cost fixed-point MAC operations.
To further reduce inference computing overhead, prior works (Kim & Smaragdis 2016 Xu et al.
2018 Guo al. 2017 Tang et al. 2017 Rastegari al. 2016; Lin et al. (017)) quantize weights and activations into multibitt bit binary codes of {-1, +1 +IJs. Rather than real rellevalued MACs, inferences
of these quantized models depend on bit-wise logic operations, i.e., XNORs and popcounts. These
2

---

Published as a conference paper at ICLR 2020
traditional quantization techniques either simply assign single QBN to the whole CNN or require domain experts to determine a QBN for each layer of CNN.
Table I:
The search space size of network quantization. QBN E [0,32], where 0 means the com- ponent is pruned. Player is the layer number of the network.
quantization granularity
search space size (weight activation)
network-
33 x 33
layer-wise
Mayer 33
kernel-wise
33 2-11" 1i Could X Mayr
Kernel-wise quantization. As Table [ shows, almost all prior works (Lin et al.]
2016 Kim &
Smaragdis 2016 Rastegari et al. 2016 Lin et al. 2017] Guo et al. 2017 Zhou et al. 2017 7017:011 Jacob
al. 2018 Tang al. 2017 Xu al.) 20 McKinstry al. 2018 Zhang et al. 20188) categorized
as the weeworkkwwie quantization focus on searching QBN E [0. 32 for all weights, and searching another QBN for all activations in a CNN. Totally, there are only 1089 combinations of the QBN
configuration for the network-wise quantization. The layer-wise quantization (Wang et al.) 2019) searches a Q BN [0, 32) for all weights of a convolutional layer, and decides another QBN for all activations of the same layer. The QBN search space size of the layer-wise quantization substantially
increases to 3 X 3eee where Faayerr is the layer number of a CNN. Recent works (Zeng
ct al. 2019 Choukroun et al.) 2019b Zhang et al. 2018 Li et al. 2019 Krishnamoorthi 2018
Sasaki et al. 2019) observe various weight kernels of a convolutional layer have different amounts of redundancy, and quantize each weight kernel independently for higher accuracy. To exploit differ- ent amounts of redundancy among different weight kemels,, these kernel- network quantization techniques should have searched QBN for each kernel of each convolutional layer, and assigned a QBN for each activation layer in a CNN. However, the search space size of the kernel-wise net-
work quantization 33 "Layer Conts 33* where Couti is the number of weight kernels (output channels) of the ith layer. No prior work tries to search such huge design space.
Table 2:
The comparison of DRL-based techniques for quantization and pruning. feature
AMC
ReLeQ HAQ
AutoQ
search for activations and weights
x
x
/
V
kernel-wise quantization
x
x
x
/
hierarchical DRL
x
x
7
shaped intrinsic reward
x
x
x
/
AutoML Recent works take advantage of DRL (Baker al. 2016 Zoph al. 2017), genetic al- gorithm (Suganuma et al. 2017] Stanley & Mükkulainen (2022 and Bayesian Optimization (Kan- dasamy al. [0188] Stewart & Stalzer, 2018) to automatically architect CNNs for higher inference accuracy. Their network architectures outperform many human-designed neural networks. The weight channel pruning is automatically conducted by DRL. (He et al., 2018) and genetic algo- rithm (Wang et al. 2018). ReLeQ (Elthakeb et al. 2018) quantizes only the weights of each layer of a CNN by DRL. while HAQ (Wang al. 2019) performs the layer-wise quantization for both weights and activations via a DRL agent. No prior quantization or pruning work relies on hier- archical DRL. Table compares AutoQ against prior DRL-based techniques for quantization and pruning. AutoQ is the first work to automatically quantize each weight kernel and each activation layer of a pre-trained CNN model for mobile devices by hierarchical DRL.
3 AUTOQ
Overview. We do not aim present a new network quantization technique, but we formulate the search of a QBN for each weight kernel and each activation layer as hierarchical DRL problem. We propose two-level hierarchical DRL. technique, AutoQ, to automatically quantize the weights in the kernel-wise manner and the activations in the layer-wise fashion. We build the state space, action and goal space, extrinsic and intrinsic reward functions and a hierarchical DRL agent for AutoQ. Although we use the state-of-the-art learned quantization technique, LQ-Nets (Zhang al.) 2018), to quantize weight kernels and activation layers with the QBNs found by AutoQ, future novel quantization techniques can be easily integrated to AutoQ to improve the inference accuracy of the quantized networks. In the extrinsic reward, besides the inference latency and energy (Wang et al. 2019) AutoQ also considers the FPGA overhead critical low-cost mobile devices.
3

---

Published as a conference paper at ICLR 2020
AutoQ Agent
HLC
machine learning
HLC
HLC
statep..
network &hardware configuration
hardware over- head estimator
|1
iRd.
Oucc
8..
power
LLC
GeRd
latency
LLC
LLC
Buc
Ouc
power latency area
quantized model
layer search
L,1
L,1
4
4
L+-
4
A[]-1][]]]]]]
-00111111 =2-bit
100 -2-bit
A[i+ M[+1]][|:11
=4-bit
=3-bit
-i-bit
bit
wernel-wiie search
K
K
Wiili-116-1: -2-bit
Wiil-- =1-bit
-3-bit
Figure 3: The working flow of AutoQ (HLC: high-level controller, LLC: low-level controller).
Working Flow. For an -layer CNN, the weight is defined as W E Riuyee X X Xwew x xha, where nlayer is the number of layers; Cout denotes the number of kernels (output channels); Cin means the number of input channels; ww indicates the kernel width; and hn is the kernel height. The activation is defined as A IR XCin xw, Xha, where Wa is the feature map width; and ha means the feature map height. The working flow of AutoQ is shown in Figure 3 AutoQ consists of high-level controller (HLC) and a low-level controller (LLC). The HLC quantizes the network layer by layer, while the LLC searches QBN for each weight kernel in a layer. O At first, Auto re- ceives an observation stater... from the environment that is the quantized network model, where stoterr., includes the information of the CNN architecture. @ The HLC makes a goal gr., that the QBN for the activation layer I,, The flow then jumps to O. Or the HICC generates a goal gr, which is the average QBN of all weight kernels in the layer I, for the LLC. O The LLC produces an action K,I. QBN, for the weight kernel K2 of the layer L, For the entire layer Li, the LLC aims to reach the goal gL, of the HLC. O The environment sends the network quantization and hardware wecchine-eearnng based hardware overhead estimator. & The
configuration to the fast and accuracy hardware overhead estimator returns the energy consumption, for the current quantization and hardware configuration. @ With the hardware overhead and infer- ence accuracy, the environment generates extrinsic reward eRdj,..K.) for AutoQ to evaluate the LLC action. ? Based on all actions of LLC for the layer Lg, the HLC provides an intrinsic reward iRdr, to tell how well the goal implemented by the LLC.
overhead and inference latency
State Space. A state state L., (observation) is represented by
(I) where L; the layer index; K2 means the weight kernel index; Cin indicates the number of input channels; Cout denotes the number of kernels; Skernet is the kernel size; Satride is the stride; feature the input feature map size; 6 binarily indicates depthwise convolution or not; bwra binarily represents weight or activation; - is the goal (average QBN) of the last layer; and "IL, is the action (QBN) of the last kernel the T. layer. For each variable state K,], we normalize to J0. 1). If the layer a fully-connected layer, we set Skernel = 1, Sstride = and bat = 0.
st,te = (Li, Kj, Cins Cout, Skernel: Setride: &feature: bdw bwoaas 9L. K(1))
Goal and Action Space. The HLC produces the average QBN for all weight kernels of each layer or the QBN for each activation layer as goal, while the LLC generates a QBN for each weight kernel in layer as action. The HIC goal gr., for the L, layer uses continuous space and can be any real value between and goalmaz. where goalmax is the maximum average QBN for layer and we set it to 8. If the L, layer is an activation layer, round the real-valued gL, the discrete value of roundup(1 + IL, (goalmer - 1)). Although the LLC action is an integer between 0 and actionmaz. it still uses continuous space to capture the relative order, i.e., 2-bit is more aggressive than 3-bit, where action is the maximum QBN for a kernel and we set to 8. For the K2 kernel of the L; layer, the LLC generates the continuous action ra,,,,K,] that is in the range of [0, 1). and round it up the discrete value (,L, roundup(rap..c action
4

---

Published as a conference paper at ICLR 2020
Extrinsic Reward. After action [] is taken, AutoQ arrives at a new state state K+11] and receives an extrinsic reward eRd from the environment. The HLC aims to maximize the accu-
mulative extrinsic reward eRd = =E Lj ReRd factor. The immediate extrinsic reward can be represented by
L, fowtst)- eRd(L..K,) where "eRa E (0,1) 1 is a decay
Cecuracy(NC)a
eRd L. L,,C = logg( [NC( HC) it (NC, HC)) Ve. area(NC, HC)
(2)
where NC is the network configuration; HC' means the hardware configuration, e.g., memory band- width; accuracy( NC) indicates the inference accuracy; lol is the inference latency of the network NC running on the hardware HC; en represents the inference energy of NC running on HC; area is the FPGA area (hardware cost) used by NC on HO; Vocce lt, U. and va are user-defined factors deciding the impact of inference accuracy, lateney, energy and FPGA area on the extrinsic reward. By different values of user-defined factors, Auto implements the resource-constraing and accuracy-gurronteed searches. For resource- constrre-on applications, e.g., low-power drones, AutoQ sets Macc = 1, 8 0, the 0 and v/o 0 to achieve the best accuracy given the maximum amount of hardware resources (latency, energy, and FPGA area). This extrinsic reward offers no incentive for lower QBNs, so AutoQ reduces the QBN by limiting the action space. AutoQ allows arbitrary action at the first few layers and starts to limit the action when it finds that the hardware resource budget is insufficient even after using the smallest QBN for all the following layers. For accuracy-guaranteed applications, e.g., fingerprint locks, AutoQB sets Vance = 2, t/1 < 1, the < and vha < 1 to obtain the shortest latency, the minimal energy, and the smallest hardware cost with no accuracy loss.
Intrinsic Reward. Based on the goal It. produced by the HLC for the L, layer, the LLC generates Cout actions K[],,Ko] ~ (our at the states state ~ state Ko..e-- AutoQ then arrives the state state Koott 1]. where it receives an intrinsic reward iRd and maximizes the accumulative intrinsic reward iRd E, Ridd iRdy,,, where Yidd E [0, 1) decay factor and k,yj indicates the intrinsic reward for the weight kernel K; of the layer L.. The LLC produces actions to help the HLC to maximize the extrinsic reward, so it should aim to complete the goal of the HLC and to maximize the extrinsic reward. But at the beginning of the Auto training, the extremely low extrinsic reward due to the random goals of the HLC prevents the LLC from efficiently learning from the environment. We propose shaped reward as the intrinsic reward for the LLC to take both the goal completion and the extrinsic reward into consideration, and to enable fine-grained low-level behavior learning. The intrinsic reward can represented by
iRdz, = (1 () (-11gL, Cout
OLL.K, +
Σ j-0
eRdL.,K,
(3)
where Ç is a user-defined factor dynamically enlarging from 0.1 to 0.8 as the number of training epochs increases. When Ç is small, the HLC has stronger influence the LLC. On the contrary, when Ç = 1, the LLC maximizes only the accumulative extrinsic reward.
Hardware Overhead Estimator. A recent work (Wang et al. 2019) estimates the hardware latency and energy by physical FPGA accelerators. However, a typical synthesis for a CNN model on a FPGA costs > 30 minutes (Gopinath et al. 2019). Invoking a FPGA synthesis for each action will make AutoQ unacceptably slow. We adopt fast and accurate FPGA latency, area (Liu & Carloni 2013) and power (Zhou et al., 2019) models to predict the inference latency, energy and FPGA area for an arbitrary configuration of network and hardware. These machine-learning-ee
models are highly accurate and can estimate the hardware overhead compute the extrinsic reward of Auto within several milliseconds.
Hierarchical DRL. AutoQ uses a Hlerarchical Reinforcement learning with Off-policy correction (HIRO) (Nachum et al., 2018), to implement the HLC and the LLC. The LLC trained by incorpo- rating IL. into the standard TD3 method (Nachum et al. 2018). So the low-level Q-value function Q.I.. is to minimize the error E E (state
It.,, 9t.,, KJ., statep.,KK, -]). which is
LLC (QbLe (stater, 92, "I,,,,) iddz, - "Miaa" Qb (statep, K511): :4.. (staterr, 85411::: )) where LLLC is trained to maximize QbCC We further augment P with Gaussian noises by collecting the actions N (p oap.,K))) where N is Gaussian distribution, and Caun is
(4)
5

---

Published as a conference paper at ICLR 2020
the variance. During the exploitation, Fap,,.k) Kyl is initialized to 0.5 and decayed after each episode exponentially, The HLC converts a series of high-level transition tuples
(8IL,Ka: IL., Ko. roue-1) e eRd Ka 11. 8(L141,Ko))
(5)
to state-goal-reward transitions
(ol,, I Σ &PRclu Ke (1
(6)
where ([L, Kot -1] denotes the sequence of Kal alL, -I]i and eRdiL,
means the sequence of eRd]t.,Ko) ~ eRd Kept 1]. Auto stores these state-goal-rewan transi- tions into the replay buffer. However, since transitions obtained from the past LLCs do not accurately reflect the actions that would occur if the same goal was used with the current LLC,, Auto has to introduce correction translating old transitions into ones that agree with the current LLC. AutoQ re-labels the high-level transition Ko) 9L.: Tua:Zen KeR -1)
(6)) with a different
I.C.C goal gł, chosen to maximize the probability MeLic (oiT., a) | Ko:Ka 1) IL, AutoQ computes 10 candidate goals sampled randomly from a Gaussian distribution centered at JL.- and selects the minimal goal re-label the experience.
Quantization and Finetuning, During a search, we quantize the model by the learned quantization technique (Zhang et al., 2018), and finetune the quantized model for ten epochs to recover the accuracy using stochastic gradient descent (SGD) with a fixed learning rate of 10 and momentum of 0.9. We randomly select 100 categories from the ImageNet to accelerate the model finetuning. After the search is done, we quantize the model with the best policy found by AutoQ and finetune it on the full dataset.
Implementation Details. An AutoQ agent, i.e., HLC or LLC, consists of an actor network and a critic network. Both share the same architecture, i.c. two hidden layers, each of which has 300 units. For the actor network, we add an additional sigmoid function producing an output in the range of [0, 1]. We use a fixed learning rate of 10 for the actor network and 10-3 for the critic network. AutoQ trains the networks with the batch size of 64 and the replay buffer size of 2000. AutoQ first explores 100 episodes with a constant noise, i.c., 6aL, K, = 0.5 for the LLC and 6 69u.) 0.5 for the HLC, and then exploits 300 episodes with exponentially decayed noise.
Storage Cost. We need record a 4-bit QBN ranging from 0 to 8 for each activation layer and each weight kernel of a convolutional layer. The storage overhead of AutoQ is ~ 0.1% of the size of various CNN models. For instance, ResNet-18 found by resource-constrained Auto requires The storage overhead of Auto is only 0.07%.
8.3MB to store its quantized model in Table 3
EXPERIMENTAL RESULTS
Experimental Settings. To evaluate AutoQ, we selected several CNN models including ResNet-18 18, ResNet-50, SqueezeNetV1 (Iandola et al. 2016) and MobileNetV2 (Sandler et al.) 2018). The CNN models are trained on ImageNet including 1.26M training images and tested on 50K test images spanning IK categories of objects. We evaluated the inference performance, energy consumption and FPGA area of the CNN models quantized by AutoQ on a Xilinx Zynq-7020 embedded FPGA. On the FPGA, we implemented temporal CNN accelerator (Umuroglu et al.) 2019b) that uses ber-serial multipliers, each of which computes with one-bit digits from multiple weights and their corresponding activations in parallel at one time, and then accumulates their partial products.
4.1 OVERALL PERFORMANCE
Resource-constrainee Quantization. We make AutoQ perform the resource-consttrined searches by imposing a latency constraint and setting Place 1, di 0, the and v.a in the extrinsic reward. With such a setting, AutoQ aims to search for the best inference accuracy given the longest latency constraint, which is set to the inference latency of the 4-bit network-wise quantized CNN models. We compare the kernel-wise AutoQ quantized models against the layer-wise Hardware- Aware Automated Quantization (HAQ) (Wang et al. (019) quantized models and the 4-bit network- wise quantized models in Table 3 We used the LQ-Nets quantization Zhang et al. (018)) to quantize and finetune the models in all three schemes. The network- wise scheme uses 4-t to quantize the whole models, while the layer-wise scheme searches a QBN for weights of each layer, and chooses another QBN for activations of the same layer. AutoQ chooses a QBN for each weight kernel, and selects another QBN for each activation layer of CNN. In Table the average QBN of weights

---

Published as a conference paper at ICLR 2020
Table 3: Network Quantization by AutoQ (A-QBN: the average QBN of activations; W-QBN: the average QBN of weights; LAT: inference latency).
resoirce-consttained
nncuracy-gurranen
model
scheme
Top-1
top-5
A-QBN
W-QBN (bit)
LAT
top-1
top-5 (m)
A-QBN
W-QBN (hit)
LAT
em(%)
(bit)
(ms)
(hir)
(ms)
network-wise
32.7
12.32
4
4
296.8
32.7
12.32
4
296.8
ResNet-18
layer-wise
31.8
11.92
3.32
4.63
290.99
32.5
11.90
3.37
3.05
189.6
kemel-wise
30.22
11.62
4.12
3.32
286.3
32.6
11.82
3.02
2.19
125.3
oraginal
30.10
16
30.10
16
Belwork-wise
27.57
9.02
4
4
616.3
27.57
9.02
4
4
616.3
ResNet-50
layer wise
26.79
4.23
3.51
27.49
4.02
kemel-wise
25.53
7.92
3.93
4.02
610.3
27.53
9.12
3.07
2.21
327.3
original
7.82
16
16
2357
25.20
7.82
16
16
2357
Betworkkwiis
45.67
23.12
4
4
43,1
45.67
23.12
4
4
43.1
SqueezeNetV1
layer-wise
44.89
21.14
3.56
4.27
42.1
45.63
23.04
3.95
3.28
25.5
kermel wine
43.51
20.89
4.05
3.76
41.6
45.34
23.02
3,29
2.32
12,5
original
43.10
20.5
16
16
127.3
43,10
20.5
16
16
127.3
network
31.75
11.67
4
37.4
31.35
11.67
4
37.4
MohileNesV2
layer-wise
30.98
10.37
3.57
4.22
36.9
31.34
10.57
3.92
3.21
23.9
kermel-wise
9.67
4.14
3.67
31.32
3.13
original
28.90
9.37
16
16
125.6
28.90
9.37
16
16
123.6
(W-QBN) can be calculated by
E Di Wei Weight-QBBN....
Coouti
where Couti the number of output channels in the layer I, and Weight QBNLL: is the QBN for the Kyth weight kernel in the layer I.. The average QBN of activations (A-QBN) is computed
Prayer E 21 Act.QBN,
as
where Act. LBNL, is the QBN for all activations of the layer L,. Compared the layer-wise quantization, AutoQ improves the top-1 inference accuracy by > 1.25% when spending almost the same inference latency. Compared to the 16-bit full-precision models, the models quantized by AutoQ degrade the inference accuracy by at most only 0.41%, but reduce the inference latency by 71.2% on average.
Plager
Accuracy-guarantee Quantization. We run Auto to do the accuracy- guarracggu searches by setting Vaec = 2, 1/2 0.5, vhe O and tha = 0 in the extrinsic reward. Such an extrinsic reward drives AutoQ to quantize the models to achieve the shortest inference latency without significant accuracy loss. Compared to the layer-wise scheme, AutoQ substantially reduces the inference la- tency by 42.2% while achieving a similar (averagely -0.1 %) top-1 inference accuracy. Compared to ResNet-18 and ResNet50, the compact models such SqueezeNetVi suffer from larger top-1 accuracy degradation, i.e., -0.3% in a accuracy- guaranteed search of AutoQ.
accuracy-guaranteed
Weight QBal
w 6 6
network -- layer
Auto
Activ. QBN
network layer
Auko
Kerne QBN
network ... layer - Autoo
10 13 19 Layer
7 10 13 16 19
50
100
150
200
250
Layer
Kernel
Figure 4: The ave. QBNs in various layers.
Figure 5: The weight kernel QBNs in a layer.
4.2 DETAILED ANALYSIS
Kernel-wise Search. AutoQ can assign a QBN to each kernel of a convolutional layer. The aver- age weight QBN and the average activation QBN of each ResNet-18 18 layer found by an accuracy- guaranteed AutoQ search are shown in Figure Both the network- wise and layer-wise quantization techniques use only one QBN to quantize all weight kernels in a convolutional layer, and quantize all activations of the layer by another QBN. On the contrary, AutoQ searches QBN for each weight kernel. Compared to a CNN model quantized by the network-wise or layer-wise quantization tech- nique, the same model quantized by the kernel-wise AutoQ can achieve similar inference accuracy but with a smaller average QBN in each layer. We also show the weight kernel QBNs of the L14 layer of ResNet-18 18 produced by resource-constrainee AutoQ searches in Figure AutoQ automati- cally identifies which weight kernel has a smaller (larger) variance and thus less (more) redundancy, so that it can assign larger (smaller) QBN the weight kernel. For instance, Figure T shows,
7

---

Published as a conference paper at ICLR 2020
compared to the 53th weight kernel (top-right), the 52th weight kernel (top-left) of ResNet- 18 has a smaller weight distribution variance. Therefore, in Figure AutoQ assigns a smaller QBN to the 52th weight kernel but provides the 53th weight kernel larger QBN.
Hierarchical DRL Agent with Shaped In- trinsic Reward. We evaluated and compared our hierarchical-DRL--assee the traditional one-level DDPG-based DRL
Inference Accuracy
DDPG
HIRO
Autoo
100%
AutoQ against
80%
60%
40%
adopted by a recent layer-wise quantization technique, HAQ [Wang et al. 2019). The re-
20%
0%
150
250
ward comparison of different techniques during
Training Episodes
the kernel-wise quantization on MobileNetV2
Figure 6: The DRL scheme comparison.
is shown in Figure 6 HAQ and AutoQ both support resource-constrale support recuracy-gaaranted searches. So their rewards are just the inference accuracy. Through the goals of the HLC and the actions of the LLC, AutoQ can find QBN for each weight kernel and achieve > 70% accuracy much faster than the DDPG-based DRL, i.e., it reaches ~ 70% accuracy after only 200 episodes. However, the DDPG-based DRL is stuck with 20% inference accuracy until 250 episodes. The Sawtarchhccl-DRL--aaaee ration of the kernel-wise network quantization. Although AutoQ uses prior hierarchical DRL agent HIRO (Nachum et al.) 2018) to search a QBN for each weight kernel, we propose a novel shaped intrinsic reward considering both the completion of the HLC goals and the extrinsic reward to ac- celerate the search. The intrinsic reward of HIRO takes only the complction of the HLC goals into consideration. The LLC of HIRO cannot directly learn from the environment. Therefore, compared to AutoQ, it takes extra 200 episodes for HIRO to reach only 60% accuracy as shown in Figure 6
searches, but HAQ cannot
Auto significantly accelerates the search space explo-
Extrinsic Reward. Unlike the reward of the DDPG- based layer-wise HAQ (Wang al. 2019) considering only the inference accuracy, the extrinsic reward of AutoQ can balance the trade-off be- tween the inference accuracy, latency, energy consumption and FPGA area by enabling the accuracy- guaranteed search. By setting Vacc = 2, 1/2 0.5, the = 0.5 and tha = 0.5, Auto takes the in- ference accuracy, latency, energy and FPGA area into considcration during an securacy-guarantee search. For instance, AutoQ can find two kernel-wise QBN configurations having similar inference accuracy, latency and energy for MobileNetV2. We cannot differentiate these two configurations by using only the HAQ reward. However, the first configuration consumes 94% of the FPGA area, while the other configuration occupies 85% the FPGA area. AutoQ can identify the second QBN configuration better choice via its extrinsic reward.
Latency (ms)
50
network
layer
kernel
sub-kernel
Energy (m)) meeagaccoo
netwerk
leyer
kemel
sub-lernel
30
20
10
SqueezeNet MobileNeiV2
SqueezeNet MobileNetV2
SgueezeNet MobileNetV2 SqueezeNet MobileNetV2
Spatial Architecture
"'emprraArrchttcctur
Spatial Architecture
Temporal Architecture
(a) Latency.
(b) Energy.
Figure 7: The comparison of latency and energy between temporal and spatial CNN accelerators.
Quantization Granularity. Besides the temporal CNN accelerator (Umuroglu et al. 2019b), the kernel-wise quantized models found by the accuracy guarraacgg Auto can reduce the inference latency spatial CNN accelerator, BitFunsion (Sharma et al.) 2018), that relies on a 2D systolic array of the fusion units spatially summing the shifted partial products of weights and activations F compared to the layer-wise quantized models, on average, the kernel-wise quantized models reduce the inference latency by 39.04% and decrease the inference energy by 33.34% on the spatial CNN accelerator. Therefore, the kernel-wise quantized models greatly reduce the inference latency and energy on both the temporal and spatial CNN accelerators. Prior works (Mellempudi al. 2017 Choukroun al. 2019a) suggest possible to divide a weight kernel into several sub- kernels and quantize each sub-kernel independently. We also use AutoQ to search QBN for each weight sub-kernel. As Figure S shows, the wub-kernel-wise quantized models cannot improve the inference latency energy on the spatial CNN accelerator consisting of systolic computing arrays. Each dot-product operation of a bub-kernel-wise quantized model has to be split into several dot- product operations to be accumulated together. A systolic computing array still has to be designed to accommodate the weight sub-kernel with the largest QBN a kerne.. Therefore. we can see that it is difficult for the fine- grained quantization schemes choosing QBN for each weight unit that is
8

---

Published as a conference paper at ICLR 2020
part of kernel to further reduce the inference latency energy both the temporal and the spatial CNN accelerators.
CONCLUSION
In this paper, we propose a Srrarchical-DRL-bee kernel-wise network quantization technique, AutoQ, consisting of a HLC and a LLC. The HLC automatically searches an average weight QBN and an average activation QBN for each convolutional layer. Based on the average weight QBN, the LLC generates a QBN for each weight kernel in each layer. We also create a state space, a goal and action space, an intrinsic reward and an extrinsic reward to support AutoQ. Particularly, our shaped intrinsic reward enables the LLC to learn efficiently from the environment by considering both the HLC goal completion and the environment extrinsic reward. Moreover, the extrinsic reward of AutoQ can balance the inference accuracy, latency. energy consumption and FPGA area. Compared to the models quantized by the state-of-the-ar DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. CoRR, abs/1611.02161 2016.
Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. CoRR, Abss/192..0822, 2019a.
Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. arXiv preprint arXiv: /902.0682, 2019b.
Emily Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In International Conference on Neural Information Processing Systems, 2014.
Ahmed T. Elthakeb, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Es- macilzadeh. Releq: A reinforcement learning approach for deep quantization of neural networks. CoRR, abs/18 81 1.01704, 2018.
Sridhar Gopinath, Nikhil Ghanathe, Vivek Seshadri, and Rahul Sharma. Compiling kb-sized ma- chine learning models to tiny iot devices. In ACM SIGPLAN Conference on Programming Lan- guage Design and Implementation, pp. 79-95, 2019.
Yiwen Guo, Anbang Yao, Hao Zhao, and Yurong Chen. Network sketching: Exploiting binary structure in deep enns. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 5955-5963. 2017.
Yihui He, Lin. Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration mobile devices. In European Conference Computer Vision, 2018.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < <Imb model size. CoRR, abs/1602.07360, 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for effi- cient integer-arithmetii-onl inference. In The IEFE Conference on Computer Vision and Pattern Recognition, June 2018.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, and Eric Xing. Neu- ral architecture search with bayesian optimisation and optimal transport. CoRR, abs/ 1802.07191, 2018.
Minje Kim and Paris Smaragdis. Bitwise neural networks, In ICML Workshop on Efficiet-Ef Machine Learning, 2016.

---

Published as a conference paper at ICLR 2020
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.0834 2018.
Rundong Li, Yan Wang. Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2810-2819, 2019.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, 2016.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 345-353. Curran Associates, Inc., 2017.
Hung- Yi Liu and Luca Carloni. On leammin- methods for design-space space exploration with high-level synthesis. In IEEE/ACM Design Automation Conference, 2013.
Jeffrey L. McKinstry, Steven K. Esser, Rathinakumar Appuswamy, Deepika Bablani, John V. Arthur, Izzet B. Yildiz, and Dharmendra S. Modha. Discovering low-precision networks close full- precision networks for efficient embedded inference. CoRR, Abs//109.04191, 2018.
Naveen Mellempudi, Abhisek Kundu, Dheevatsa Mudigere, Dipankar Das, Bharat Kaul, and Pradeep Dubey. Ternrry neural networks with fine-grained quantization. CoRR, abs/1705.01462, 2017.
Ofir Nachum, Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2018
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In IEEE European Conference on Com- puter Vision, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
S. Sasaki, A. Maki, D. Miyashita, and J. Deguchi. Post training weight compression with distribution- filter-wise quantization step. In JEEE Symposium in Low-Power and High- Speed Chips, pp. 1-3, 2019.
H. Sharma, J. Park, N. Suda, L. Lai, B. Chau, V. Chandra, and H. Esmaeilzadeh. Bit fusion: Bit- level dynamically composable architecture for accelerating deep neural network. ACM/IEEE International Symposium Computer Architecture, pp. 764-775, 2018.
2018.
Kenneth O. Stanley and Risto Mikkulainen. Evolving neural networks through augmenting topolo- gies. Journal Evolutionary Computation, 10(2):99-127, June 2002. ISSN 1063-6560.
Lawrence Stewart and Mark Stalzer. Bayesian optimization for parameter tuning of the xor neural network. CoRR, abs/ 711110990009 009447 2018.
Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic programming approach to designing convolutional neural network architectures. In ACM Genetic and Evolutionary Com- putation Conference, pp. 497-504, 2017.
Wei Tang, Gang Hua, and Liang Wang- How to train compact binary neural network with high accuracy? In AAAI Conference on Artificial Intelligence, pp. 2625-2631, 2017.
Yaman Umuroglu, Davide Conficconi, Lahiru Rasnayake, Thomas B. Preusser, and Magnus Själander. Optimizing bit-serial matrix multiplication for reconfigurable computing. ACM Trans- actions Reconfigurable Technology and Systems, 12(3), August 2019a.
Yaman Umuroglu, Davide Conficconi, Lahiru Rasnayake, Thomas B. Preusser, and Magnus Själander. Optimizing bit-serial matrix multiplication for reconfigurable computing, ACM Trans- actions Reconfigurable Technology and Systems, 12(3), August 2019b.
10

---

Published as a conference paper at ICLR 2020
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-awart automated quan- tization with mixed precision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Yunhe Wang, Chang Xu, Jiayan Qiu, Chao Xu, and Dacheng Tao. Towards evolutionary compres- sion. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2476-2485 2018.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. In International Conference on Learning Representations, 2018.
Linghua Zeng, Zhangcheng Wang, and Xinmei Tian. Kenn: Kernel-wise quantization to remarkably decrease multiplications in convolutional neural network. In International Joint Conference Artificial Intelligence, pp. 4234-4242, 2019.
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. European Conference on Computer Vision, 2018.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights, In International Conference on Learning Representations, 2017.
Yuan Zhou, Haoxing Ren, Yanqing Zhang, Ben Keller, Brucek Khailany, and Zhiru Zhang. Primal: Power inference using machine learning. In JEEE/ACM Design Automation Conference, 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. CoRR, ahs/1707700001 2017.


Reviews:
[{"review_id": "HyggGAw6YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary of the work\n\nThis paper proposes to automatically search quantization scheme for each kernel in the neural network. Importantly, due to a large amount of search space, hierarchical RL was used to guide the search.\n\nStrength\n\nFirst of all, I really liked the idea about more detailed search for kernel-wise configurations. It is interesting to see that different kernel settings has different best bitwidth. More importantly, it is great to provide cost model based estimations for cost/latency.\n\nThe authors also provided detailed discussion about weight bit distributions and average quantization bits among the layers. \n\nThe author provided implementation on real world FPGAs using bit-serial spatial architecture, which plays particularly nice with the algorithm. This something that should be highlighted in the discussion.\n\nWeakness\n\nFirst of all, I would love to see more detailed discussion about the corresponding hardware support implications. In particular, given the need of re-using computing resources, kernel-wise quantization has to use the bit-serial version of architecture(otherwise the MAC cannot be reused and we have to layout the entire network on FPGA), which may limit the applicability of the methods to higher number of bits.\n\nThe second potential weakness is the close relation between the method and HAQ. The method feels like a straight-forward extension(with DRL added). What would happen if you directly apply HAQ’s method to the kernel-wise search space?\n\nFinally it would be great if the authors can clarify more about the FPGA setups(number of accumulators being used, whether there is reuse of compute unit in FPGA, or did you just layout everything spatially).\n\nQuestion:\n\nHow do you handle layers like BatchNorm(which normally need floating pt)?\n\nOverall, I find this paper provides interesting insights and solid evaluation and should be accepted to ICLR.\n\n\n---\ni have read the authors' response. \n\nPlease do note that because the different kernel-wise precision, the accelerator has to resort to bit-serial computation, which somewhat limits the structure of the accelerator. e.g. we cannot simply build 8bit MAC along with 4 bit ones, but have to use 4bit ones to bit-serially accumulate the 8 bit ones. This will somewhat limit the applicability of the model, and i think the author should add a discussion section to the paper about this limitation.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxJ0SuRtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a method for network quantization. Similar with the work of \"HAQ: Hardware-Aware Automated Quantization with Mixed Precision\"(CVPR 2019), the proposed method is based on reinforcement learning. The contribution of the work is on the kernel-wise quantization, i.e., assigning different bitwidth to different kernels in one layer. And in the experiments, the proposed method clearly outperformed the state-of-arts of network-wise and layer-wise quantization methods.\nAlthough the high level idea is presented very well, some essential parts of the paper is a little bit hard to follow.  The motivation of using the hierarchical DRL is unclear.\n\nQuestions:\n\nWhy a hierarchical DRL agent is desired for kernel-wise quantization? Is it possible to modify the definitions of state and action of HAQ for kernel-wise quantization, which seems to be a much simpler solution for the task?\n\n\nWhat is the definition of iRd [L_i,K_j] in the Intrinsic Reward?\n\n\nIt seems the original HAQ used simple quantization way instead of LQ-Nets, which is different with the experiment setting in this paper. If I'm correct, does the change affect the HAQ's performance for fair comparison?\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ebnpXOnH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper proposes a network quantization method. Different from previous methods focusing on network-level or layer-lever quantization, this work pays attention to kernel-level quantization. Specifically, they use a hierarchical reinforcement learning framework to search in the search space related with quantization. The experiment result validates the significance of the work.\n\nStrength\nThe paper provides us with a new insight into network quantization. Even though the extension from layer-level to kernel-level is straightforward, the improvement is significant and meaningful. The experiment result demonstrates its efficiency in real applications.\n\nWeakness\n1. The algorithm of the paper is similar with previous work HAQ, which is based on DRL to guide the search procedure. Thus, the novelty of algorithm is somewhat weak.\n \n2. For kernel-level quantization, this paper proposes a hierarchical DRL method. However, I didn't see the importance of the hierarchy. The author may discusses more about this and compare it with flatten algorithms for ablation study. \n\n3. The paper didn't compare their methods with other baselines on the same level. I think some algorithms can be applied into kernel-level quantization directly. Based on the same level comparison, the efficiency of your method can be seen more directly. \n\n4. The description was not written well. For example, the detail of the experiment and the hardware settings are unclear.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkeC2ghiFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for quantizing neural network weights and activations that uses deep reinforcement learning to select the appropriate bitwidth for individual kernels in each layer. The algorithm uses a reward function that weights accuracy of the quantized model with latency, energy, and FPGA area, and leverages a high level and low-level controller to create quantized models that can take into account these factors. Compared to prior approaches taht only perform layer-wise instead of kernel-wise quantization, the quantized models can achieve better performance, or latency.\n\nWhile the problem of more effectively quantizing neural network weights and activations is interesting, I found this paper hard to follow and evaluate. The proposed hierarchical deep RL method mentions a number of tricks and design decisions that are not ablated, and the notation and text around the method were difficult to follow. There were also no baselines comparing to other approaches for performing kernel-wise quantization (e.g. random search, quantize based off variance, etc.). Without improved baselines and text I would recommend rejecting this paper.\n\nMajor comments:\n* Based on the analysis of kernel-wise results, it seems that a very simple strategy that chooses QBNs based on weight variance could be sufficient to achieve good performance. However, there’s no comparisons to these kinds of heuristics or even to random search over QBN per-kernel. There are also no ablations that study whether the hierarchical-RL based approach beats a baseline that just directly predicts the kernel-wise QBNs independently.  Without these baselines, it’s hard to know how well AutoQ works.\n* The text that details the hierarchical RL approach with multiple controllers that is core to the new AutoQ approach is extremely hard to follow, with many undefined symbols.\n\nMinor comments:\n* FIg 1: where in the neightwork do these weight kernels come from? What network/dataset?\n* You repeatedly state that ML experts obtain only sub-optimal results at selecting QBNs, can you cite something for this? What if you give experts access to additiional information like visualizations of weight distributions?\n* What’s the tradeoff between using different QBNs for each kernel and specifying this number? Is there additional memory overhead?\n* Table 1: hard to read the exponent in kernel-wise math, add parentheses\n* Please use \\citep vs. \\citet where appropriate\n* Why not also search for kernel-specific activation quantization? \n* The notation in Fig 3 and surrounding text is really hard to follow, e.g. w_w, h_w, and indices into states (roundup could be ceil, names are confusing iRd, eRd?).\n* Eqn 2 comes out of nowhere… why optimize for log(accuracy) vs accuracy? How do you choose the user-defined constants? Decay factor?\n* Hardware overhead estimator: are these models accurate independent of QBN?\n* Why do you have to perform Gaussian augmentation? I didn’t follow the re-labeling of transitions after Eqn. 6, is this needed?\n* Define \\mu when you first use it (Eqn 4?)\n* Is \\delta_a in implementation details the same as \\sigma around Eqn 4?\n* Table 3: Is there variance in these results based off fine-tuning? Could you include error bars or stderr on these estimates?\n* Figure 6: are the search spaces different for these different approaches? I.e. does AutoQ perform better due to the upper/lower bounds you set on QBN?\n\n=============\n\nThank you to the authors for addressing many of my concerns. The updated paper still does not present any baselines for methods that can search or learn kernel-wise quantization parameters, e.g. run random search and select the best-performing models according to your reward function. Additionally, there were no revisions made to the text to improve the presentation, and thus I continue to recommend rejection of this work.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a network quantization method which is based on kernel-level quantization. The extension from layer-level to kernel-level is straightforward, and so the novelty is somewhat limited given its similarity with HAQ. Nevertheless, experimental results demonstrate its efficiency in real applications. The paper can be improved by clarifying some experimental details, and have further discussions on its relationship with HAQ.
Decision: Accept (Poster)