AutoQ: Automated Kernel-Wise Neural Network Quantization 

1 INTRODUCTION . Although convolutional neural networks ( CNNs ) have been the dominant approach ( Sandler et al. , 2018 ) to solving a wide variety of problems such as computer vision and recommendation systems , it is challenging to deploy CNNs to mobile devices having only limited hardware resources and tight power budgets , due to their huge essential computing overhead , e.g. , an inference of MobileNetV2 ( Sandler et al. , 2018 ) involves 6.9M weights and 585M floating point operations . Several approaches such as pruning ( He et al. , 2018 ) and low-rank approximation ( Denton et al. , 2014 ) are proposed to reduce the inference computing overhead of CNNs . Network quantization ( Wang et al. , 2019 ; Lin et al. , 2017 ) becomes one of the most hardware friendly CNN acceleration techniques by approximating real-valued weights and activations to QBN -bit fixed-point representations , and performing inferences using cheaper fixed-point multiple-accumulation ( MAC ) operations , where QBN is the quantization bit number . Instead of using one QBN for the whole CNN , the layer-wise network quantization ( Wang et al. , 2019 ; Elthakeb et al. , 2018 ) assigns a QBN to the weights of each convolutional layer , and searches another QBN for the activations of the same layer to decrease the inference computing overhead . But the inference cost of the layer-wise quantized CNNs is still prohibitive for low-power mobile devices powered by batteries . Recent works ( Zeng et al. , 2019 ; Choukroun et al. , 2019b ; Zhang et al. , 2018 ; Li et al. , 2019 ; Krishnamoorthi , 2018 ; Sasaki et al. , 2019 ) find that various weight kernels of a ∗This work was supported in part by NSF CCF-1908992 and CCF-1909509 . 0 0.5-0.5 250 C ou nt 00 250 C ou nt -0.25 0 0.250 250 C ou nt 0.0 0.50 250 C ou nt mean=-0.012 std=0.127 mean=-0.052 std=0.198 mean=-0.022 std=0.126 mean=0.070 std=0.200 outliers outliers -1 Figure 1 : The weight distribution of kernels . convolutional layer ( ResNet-18 ) exhibit different variances shown in Figure 1 and hence have different amounts of redundancy . Therefore , they quantize each weight kernel independently for higher accuracy by calculating a QBN -element scaling factor vector for each kernel , rather than globally quantize all the kernels of a layer as a whole . To reduce different amounts of redundancy among different weight kernels , these kernel-wise network quantization techniques should have searched a QBN for each kernel of each layer in a CNN . However , the search space of choosing a QBN for each weight kernel is too large , so prior kernel-wise network quantization ( Zeng et al. , 2019 ; Choukroun et al. , 2019b ; Zhang et al. , 2018 ; Li et al. , 2019 ; Krishnamoorthi , 2018 ; Sasaki et al. , 2019 ) still uses the same QBN for the entire CNN . As Figure 2 shows , compared to the layer-wise quantized model , on the same FPGA accelerator ( Umuroglu et al. , 2019a ) , the kernel-wise quantized model ( assigning a QBN to each weight kernel and choosing a QBN for each activation layer ) improves the inference accuracy by ∼ 2 % ( ImageNet ) with the same computing overhead ( inference latency ) . How to decide a QBN for each weight kernel is the most important task of the kernel-wise network quantization , since the QBNs have a large impact on the inference accuracy , latency and hardware overhead . Determining a QBN for each weight kernel via hand-crafted heuristics is so sophisticated that even machine learning experts can obtain only sub-optimal results . Recent works ( Wang et al. , 2019 ; Elthakeb et al. , 2018 ) automatically select a QBN for each layer of a CNN through a deep reinforcement learning ( DRL ) agent without human intervention . However , it is still difficult for low-power mobile devices such as drones and smart glasses to adopt the layer-wise quantized CNN models . These mobile devices are very sensitive to the bit-width of fixed-point MAC operations and memory access during inferences due to their limited battery lifetime and hardware resources . Kernel-wise network quantization assigning a QBN to each weight kernel and searching a QBN for each activation layer of a CNN becomes a must to enable the efficient deployment of deep CNNs on mobile devices by reducing the inference computing overhead . Although it is straightforward to perform kernel-wise quantization via DRL , it takes ultra-long time for a DRL agent to find a proper QBN for each weight kernel of a CNN . As CNN architectures are becoming deeper , it is infeasible to employ rule-based domain expertise or conventional DRL-based techniques to explore the exponentially enlarging search space of kernel-wise network quantization . In this paper , we propose a hierarchical-DRL-based agent , AutoQ , to automatically and rapidly search a QBN for each weight kernel and choose a QBN for each activation layer of a CNN for accurate kernel-wise network quantization . AutoQ comprises a high-level controller ( HLC ) and a low-level controller ( LLC ) . The HLC chooses a QBN for each activation layer and generates a goal , the average QBN for all weight kernels of a convolutional layer , for each layer . Based on the goal , the LLC produces an action , QBN , to quantize each weight kernel of the layer . The HLC and LLC simultaneously learn by trials and errors , i.e. , penalizing inference accuracy loss while rewarding a smaller QBN . We also build a state space , a goal and an action space , an intrinsic reward and an extrinsic reward for AutoQ . Instead of proxy signals including FLOPs , number of memory access and model sizes , we design the extrinsic reward to take the inference latency , energy consumption and hardware cost into consideration . 2 BACKGROUND AND RELATED WORK . Quantization . Recent works ( Lin et al. , 2016 ; Zhou et al. , 2017 ; Jacob et al. , 2018 ; McKinstry et al. , 2018 ; Zhang et al. , 2018 ) quantize the real-valued weights and activations to fixed-point representations , so that the model size is reduced and inferences can use low-cost fixed-point MAC operations . To further reduce inference computing overhead , prior works ( Kim & Smaragdis , 2016 ; Xu et al. , 2018 ; Guo et al. , 2017 ; Tang et al. , 2017 ; Rastegari et al. , 2016 ; Lin et al. , 2017 ) quantize weights and activations into multi-bit binary codes of { -1 , +1 } s. Rather than real-valued MACs , inferences of these quantized models depend on bit-wise logic operations , i.e. , XNORs and popcounts . These traditional quantization techniques either simply assign a single QBN to the whole CNN or require domain experts to determine a QBN for each layer of a CNN . Kernel-wise quantization . As Table 1 shows , almost all prior works ( Lin et al. , 2016 ; Kim & Smaragdis , 2016 ; Rastegari et al. , 2016 ; Lin et al. , 2017 ; Guo et al. , 2017 ; Zhou et al. , 2017 ; Jacob et al. , 2018 ; Tang et al. , 2017 ; Xu et al. , 2018 ; McKinstry et al. , 2018 ; Zhang et al. , 2018 ) categorized as the network-wise quantization focus on searching a QBN ∈ [ 0 , 32 ] for all weights , and searching another QBN for all activations in a CNN . Totally , there are only 1089 combinations of the QBN configuration for the network-wise quantization . The layer-wise quantization ( Wang et al. , 2019 ) searches a QBN ∈ [ 0 , 32 ] for all weights of a convolutional layer , and decides another QBN for all activations of the same layer . The QBN search space size of the layer-wise quantization substantially increases to 33nlayer × 33nlayer , where nlayer is the layer number of a CNN . Recent works ( Zeng et al. , 2019 ; Choukroun et al. , 2019b ; Zhang et al. , 2018 ; Li et al. , 2019 ; Krishnamoorthi , 2018 ; Sasaki et al. , 2019 ) observe various weight kernels of a convolutional layer have different amounts of redundancy , and quantize each weight kernel independently for higher accuracy . To exploit different amounts of redundancy among different weight kernels , these kernel-wise network quantization techniques should have searched a QBN for each kernel of each convolutional layer , and assigned a QBN for each activation layer in a CNN . However , the search space size of the kernel-wise network quantization is 33 ∑nlayer i=1 couti × 33nlayer , where couti is the number of weight kernels ( output channels ) of the ith layer . No prior work tries to search such huge design space . AutoML . Recent works take advantage of DRL ( Baker et al. , 2016 ; Zoph et al. , 2017 ) , genetic algorithm ( Suganuma et al. , 2017 ; Stanley & Miikkulainen , 2002 ) and Bayesian Optimization ( Kandasamy et al. , 2018 ; Stewart & Stalzer , 2018 ) to automatically architect CNNs for higher inference accuracy . Their network architectures outperform many human-designed neural networks . The weight channel pruning is automatically conducted by DRL ( He et al. , 2018 ) and genetic algorithm ( Wang et al. , 2018 ) . ReLeQ ( Elthakeb et al. , 2018 ) quantizes only the weights of each layer of a CNN by DRL , while HAQ ( Wang et al. , 2019 ) performs the layer-wise quantization for both weights and activations via a DRL agent . No prior quantization or pruning work relies on hierarchical DRL . Table 2 compares AutoQ against prior DRL-based techniques for quantization and pruning . AutoQ is the first work to automatically quantize each weight kernel and each activation layer of a pre-trained CNN model for mobile devices by hierarchical DRL . 3 AUTOQ . Overview . We do not aim to present a new network quantization technique , but we formulate the search of a QBN for each weight kernel and each activation layer as a hierarchical DRL problem . We propose a two-level hierarchical DRL technique , AutoQ , to automatically quantize the weights in the kernel-wise manner and the activations in the layer-wise fashion . We build the state space , action and goal space , extrinsic and intrinsic reward functions and a hierarchical DRL agent for AutoQ . Although we use the state-of-the-art learned quantization technique , LQ-Nets ( Zhang et al. , 2018 ) , to quantize weight kernels and activation layers with the QBNs found by AutoQ , future novel quantization techniques can be easily integrated to AutoQ to improve the inference accuracy of the quantized networks . In the extrinsic reward , besides the inference latency and energy ( Wang et al. , 2019 ) , AutoQ also considers the FPGA area overhead critical to low-cost mobile devices . Working Flow . For an nlayer-layer CNN , the weight is defined as W ∈ Rnlayer×cout×cin×ww×hw , where nlayer is the number of layers ; cout denotes the number of kernels ( output channels ) ; cin means the number of input channels ; ww indicates the kernel width ; and hw is the kernel height . The activation is defined as A ∈ Rnlayer×cin×wa×ha , where wa is the feature map width ; and ha means the feature map height . The working flow of AutoQ is shown in Figure 3 . AutoQ consists of a high-level controller ( HLC ) and a low-level controller ( LLC ) . The HLC quantizes the network layer by layer , while the LLC searches a QBN for each weight kernel in a layer . ¶ At first , AutoQ receives an observation state [ Li , Kj ] from the environment that is the quantized network model , where state [ Li , Kj ] includes the information of the CNN architecture . · The HLC makes a goal gLi that is the QBN for the activation layer Li . The flow then jumps to ¹ . Or the HLC generates a goal gLi which is the average QBN of all weight kernels in the layer Li for the LLC . ¸ The LLC produces an action a [ Li , Kj ] , QBN , for the weight kernelKj of the layer Li . For the entire layer Li , the LLC aims to reach the goal gLi of the HLC . ¹ The environment sends the network quantization and hardware configuration to the fast and accuracy machine-learning-based hardware overhead estimator . º The hardware overhead estimator returns the energy consumption , area overhead and inference latency for the current quantization and hardware configuration . » With the hardware overhead and inference accuracy , the environment generates an extrinsic reward eRd [ Li , Kj ] for AutoQ to evaluate the LLC action . ¼ Based on all actions of LLC for the layer Li , the HLC provides an intrinsic reward iRdLi to tell how well the goal is implemented by the LLC . State Space . A state state [ Li , Kj ] ( observation ) is represented by state [ Li , Kj ] = ( Li , Kj , cin , cout , skernel , sstride , sfeature , bdw , bw/a , gLi−1 , a [ Li , Kj−1 ] ) ( 1 ) where Li is the layer index ; Kj means the weight kernel index ; cin indicates the number of input channels ; cout denotes the number of kernels ; skernel is the kernel size ; sstride is the stride ; sfeature is the input feature map size ; bdw binarily indicates depthwise convolution or not ; bw/a binarily represents weight or activation ; gLi−1 is the goal ( average QBN ) of the last layer ; and a [ Li , Kj−1 ] is the action ( QBN ) of the last kernel in the Li layer . For each variable in state [ Li , Kj ] , we normalize it to [ 0 , 1 ] . If the layer is a fully-connected layer , we set skernel = 1 , sstride = 0 , and bdw = 0 . Goal and Action Space . The HLC produces the average QBN for all weight kernels of each layer or the QBN for each activation layer as a goal , while the LLC generates a QBN for each weight kernel in a layer as an action . The HLC goal gLi for the Li layer uses a continuous space and can be any real value between 1 and goalmax , where goalmax is the maximum average QBN for a layer and we set it to 8 . If the Li layer is an activation layer , we round the real-valued gLi to the discrete value of roundup ( 1 + gLi · ( goalmax − 1 ) ) . Although the LLC action is an integer between 0 and actionmax , it still uses a continuous space to capture the relative order , i.e. , 2-bit is more aggressive than 3-bit , where actionmax is the maximum QBN for a kernel and we set it to 8 . For the Kj kernel of the Li layer , the LLC generates the continuous action ra [ Li , Kj ] that is in the range of [ 0 , 1 ] , and round it up to the discrete value a [ Li , Kj ] = roundup ( ra [ Li , Kj ] · actionmax ) . Extrinsic Reward . After an action a [ Li , Kj ] is taken , AutoQ arrives at a new state state [ Li , Kj+1 ] and receives an extrinsic reward eRd from the environment . The HLC aims to maximize the accumulative extrinsic reward eRd = ∑ i ∑ j γ ∑ i couti+j−1 eRd eRd [ Li , Kj ] , where γeRd ∈ [ 0 , 1 ) is a decay factor . The immediate extrinsic reward can be represented by eRd [ Li , Kj ] ( NC , HC ) = log ( accuracy ( NC ) ψacc lat ( NC , HC ) ψl · en ( NC , HC ) ψe · area ( NC , HC ) ψa ) ( 2 ) whereNC is the network configuration ; HC means the hardware configuration , e.g. , memory bandwidth ; accuracy ( NC ) indicates the inference accuracy ; lat is the inference latency of the network NC running on the hardware HC ; en represents the inference energy of NC running on HC ; area is the FPGA area ( hardware cost ) used by NC on HC ; ψacc , ψl , ψe and ψa are user-defined factors deciding the impact of inference accuracy , latency , energy and FPGA area on the extrinsic reward . By different values of user-defined factors , AutoQ implements the resource-constrained and accuracy-guaranteed searches . For resource-constrained applications , e.g. , low-power drones , AutoQ sets ψacc = 1 , ψl = 0 , ψe = 0 and ψa = 0 to achieve the best accuracy given the maximum amount of hardware resources ( latency , energy , and FPGA area ) . This extrinsic reward offers no incentive for lower QBNs , so AutoQ reduces the QBN by limiting the action space . AutoQ allows arbitrary action at the first few layers and starts to limit the action when it finds that the hardware resource budget is insufficient even after using the smallest QBN for all the following layers . For accuracy-guaranteed applications , e.g. , fingerprint locks , AutoQB sets ψacc = 2 , ψl < 1 , ψe < 1 and ψa < 1 to obtain the shortest latency , the minimal energy , and the smallest hardware cost with no accuracy loss . Intrinsic Reward . Based on the goal gLi produced by the HLC for the Li layer , the LLC generates cout actions a [ Li , K0 ] ∼ a [ Li , Kcout−1 ] at the states state [ Li , K0 ] ∼ state [ Li , Kcout−1 ] . AutoQ then arrives the state state [ Li , Kcout−1 ] , where it receives an intrinsic reward iRd and maximizes the accumulative intrinsic reward iRd = ∑ j γ j−1 iRd iRd [ Li , Kj ] , where γiRd ∈ [ 0 , 1 ) is a decay factor and iRd [ Li , Kj ] indicates the intrinsic reward for the weight kernel Ki of the layer Li . The LLC produces actions to help the HLC to maximize the extrinsic reward , so it should aim to complete the goal of the HLC and to maximize the extrinsic reward . But at the beginning of the AutoQ training , the extremely low extrinsic reward due to the random goals of the HLC prevents the LLC from efficiently learning from the environment . We propose a shaped reward as the intrinsic reward for the LLC to take both the goal completion and the extrinsic reward into consideration , and to enable fine-grained low-level behavior learning . The intrinsic reward can be represented by iRdLi = ( 1− ζ ) · ( −||gLi · cout − cout−1∑ j=0 aLi , Kj ||2 ) + ζ · cout−1∑ j=0 eRdLi , Kj ( 3 ) where ζ is a user-defined factor dynamically enlarging from 0.1 to 0.8 as the number of training epochs increases . When ζ is small , the HLC has stronger influence on the LLC . On the contrary , when ζ = 1 , the LLC maximizes only the accumulative extrinsic reward . Hardware Overhead Estimator . A recent work ( Wang et al. , 2019 ) estimates the hardware latency and energy by physical FPGA accelerators . However , a typical synthesis for a CNN model on a FPGA costs > 30 minutes ( Gopinath et al. , 2019 ) . Invoking a FPGA synthesis for each action will make AutoQ unacceptably slow . We adopt fast and accurate FPGA latency , area ( Liu & Carloni , 2013 ) and power ( Zhou et al. , 2019 ) models to predict the inference latency , energy and FPGA area for an arbitrary configuration of network and hardware . These machine-learning-based models are highly accurate and can estimate the hardware overhead to compute the extrinsic reward of AutoQ within several milliseconds . Hierarchical DRL . AutoQ uses a HIerarchical Reinforcement learning with Off-policy correction ( HIRO ) ( Nachum et al. , 2018 ) , to implement the HLC and the LLC . The LLC is trained by incorporating gLi into the standard TD3 method ( Nachum et al. , 2018 ) . So the low-level Q-value function QLLCθLLC is to minimize the error εLLC ( state [ Li , Kj ] , gLi , a [ Li , Kj ] , state [ Li , Kj+1 ] ) , which is ( Q LLC θLLC ( state [ Li , Kj ] , gLi , a [ Li , Kj ] ) − iRdLi − γiRd ·Q LLC θLLC ( state [ Li , Kj+1 ] , gLi , µ LLC φLLC ( state [ Li , Kj+1 ] , gLi ) ) ) 2 ( 4 ) where µLLCφLLC is trained to maximize Q LLC θLLC . We further augment µLLCφLLC with Gaussian noises by collecting the actions as N ( µLLCφLLC , σa [ Li , Kj ] ) , where N is a Gaussian distribution , and σa [ Li , Kj ] is the variance . During the exploitation , σa [ Li , Kj ] is initialized to 0.5 and decayed after each episode exponentially . The HLC converts a series of high-level transition tuples ( s [ Li , K0 : Kcout−1 ] , gLi , a [ Li , K0 : Kcout−1 ] , eRd [ Li , K0 : Kcout−1 ] , s [ Li+1 , K0 ] ) ( 5 ) to state-goal-reward transitions ( s [ Li , K0 ] , gLi , ∑ eRd [ Li , K0 : Kcout−1 ] , s [ Li+1 , K0 ] ) ( 6 ) where a [ Li , K0 : Kcout−1 ] denotes the sequence of a [ Li , K0 ] ∼ a [ Li , Kcout−1 ] ; and eRd [ Li , K0 : Kcout−1 ] means the sequence of eRd [ Li , K0 ] ∼ eRd [ Li , Kcout−1 ] . AutoQ stores these state-goal-reward transitions into the replay buffer . However , since transitions obtained from the past LLCs do not accurately reflect the actions that would occur if the same goal was used with the current LLC , AutoQ has to introduce a correction translating old transitions into ones that agree with the current LLC . AutoQ re-labels the high-level transition ( s [ Li , K0 ] , gLi , ∑ eRd [ Li , K0 : Kcout−1 ] , s [ Li+1 , K0 ] ) with a different goal ˜gLi chosen to maximize the probability µ LLC φLLC ( a [ Li , K0 : Kcout−1 ] |s [ Li , K0 : Kcout−1 ] , ˜gLi ) . AutoQ computes 10 candidate goals sampled randomly from a Gaussian distribution centered at gLi , and selects the minimal goal to re-label the experience . Quantization and Finetuning . During a search , we quantize the model by the learned quantization technique ( Zhang et al. , 2018 ) , and finetune the quantized model for ten epochs to recover the accuracy using stochastic gradient descent ( SGD ) with a fixed learning rate of 10−3 and momentum of 0.9 . We randomly select 100 categories from the ImageNet to accelerate the model finetuning . After the search is done , we quantize the model with the best policy found by AutoQ and finetune it on the full dataset . Implementation Details . An AutoQ agent , i.e. , HLC or LLC , consists of an actor network and a critic network . Both share the same architecture , i.e. , two hidden layers , each of which has 300 units . For the actor network , we add an additional sigmoid function producing an output in the range of [ 0 , 1 ] . We use a fixed learning rate of 10−4 for the actor network and 10−3 for the critic network . AutoQ trains the networks with the batch size of 64 and the replay buffer size of 2000 . AutoQ first explores 100 episodes with a constant noise , i.e. , δa [ Li , Kj ] = 0.5 for the LLC and δg [ Li ] = 0.5 for the HLC , and then exploits 300 episodes with exponentially decayed noise . Storage Cost . We need to record a 4-bit QBN ranging from 0 to 8 for each activation layer and each weight kernel of a convolutional layer . The storage overhead of AutoQ is ∼ 0.1 % of the size of various CNN models . For instance , ResNet-18 found by resource-constrained AutoQ requires 8.3MB to store its quantized model in Table 3 . The storage overhead of AutoQ is only 0.07 % .

Summary: This paper proposes a new method for quantizing neural network weights and activations that uses deep reinforcement learning to select the appropriate bitwidth for individual kernels in each layer. The algorithm uses a reward function that weights accuracy of the quantized model with latency, energy, and FPGA area, and leverages a high level and low-level controller to create quantized models that can take into account these factors. Compared to prior approaches taht only perform layer-wise instead of kernel-wise quantization, the quantized models can achieve better performance, or latency.