Simple Augmentation Goes a Long Way: ADRL for DNN Quantization
Authors: Lin Ning
Guoyang Chen
Weifeng Zhang
Xipeng Shen
Venue: ICLR 2021 Poster (2021)
Keywords: Reinforcement Learning
Quantization
mixed precision
augmented deep reinforcement learning
DNN

Abstract: Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values. Searching for the optimal bit-width for each layer, however, remains a challenge. Deep Reinforcement Learning (DRL) shows some recent promise. It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed-precision quantization problem. This paper proposes augmented DRL (ADRL) as a way to alleviate these issues. This new strategy augments the neural networks in DRL with a complementary scheme to boost the performance of learning. The paper examines the effectiveness of ADRL both analytically and empirically, showing that it can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64 times. 

Published as a conference paper at ICLR 2021
SIMPLE AUGMENTATION GOES A LONG WAY: ADRL FOR DNN QUANTIZATION
Lin Ning"
Guoyang Chen
North Carolina State University Raleigh, NC, USA Iningencsu.edu
Alibaba Group
Sunnyvale, CA, USA
g. wwwhhhenilaaaaee -enninnc.o com
Weifeng Zhang Alibaba Group Sunnyvale, CA, USA weifeng z@al waabbbbaaan...... com
Xipeng Shen
North Carolina State University Raleigh, NC, USA Secu..@ncsu.edu
ABSTRACT
Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values. Searching for the optimal bit-width for each layer, however, remains a challenge. Deep Reinforcement Learning (DRL) shows some recent promise. It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed precision quantization problem. This paper proposes augmented DRL (ADRL) as a way to alleviate these issues. This new strategy augments the neural networks in DRL with complementary scheme to boost the performance of learning. The paper examines the effectiveness of ADRL both analytically and empirically, showing that can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64x.
INTRODUCTION
By reducing the number of bits needed to represent model parameter of Deep Neural Networks
(DNN), quantization (Lin et al. 2016 Park et al. 2017 Han et al. 2015 Zhou et al. 2018 Zhu et al.
2016 Hwang& & Sung 2014 Wu al., 2016 Zhang et al. 2018 Köster al. 201 Ullrich et al.
2017 Hou & Kwok] 2018 1018/accot Jacob et al. 2018) is an important way to reduce the size and improve
the energy efficiency and speed of DNN. Mixed precision quantization selects proper bit-width for each layer of DNN, offering more flexibility than fixed precision quantization.
major challenge to mixed precision quantization (Micikevicius et al.) 2017 Cheng et al., 2018 is the configuration search problem, that is, how to find the appropriate bit-width for each DNN layer efficiently. The search space grows exponentially as the number of layers increases, and assessing each candidate configuration requires a long time of training and evaluation of the DNN.
Research efforts have been drawn to mitigate the issue for help better tap into the power of mixed precision quantization. Prior methods mainly fall into two categories: (i) automatic methods, such as
reinforcement learning (RL) (Lou et al. 2019 Gong et al. 2019 Wang et al. 2018 Yazdanbakhsh
et al. 2018 Cai et al. 2020) and neural architecture search (NAS) Wu al. 2018 Li al. 2022)), to learn from feedback signals and automatically determine the quantization configurations; (ii) heuristic methods to reduce the search space under the guidance of metrics such weight loss Hessian
spectrum (Dong et al. 2019 Wu et al. 2018 Zhou et al., 2018 8: Park et al. 2017) of each layer.
Comparing the heuristic method, automatic methods, especially Deep Reinforcement Learning (DRL), require little human effort and give the state-of-the-art performance (e.g., via actor-critic set-
"This work was done when Lin Ning was an intern at Alibaba Group.
Now at Google Research.

---

Published as a conference paper at ICLR 2021
ting (Whiteson et al. 2011 Zhang et al. 2016 Henderson et al. 2018 Wang al. 2018 ). however
suffers from overestimation bias, high variance of the estimated value, and hence slow convergence and suboptimal results. The problem is fundamentally due to the poor function approximations given out by the DRL, agent, especially during the early stage of the DRL learning process (Thrun & Schwartz 1993] Anschel et al.) 2017; Fujimoto al. (018)) when the neural networks used in the DRL are of low quality. The issue prevents DRL from serving as a scalable solution to DNN quantization DNN becomes deeper and more complex.
This paper reports that simple augmentations can bring some surprising improvements to DRL for DNN quantization. We introduce augmented DRL (ADRL) as a principled way to significantly magnify the potential of DRL for DNN quantization. The principle of ADRL is to augment the neural networks DRL with a complementary scheme (called augmentation scheme) to complement the weakness of DRI. policy approximator. Analytically, we prove the effects of such a method in reducing the variance and improving the convergence rates of DRL. Empirically, we exemplify ADRL with two example augmentation schemes and test on four popular DNNs. Comparisons with four prior DRL methods show that ADRL can shorten the quantization process by 4.5-64x while improving the model accuracy substantially. It is worth mentioning that there is some prior work trying to increase the scalability of DRL. Dulac-Aroo et al. (2016), for instance, addresses large discrete action spaces by embedding them into continuous spaces and leverages nearest neighbor to find closest actions. Our focus is different, aiming to enhance the learning speed of DRI. by augmenting the weak policy approximator with complementary schemes.
BACKGROUND
Deep Deterministic Policy Gradient (DDPG) A standard reinforcement learning system consists of an agent interacting with an environment & At each time step f. the agent receives an observation Tr, takes an action de and then receives an award rN. Modeled with Markov decision process (MDP) with a state space S and an action space A, an agent's behavior is defined by a policy π : S - A. I4) when the environment is (partially) observed. For DNN quantization, the environment is assumed to be fully observable (8, = 24)). The return from a state at time is defined as the future discounted return R₂ = 7 a;) with a discount factor y. The goal of the agent is to learn a policy that maximizes the expected return from the start state [(T) E[R, |-] An RL agent in continuous action spaces can be trained through the actor-critic algorithm and the deep deterministic policy gradient (DDPG). The parameterized actor function pts (0+) specifies the current policy and deterministically maps state 8 to an action a. The critic network Q( a) is a neural network function for estimating the vvluue E,R, s = s, a = a, T]; parameterized with 8 and is learned using the Bellman equation as Q-learming. The critic is updated by minimizing the loss
state defined as sequence of actions and observations st = (x1 (21.02, 02. ... a;-1,
L(0 = B[(y - Q(s,, ag ) where y1- = r(s,, 0z) + 7 Q((++1 (aa_,,,(((1) 1()==10")100) (1)
The actor is updated by applying the chain rule to the expected return J with respect its parameters:
V J = V. )l
(s(0"))
(2)
DDPG for Mixed Precision Quantization To apply the DRI. to mixed precision quantization, previous work, represented by HAQ (Wang et all.)2018) uses DDPG as the agent learning policy. The environment is assumed to be fully observed so that
Ht, where the observation æ, is
defined as xe = (1, Cin, Couts Skernel, Sstride; Sfeat, Uparama, idw:/w/a a;-1) for convolution layers and x2 = (I, han, hout, 1, 0, 8 feats Nparams: 0, 0, a111) for fully connected layers. Here, denotes the layer index, Cin. and Cout are the numbers of input and output channels for the convolution layer, Skernel and Sstride are the kernel size and stride for the convolution layer, hin and hout are the numbers of input and output hidden units for the fully connected layer, Nparems is the number of parameters, ddw and iwra are binary indicators for depth-wise convolution and weight/activation, and a--1 is the action given by the agent from the previous step.
In time step t 1, the agent gives an action dt-1 for layer - 1, leading to an observation Zy. Then the agent gives the action a, for layer at time step t given ø4. The agent updates the actor and critic networks after one episode following DDPG, which is a full pass of all the layers in the target neural network for quantization. The time step l and layer index are interchangeable in this scenario.
2

---

Published as a conference paper at ICLR 2021
argmax
generation
refinement
Q-value Indicator
state
actor
critic
latton
candidate actions
Augmented Policy Approximator
Figure I: Illustration of the augmented policy approximator
The systems use continuous action space for precision prediction. At each time step f, the continuous action 0r is mapped to the discrete bit value by for layer & using by round(bmis - 0.5 + dg X (bmar - bmin + 1)). The reward function is computed using R À x (40Cquant aCCorigin).
AUGMENTED DEEP REINFORCEMENT LEARNING (ADRL)
This section explains our proposal, ADRI.. It builds upon the default actor-critic algorithm and is trained with DDPG. In the original actor-critic algorithm, the policy approximator is the actor networks, which generates one action and feeds it into the critic networks. The essence of ADRL is augment the policy approximator with a supplementary scheme. The scheme may be in an arbitrary form, constructed with domain knowledge or other mechanisms. It should boost the weak policy approximator in accuracy especially when it is not yet well trained, and hence help efficiently reduce the variance and accelerate the convergence of the DRL algorithm.
3.1 DESIGN
Components of ADRL Figure iillustrates the general flow ADRL that employs post augmen- tation. The augmented policy approximator consists of two main components: an expanded actor network and refinement function. The expansion of the actor network makes output multiple rather than one candidate action: the refinement function derives the most promising candidate by processing those actions and feeds to the critic networks. The two components can be formally described as follows.
Actions generation The expanded actor function, Aga : + R*xn, maps a state from the state space Rm to ls (k>1) actions in continuous actions space R": A p(s)@") where Aş ....... [a],... a
The expansion can be done by modifying the last layer of the actor network (Sec (.3)..3). The outputs of this actor function serve the candidate actions the refinement component.
Action refinement. The action refinement function, g R - R" derives promising action from the candidate actions. A simple form of derivation is selection, that is, a" arg maxa=4.,- Q(s,, a) Depending on how well the critic network is trained, the critic may not be able to consistently give good estimation of Q(s, a), especially the early training stage. The augmented policy may use Q-value indicator (a(a) whose output depends on only the action and the property of the environment. Hence, ((A)) arg maxa= (a The choice of Qe (a) also depends on the specific tasks the ADRL trying solve.
Combining the generation function and the refinement function, the augmented policy estimator can be represented as Tii (s) g AAga.
Training with DDPG We train parameters for the actor and the critic networks using DDPG. Although the augmented policy тya consists of an actor network Aga and a refinement function the training of the full policy follows the policy gradient of igga because the effects of are deterministic aspects of the environment & (Dulac-Arold et al. 2015).
The actor and the critic network are trained with variations of formulae 1 and 2p:
T El(v Q( (s (00))0))00)))) where ye r(8,, Fop (st)) + 7 f6 100) 109)
3

---

Published as a conference paper at ICLR 2021
Algorithm I Augmented Policy
State s previously received from the environment &
2: A = {(s(0r) (generating A candidate actors)
3: g(A+)(a) (refines the choice of the action with g(As) arg ma Qa(a))
4: Apply to environment; receive
V x A l=m )o
(3)
3.2 EFFECTS ON VARIANCE AND LEARNING SPEED
Variance Reduction The actor-critic algorithm for mixed-precision quantization is a combination of Q-learning and function approximation. Therefore, suffers from various types of function approximation errors which affect the training stability. Analysis below follows the terminology from previous work (Thrun & Schwartz) 1993).
Definition If actions derived by the augmented policy approximator leads to higher rewards com- paring to actions given by the default actor network, the augmentation is effective augmentation.
Definition 2 Let Q(s ((s.a,)82)) be the value function of DRL at time step Q" a.)
m Q (&t. 04) be the optimal value function, and Ve.a = r(st, at) + + ax+1 109) be the DRL target. Then the function approximation error denoted as ô, can be decomposed as A; = (st, ar109)- Q2
Q* (s1, az) Q(st, as (62) ye + ty Q2 (5t,a)) The first two items form the Target Approximation Error (TAE), denoted as Z = -[(s,,, -V',ar
Proposition 1 Assuming that TAE is random process (Thrun & Schwwrtz 1993) such that F = 0, Tor(Zi.al = d². and Cool s,a: CoolZ.; 7:3,a'1 = for l F 12, then an effective aug- mentation leads to a smaller variance of the estimated Q value.
Proof: The estimated Q value function can be expressed as Qss. = Zg + r(s,, + 1((441, = 7 + r(st,a)) a.) + 7/Z + r(81+1,, + 10(1422 Et "r(s(r a:) + Z It is function of rewards and TAEs. Therefore, the variance of the estimated Q value will be proportional
the variance of the reward and TAE, such that VarlQ(s ar (19)) E y "Var( (s;; a;)] + Var[Zi,.) Let [rmnn T'marl and ['min Faaz] be the range of rewards associated with the default actor network and the augmented policy approxi- mator respectively. With an effective augmentation, we have Tmin < Fmmns and Tmar Tman- easy to derive that Var [7] < Var[r). Since Var a is the same for both of the cases, effective augmentation leads to a smaller variance the estimated Q value due to smaller reward variance. D
In ADRL, accuracy, leading to a higher reward. Therefore, helps reduce the variance the estimated Q value, confirmed by the experiment results in Section
Q-value indicator to help choose actions that tend to give higher quantization
Effects on Learning Speed
Proposition 2 An effective augmentation leads to larger step size and hence faster updates of the critic network.
Proof: The critic network is trained by minimizing loss as shown in Eq. Let 8, 3: -(s,, a 109). ) By differentiating the loss function with respect to 1 we get the formula for updating the critic network as 02 0 + aggo 6, V T.()).. Let gae the augmented policy approximator and Haa be the default actor network. With effective augmentation, the augmented policy approximator should generate an action that leads to higher rewards such that r(st, Tga (s:)) >= r(st, per (s:)). and has larger Q(5++1, a.+1). Assuming the same starting point for both т&a and Mar. which means that Q(st, ax) the same for both of the two policies, get 85 8:tm Therefore, tha leads to
larger aga 8, oe L(B9), and thus a larger step size and hence faster updates to 8?, the parameters in the critic network. 0
4

---

Published as a conference paper at ICLR 2021
Meanwhile, an effective augmentation helps DRL choose better- perterrperforming actions especially during the initial phases of the learning. Together with the faster updates, it tends to make the learning converge faster, consistently observed in our experiments reported in Section 4
3.3 APPLICATION ON MIXED-PRECISION QUANTIZATION
We now present how ADRL applies to mixed precision quantization. The key question is how to design the augmented policy approximator for this specific task. Our design takes the HAQ (Wang) et al. 2018) as the base. For the actor networks, our expansion is to the last layer. Instead of having only one neuron for the last layer, our action generator has & neurons corresponding to k candidate actions. We give detailed explanation next.
Q-value Indicator The augmentation centers around a Q-value indicator that we introduce to help select the most promising action from the candidates generated by the expanded actor function.
To fulfill the requirement of the ADRL., the Q-value indicator should be independent from the learnable parameters of the actor and critic networks. Its output shall be determined by the action and the environment E. Since the Q-value is the expected return, it is directly connected with the inference accuracy after quantization (aCCquant). this work, we experiment with two forms of Q-value indicators. They both estimate the relative fitness of the candidate quantization levels in terms of the influence on aCCquant- Their simplicities make them promising for actual adoptions.
Profiling-haser based indicator: The first form is to use the inference accuracy after quantization (aCCquant) such that (a) ~ qu M(x)))- At each time step f,, the action generator generates k action candidates for layer I, where 1 = t. For each action = Ag, the Q-value indicator computes the corresponding bit value 6, and quantizes layer to b, bits while keeping all other layers unchanged. The quantized network is evaluated using test dataset and the resulting accuracy is used as the indication of the fitness of the quantization level. The candidate giving the highest accuracy is selected by the action refinement function g. When there is tie, the one that leads to smaller network is selected by g-
Distance-based indicator: The second form is the distance between the original weight and the quantized weight given the bit value b(a)) for layer 1. We explore two forms of distance: 1.2 norm (Qæ ||W - quantize(W;, b(a:)l)) and KL-divergence (Wikipedia contributors 2020) (Qe ~ Dx (Wi) Will|uuntizeeWW b(a)))). Both of them characterize the distance between the two weight distributions, and hence how sensitive a certain layer with respect to quantization level. As the distance depends only on the quantized bit value by and the original weight, all the computations can be done offline ahead of time. During the search, the refinement function g selects the action that gives the smallest KI. -l.-diveeeeece for the corresponding layer from the candidate pool.
Implementation Details The entire process of peeeed--reeision quantization has three main stages: searching, finetuning, and evaluation. The main focus of this work is the search stage, where the ADRL for several episodes to find a quantization configuration that satisfies our requirements (e.g., weight compression ratio). Then we finetune the model with the optimal configuration for a number of epochs until the model converges. A final evaluation step gives the accuracy of the quantized model achieves.
In each episode of the search stage, the agent generates a quantization configuration with the guidance of both the quantized model accuracy and the model compression ratio. First, the Q-value indicator selects the candidate network that gives the highest accuracy for each layer. After selecting the bit values for all the layers, the agent checks whether the quantized network fulfills the compression ratio requirement. If not, it will decrease the bit value layer by layer starting from the one that gives the least accuracy loss to generate quantized model that fulfills the compression ratio requirement. The accuracy of this quantized model is used to compute the reward and to train the agent.
Memoization. When using the profiling based indicator, the agent needs to run inference on a subset of test dataset for each candidate action. To save time, we introduce memoization into the process. We use a dictionary to store the inference accuracy given by different bit-width values for each layer. Given candidate action a, for layer 1, the agent maps it to discrete bit value b, and checks if the inference accuracy of quantizing weights of layer to b, bits is stored in the corresponding dictionary. If so, the agent simply uses the stored value as the indicated Q-value for action a;- Otherwise, it.
5

---

Published as a conference paper at ICLR 2021
quantizes layer to b; bits, evaluates the network to get the accuracy the indicated Q Q-value, and adds the b; aCCquant(b:) pair into the dictionary.
Early termination. A typical design for the searching stage runs the RL agent for fixed number of episodes and outputs the configuration that gives the highest reward. We observed that in the process of searching for optimal configurations, the time it takes for the aCCquant reach stable status could be much less than the pre-set number of episodes. Especially when using ADRL, the agent generates actions with high rewards at very early stage. Therefore, we introduce an early termination strategy, which saves much computation time spending on the search process.
The general idea for early termination is to stop the search process when the inference accuracy has small variance among consecutive runs. We use the standard derivation among NN runs to measure the variance. If it less than a threshold (detailed in See 4). the search process terminates.
Time complexity. Let gen be the time of generating actions for each layer and Tevat be the time spending on evaluating the quantized network once. The time complexities of the original DRL method and ADRL without memorization are N-(L-T +Teval) and -L- + ----eaa where N and N' are the number of search episodes for the original DRL. method and ADRL, is the number of candidate actions in ADRL for each layer and L is the number of layers of the target network. The overhead brought by the Q-value indicator is N' Teval since needs to evaluate the network accuracy for each candidate action in each layer. With memorization, assuming is the number of possible bit values in the search space, this overhead is largely reduced as the Q-value indicator only needs to evaluate the network times for each layer during the entire search stage. Therefore, if the network needs to be quantized to at most bits, the overhead becomes 8L Teval- The time complexity is hence reduced to N' (L-Tgen + Teval) + SL Tevel. With early termination, N' is much smaller than N. Therefore, the total time spending on the search stage with ADRL is much smaller than that of the original DRL based method, which is demonstrated in the experiment results as shown in Sec
4 EXPERIMENTAL RESULTS
To evaluate the efficiency of augmented DRL on mixed precision quantization of DNNs, we ex-
periment with four different networks: CifarNet (Krizhevsky) 2012). ResNet20 (He et al.) 2016). AlexNet (Krizhevsky et al. 2012) and ResNet50 (He et al. (016)). The first two networks work on Cifar10 (Krizhevsky) 2012) dataset while the last two work on the ImageNet (Deng et al. 2009)
dataset. The experiments are done on a server with an Intel(R) Xeon(R) Platinum 8168 Processor, 32GB memory and 4 NVIDIA Tesla V100 32GB GPUs.
The entire process of DRL based mixed precissio quantization consists of three main stages: search, finetuning, and evaluation. The search stage has E episodes. Each episode consists of T time steps, where T equals the number of layers of the neural network. At each time step 4, the RL algorithm selects bit- tittw value for layer / t. After finishing 1 episode, it generates mixed-preciste configuration with the bit-width values for all the layers, quantizes the network accordingly, and evaluates the quantized network calculate the reward. It then updates the actor and critic parameters and resets the target DNN network to the original non-quantized one. The finetuning stage follows the search stage. In this stage, we quantize the DNN network using the -exeed-precision configuration selected by the RL algorithm at the end of the search stage and finetune the network for N epochs. The final stage is evaluation, which runs the finetuned quantized model on the test dataset. The resulting inference accuracy is the metric for the quality of the quantized network.
We compare our ADRL with four prior muixed-precisio quantization studies. Besides the already introduced HAQ (Wang et al. 2018), the prior studies include ReLeQ bakhhanbakhsh et al.) 2018). HAWQ (Dong et al. 2019), and ZeroQ (Cai et al., 2020). ReL.e (Yazdanbakhsh et al. (20T8) uses a unique reward formulation and shaping in its RL. algorithm so that it can simultaneously optimize for two objectives (accuracy and reduced computation with lower bit width) with a unified RL process. HAWQ (Dong et al. 2019) automatically selects the relative quantization precision of each layer based on the layer's 's Hessiam spectrum and determines the fine-tuning order for quantization layers based second-order information. ZeroQ (Cai et al. 2020) enables mix-precision quantization without any access to the training or validation data. uses a Pareto frontier based method to automatically determine the precdipre bit setting for all layers with no manual search involved.

---

Published as a conference paper at ICLR 2021
Our experiments use layer-wise quantization, but note that ADRL potentially augment policy approximators in quantizations at other levels (e.g. kernel-wise Lou et al. (2019)) in a similar way.
For ADRL, we experimented with both P-ADRL (profiling- and D-ADRL (isstacee-bseed) P-ADRL outperforms D-ADRL. We hence focus the overall results on P-ADRL, but offer the results of D-ADRL in the detailed analysis.
As ADRL is implemented based on HAQ, we collect detailed results on all experimented networks on HAQ (code downloaded from the original work (Wang et al.] 2018)) to give head-to-head comparison. For the other three methods, we cite the numbers reported in the published papers. The default setting of HAQ is searching for 600 episodes and finetuning for 100 epochs. We also apply early termination to HAQ (HAQ+E), where, the termination criterion is that the coefficient of variance of the averaged inference accuracy over 10 episodes is smaller than 0.01 for at least two consecutive 10-episode windows. In all experiments, k (number of candidate actions) is 3 for ADRL. We report the quality of the quantization first, and then focus on the speed and variance of the RL learning.
Table 1: Comparison of quantization results. (Comp: compression ratio; Acc A: accuracy loss; "-1 no results reported in the original paper)
ResNet20
AlexNet
ResNet50
Comp
Acc A
Comp
Acc A
Comp
Acc
ADRL (i.e., P-ADRL)
10.3X
10.1X
10.0X
0
HAQ (Wang et al. 2018
10.1X
0.41
10.8X
0
10.1X
1.168
ReLeQ Yazzanbakhsh et al. 2018
1.88X
0.12
3,56X
0,08
HAWQ Dong et al 2019)
13.1X
0.15
12.8X
1.91
ZeroQ Cal et al.
2020
8.4X
1.64
Compression Ratios and Accuracy Table comppree the compression ratios and accuracy. For the effects of the augmentation of the approximator, ADRL is the only method that delivers a comparable compression ratio without causing any accuracy loss on all three networks. Note that ADRL is more as a complement rather than a competitor to these work: The augmentation strategy it introduces can help complement the approximators the existing DRL especially when they are not yet well trained.
Learning Speed Table T reports the learning speed in each of the two major stages (search, finetune) and the overall learning process. The reported time is the total time before convergence. For the search stage, the time includes the overhead brought by the Q-value indicator. As shown in column and 8, the search stage is accelerated by 6-66x, while the finetune stage by up to 20x. CifarNet is an exception. For its simplicity, all methods converge in only one epoch in finetuning; because HAQ has larger compression ratio shown in Figure 2 neither P-ADRL nor HAQ+E has a speedup in finetuning over HAQ. But they take much fewer episodes to search and hence still get large overall speedups. Overall, P-ADRL shortens RDL learning by HAQ by factor of 4.5 to 64.1, while HAQ+E gets only 1.2 to 8.8.
The reasons for the speedups come from the much reduced numbers of episodes needed for the DRL to converge in search, and the higher accuracy (and hence earlier termination) that P-ADRL gives in the finetuning process. We next give detailed analysis by looking into each of the two stages in further depth.
Search Stage Figure shows the inference accuracy (acc), the coefficient of variance (CV) of the inference accuracy. and the compression ratio in terms of the weight ratio (wratio) of the network for the search stage. The first row of Figure gives the inference accuracy results in the search stage. we can see that the configurations found by P-ADRL give comparable inference accuracy for CifarNet and ResNet20 on Cifar dataset and better accuracy for AlexNet and ResNet50 on ImageNet dataset comparing to that of HAQ. Also, with early termination, our strategy terminates with much smaller number of search episodes comparing to HAQ. As shown in column 3 of Table 2 the numbers of search episodes of P-ADRL are 19.1% 19 34.5% of that of HAQ+E and 2.2% 12.3% of that of the original HAQ. The performance of D-ADRL is between that of P-ADRL and HAQ on CifarNet, ResNet20 and ResNet50 and the worst AlexNet. P-ADRL is more promise than D-ADRL does.
7

---

Published as a conference paper at ICLR 2021
Table 2: Execution time of each stage of original HAQ, HAQ with early termination (HAQ+E) and profiling based augmented DRL (P-ADRL) and the speedups of HAQ+E and P-ADRL over HAQ.
Search
Finetune
Overall
episode
time (s)
speedup
epoch
time(s)
speedup
speedup
HAQ
600
22464
CifarNet
HAQ+E
103
3856
5.8X
1
22
0.59X
5.8X 23.8X
P-ADRI.
30
935
24X
1
17
0.76X
HAQ
600
22680
100
2900
ResNer20
HAQ+E
68
2570
8.8X
100
2900
IX
8.8X
P-ADRI.
13
342
66.4X
5
140
20.7X
64.1X
HAQ
600
37800
43
19049
AlexNet
HAQ+E
277
17451
2.16X
100
44200
0.43X
1.16X
P-ADRL
74
6147
6.15X
35
16765
1.14X
4.49X
HAQ
600
51900
100
302800
ResNet50
HAQ+E
215
18597
2.79X
100
312600
0.97X
1.16X
P-ADRL
74
8580
6.04X
8
23360
12.9X
12.7X
cifarnet
resnet20
alexnet
resnet50
90
HAQ
D-ADRL
80
85
80-
HAQ P.ADR
HAQ
D-ADRL
60-
HAQ
P-ADRL
50
P-ADRL
75
200
400
600
200
400
600
200
400
600
0
200
400
500
0.06
0.10
0.05
HAQ
0.05
HAQ
0.08
HAQ
0.08
HAD
P.ADRL
P-ADRI
0.06
P-ADRI
0.06
P-ADRL
2 0.03
0.03
0.04
0.04
0.01
0.01
0.02
0.02
0.00
0.00
0.00
0.00
400
wrato
0.09
0.09
0.09
0.09
HAQ
HMQ
HNQ
HWQ
D-ADRI
D-ADRL
D-ADRL
D-ADRL
P-DDAL
P-ADLL
P-ADRL
P-ADRL
0.07
0.07
0.07
0.07
200
400
800
600
200
400
400
600
episode
opisode
episade
opisode
Figure 2: Results for search stage. Top Row: Accuracy. Middle Row: Coefficient Variance (CV). Bottom Row: Weight Ratio (wratio) = Weight size under current configuration / Original weight size. The horizontal dashed light blue line is the accuracy the original network before the quantization. The blue and green lines the accuracy result for our strategies (P-ADRL and D-ADRL) while the dashed red line shows the result for HAQ. The two vertical lines mark the episodes at which the strategies terminated with early termination.
The accuracy given by P-ADRL is much stabler than that of the HAQ. We use the coefficient of variance (CV) to measure the stability of the inference accuracy. As shown in the second row of Figure 2 P-ADRL achieves a small CV much faster than HAQ on all four networks. This is the reason that P-ADRL terminates in fewer episodes than HAQ and HAQ+E do; this conclusion holds regardless of what variance threshold is used in the early termination as the CVs P-ADRL are consistently smaller than those of HAQ.
8

---

Published as a conference paper at ICLR 2021
The third row of Figure shows shows the compression ratio after quantization each episode for each of the three algorithms. The target ratio set to 0.1. All three strategies fulfill the size limit requirement. By the end of the search stage, P-ADRL achieves compression ratio similar to HAQ on large networks (ResNer20, AlexNet and ResNet50). On a small network CifarNet, the compression ratio given by P-ADRL is slightly larger than that of HAQ, but still smaller than 0.1.
Finetuning Stage After getting network precision from the search stage, quantize the networks and finetune for 100 epochs. The learning rate is set as 0.001, which is the default setting from HAQ.
cifarnet
resne
alexnet
resnet50
76
92
56 Aubae
75
90
74
3 54
888-
73
86
P-ADRL
HAQ+E
87
HAQ+E P-ADRL
52
HAQ+E P-ADRL
72
HAQ+E
P-ADRL
71
84
85
50
70
25
25
100
100
epoch
epoch
epoch
epoch
Figure 3: Finetune accuracy. (Horizontal dashed line: the original accuracy without quantization.)
Figure 3 iilustrates the accuracy results of the finetune stages for three different precision settings: the one selected by P-ADRL; the one selected by HAQ+E and the one selected by the original HAQ. For CifarNet, all three achieve similar accuracies, higher than that of the original. For the other three networks, P-ADRL gets the largest accuracy, fully recovering the original accuracy. HAQ+E performs the worst, suffering at least 1% accuracy loss.
The sixth column of Table Sableegiven gives the number of epochs each strategy needs for finetuning before reaching the accuracy of the original network; '100' means that the strategy cannot recover the original accuracy in the 100-epoch finetuning stage. P-ADRL takes 1-35 epochs to converge to the original accuracies, much fewer than HAQ and HAQ+E Finetuning hence stops earlier.
CONCLUSION
This paper has demonstrated that simple augmentation goes long way in boosting DRL for DNN mixed precision quantization. It is easy to apply ADRL to deeper models such ResNet-101 and Resnet-152 In principle, the basic ideas of the reinforcement learning and the Q-value indicator are independent of the detailed architecture of the network model. Although the design of ADRL is motivated by DNN quantization, with appropriate augmentation schemes, the concept is potentially applicable to other RL applications a direction worth future exploration.
also possible to extend the method to latest model architectures
REFERENCES
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, 2017.
Yaohui Cai, Zhewei Yao, Zengchuan Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. ArXiv, abs/2001.00281, 2020.
Hsin-Pai Cheng, Yuanjun Huang, Xuyang Guo, Yifei Huang, Feng Yan, Hai Li, and Yiran Chen. Differentiable fine-grane grained quantization for deep neural network compression. arXiv preprint 1rXiv:18101103511 2018
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scage hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.

---

Published as a conference paper at ICLR 2021
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision In The IEEE International Conference Computer Viston (ICCV), October 2019.
Gabriel Dulac-Amold, Richard Evans, Peter Sunehag, and Ben Coppin. Reinforcement learning in large discrete action spaces, ArXiv, abs/1512.07677 2015.
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. ArXiv, arXiv:1512.0761: 2016.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprini arXiv: 010...00299977 2018.
Chengyue Gong, Zixuan Jiang, Dilin Wang, Yibo Lin, Qiang Liu, and David Pan. Mixed precision neural architecture search for energy efficient deep learning. In 2019 IEEE/ACM International Conference on Computer-Aee Aided Design (ICCAD), pp. 1-7. IEEE, 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 2015.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pincau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI, 2018.
Lu Hou and James Kwok. Loss-aware weight quantization of deep networks. arXiv preprint arXiv:/802.00633. 2018.
Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, O, and- 1. In 20/4 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2014.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-rrithmetic-olly inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713 2018
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Hornof, et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in neural information processing systems, pp. 1742-1752 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks, In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, pp. 1097-1105, USA, 2012. Curran Associates Inc.
Bapu Li, Yanwen Fan, Zhiyu Cheng, and Yingze Bao. Cursor based adaptive quantization for deep neural network, 2020. URL
htt.:/nttfoorta d= gL3RVtrr
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-2858, 2016.
Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang. Autoq: Automated kernel-wise neural network quantization. ArXiv, arXiv:1902.05665 2019.
Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 1710.03740, 2017.
10

---

Published as a conference paper at ICLR 2021
Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-entropy based quantization for deep neu- ral networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5456--6664, 2017.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionisl Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv: 1702.04000000 2017.
Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In CVPR, 2018.
S. Whiteson, B. Tanner, M. E. Taylor, and P. Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), 2011.
Wikipedia contributors. Kullback-leibler divergence -
Wikipedia, the free encyclopedia, wuwpppttleptuullllll.. 1backkkl [Online; accessed 22-Jamuayy-2020)
2020. URL atttppss//e..wiippedaaaaaa.a.a 80893Leibler www.ergeeceeollllaa---33333
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint 1812.009990000 2018.
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1820-4828, 2016.
Amir Yazdanbakhsh, Ahmed Elthakeb, Prannoy Pilligundla, FatemehSadat Mireshghallah, and Hadi Esmaeilzadeh. Releq: A reinforcement learning approach for deep quantization of neural networks. ArXiv, 0bbs/1811.01000 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1rXiv:161..035300 2016.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 365-382, 2018
Yiren Zhou, Seyed- Moosavi-Dezfool Ngai-Man Cheung, and Pascal Frossard. Adaptive quantization for deep neural network. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint 1rX1v:612..0104, 2016.

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILS OF THE MIXED QUANTIZATION BITS FOR EACH MODEL
Figure shows the comparison between the quantized networks selected using ADRL and HAQ. In each plot, the upper half shows the bit value being used for each layer for the network quantized with ADRL. The bottom half are the quantization configurations of the networks being quantized with HAQ.
#30 (ADRL)
abt (HAI)
3bc (HAR)
layer
layer
layer
(a) CifarNet and AlexNet
(b) ResNet20
#bit (HAO) #biR LADRR
layer
ResNet50
Figure 4: The bit value of each layer for (a) CifarNet (left) and AlexNet (right), (b) ResNet20 and (c) ResNet50. The methods for quantization are ADRL and HAQ.
A.2 EVOLUTION OF THE SELECTED BIT VALUES
cifarnet
resnet20
alexnet
resnet50
byeer
bayer
#PbIc
#bit
#bil
#bC
episode
episode
episode
episode
Figure 5: The bit values being selected in each search episode for the four networks.
Figure 5 Biillustrates how the selected bit values change during the search stage for layers of the four networks. The value being selected for the same layer changes from one episode to the next. It spans well in the action search space.
12


Reviews:
[{"review_id": "Uhmaq_TLLcX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an improved way to determine weight quantization bit lengths using reinforcement learning, by injecting model evaluation directly into action selection.  Building upon a DRL setup where the action at each timestep corresponds to selecting a bit value for each layer, the method adds a \"Q-value indicator\" function Q~ that selects among a set of candidate actions, and filters based on the model performance quantizing the layer to each level.  This seems to form a hybrid between DRL and greedy search, using a greedy criteria Q~ to filter proposals made by the DRL agent.  Experiments show very good performance, with similar or better quantization levels and accuracy as other DRL-based methods and much faster runtime.\n\nThe method is well described overall, though I would have liked more details on the mu and Q networks including their initializations and how well the mu proposals span the action space initially (see questions below).\n\nLikewise, I wonder if the mechanism of improvement is mostly through the initial guess provided by greedy search with Q~ at the beginning of training.  An alternative may be to edit the action network mu using a guess made by greedy profiling (see question 3 below) -- was anything like this explored?\n\nAlthough I wonder whether the method may be able to be further simplified, I still find this a good paper overall, offering an effective way to reduce time to produce quantized models with no accuracy hit.  A few additional baselines of even simpler systems, including greedy search and greedy policy initialization, could help provide more context and assess the use of Q~ as a filter.\n\n\n\nAdditional questions:\n\n1.  The descretized action space only has 8 values per layer.  What happens if one evaluates all 8 with the Q~ indicator and remove the candidate generation network?  Is this equivalent to a layerwise greedy search?  What is the final accuracy and quantization speed of this baseline?\n\n2.  How is the last layer of the expanded actor function mu^ initialized?  Randomly, or in such a way that the initial outputs correspond to different discrete bit sizes?  How many candidate actions are there?  How well do the candidate values span the action space to allow for guidance by Q~, particularly in the beginning episodes?\n\n3.  Instead of having multiple candidates, is it possible to introduce an output bias per layer in the actor function mu, initialized to the bit sizes determined by greedy profiler search with Q~ ?  That is, to use a function mu_l(s) = mu(s) + b(l), where b(l) = argmax_a Q~(a) for each layer l, then use regular DRL (not ADRL) with this mu_l that has an initial guess using greedy search from the profiler?\n\n4.  I don't see where the system is encouraged to use fewer bits.  If more bits generally leads to higher accuracy, what makes the system learn to output smaller bits in its actions?  Both Q~ augmentation and the reward R = acc_quant - acc_orig depend only on accuracy.\n\n5.  How different are the proposal actions from one another (do they span the space of possible bit lengths), and how do they change over the course of model selection?  Is the initial max according to Q~ used most of the time or are there times when the proposal index selected by Q~ changes from one episode to the next?", "extra_scores": {}, "extra_text": {}}, {"review_id": "UYpoZsXW1zs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "========\nSummary\n\nThis paper studies the DNN quantization using deep reinforcement learning. The paper proposes an augmented DRL which introduces a Q-value indicator to refine action selection. The proposed approach has been applied to several image classification baselines and has compared with several recent DRL based quantization approach, achieving a similar compression rate without accuracy decrease.  In addition, compared to previous methods, the learning speed has been improved by 4.5-64x. \n\n=========\nPros\n\n1. The paper is well written and easy to follow.  It is very clear even for an audience who may not be an expert on DNN quantization. \n2. The idea is simple and reasonable to introduce an additional Q-value indicator to refine action selection.  Given the improvement in performance,  despite the simplicity, the method does provide great performance. \n\n\n==========\nConcerns/Comments\n\n1.  Is there any result about the final quantization configuration of the compressed model? Is there any takeaway about the quantization pattern? \n\n2.  What is the action space used in the experiments? How many bits were the model used? \n\n3. In Table 1, could you provide the original accuracy of the models rather than the accuracy delta?  In addition, is that possible to quantize deeper models like ResNet-101 or ResNet 152? Is the method extendable to the latest model architectures such as ResNeXt, DenseNet, visual transformers?\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SppWcOiYOAX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work attempts to improve DRL based mixed-precision network compression. Compared to the standard action-critic method, he proposed method inserts an additional candidate evaluation step on k proposed actions as shown in Algorithm 1. As the paper title suggests, the proposed augmentation is “simple” yet practically brings non-trivial performance improvement. \n\nI am not a real expert on RL. In general, I appreciate the idea in this work. It generates higher-quality action proposals by heuristically evaluating a few of them, either using a quick calculation of inference accuracy or direct quantization loss. The proposed idea is intuitive and well validated by experiments. \n\nI have several suggestions for improving this work: the distance-based indicator is proved to be inferior in terms of performance, mentioned by authors. It is still suggested to report its accuracies in Tables 1 and 2, for a better view of two variants.  A few issues in experiments need clarification: is the reported time the total before convergence, or the per-episode time? The proposed method takes much fewer episodes to search but an additional q-value indicator brings more computations. More analysis regarding the complexity is required. There are a few ad hoc treatments in the method, such as memorizing the accuracy of specific bit value at a layer. Does it consider the evolution of the entire network (since the change of other layers have impact)? The theoretic analysis is based on assumptions that can be only empirically verified (yet not convincingly verified in the experimental sections). It is not clear to me the true value of section 3.2.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple yet effective approach for determining weight quantization bit lengths using RL. All the reviewers agree that the simplicity and performance improvements are a strong plus point. There are some concerns on applicability which have been sufficiently handled by rebuttal. AC recommends accepting the paper.
Decision: Accept (Poster)