SIMPLE AUGMENTATION GOES A LONG WAY: ADRL FOR DNN QUANTIZATION
Year: 2021

Abstract:
Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values. Searching for the optimal bit-width for each layer, however, remains a challenge. Deep Reinforcement Learning (DRL) shows some recent promise. It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed precision quantization problem. This paper proposes augmented DRL (ADRL) as a way to alleviate these issues. This new strategy augments the neural networks in DRL with a complementary scheme to boost the performance of learning. The paper examines the effectiveness of ADRL both analytically and empirically, showing that it can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64×.

heading: 1 INTRODUCTION
text: By reducing the number of bits needed to represent a model parameter of Deep Neural Networks (DNN), quantization (Lin et al., 2016; Park et al., 2017; Han et al., 2015; Zhou et al., 2018; Zhu et al., 2016; Hwang & Sung, 2014; Wu et al., 2016; Zhang et al., 2018; Köster et al., 2017; Ullrich et al., 2017; Hou & Kwok, 2018; Jacob et al., 2018) is an important way to reduce the size and improve the energy efficiency and speed of DNN. Mixed precision quantization selects a proper bit-width for each layer of a DNN, offering more flexibility than fixed precision quantization.
A major challenge to mixed precision quantization (Micikevicius et al., 2017; Cheng et al., 2018) is the configuration search problem, that is, how to find the appropriate bit-width for each DNN layer efficiently. The search space grows exponentially as the number of layers increases, and assessing each candidate configuration requires a long time of training and evaluation of the DNN.
Research efforts have been drawn to mitigate the issue for help better tap into the power of mixed precision quantization. Prior methods mainly fall into two categories: (i) automatic methods, such as reinforcement learning (RL) (Lou et al., 2019; Gong et al., 2019; Wang et al., 2018; Yazdanbakhsh et al., 2018; Cai et al., 2020) and neural architecture search (NAS) (Wu et al., 2018; Li et al., 2020), to learn from feedback signals and automatically determine the quantization configurations; (ii) heuristic methods to reduce the search space under the guidance of metrics such as weight loss or Hessian spectrum (Dong et al., 2019; Wu et al., 2018; Zhou et al., 2018; Park et al., 2017) of each layer.
Comparing to the heuristic method, automatic methods, especially Deep Reinforcement Learning (DRL), require little human effort and give the state-of-the-art performance (e.g., via actor-critic set∗This work was done when Lin Ning was an intern at Alibaba Group. †Now at Google Research.
ting (Whiteson et al., 2011; Zhang et al., 2016; Henderson et al., 2018; Wang et al., 2018)). It however suffers from overestimation bias, high variance of the estimated value, and hence slow convergence and suboptimal results. The problem is fundamentally due to the poor function approximations given out by the DRL agent, especially during the early stage of the DRL learning process (Thrun & Schwartz, 1993; Anschel et al., 2017; Fujimoto et al., 2018) when the neural networks used in the DRL are of low quality. The issue prevents DRL from serving as a scalable solution to DNN quantization as DNN becomes deeper and more complex.
This paper reports that simple augmentations can bring some surprising improvements to DRL for DNN quantization. We introduce augmented DRL (ADRL) as a principled way to significantly magnify the potential of DRL for DNN quantization. The principle of ADRL is to augment the neural networks in DRL with a complementary scheme (called augmentation scheme) to complement the weakness of DRL policy approximator. Analytically, we prove the effects of such a method in reducing the variance and improving the convergence rates of DRL. Empirically, we exemplify ADRL with two example augmentation schemes and test on four popular DNNs. Comparisons with four prior DRL methods show that ADRL can shorten the quantization process by 4.5-64× while improving the model accuracy substantially. It is worth mentioning that there is some prior work trying to increase the scalability of DRL. Dulac-Arnold et al. (2016), for instance, addresses large discrete action spaces by embedding them into continuous spaces and leverages nearest neighbor to find closest actions. Our focus is different, aiming to enhance the learning speed of DRL by augmenting the weak policy approximator with complementary schemes.

heading: 2 BACKGROUND
text: Deep Deterministic Policy Gradient (DDPG) A standard reinforcement learning system consists of an agent interacting with an environment E . At each time step t, the agent receives an observation xt, takes an action at and then receives an award rt. Modeled with Markov decision process (MDP) with a state space S and an action space A, an agent’s behavior is defined by a policy π : S → A. A state is defined as a sequence of actions and observations st = (x1, a2, · · · , at−1, xt) when the environment is (partially) observed. For DNN quantization, the environment is assumed to be fully observable (st = xt). The return from a state s at time t is defined as the future discounted return Rt = ∑T i=t γ
i−tr(si, ai) with a discount factor γ. The goal of the agent is to learn a policy that maximizes the expected return from the start state J (π) = E[R1|π] . An RL agent in continuous action spaces can be trained through the actor-critic algorithm and the deep deterministic policy gradient (DDPG). The parameterized actor function µ(s|θµ) specifies the current policy and deterministically maps a state s to an action a. The critic networkQ(s, a) is a neural network function for estimating the action-value E[Rt|st = s, at = a, π]; it is parameterized with θQ and is learned using the Bellman equation as Q-learning. The critic is updated by minimizing the loss
L(θQ) = E[(yt −Q(st, at|θQ))2], where yt = r(st, at) + γQ(st+1, µ(st+1|θµ)|θQ). (1)
The actor is updated by applying the chain rule to the expected return J with respect to its parameters:
∇θµJ ≈ E[∇aQ(s, a|θQ)|s=st,a=µ(st)∇θµµ(s|θ µ)|s=st ]. (2)
DDPG for Mixed Precision Quantization To apply the DRL to mixed precision quantization, previous work, represented by HAQ (Wang et al., 2018), uses DDPG as the agent learning policy. The environment is assumed to be fully observed so that st = xt, where the observation xt is defined as xt = (l, cin, cout, skernel, sstride, sfeat, nparams, idw, iw/a, at−1) for convolution layers and xt = (l, hin, hout, 1, 0, sfeat, nparams, 0, iw/a, at−1) for fully connected layers. Here, l denotes the layer index, cin and cout are the numbers of input and output channels for the convolution layer, skernel and sstride are the kernel size and stride for the convolution layer, hin and hout are the numbers of input and output hidden units for the fully connected layer, nparams is the number of parameters, idw and iw/a are binary indicators for depth-wise convolution and weight/activation, and at−1 is the action given by the agent from the previous step.
In time step t− 1, the agent gives an action at−1 for layer l − 1, leading to an observation xt. Then the agent gives the action at for layer l at time step t given xt. The agent updates the actor and critic networks after one episode following DDPG, which is a full pass of all the layers in the target neural network for quantization. The time step t and layer index l are interchangeable in this scenario.
The systems use a continuous action space for precision prediction. At each time step t, the continuous action at is mapped to the discrete bit value bk for layer k using bk = round(bmin − 0.5 + ak × (bmax − bmin + 1)). The reward function is computed usingR = λ× (accquant − accorigin).

heading: 3 AUGMENTED DEEP REINFORCEMENT LEARNING (ADRL)
text: This section explains our proposal, ADRL. It builds upon the default actor-critic algorithm and is trained with DDPG. In the original actor-critic algorithm, the policy approximator is the actor networks, which generates one action and feeds it into the critic networks. The essence of ADRL is to augment the policy approximator with a supplementary scheme. The scheme may be in an arbitrary form, constructed with domain knowledge or other mechanisms. It should boost the weak policy approximator in accuracy especially when it is not yet well trained, and hence help efficiently reduce the variance and accelerate the convergence of the DRL algorithm.

heading: 3.1 DESIGN
text: Components of ADRL Figure 1 illustrates the general flow of ADRL that employs post augmentation. The augmented policy approximator consists of two main components: an expanded actor network and a refinement function. The expansion of the actor network makes it output multiple rather than one candidate action; the refinement function derives the most promising candidate by processing those actions and feeds it to the critic networks. The two components can be formally described as follows.
Actions generation The expanded actor function, µ̂θµ̂ : S → Rk×n, maps a state from the state space Rm to k (k>1) actions in a continuous actions space Rn: Ak = µ̂(s|θµ̂) where Ak = [aT1 , · · · ,aTk ]T . The expansion can be done by modifying the last layer of the actor network (Sec 3.3). The outputs of this actor function serve as the candidate actions to the refinement component.
Action refinement. The action refinement function, g : Rk×n → Rn, derives a promising action from the candidate actions. A simple form of derivation is selection, that is, a∗ = arg maxa=Ak,i Q(s,a) Depending on how well the critic network is trained, the critic may not be able to consistently give a good estimation of Q(s,a), especially at the early training stage. The augmented policy may use a Q-value indicator Q̃E(a) whose output depends on only the action and the property of the environment. Hence, g(Ak) = arg maxa=Ak,i Q̃E(a) The choice of Q̃E(a) also depends on the specific tasks the ADRL is trying to solve.
Combining the generation function and the refinement function, the augmented policy estimator can be represented as πθµ̂(s) = g ◦ µ̂θµ̂ .
Training with DDPG We train parameters for the actor and the critic networks using DDPG. Although the augmented policy πθµ̂ consists of an actor network µ̂θµ̂ and a refinement function g, the training of the full policy follows the policy gradient of µ̂θµ̂ because the effects of g are deterministic aspects of the environment E (Dulac-Arnold et al., 2015). The actor and the critic network are trained with variations of formulae (1) and (2):
L(θQ) = E[(yt −Q(st, µ̂(st|θµ̂)|θQ))2], where yt = r(st, πθµ̂(st)) + γQ(st+1, µ̂(st+1|θµ̂)|θQ)
Algorithm 1 Augmented Policy 1: State s previously received from the environment E 2: Ak = µ(s|θµ) (generating k candidate actors) 3: a = g(Ak)(a) (refines the choice of the action with g(Ak) = arg maxa=Ak,i Q̃E(a)) 4: Apply a to environment; receive r, s′
∇θµ̂J ≈ E[∇AkQ(s,Ak|θQ)|s=st,Ak=µ̂(st) · ∇θµ̂ µ̂(s|θ µ̂)|s=st ]. (3)

heading: 3.2 EFFECTS ON VARIANCE AND LEARNING SPEED
text: Variance Reduction The actor-critic algorithm for mixed-precision quantization is a combination of Q-learning and function approximation. Therefore, it suffers from various types of function approximation errors which affect the training stability. Analysis below follows the terminology from previous work (Thrun & Schwartz, 1993).
Definition 1 If actions derived by the augmented policy approximator leads to higher rewards comparing to actions given by the default actor network, the augmentation is an effective augmentation.
Definition 2 Let Q(st, at|θQ) be the value function of DRL at time step t, Q∗(st, at) = maxπ Q
π(st, at) be the optimal value function, and yts,a = r(st, at) + γQ(st+1, at+1|θQ) be the DRL target. Then the function approximation error denoted as δt can be decomposed as ∆t = Q(st, at|θQ)−Q∗(st, at) = Q(st, at|θQ)− yts,a + yts,a −Q∗(st, at) The first two items form the Target Approximation Error (TAE), denoted as Zts,a = Q(st, at|θQ)− yts,a.
Proposition 1 Assuming that TAE is a random process (Thrun & Schwartz, 1993) such that E[Zts,a] = 0, V ar[Z t s,a] = σ 2 s , and Cov[Z t1 s,a, Z t2 s′,a′ ] = 0 for t1 6= t2, then an effective augmentation leads to a smaller variance of the estimated Q value.
Proof: The estimated Q value function can be expressed as Q(st, at|θQ) = Zts,a + r(st, at) + γQ(st+1, at+1|θQ) = Zts,a + r(st, at) + γ[Zt+1s,a + r(st+1, at+1) + γQ(st+2, at+2|θQ)] =∑T i=t γ i−t[r(si, ai) + Z i s,a]. It is a function of rewards and TAEs. Therefore, the variance of the estimated Q value will be proportional to the variance of the reward and TAE, such that V ar[Q(st, at|θQ)] = ∑T i=t γ i−t(V ar[r(si, ai)] + V ar[Z i s,a]). Let [rmin, rmax] and [r̃min, r̃max] be the range of rewards associated with the default actor network and the augmented policy approximator respectively. With an effective augmentation, we have rmin < r̃min, and rmax = r̃max. It is easy to derive that V ar[r̃] < V ar[r]. Since V ar[Zis,a] is the same for both of the cases, an effective augmentation leads to a smaller variance of the estimated Q value due to smaller reward variance.
In ADRL, we use a Q-value indicator to help choose actions that tend to give higher quantization accuracy, leading to a higher reward. Therefore, it helps reduce the variance of the estimated Q value, as confirmed by the experiment results in Section 4.
Effects on Learning Speed
Proposition 2 An effective augmentation leads to larger step size and hence faster updates of the critic network.
Proof: The critic network is trained by minimizing loss as shown in Eq. 1. Let δt = yt−Q(st, at|θQ). By differentiating the loss function with respect to θQ, we get the formula for updating the critic network as θQt+1 = θ Q t + αθQ · δt · ∇θQL(θQ). Let πθµ̂ be the augmented policy approximator and µθµ be the default actor network. With an effective augmentation, the augmented policy approximator should generate an action that leads to higher rewards such that r(st, πθµ̂(st)) >= r(st, µθµ(st)), and has a larger Q(st+1, at+1). Assuming the same starting point for both πθµ̂ and µθµ , which means that Q(st, at) are the same for both of the two policies, we get δ π θµ̂ t > δ µθµ t . Therefore, πθµ̂ leads to larger αθQ · δt · ∇θQL(θQ), and thus a larger step size and hence faster updates to θQ, the parameters in the critic network.
Meanwhile, an effective augmentation helps DRL choose better-performing actions especially during the initial phases of the learning. Together with the faster updates, it tends to make the learning converge faster, as consistently observed in our experiments reported in Section 4.

heading: 3.3 APPLICATION ON MIXED-PRECISION QUANTIZATION
text: We now present how ADRL applies to mixed precision quantization. The key question is how to design the augmented policy approximator for this specific task. Our design takes the HAQ (Wang et al., 2018) as the base. For the actor networks, our expansion is to the last layer. Instead of having only one neuron for the last layer, our action generator has k neurons corresponding to k candidate actions. We give a detailed explanation next.
Q-value Indicator The augmentation centers around a Q-value indicator that we introduce to help select the most promising action from the candidates generated by the expanded actor function.
To fulfill the requirement of the ADRL, the Q-value indicator should be independent from the learnable parameters of the actor and critic networks. Its output shall be determined by the action a and the environment E . Since the Q-value is the expected return, it is directly connected with the inference accuracy after quantization (accquant). In this work, we experiment with two forms of Q-value indicators. They both estimate the relative fitness of the candidate quantization levels in terms of the influence on accquant. Their simplicities make them promising for actual adoptions.
Profiling-based indicator. The first form is to use the inference accuracy after quantization (accquant) such that QE(a) ∼ accquant|quantize(Wl,b(al)). At each time step t, the action generator generates k action candidates for layer l, where l = t. For each action ai = Ak,i, the Q-value indicator computes the corresponding bit value bi and quantizes layer l to bi bits while keeping all other layers unchanged. The quantized network is evaluated using a test dataset and the resulting accuracy is used as the indication of the fitness of the quantization level. The candidate giving the highest accuracy is selected by the action refinement function g. When there is a tie, the one that leads to a smaller network is selected by g.
Distance-based indicator. The second form is the distance between the original weight and the quantized weight given the bit value b(al) for layer l. We explore two forms of distance: L2 norm (QE(a) ∼ ||Wl − quantize(Wl, b(al)||) and KL-divergence (Wikipedia contributors, 2020) (QE(a) ∼ DKL(Wl||quantize(Wl, b(al)))). Both of them characterize the distance between the two weight distributions, and hence how sensitive a certain layer is with respect to a quantization level. As the distance depends only on the quantized bit value bl and the original weight, all the computations can be done offline ahead of time. During the search, the refinement function g selects the action that gives the smallest KL-divergence for the corresponding layer from the candidate pool.
Implementation Details The entire process of mixed-precision quantization has three main stages: searching, finetuning, and evaluation. The main focus of this work is the search stage, where we run the ADRL for several episodes to find a quantization configuration that satisfies our requirements (e.g., weight compression ratio). Then we finetune the model with the optimal configuration for a number of epochs until the model converges. A final evaluation step gives the accuracy of the quantized model achieves.
In each episode of the search stage, the agent generates a quantization configuration with the guidance of both the quantized model accuracy and the model compression ratio. First, the Q-value indicator selects the candidate network that gives the highest accuracy for each layer. After selecting the bit values for all the layers, the agent checks whether the quantized network fulfills the compression ratio requirement. If not, it will decrease the bit value layer by layer starting from the one that gives the least accuracy loss to generate a quantized model that fulfills the compression ratio requirement. The accuracy of this quantized model is used to compute the reward and to train the agent.
Memoization. When using the profiling based indicator, the agent needs to run inference on a subset of test dataset for each candidate action. To save time, we introduce memoization into the process. We use a dictionary to store the inference accuracy given by different bit-width values for each layer. Given a candidate action ai for layer l, the agent maps it to a discrete bit value bi and checks if the inference accuracy of quantizing weights of layer l to bi bits is stored in the corresponding dictionary. If so, the agent simply uses the stored value as the indicated Q-value for action ai. Otherwise, it
quantizes layer l to bi bits, evaluates the network to get the accuracy as the indicated Q-value, and adds the bi : accquant(bi) pair into the dictionary.
Early termination. A typical design for the searching stage runs the RL agent for a fixed number of episodes and outputs the configuration that gives the highest reward. We observed that in the process of searching for optimal configurations, the time it takes for the accquant to reach a stable status could be much less than the pre-set number of episodes. Especially when using ADRL, the agent generates actions with high rewards at a very early stage. Therefore, we introduce an early termination strategy, which saves much computation time spending on the search process.
The general idea for early termination is to stop the search process when the inference accuracy has small variance among consecutive runs. We use the standard derivation among N runs to measure the variance. If it is less than a threshold (detailed in Sec 4), the search process terminates.
Time complexity. Let Tgen be the time of generating actions for each layer and Teval be the time spending on evaluating the quantized network once. The time complexities of the original DRL method and ADRL without memorization areN ·(L·Tgen+Teval) andN ′ ·(L·Tgen+(kL+1)·Teval), where N and N ′ are the number of search episodes for the original DRL method and ADRL, k is the number of candidate actions in ADRL for each layer and L is the number of layers of the target network. The overhead brought by the Q-value indicator is N ′kL · Teval since it needs to evaluate the network accuracy for each candidate action in each layer. With memorization, assuming n is the number of possible bit values in the search space, this overhead is largely reduced as the Q-value indicator only needs to evaluate the network n times for each layer during the entire search stage. Therefore, if the network needs to be quantized to at most 8 bits, the overhead becomes 8L · Teval. The time complexity is hence reduced to N ′ · (L ·Tgen +Teval) + 8L ·Teval. With early termination, N ′ is much smaller than N . Therefore, the total time spending on the search stage with ADRL is much smaller than that of the original DRL based method, which is demonstrated in the experiment results as shown in Sec 4.

heading: 4 EXPERIMENTAL RESULTS
text: To evaluate the efficiency of augmented DRL on mixed precision quantization of DNNs, we experiment with four different networks: CifarNet (Krizhevsky, 2012), ResNet20 (He et al., 2016), AlexNet (Krizhevsky et al., 2012) and ResNet50 (He et al., 2016). The first two networks work on Cifar10 (Krizhevsky, 2012) dataset while the last two work on the ImageNet (Deng et al., 2009) dataset. The experiments are done on a server with an Intel(R) Xeon(R) Platinum 8168 Processor, 32GB memory and 4 NVIDIA Tesla V100 32GB GPUs.
The entire process of DRL based mixed-precision quantization consists of three main stages: search, finetuning, and evaluation. The search stage has E episodes. Each episode consists of T time steps, where T equals the number of layers of the neural network. At each time step t, the RL algorithm selects a bit-width value for layer l = t. After finishing 1 episode, it generates a mixed-precision configuration with the bit-width values for all the layers, quantizes the network accordingly, and evaluates the quantized network to calculate the reward. It then updates the actor and critic parameters and resets the target DNN network to the original non-quantized one. The finetuning stage follows the search stage. In this stage, we quantize the DNN network using the mixed-precision configuration selected by the RL algorithm at the end of the search stage and finetune the network for N epochs. The final stage is evaluation, which runs the finetuned quantized model on the test dataset. The resulting inference accuracy is the metric for the quality of the quantized network.
We compare our ADRL with four prior mixed-precision quantization studies. Besides the already introduced HAQ (Wang et al., 2018), the prior studies include ReLeQ (Yazdanbakhsh et al., 2018), HAWQ (Dong et al., 2019), and ZeroQ (Cai et al., 2020). ReLeQ (Yazdanbakhsh et al., 2018) uses a unique reward formulation and shaping in its RL algorithm so that it can simultaneously optimize for two objectives (accuracy and reduced computation with lower bit width) with a unified RL process. HAWQ (Dong et al., 2019) automatically selects the relative quantization precision of each layer based on the layer’s Hessian spectrum and determines the fine-tuning order for quantization layers based on second-order information. ZeroQ (Cai et al., 2020) enables mix-precision quantization without any access to the training or validation data. It uses a Pareto frontier based method to automatically determine the mixed-precision bit setting for all layers with no manual search involved.
Our experiments use layer-wise quantization, but we note that ADRL can potentially augment policy approximators in quantizations at other levels (e.g. kernel-wise Lou et al. (2019)) in a similar way.
For ADRL, we experimented with both P-ADRL (profiling-based) and D-ADRL (distance-based). P-ADRL outperforms D-ADRL. We hence focus the overall results on P-ADRL, but offer the results of D-ADRL in the detailed analysis.
As ADRL is implemented based on HAQ, we collect detailed results on all experimented networks on HAQ (code downloaded from the original work (Wang et al., 2018)) to give a head-to-head comparison. For the other three methods, we cite the numbers reported in the published papers. The default setting of HAQ is searching for 600 episodes and finetuning for 100 epochs. We also apply early termination to HAQ (HAQ+E), where, the termination criterion is that the coefficient of variance of the averaged inference accuracy over 10 episodes is smaller than 0.01 for at least two consecutive 10-episode windows. In all experiments, k (number of candidate actions) is 3 for ADRL. We report the quality of the quantization first, and then focus on the speed and variance of the RL learning.
Compression Ratios and Accuracy Table 1 compares the compression ratios and accuracy. For the effects of the augmentation of the approximator, ADRL is the only method that delivers a comparable compression ratio without causing any accuracy loss on all three networks. Note that ADRL is more as a complement rather than a competitor to these work: The augmentation strategy it introduces can help complement the approximators in the existing DRL especially when they are not yet well trained.
Learning Speed Table 2 reports the learning speed in each of the two major stages (search, finetune) and the overall learning process. The reported time is the total time before convergence. For the search stage, the time includes the overhead brought by the Q-value indicator. As shown in column 5 and 8, the search stage is accelerated by 6–66×, while the finetune stage by up to 20×. CifarNet is an exception. For its simplicity, all methods converge in only one epoch in finetuning; because HAQ has a larger compression ratio as shown in Figure 2, neither P-ADRL nor HAQ+E has a speedup in finetuning over HAQ. But they take much fewer episodes to search and hence still get large overall speedups. Overall, P-ADRL shortens RDL learning by HAQ by a factor of 4.5 to 64.1, while HAQ+E gets only 1.2 to 8.8.
The reasons for the speedups come from the much reduced numbers of episodes needed for the DRL to converge in search, and the higher accuracy (and hence earlier termination) that P-ADRL gives in the finetuning process. We next give a detailed analysis by looking into each of the two stages in further depth.
Search Stage Figure 2 shows the inference accuracy (acc), the coefficient of variance (CV) of the inference accuracy, and the compression ratio in terms of the weight ratio (wratio) of the network for the search stage. The first row of Figure 2 gives the inference accuracy results in the search stage. we can see that the configurations found by P-ADRL give comparable inference accuracy for CifarNet and ResNet20 on Cifar10 dataset and better accuracy for AlexNet and ResNet50 on ImageNet dataset comparing to that of HAQ. Also, with early termination, our strategy terminates with a much smaller number of search episodes comparing to HAQ. As shown in column 3 of Table 2, the numbers of search episodes of P-ADRL are 19.1% - 34.5% of that of HAQ+E and 2.2% - 12.3% of that of the original HAQ. The performance of D-ADRL is between that of P-ADRL and HAQ on CifarNet, ResNet20 and ResNet50 and the worst on AlexNet. P-ADRL is more promise than D-ADRL does.
The accuracy given by P-ADRL is much stabler than that of the HAQ. We use the coefficient of variance (CV) to measure the stability of the inference accuracy. As shown in the second row of Figure 2, P-ADRL achieves a small CV much faster than HAQ on all four networks. This is the reason that P-ADRL terminates in fewer episodes than HAQ and HAQ+E do; this conclusion holds regardless of what variance threshold is used in the early termination as the CVs of P-ADRL are consistently smaller than those of HAQ.
The third row of Figure 2 shows the compression ratio after quantization at each episode for each of the three algorithms. The target ratio is set to 0.1. All three strategies fulfill the size limit requirement. By the end of the search stage, P-ADRL achieves a compression ratio similar to HAQ on large networks (ResNet20, AlexNet and ResNet50). On a small network CifarNet, the compression ratio given by P-ADRL is slightly larger than that of HAQ, but still smaller than 0.1.
Finetuning Stage After getting network precision from the search stage, we quantize the networks and finetune for 100 epochs. The learning rate is set as 0.001, which is the default setting from HAQ.
Figure 3 illustrates the accuracy results of the finetune stages for three different precision settings: the one selected by P-ADRL; the one selected by HAQ+E and the one selected by the original HAQ. For CifarNet, all three achieve similar accuracies, higher than that of the original. For the other three networks, P-ADRL gets the largest accuracy, fully recovering the original accuracy. HAQ+E performs the worst, suffering at least 1% accuracy loss.
The sixth column of Table 2 gives the number of epochs each strategy needs for finetuning before reaching the accuracy of the original network; ’100’ means that the strategy cannot recover the original accuracy in the 100-epoch finetuning stage. P-ADRL takes 1–35 epochs to converge to the original accuracies, much fewer than HAQ and HAQ+E. Finetuning hence stops earlier.

heading: 5 CONCLUSION
text: This paper has demonstrated that simple augmentation goes a long way in boosting DRL for DNN mixed precision quantization. It is easy to apply ADRL to deeper models such as ResNet-101 and Resnet-152. In principle, it is also possible to extend the method to latest model architectures as the basic ideas of the reinforcement learning and the Q-value indicator are independent of the detailed architecture of the network model. Although the design of ADRL is motivated by DNN quantization, with appropriate augmentation schemes, the concept is potentially applicable to other RL applications—a direction worth future exploration.

heading: A APPENDIX
text: A.1 DETAILS OF THE MIXED QUANTIZATION BITS FOR EACH MODEL
Figure 4 shows the comparison between the quantized networks selected using ADRL and HAQ. In each plot, the upper half shows the bit value being used for each layer for the network quantized with ADRL. The bottom half are the quantization configurations of the networks being quantized with HAQ.
A.2 EVOLUTION OF THE SELECTED BIT VALUES
Figure 5 illustrates how the selected bit values change during the search stage for some layers of the four networks. The bit value being selected for the same layer changes from one episode to the next. It spans well in the action search space.

Review summaries:
This paper describes an improved way to determine weight quantization bit lengths using reinforcement learning, by injecting model evaluation directly into action selection.  Building upon a DRL setup where the action at each timestep corresponds to selecting a bit value for each layer, the method adds a "Q-value indicator" function Q~ that selects among a set of candidate actions, and filters based on the model performance quantizing the layer to each level.  This seems to form a hybrid between DRL and greedy search, using a greedy criteria Q~ to filter proposals made by the DRL agent.  Experiments show very good performance, with similar or better quantization levels and accuracy as other DRL-based methods and much faster runtime.

This paper studies the DNN quantization using deep reinforcement learning. The paper proposes an augmented DRL which introduces a Q-value indicator to refine action selection. The proposed approach has been applied to several image classification baselines and has compared with several recent DRL based quantization approach, achieving a similar compression rate without accuracy decrease.  In addition, compared to previous methods, the learning speed has been improved by 4.5-64x.