DQA: An Efficient Method for Deep Quantization of Deep Neural Network Activations

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-11 15:52:30

Abstract: Quantization of Deep Neural Network (DNN) activations is a commonly used technique to reduce the memory demand during inference, which can be particularly beneficial on edge devices with limited computational resources. To achieve high accuracy, existing methods rely on expensive mathematical computations or perform extensive online searches for the best hyper-parameters. However, these expensive operations are impractical on edge devices due to their restricted computation capabilities, memory capacities, and limited energy budgets. Furthermore, many existing methods do not focus on sub-6-bit (or deep) quantization, which is more relevant to resource-constrained edge devices. 

To fill these gaps, we propose DQA (Deep Quantization of DNN Activations), a new method that focuses on sub-6-bit quantization of activations. DQA leverages simple shifting-based operations and Huffman coding to achieve high accuracy. 
We evaluate DQA with 3, 4, and 5-bit levels of quantization and three different DNN models on two different datasets for two different tasks, image classification and image segmentation. DQA shows significantly better accuracy (up to 29.28\%) compared to a direct method and NoisyQuant for sub-6-bit quantization.

Main paper: 
 introduction: 
 Quantization is a popular compression method for reducing the cost (i.e., compute and memory) of Deep Neural Networks (DNNs). During inference, DNN weights are typically fixed, so they can be quantized offline. However, activations are dynamically generated. This means that effective quantization methods must dynamically quantize activations online.There are many methods for quantizing activations. The most straightforward one is to directly use the mathematical definition of quantization [6; 12] . Beyond that, to get higher accuracy, more sophisticated techniques are used. For example, Liu at al. [14] injects noise in the activations before quantization and removes the noise after the de-quantization. However, these methods are not specifically designed for resource-constrained edge devices, where the computational cost of the expensive mathematical operations (e.g. matrix level multiplication) or large online search spaces that are required in these methods are impractical. Furthermore, many of these methods do not focus on deep quantization like sub-6-bit quantization (i.e., quantizing to less than 6 bits), which is preferred for edge devices with limited memory.To fill these gaps, in this paper we propose DQA (Deep Quantization of DNN Activations), a new method to deeply quantize DNN activations, suited for resource-constrained edge devices, that provides high accuracy. DQA first determines the importance of each activation channel offline using training or calibration datasets. Based on this knowledge, some activation channels are then handled differently than others. For important channels, DQA quantizes their values with m extra bits first and then right-shifts the m bits to achieve the target number of bits while storing the shift-out errors using Huffman coding. Then, the shift-out errors are decoded and added back to the corresponding channels during the de-quantization phase. For unimportant channels, DQA uses a direct method to quantize and de-quantize the activations, i.e., it simply applies the mathematical definition of quantization/de-quantization. By learning the importance of channels offline, DQA avoids the need to separate the channels online, saving significant computational resources; by using an m more bits quantization approach, DQA can reduce the rounding error exponentially and can efficiently shift back to the target bits without expensive mathematical computations. It is important to note that, by right-shifting, all the quantized values can ultimately be stored using the same bit length. This makes DQA to avoid hardware inefficiencies such as wasted storage [13] . By applying Huffman coding to the shift-out errors, DQA can reduce the memory overhead.We evaluated DQA with three sub-6-bit quantization levels (3, 4 , and 5 bits) on two different datasets (CIFAR-10 [11], and CityScapes [3] ) for three different DNN models (ResNet-32 [7] , Mo-bileNetV2 [8] and U-Net [16] ) for two different tasks (image classification and image segmentation).Overall, DQA shows significantly better accuracy (up to 29.28%) than two existing methods: direct quantization/de-quantization and NoisyQuant [14] .The main contributions of this paper are as follows:• We propose DQA, a deep (sub-6-bit) quantization method for DNN activations which is especially relevant for resource constrained edge devices. DQA deals with important activation channels separately and leverages Huffman coding to optimize the memory overhead.• We explore the patterns of shift-out errors in DQA to justify our choice of Huffman coding.• We conduct a detailed evaluation of DQA on two different datasets (CIFAR-10, and CityScapes) with three different models (ResNet-32, MobileNetV2, and U-Net) for two different tasks (image classification and image segmentation). We compare DQA with two existing methods: direct quantization/de-quantization and NoisyQuant [14] . 
background and related work: 
 Quantization is a well-known compression method for optimizing parameterized systems [6; 10] .The parameters are converted into a range of integers which require fewer bits than the original floating point formats. The process normally includes clipping and rounding operations. The clipping operation restricts the minimum and maximum values of the quantization, and the rounding operation rounds the values into the closest integers, which causes rounding errors that cannot be recovered during de-quantization. De-quantization is the opposite operation that approximately recovers the quantized values back to floating point. Quantization is typically divided into uniform and nonuniform. In uniform quantization, quantized values are spaced uniformly, while in non-uniform quantization they are not [6] . Uniform quantization can be further divided into symmetric and asymmetric quantization. In symmetric quantization, the clipping range is symmetric with respect to the origin while in asymmetric quantization it is not [6] . In this paper, we use uniform symmetric quantization due to its popularity and relatively low cost of computation [6] , and we define the direct uniform symmetric quantization and de-quantization as the Direct method (this is same as the direct methods for unimportant channels in Section 1).Previous works on DNN quantization apply it to the model weights and/or activations. For weights, AWQ [12] checks the importance of the weight channels for each layer and scales up the important channels before quantization to reduce the rounding errors. For activations, quantization is difficult due to its dynamic nature, i.e., the activation values are typically not known before the inference happens. Therefore, to achieve high accuracy, expensive mathematical operations (e.g., matrix level multiplication) or large online search spaces to find the best hyper-parameters are required by many methods. For example, NoisyQuant [14] injects noise, obtained by online search within the given search spaces, to the activations before quantization and removes the noise after de-quantization.However, these methods do not focus on deep quantization like sub-6-bit quantization (i.e., lower than (1) Activation Channels A, (2) Target Bits n, (3) Extra Bits m, (4) Important Channels I, (5) Huffman Encoder H(.)Output: Quantized A, encoded shift-errors e e ← empty for a ∈ A do if a ∈ I then ∆n+m ← |max(A layer )| 2 n+m-1 amore ← Round( a ∆ n+m ) a ← RightShif t(amore, m) e ← e + H(amore -a) else ∆n ← |max(A layer )| 2 n-1 a ← Round( a ∆n ) return A, eAlgorithm 2 DQA Layer Activation De-quant Input:(1) Quantized Activation Channels Aq, (2) Target Bits n, (3) Encoded Shifting Errors e, (4) Important Channels I, (5) Huffman Dencoder with Values Converting H(.)Output: De-Quantized activation channels A for a ∈ Aq do ∆n = |max(A layer )| 2 n-1 if a ∈ I then a = ∆n • (a + H(e)) else a = ∆n • a return AGiven a target n of quantization bits, DQA starts by quantizing the given activations to n bits. As shown in Figure 1: 1 DQA first uses the training or calibration data to find the rank of the activation channels by an offline greedy search as described in Section 3.1; 2 then uses the knowledge of the channels' rank to determine the activation channels. During quantization, for a pre-selected amount of important activation channels in each layer, DQA quantizes them using m more (extra) bits than the rest of channels, which are only quantized to the target n bits. The higher the accuracy required, the more important channels are needed. Then DQA right shifts the important channels by m bits while retaining the shifted-out bits of each value. Since there are only 2 m possible arrangements of the shift-out m bits which are corresponding to the shift-out errors, they can be easily converted to decimal floating point values by a pre-computed map. For non-important channels, DQA uses the direct method to quantize; 3 DQA encodes the shift-out bits using Huffman coding [17] (Section 3.2).During the de-quantization phase, and immediately before the de-quantization operation, DQA simply decodes the encoded shift-out bits and converts them to decimal floating point error values; 4 finally DQA compensates the information loss of the corresponding channels by adding the error values to them.Algorithm 1 and Algorithm 2 describe the steps in more details. Note that the Important Channels I is logically equivalent to reading top channels from the general channel ranks with important channel ratios in practice. Similarly, the important channel ratios depends on the accuracy requirements. The higher the final accuracies are required, the higher important channel ratios are needed. 
greedy-search-based channel importance ranking: 
 We hypothesize that there are important activation channels that can be treated differently during quantization, similar to how AWQ [12] processes weights. We define the important activation channels as those channels for which skipping quantization yields significantly better accuracy than skipping others. Since activations are dynamically generated, it is expensive to find the ranks of the activation channels online. As a result, we leverage the offline activation ranks from training or calibration data. Assuming the training/calibration data and the inference data are drawn from sufficiently similar distributions, the ranks can be reused online during inference.To find the ranks of the activation channels, we avoid brute force and dynamic programming approaches due to their impractical time complexity; instead we use greedy search which only requires O(LN ), where N is the number of channels and L is the number of layers. In greedy search, from the first layer, we iterate the channels. For each channel, we skip the quantization on it and quantize the remaining channels in the layer. Then we measure the model accuracy on validation data. The skipped channels that give the highest accuracies are considered as the most important channels for this layer. We repeat this process for every layer, first quantizing the activation channels of the previous layers and skipping their most important channels.The important activation channels are assumed to contain more valuable information. To preserve this information, we quantize them with m more bits, where m is suggested to be less than or equal to the target number of bits n. Then, we right-shift the activations by m bits to reach the target bit length. Note that this operation allows the devices to store the quantized values with same bit length, which helps DQA to avoid hardware inefficiencies like wasted storage [13] . Also note that, right-shifting is computationally much cheaper than multiplication [5] . Since right-shifting will lead to information loss for the values, we store the shift-out errors after the operations and add them back during de-quantization phase to compensate the information loss.We now formally analyze the expected Round-Shift quantization error for the important channels. We denote A as the important activation channels. The most straightforward quantization/de-quantization approach to the important activation channels can be expressed as [12] :EQUATIONThe quantization error of this method is (RE means the Rounding Error):Error = |A -Q(A)| = |A -∆N • ( A ∆N + RE)| = ∆N • RE (2)The average Rounding Error is 0.25 [12] . With our method, the A will be quantized to m more bits first, and then do m bits right shifting. Then, the dequantized important activation channels can be expressed as (SE means Shifting Error):EQUATIONThen the quantization error becomes:EQUATIONFrom equations 2 and 4, it is clear that Round-Shift exponentially reduces the quantization error. This is expected since we retain m more bits of information.To reduce the memory overhead created by the shift-out errors, we empirically explore the pattern of the shift-out errors that can be used to reduce the shift-out memory overhead. We now study the frequency distribution of the shift-out errors of all channels, for ResNet-32 on the CIFAR-10 dataset, by considering all the channels as important and setting m = 3 in DQA. In Figure 2 we observe that the distributions of the shift-out errors are not uniform. This indicates that they can be compressed by Huffman coding [17] which is able to encode the more frequent errors with fewer bits and less frequent errors with more bits. In general, it could generate shorter encoding bits compared to directly storing m bits. This justifies our choice of using Huffman coding to compress the shift-out errors. In our experiments 4.3, on average, the Huffman coding compression ratios on experiments with the CIFAR-10 dataset are up to 1.11 and in most cases (6 out of 9) can compress the shift-out errors of our experiments in Section 4.We also calculated the per-image DQA memory overhead ratios for the weights. At most, on average, in our experiments 4, the ratio is 1.91% for ResNet-32 on CIFAR-10, 0.72% for MobileNetV2 on CIFAR-10, and 1.17% for the U-Net model on CityScapes. Therefore, for inference of one image (which is a common batch size for edge devices [1; 2] ), the memory overheads of DQA are very small compared to the memory of weights (< 2%). This small additional amount of data can be stored in on-chip caches for faster access [9] . 
experimental setup: 
 The image classification task is based on the CIFAR-10 dataset [11] . The models for image classification are ResNet-32 [7] and MobileNetV2 [8] . For the image segmentation task, we use the CityScapes [3] dataset. The model used is U-Net [16] , which stores the activations of encoder layers before concatenating them with activations of decoder layers. U-Net is a typical example where activations need to be stored for a relatively long term and need to be compressed for a manageable peak memory usage, e.g., when targeting edge devices. We implement our experiments using PyTorch [15] and run them on a Nvidia RTX 3090 GPU. For each layer, we quantize/de-quantize the activations channel-wise batch by batch. Note that for simplicity, we only apply the operations on activations of the main branch of main blocks (i.e. not the blocks on the shortcut branches). Every experiment is run 5 times for each target quantization bit number and we take the average value in each case.We compare DQA with two methods. The first method is the direct method defined in Section 2.The second method is NoisyQuant [14] which is state-of-the-art. Even though this method is designed for Vision Transformers [4] , it should still work for other types of DNNs. Note that all our implementations are only for activations, i.e., without considering weights.We first created 5 rank maps (every map includes the ranks of the activation channels for each main block of the whole model) with 5 random subsets of the training data (with size 5000 for each subset) using greedy search (see Section 3.1). We select important channels with ratios of 40% of all channels in all the blocks. The batch size is 128 for all the experiments, including the direct method and NoisyQuant experiments.In Table 1a and Table 1b we observe that DQA with m = 3 can achieve up to 29.28% higher accuracy than the direct method and NoisyQuant. We also see that the lower the quantization bits are, the more accuracy advantages DQA can provide.Similarly as in Section 4.3, we first created 5 rank maps with 5 random subsets of the training data (with size 50 for each subset) using rank greedy search (see Section 3.1). We select important channels with ratios of 50% of all channels in all the blocks. The batch size is 4 for all the experiments including the direct method and NoisyQuant experiments.From Table 1c , we observe that the DQA with m = 3 can provide higher accuracy (up to 0.9%) than the direct method and NoisyQuant after quantization when the quantization bits are equal to 4 and similar accuracy when the quantization bits are equal to 5. 
conclusion: 
 In this paper we proposed DQA, an efficient method that applies deep quantization to DNN activations using cheap operations such as right-shfitng and low memory overheads (< 2% of the memory usage of the weights per-image) while providing high accuracy, thus being suitable for resource constrained edge devices. We evaluate DQA on both image classification and image segmentation tasks, showing up to 29.28% accuracy improvement compared with a direct quantization method and the state-of-theart. In future work, we will co-design the edge device hardware that can further exploit the benefits of DQA [18] . 
 
 Appendix: bit quantization), thus being less suitable for resource-constrained edge devices. Since resources are very limited on edge devices, the requirement for expensive memory and computation is not practical. To quantize the activation values with high performance on edge devices, we need cheaper but effective methods. 



Reviews: review: title: title: Official Review
main_review: main_review: Paper Content Summary : the paper claims deep quantization (i.e. sub 6-bit quantization) of DNN (i.e. ResNet, MobileNet and U-Net) is lacking in the current state, where the only few exisiting solutions requires expensive process on device to make the quantized model accurate. Then the paper proposes a process that 1) identifies important channels in activations 2) quantize those with higher precision (n+m bits) 3) right shift to lower the precision to match target compression precision (n bits) 3) bookkeep the right shift errors and compress them using huffman coding. 

Reasons to Reject:

First of all, the paper is hard to parse, with lots of repetitive content in abstract, intro, proposed method, result and conclusion. This makes the paper lack depth. I don’t feel like learning a lot or get inspired after reading the paper. But I do think with enough rewriting this can be improved. 

Secondly, it is not convincing that the method proposed is superior or novel than existing methods. The paper states that few works compress DNN to sub-6bit, but this is not true [1][2]. The paper studies an outdated problem and the solution proposed is not significantly better in terms of result or conceptual insights. 

- In terms of results, the accuracy was claimed to be up to 29% better than others, however the result table (Table 1) shows that this only applies to a single case among the 9 settings tested; for other cases the improvement is trivial or even worse than baselines.
- Conceptually, the method is overly complicated, leading to potential slowdown in inference. For example the different processing of channels disables parallelized matrix operation of quantization. Additional huffman coding and decoding steps are costly too. In addition, the compression rate of huffman coding is shown to be only 1.1 in the paper - leaving the "important" channels essentially taking "m-bits" more than the target n-bit precision. All these overhead defeat the purpose of deep quantization.

Suggestions to Improve the Paper:

- back up the problem statement and motivation, as the problem and the model scope in current paper is studied by many other works already. show the reason why your work has impact.
- address the questions on efficiency and inefficiencies of your algorithm. in various places in the paper, you tried to claim the method is more efficient - but no quantification or evidence to support it. (how expensive online search is? how much more efficient is your approach compared to, say, NoisyQuant? why would you compare the memory consumption to weights? if activation tensor is small to start with, in batch size 1 setting, why bother quantizing it?)
- In terms of writing, I'd suggest keeping abstract and introduction high level, leaving all the algorithm details to specific sections. The first few sections should well support the motivation and impact of this work and leaving the "how to" later. It was very confusing when I read about method details with all the custom terms in abstract and intro without seeing Figure 1 first. Figure 1 was very helpful though. 

(final note: I spent a long time parsing the paper trying to find reasons to accept it, and not be rejecting due to writing quality, but this is all I got so far, I can't figure out enough contributions by the paper to support accepting. I hope these revise suggestions could help, now or in the long run.)

[1] Abdolrashidi, AmirAli, et al. "Pareto-optimal quantized resnet is mostly 4-bit." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2021

[2] Yang, Bin, et al. "2-bit model compression of deep convolutional neural network on ASIC engine for image retrieval." *arXiv preprint arXiv:1905.03362* (2019).
title: title: A "dual-mode" quantization scheme that still needs some investigation
main_review: main_review: # Summary
The paper presents a novel quantization approach for activation that leverages additional encoded biases for "important" channels. The method exploits sub-6-bit quantization, is tested on three CV tasks, and achieves better accuracy than standard quantization methods.

The paper is generally promising, but it would benefit from a deep evaluation of the memory/accuracy tradeoff.

# Pros
- The paper topic is engaging.
- Method seems general and could potentially be used in many contexts.
- Preliminary results show the effectiveness of the method.

# Cons
- Memory overheads/accuracy tradeoff is not discussed in depth.
- Channel selection tradeoff (how many channels) and costs (in terms of time) are not discussed.
- Execution time overheads for runtime quantization and de-quantization are not discussed.

# Comments
The paper presents an interesting hybrid solution for low-bit quantization.
The paper, at this stage, could be strongly improved by a more detailed evaluation of the accuracy/memory tradeoff enabled by the proposed method:
- First, channel selection and the number of channels selected are important because they impact both the memory overheads and the accuracy of your method. An ablation study would be very appreciated.
- Second, memory overheads and accuracy with respect to the FP32 model should be added in Table 1.
- Third, the additional additive operations at runtime for quantization and de-quantization must be analyzed using realistic devices.

Your background and related work should also consider mixed quantization methods because your quantization scheme ultimately results in a channel-wise dual quantization process.
title: title: DQA presents a simple yet efficient way to quantize activations to sub-6bit representations while maintaining high performance
main_review: main_review: This is a nice paper. DQA determines the importance of each activation, and depending on the importance, will handle the quantization differently. I've seen lots of works in quantizing weights, but this one tackles the challenging problem of quantizing activations. This is usually difficult since the outputs are not usually known before inference. DQA explains their methodology well. They follow with a number of experiments supporting their claims and how their method is simple and performs better than NoisyQuant, their most related work. 

A few minor questions:
- When storing the shift errors using Huffman coding, what is the effective number of bits per value? This might be a nice number to include. I see the authors mention that the overhead is <2% of the memory for model weights but thinking about it in terms of number of effective bits helps for comparison. How does storing the full activation value using Huffman coding compare?
- when trying other values of m, did you see any trends as you increased or decreased m? How should researchers choose m?
- in the table 1a-c, would you be able to add in the accuracies for the original models with no quantization? It would be good to know what the cap is.
- a little typo in figure 2's caption and line 207.


Overall, very nice.