Title: No title found

Abstract: Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators. Accordingly, an efficient feature map compression method can result in substantial performance gains. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way that allows better compression of feature maps during inference. Our method trains the model to achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods. CAT significantly improves the state-of-the-art results reported for quantization. For example, on ResNet-34 we achieve 73.1% accuracy (0.2% degradation from the baseline) with an average representation of only 1.79 bits per value. Reference implementation accompanies the paper.

1. I N T R O D U C T I O N:

Deep Neural Networks (DNNs) have become a popular choice for a wide range of applications such as computer vision, natural language processing, autonomous cars, etc. Unfortunately, their vast demands for computational resources often prevents their use on power-challenged platforms. The desire for reduced bandwidth and compute requirements of deep learning models has driven research into quantization (Hubara et al., 2016; Yang et al., 2019b; Liu et al., 2019; Gong et al., 2019) , pruning (LeCun et al., 1990; Li et al., 2017; Molchanov et al., 2019) , and sparsification (Gale et al., 2019; Dettmers & Zettlemoyer, 2019) .

In particular, quantization works usually focus on scalar quantization of the feature maps: mapping the activation values to a discrete set {q i } of size L. Such a representation, while being less precise, is especially useful in custom hardware, where it allows more efficient computations and reduces the memory bandwidth. In this work, we focus on the latter, which has been shown to dominate the energy footprint of CNN inference on custom hardware (Yang et al., 2017) . We show that the quantized activation values {q i } can further be coded to reduce memory requirements.

The raw quantized data require log 2 (L) bits per value for storage. This number can be reduced by compressing the feature maps. In particular, in the case of element-wise compression of independent identically distributed values, the lower bound of amount of bits per element is given by the entropy (Shannon, 1948) :

EQUATION

of the quantized values {q i }, where p(q i ) denotes the probability of q i .

In this work, we take a further step by manipulating the distribution of the quantization values so that the entropy H(q) is minimized. To that end, we formulate the training problem by augmenting the regular task-specific loss (the cross-entropy classifier loss in our case) with the feature map entropy serving as a proxy to the memory rate. The strength of the latter penalty is controlled through a Effect of entropy regularization. Without regularization, the distribution has much heavier tails and thus has higher entropy. With increasing regularization, the probability of extreme values is significantly reduced. The entropy penalty λ was selected so that the overall accuracy is not affected. The compression ratio in the strongly regularized case is 2.23 higher compared to the unregularized the baseline.

parameter λ > 0. Fig. 1 demonstrates the effect of the entropy penalty on the compressibility of the intermediate activations.

Contributions. Our paper makes several contributions. Firstly, we introduce Compression-Aware Training (CAT), a technique for memory bandwidth reduction. The method works by introducing a loss term that penalizes the entropy of the activations at training time and applying entropy encoding (e.g., Huffman coding) on the resulting activations at inference time. Since the only overhead of the method at inference time is entropy encoding that can be implemented efficiently, the improvement is universal for any hardware implementation, being especially efficient on computationally optimized ones, where memory I/O dominates the energy footprint (Yang et al., 2017; Jouppi et al., 2017) . We demonstrate a 2 to 4-fold memory bandwidth reduction for multiple architectures: MobileNetV2 and ResNet on the ImageNet visual recongition task, and SSD512 on the PASCAL VOC object detection task. We also investigate several differentiable loss functions which lead to activation entropy minimization and show a few alternatives that lead to the same effect.

Finally, we analyze the rate-distortion tradeoff of the method, achieving even stronger compression at the expense of minor accuracy reduction: for ResNet-18, we manage to achieve entropy inferior to one bit per value, at the expense of losing 2% of the top-1 accuracy.

2. R E L AT E D W O R K:

Recent studies (Yang et al., 2017; Wang et al., 2019) have shown that almost 70% of the energy footprint on custom hardware is due to the data movement to and from the off-chip memory. Nonetheless, the techniques for memory bandwidth reduction have not yet received significant attention in the literature. One way to improve memory performance is by fusing convolutional layers (Xiao et al., 2017; Xing et al., 2019) , reducing the number of feature maps transfers. This reduces both runtime and energy consumption of the hardware accelerator. Another option is to use on-chip cache instead of external memory. Morcel et al. (2019) have shown order of magnitude improvement in power consuption using this technique. Another important system parameter dominated by the memory bandwidth is latency. Jouppi et al. (2017) and Wang et al. (2019) showed that the state-of-the-art DNN accelerators are memory-bound, implying that increasing computation throughput without reducing the memory bandwidth barely affects the total system latency.

Quantization reduces computation and memory requirements; 16-bit fixed point has become a de facto standard for fast inference. In most applications, weights and activations can be quantized down to 8 bits without noticeable loss of precision (Lee et al., 2018; Yang et al., 2019a) . Further quantization to lower precision requires non-trivial techniques (Mishra et al., 2018; Zhang et al., 2018) , which are currently capable of reaching around 3-4 bits per entry without compromising precision (Choi et al., 2018b; a; Dong et al., 2019) .

Another way to reduce memory bandwidth is by compressing the intermediate activations prior to their transfer to memory with some computationally cheap encoding, such that Huffman (Chandra, 2018; Chmiel et al., 2019) or run-length (RLE) encoding (Cavigelli et al., 2019) . A similar approach of storing only nonzero values was utilized by Lin & Lai (2018) . Chmiel et al. (2019) 

3. M E T H O D:

We consider a feed-forward DNN F composed of L layers; each subsequent layer processing the output of the previous one:

x i = F i x i-1

, using the parameters w ∈ R N . We denote by x 0 = x and x L = y the input and the output of the network, respectively.

The parameters w of the network are learned by minimizing L(x, y; w) + λR(w), with the former term L being the task loss, and the latter term R being a regularizer (e.g., w 2 ) inducing some properties on the parameters w.

3 . 1 E N T R O P Y E N C O D I N G A N D R AT E R E G U L A R I Z AT I O N

Entropy encoders are a family of lossless data encoders which compress each symbol independently. In this case, assuming i.i.d. distribution of the input, it has been show that optimal code length is -log b p, where b is number of symbols and p i is the probability of i th symbol (Shannon, 1948) . Thus, for a discrete random variable X we define an entropy H(X) = -E log 2 X = i p(x i ) log 2 p(x i ), which is a lower bound of the amount of information required for lossless compression of X. The expected total space required to encode the message is N • H, where N is number of symbols. Since we encode the activations with entropy encoder before writing them into memory, we would like to minimize the entropy of the activations to improve the compression rate.

One example of entropy encoder is Huffman coding -a prefix coding which assigns shorter codes to the more probable symbols. It is popular because of simplicity along with high compression rate (Szpankowski, 2000) bounded by H(X) ≤ R ≤ H(X) + 1 (the comparison of Huffman coding rates to entropy is shown in Fig. 3 ). Another entropy encoder is arithmetic coding -highly efficient encoder which achieves optimal rates for big enough input but requires more computational resources for encoding.

3. . 2 D:

I F F E R E N T I A B L E E N T R O P Y-R E D U C I N G L O S S

Since the empirical entropy is a discrete function, it is not differentiable and thus cannot be directly minimized with gradient descent. Nevertheless, there exist a number of differentiable functions which either approximate entropy or have same minimizer. Thus we optimize

EQUATION

where L p is a target loss function and L H is some regularization which minimizes entropy.

Soft entropy First, we consider the differentiable entropy estimation suggested by Agustsson et al. (2017) . We start from definition of the entropy,

EQUATION

EQUATION

where q is a vector of quantized values. Let m be an index of the bin the current value is mapped to, and Q a one-hot encoding of this index, i.e.,

EQUATION

EQUATION

where δ im denotes Kroneker's delta. To make the latter expression differentiable, we can replace argmax with softmax:

EQUATION

)

where T is the temperature parameter, and

Q(x) → Q(x) as T → ∞.

Finally, the soft entropy Ĥ is defined as

EQUATION

To improve both memory requirements and time complexity of the training, we calculate soft entropy only on part of the batch, reducing the amount of computation and the gradient tensor size. We empirically confirm that this choice gives a reasonable approximation of the real entropy (Appendix B).

Compressibility loss for entropy reduction An alternative loss promoting entropy reduction was proposed by Aytekin et al. (2019) under the name of compressibility loss and based on earlier work by Hoyer (2004) :

EQUATION

This loss has the advantage of computational simplicity, and has been shown both theoretically and practically to promote sparsity and low entropy in input vectors. While originally applied to the weights of the network, here we apply the same loss to the activations.

As shown in Section 4.1, both the soft entropy and the compressibility loss lead to similar results.

Our method for reducing memory bandwidth can be described as follows: at training time, we fine-tune (training from scratch should also be possible, but we have not tested it) the pre-trained network F with the regularized loss ( 2), where we use L H = Ĥ(x i ) in case of differentiable entropy and L H = L c (x i ) in case of compressability loss. At test time, we apply entropy coding on the activations before writing them to memory, thus reducing the amount of memory transactions. In contrast to Chmiel et al. (2019) , who avoided fine-tuning by using test time transformation in order to reduce entropy, our method does not requires complex transformations at test time by inducing low entropy during training.

4. E X P E R I M E N TA L R E S U LT S:

We evaluate the proposed scheme on common CNN architectures for image classification on ImageNet (ResNet-18/34/50, MobileNetV2), and object detection on Pascal VOC dataset (SSD5121 (Liu et al., 2016) ). The weights were initialized with pre-trained model and quantized with uniform quantization (Hubara et al., 2016; Rastegari et al., 2016) . The activation were clipped with a learnable parameter and then uniform quantized as suggested by Baskin et al. (2018) . Similarly to previous works (Zhou et al., 2016; Rastegari et al., 2016) , we used the straight-through estimator (Bengio et al., 2013) to approximate the gradients. We quantize all layers in the network, in contrast to the common practice of leaving first and last layer in high-precision (Zhou et al., 2016; Baskin et al., 2018) . Since the weights are quantized only to 8 bit, we have noticed small to no difference between the quantized and non-quantized weights.

For optimization, we use SGD with a learning rate of 10 -4 , momentum 0.9, and weight decay 4 × 10 -5 for up to 30 epochs (usually, 10 -15 epochs were sufficient for convergence). Our initial choice of temperature was T = 10, which performed well. We tried, following the common approach (Jang et al., 2016) , to apply exponential scheduling to the temperature, but it did not have any noticeable effect on the results. In Fig. 2 we compare our method with EPBC (Cavigelli et al. (2019) ) and GF (Gudovskiy et al. (2018) ). EPBC is based on a lossless compression method that maintains the full precision accuracy while reducing bit rate to approximately 3.5 bits/value in both models. GF, on the other hand, provides strong compression at the expense of larger accuracy degradations. In addition, Gudovskiy et al. (2018) have compressed only part of the layers. Unlike these two methods, CAT allows more flexible tradeoff between compression and accuracy. CAT shows better results in ResNet-34 and show either better accuracy or compression for MobileNetV2. We also run our method on additional architectures: ResNet-18, ResNet-50, and SSD512 with VGG backbone; the results are listed in Table 1 . Even though we can not directly compare detection results with Gudovskiy et al. (2018) , the drop in accuracy is lower in our case.

4. . 1 A B L AT I O N S T U D Y:

Rate-Accuracy Tradeoff The proposed CAT algorithm tries to balance between minimizing the rate and maximizing the accuracy of the network by means of the parameter λ in Eq. ( 2). To check this tradeoff, we run the same experiment in ResNet-18 and MobileNetV2 with different values of λ in the range of 0 -0.3, with results shown in Fig. 3 . We show the results of the theoretical entropy and of the Huffman coding, which was chosen for its simplicity but more efficient encoders can be combined with our method. For higher rate Huffman coding is close (∼3% overhead) to the theoretical entropy, while for lower entropy the difference is higher and is bounded by 1 bit per value -in the latter case, different lossless coding schemes such as arithmetic coding can provide better results.

Robustness For checking the robustness of our method, we performed several runs with the same hyper-parameters and a different random seed. Statistics reported in Table 2 suggest that the method is robust and stable under random intialization.

Soft entropy vs. compressibility loss Replacing the soft entropy with a different loss which minimizes the entropy almost did not affect the results, as shown in Table 3 . We conclude that the desired effect is a result of entropy reduction rather than a particular form of regularization promoting it. (Hubara et al., 2016) , or as a fine-tuning step from a pre-trained floating point model (Han et al., 2016) . Training usually compensates significantly for model's accuracy loss due to quantization.

In this work, we take a further step and propose a compression-aware training method to aggressively compress activations to as low as 2 average bit/value representations without harming accuracy. Our method optimizes the average-bit per value needed to represent activation values by minimizing the entropy. We demonstrate the applicability of our approach on classification tasks using the models MobileNetV2, ResNet, and object detection task using the model SSD512. Due to the low overhead, the method provides universal improvement for any custom hardware, being especially useful for the accelerators with efficient computations, where memory transfers are a significant part of the energy budget. We show that the effect is universal among loss functions and robust to random initialization. 

:

Our code is based on implementation byLi (2018).

Figures and Tables:

Figure 1: Pre-activation distributions of one layer in ResNet-18. (a) Evolution at different epochs. As training progresses, the probability of non-positive pre-activation values increases, zeroing more post-ReLU values. The sharp peak at zero reduces entropy and thus improves compressibility. (b)Effect of entropy regularization. Without regularization, the distribution has much heavier tails and thus has higher entropy. With increasing regularization, the probability of extreme values is significantly reduced. The entropy penalty λ was selected so that the overall accuracy is not affected. The compression ratio in the strongly regularized case is 2.23 higher compared to the unregularized the baseline.

used linear dimensionality reduction (PCA) to increase the effectiveness of Huffman coding, while Gudovskiy et al. (2018) proposed to use nonlinear dimensionality reduction techniques. Lossless coding was previously utilized in a number of ways for DNN compression: Han et al. (2016) and Zhao et al. (2019) used Huffman coding to compress weights, while Wijayanto et al. (2019) used more complicated DEFLATE (LZ77 + Huffman) algorithm for the same purpose. Aytekin et al. (2019) proposed to use compressibility loss, which induces sparsity and has been shown (empirically) to reduce entropy of the non-zero part of the activations.

Figure 2: Comparison with other methods: EPBC (Cavigelli et al. (2019)) and GF (Gudovskiy et al. (2018)) in (a) ResNet-34 and (b) MobilenetV2. Different marker size refers to different activation bitwidths before compression. For GF, compression rate was averaged only over compressed layers.

Figure 3: Tradeoff between rate and accuracy for different values of λ (ranged between 0 and 0.3) in (a) ResNet-18 and (b) MobileNetV2. In ResNet-18 the activations are quantized to 5 bits, in MobileNet we show results for activation quantized to 6 and 8 bits.

S A M P L E S I Z E I N S O F T E N T R O P Y C A L C U L AT I O NTo check whether the amount of values used to calculate soft entropy is enough, we ran a soft entropy evaluation on a single tensor and compared it to real values. Since the tensors are large (hundreds of thousands of elements), even 5% of the values already provides reasonable approximation of the real entropy, as shown in Fig. B.1.

Results for ResNet-18, ResNet-50, and SSD512. For comparison we include the results obtained byGudovskiy et al. (2018) for the SSD512 model on the same task but with a different backbone, for which we obtain a better compression rate with a lower accuracy degradation. Compute denotes activation bitwidth used for arithmetic operations. Memory denotes average number of bits for memory transactions (after compression). Compression ratio denotes the reduction in representation size. Weight bitwidth is 8 except the full-precision experiments. Additional experimental results are provided in Appendix A.

Table content: <table><tr><td>Architecture</td><td colspan="3">Compute Memory Compression</td><td>Top-1</td></tr><tr><td/><td>(bits)</td><td>(bits)</td><td>ratio</td><td>accuracy (%)</td></tr><tr><td/><td>32</td><td>32</td><td>1</td><td>69.70</td></tr><tr><td>ResNet-18, CAT</td><td>5</td><td>1.5</td><td>3.33</td><td>69.20</td></tr><tr><td/><td>4</td><td>1.51</td><td>2.65</td><td>68.08</td></tr><tr><td/><td>32</td><td>32</td><td>1</td><td>76.1</td></tr><tr><td>ResNet-50, CAT</td><td>5</td><td>1.60</td><td>3.125</td><td>74.90</td></tr><tr><td/><td>4</td><td>1.78</td><td>2.25</td><td>74.50</td></tr><tr><td>SSD512-SqueezeNet (Gudovskiy et al., 2018)</td><td>32 8 6</td><td>32 2 2</td><td>1 4 3</td><td>68.12 64.39 62.09</td></tr><tr><td/><td>32</td><td>32</td><td>1</td><td>80.72</td></tr><tr><td>SSD512-VGG, CAT</td><td>6</td><td>2.334</td><td>2.57</td><td>77.49</td></tr><tr><td/><td>4</td><td>1.562</td><td>2.56</td><td>77.43</td></tr><tr><td colspan="5">using the shadow weights, i.e. applying updates to a full precision copy of quantized weights</td></tr></table>

Mean and standard deviation over multiple runs of

Table content: <table><tr><td colspan="4">Architecture Compute, bits Runs Accuracy, %</td><td>Memory, bits</td></tr><tr><td/><td/><td/><td>(mean±std)</td><td>(mean±std)</td></tr><tr><td>ResNet-18</td><td>5</td><td>5</td><td colspan="2">69.122±0.016 1.5150±0.0087</td></tr><tr><td>ResNet-34</td><td>5</td><td>4</td><td colspan="2">73.025±0.095 1.7875±0.033</td></tr></table>

Performance of soft entropy (8) and compressibility loss (10) on ResNet-18.Quantization of activations reduces memory access costs which are responsible for a significant part of energy footprint in NN accelerators. Conservative quantization approaches, known as post-training quantization, take a model trained for full precision and directly quantize it to 8-bit precision. These methods are simple to use and allow for quantization with limited data. Unfortunately, post-training quantization below 8 bits usually incurs significant accuracy degradation. Quantization-aware training approaches involve some sort of training either from scratch

Table content: <table><tr><td>Compute, bits</td><td>Loss</td><td colspan="2">Accuracy Memory, bits</td></tr><tr><td>4</td><td>entropy</td><td>67.86%</td><td>1.43</td></tr><tr><td>4</td><td>compressability</td><td>67.84%</td><td>1.50</td></tr><tr><td>5</td><td>entropy</td><td>69.49%</td><td>1.79</td></tr><tr><td>5</td><td>compressability</td><td>69.36%</td><td>1.73</td></tr><tr><td colspan="4">Batch size We have noticed that training ResNet-50 on a single GPU mandated the use of small</td></tr><tr><td colspan="4">batches, leading to performance degradation. Increasing the batch size from 16 to 64 without changing</td></tr><tr><td colspan="4">other hyperparameters increased accuracy by more than 0.5% with entropy increase by less than 0.1</td></tr><tr><td>bits/value.</td><td/><td/><td/></tr><tr><td>5 D I S C U S S I O N</td><td/><td/><td/></tr></table>

Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference on Computer Vision (ECCV), 2018. (cited on p. 3) Yiren Zhao, Xitong Gao, Daniel Bates, Robert Mullins, and Cheng-Zhong Xu. Efficient and effective quantization for sparse dnns. arXiv preprint arXiv:1903.03046, 2019. URL http: //arxiv.org/abs/1903.03046. (cited on p. 3) Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606.06160, 2016. URL http://arxiv.org/abs/1606.06160. (cited on p. 5) A A D D I T I O N A L R E S U LT S We list results of the additional experiments we performed in Tables A.1 and A.2. Table A.1: Experimental results for ResNet.

Table content: <table><tr><td colspan="2">Architecture Batch size</td><td>lr</td><td>λ</td><td colspan="3">Compute Memory Accuracy, %</td></tr><tr><td/><td/><td/><td>0</td><td/><td>2.050</td><td>68.000</td></tr><tr><td/><td/><td/><td>0.05</td><td>4</td><td>1.540</td><td>67.950</td></tr><tr><td/><td/><td/><td>0.08</td><td/><td>1.430</td><td>67.860</td></tr><tr><td/><td/><td/><td>0.05</td><td/><td>1.790</td><td>69.490</td></tr><tr><td/><td/><td/><td>0.05</td><td/><td>1.750</td><td>69.400</td></tr><tr><td/><td/><td/><td>0.1</td><td/><td>1.410</td><td>69.120</td></tr><tr><td/><td/><td/><td>0.12</td><td/><td>1.361</td><td>68.900</td></tr><tr><td/><td/><td/><td>0.15</td><td>5</td><td>1.280</td><td>68.914</td></tr><tr><td>ResNet-18</td><td>96</td><td>0.001</td><td>0.18 0.2</td><td/><td>1.120 1.110</td><td>68.160 68.300</td></tr><tr><td/><td/><td/><td>0.25</td><td/><td>1.040</td><td>67.800</td></tr><tr><td/><td/><td/><td>0.3</td><td/><td>0.974</td><td>67.700</td></tr><tr><td/><td/><td/><td>0</td><td/><td>3.100</td><td>70.000</td></tr><tr><td/><td/><td/><td>0.05</td><td>6</td><td>1.930</td><td>69.710</td></tr><tr><td/><td/><td/><td>0.08</td><td/><td>1.700</td><td>69.500</td></tr><tr><td/><td/><td/><td>0.05</td><td>7</td><td>2.280</td><td>69.660</td></tr><tr><td/><td/><td/><td>0</td><td/><td>5.100</td><td>69.900</td></tr><tr><td/><td/><td/><td>0.05</td><td>8</td><td>2.460</td><td>69.820</td></tr><tr><td/><td/><td/><td>0.08</td><td/><td>2.410</td><td>69.110</td></tr><tr><td/><td/><td/><td>0.05</td><td>8</td><td>2.750</td><td>73.200</td></tr><tr><td>ResNet-34</td><td>96</td><td>0.001</td><td>0.05</td><td>6</td><td>2.000</td><td>73.200</td></tr><tr><td/><td/><td/><td>0.05</td><td>5</td><td>1.790</td><td>73.100</td></tr><tr><td/><td>16</td><td/><td>0</td><td/><td>2.500</td><td>73.700</td></tr><tr><td/><td>16 64</td><td/><td>0.05 0.05</td><td>4</td><td>1.720 1.78</td><td>73.800 74.5</td></tr><tr><td>ResNet-50</td><td>48</td><td>0.0001</td><td>0.08</td><td/><td>1.67</td><td>74.2</td></tr><tr><td/><td>16</td><td/><td>0</td><td/><td>2.950</td><td>75.500</td></tr><tr><td/><td>16 16</td><td/><td>0.05 0.08</td><td>5</td><td>1.920 1.700</td><td>75.460 75.200</td></tr><tr><td/><td>16</td><td/><td>0.1</td><td/><td>1.600</td><td>74.900</td></tr></table>

Peer review (iclr2020):
Summary: The authors propose a method for training easy-to-quantize models that are quantized after training (post-training quantization). They do so by regularizing by the entropy, thereby forcing the weight distribution to be more compressible. They further compress the weights using entropy coding. Strengths of the paper: - The paper presents strong experimental results on ResNet, SqueezeNet VGG and MobileNet architectures and provides the code, which looks sensible. Weaknesses of the paper: - The authors could have applied CAT to other tasks such as Image Detection, while proving inference times on CPUs. Indeed, it is unclear to me what would be the influence of the entropic decoder which is claimed to be fast for *efficient implementations* by the authors. - The idea of regularizing by the entropy is not novel (see for instance *Entropic Regularization*, Grandvalet et a.l), as well as the idea of further encoding the weights using entropic coders (as in *Deep Compression*, Han et al.). Justification of rating: The authors present an intuitive method (yet not novel) for quantizing the weights of a neural network. My main concern would be about the inference time but I consider that the experimental results suggest strong evidence that CAT performs well on a wide variety of architectures.