CAT: Compression-Aware Training for bandwidth reduction
Authors: Chaim Baskin
Brian Chmiel
Evgenii Zheltonozhskii
Ron Banner
Alex M. Bronstein
Avi Mendelson
Venue:  (2020)
Keywords: compression
quantization
efficient inference
memory bandwidth
entropy
compression-aware training
Huffman
variable length coding

Abstract: Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators. Accordingly, an efficient feature map compression method can result in substantial performance gains. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way that allows better compression of feature maps during inference. Our method trains the model to achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods. CAT significantly improves the state-of-the-art results reported for quantization. For example, on ResNet-34 we achieve 73.1% accuracy (0.2% degradation from the baseline)  with an average representation of only 1.79 bits per value. Reference implementation accompanies the paper.

Under review as a conference paper at ICLR 2020
CAT: COMPRESSION-ANE - TRAINING FOR BANDWIDTH REDUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One the major obstacles hin- dering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and through- put bottleneck in hardware accelerators. Accordingly, compression method can result in substantial performance gains. Inspired by compiittionn
efficient feature map
quantization-awarr training approaches, we propose compression-awar train-
ing (CAT) method that involves training the model in way that allows better compression of feature maps during inference. Our method trains the model achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods. CAT significantly improves the sate-of-the-ann results reported for quantization. For example, on ResNet-34 we achieve 73.1% % accuracy (0.2% degradation from the baseline) with an average representation of only 1.79 bits per value. Reference implementation accompanies the paper.
1 INTRODUCTION
Deep Neural Networks (DNNs) have become a popular choice for wide range of applications such as computer vision, natural language processing, autonomous cars, etc. Unfortunately. their vast demands for computational resources often prevents their use power-challenged platforms. The desire for reduced bandwidth and compute requirements of deep learning models has driven research into quantization (Hubara et al., 2016; Yang al., 2019b; Liu et al., 2019; Gong et al., 2019), pruning (LeCun et al., 1990; Li et al., 2017; Molchanov et al., 2019), and sparsification (Gale et al., 2019; Dettmers & Zettlemoyer, 2019).
In particular, quantization works usually focus on scalar quantization of the feature maps: mapping the activation values to discrete set {9:} of size L. Such representation, while being less precise, especially useful in custom hardware, where it allows more efficient computations and reduces the memory bandwidth. In this work, focus the latter, which has been shown to dominate the energy footprint of CNN inference on custom hardware (Yang et al., 2017). We show that the quantized activation values {q:} can further be coded to reduce memory requirements.
The raw quantized data require [log2 (L)] bits per value for storage. This number can be reduced by compressing the feature maps. particular, in the case of element-wise compression of independent identically distributed values, the lower bound of amount of bits per element is given by the entropy (Shannon, 1948):
L. H(q) = - -200--Eroe p(a) loga p(g:)
(1)
of the quantized values {q:}, where p(q:) denotes the probability of
In this work, we take further step by manipulating the distribution of the quantization values so that the entropy H(g) is minimized. To that end, formulate the training problem by augmenting the regular -ask-specific loss (the cross-entropy classifier loss in our case) with the feature map entropy serving proxy to the memory rate. The strength of the latter penalty is controlled through

---

Under review as a conference paper at ICLR 2020
Epoch Epoch
Regulrication Medium regularization
6.35
Strong regulariration
0.30
Probability
Probability
0.20
0.10
0.30
0.65
0.00
-20
-15
-1.0
-0.5
0.0
05
1.0
1.5
20
-20
-15
-1.
-65
0.0
0.5
1.0
20
Value
Value
(a)
(b)
Figure I: Pre-activation distributions of one layer in ResNet-18. (a) Evolution at different epochs. As training progresses, the probability of non-positive pre-activation values increases, zeroing more Post-ReLU values. The sharp peak at zero reduces entropy and thus improves compressibility. (b) Effeet of entropy regularization. Without regularization, the distribution has much heavier tails and thus has higher entropy. With increasing regularization, the probability of extreme values is significantly reduced. The entropy penalty À was selected so that the overall accuracy is not affected. The compression ratio in the strongly regularized case is 2.23 higher compared to the unregularized baseline.
parameter À > O. Fig. demonstrates the effect of the entropy penalty on the compressibility of the intermediate activations.
Contributions. Our paper makes several contributions. Firstly, we introduce Compression-Awan Training (CAT), technique for memory bandwidth reduction. The method works by introducing loss term that penalizes the entropy the activations at training time and applying entropy encoding (e.g., Huffman coding) on the resulting activations at inference time. Since the only overhead of the method at inference time is entropy encoding that can be implemented efficiently, the improvement is universal for any hardware implementation, being especially efficient computationally optimized ones, where memory 1/O dominates the energy footprint (Yang et al., 2017; Jouppi et al., 2017). We demonstrate
4-fold memory bandwidth reduction for multiple architectures: MobileNetV2 and ResNet on the ImageNet visual recongition task, and SSD512 on the PASCAL VOC object detection task. We also investigate several differentiable loss functions which lead activation entropy minimization and show few alternatives that lead to the same effect.
stronger compression the expense of minor accuracy reduction: for ResNet-18, we manage to achieve entropy inferior to one bit per value, the expense of losing 2% of the top-1 accuracy.
Finally, analyze the rate-distortion tradeoff of the method, achieving
2 RELATED WORK
Recent studies (Yang et al., 2017: Wang et al., 2019) have shown that almost 70% of the energy foot- print on custom hardware due to the data movement to and from the off-chip memory. Nonetheless, the techniques for memory bandwidth reduction have not yet received significant attention in the literature. One way to improve memory performance by fusing convolutional layers (Xiao et al., 2017; Xing et al., 2019), reducing the number of feature maps transfers. This reduces both runtime and energy consumption of the hardware accelerator. Another option to use on-chip cache instead of external memory. Morcel et al. (2019) have shown order of magnitude improvement in power consuption using this technique. Another important system parameter dominated by the memory bandwidth is latency. Jouppi al. (2017) and Wang al. (2019) showed that the satte-of-the-rtt DNN accelerators are memory-bound, implying that increasing computation throughput without reducing the memory bandwidth barely affects the total system latency.
2

---

Under review as a conference paper at ICLR 2020
Quantization reduces computation and memory requirements; 16-bit fixed point has become de facto standard for fast inference. In most applications, weights and activations can be quantized down to 8 bits without noticeable loss of precision (Lee et al., 2018; Yang al., 2019a). Further quantization to lower precision requires non-trivial techniques (Mishra et al., 2018; Zhang et al., 2018), which are currently capable of reaching around 3-4 bits per entry without compromising precision (Choi et al., 2018b;a; Dong et al., 2019).
Another way reduce memory bandwidth by compressing the intermediate activations prior to their transfer to memory with some computationally cheap encoding, such that Huffman (Chandra, 2018; Chmiel et al., 2019) or run-length (RLE) encoding (Cavigelli et al., 2019). A similar approach of storing only nonzero values was utilized by Lin & Lai (2018). Chmiel et al. (2019) used linear dimensionality reduction (PCA) to increase the effectiveness of Huffman coding, while Gudovskiy et al. (2018 8) proposed to use nonlinear dimensionality reduction techniques.
Lossless coding was previously utilized in a number of ways for DNN compression: Han al. (2016) and Zhao et al. (2019) used Huffman coding to compress weights, while Wijayanto al. (2019) used more complicated DEFLATE (LZ77 + Huffman) algorithm for the same purpose. Aytekin et al. (2019) proposed to use compressibility loss, which induces sparsity and has been shown (empirically) to reduce entropy of the non-zero part of the activations.
3 METHOD
We consider a feed-forward DNN F composed of I. layers; each subsequent layer processing the output of the previous one: x' Fi (z using the parameters w RR We denote by x. and _ = y the input and the output of the network, respectively.
The parameters of the network are learned by minimizing ((x, ys w) + XR(w), with the former term being the task loss, and the latter term R. being a regularizer (e.g., [[w]])) inducing some properties on the parameters w.
3.1 ENTROPY ENCODING AND RATE REGULARIZATION
Entropy encoders are a family of lossless data encoders which compress each symbol indepen- dently. In this case, assuming i.i.d. distribution of the input, it has been show that optimal code length is log, P. where
number of symbols and p; is the probability of 1th symbol (Shan- non, 1948). Thus, for discrete random variable X we define entropy ((X) Eog, X - - p(x;) log, p(x.), which is a lower bound of the amount of information required for lossless compression of X. The expected total space required to encode the message is N H. where N is number of symbols. Since we encode the activations with entropy encoder before writing them into memory, we would like to minimize the entropy of the activations to improve the compression rate.
One example of entropy encoder is Huffman coding - prefix coding which assigns shorter codes popular because simplicity along with high compression rate (Szpankowski, 2000) bounded by (()) < R < H(X) 1 (the comparison of Huffman coding rates entropy is shown Fig. 3). Another entropy encoder is arithmetic coding highly efficient encoder which achieves optimal rates for big enough input but requires more computational resources for encoding.
the more probable symbols.
3.2 DIFFERENTIABLE ENTROPY-REDUCIIS LOSS
Since the empirical entropy is discrete function, it is not differentiable and thus cannot be directly minimized with gradient descent. Nevertheless, there exist number differentiable functions which either approximate entropy have same minimizer. Thus optimize
L= L. + XLH,
(2)
where Ly is a target loss function and Lu is some regularization which minimizes entropy.
3

---

Under review as a conference paper at ICLR 2020
Soft entropy First, we consider the differentiable entropy estimation suggested by Agustsson et al. (2017). We start from definition of the entropy,
H(X) Σ log(((x))
(3)
Pj = Mar [+]1
(4)
N
where q is vector of quantized values. Let m be an index of the bin the current value is mapped to, and Q one-hot encoding of this index, i.e.,
fm arg min x 4.E0
9hl arg max (-lar (:l))
(5)
9.
Q: 8pm:
(6)
where dim denotes Kroneker's delta. To make the latter expression differentiable, we can replace argmax with softmax:
(xx) softmax(- ql.
where 7' is the temperature parameter, and Qxx) + Q(x) as T
Finally, the soft entropy H is defined
(x) = Σ (og((a))(((
(8)
A E, (z,)
To improve both memory requirements and time complexity of the training, we calculate soft entropy only on part of the batch, reducing the amount of computation and the gradient tensor size. We empirically confirm that this choice gives a reasonable approximation of the real entropy (Appendix B).
Compressibility loss for entropy reduction An alternative loss promoting entropy reduction was proposed by Aytekin al. (2019) under the name of compressibility loss and based earlier work by Hoyer (2004):
X |
L.
(10)
This loss has the advantage of computational simplicity, and has been shown both theoretically and practically promote sparsity and low entropy in input vectors. While originally applied to the weights the network, here we apply the same loss to the activations.
As shown in Section 4.1, both the soft entropy and the compressibility loss lead to similar results.
Our method for reducing memory bandwidth can be described as follows: at training time, we fine-tune (training from seratch should also be possible, but we have not tested it) the pre-trained network with the regularized loss (2), where we use L# Σ H (x;) in case of differentiable entropy and Lu = Σ (x,)) in case of compressability loss. At test time, we apply entropy coding on the activations before writing them to memory, thus reducing the amount memory transactions. In contrast to Chmiel et al. (2019), who avoided fine-tuning by using test time transformation in order to reduce entropy, our method does not requires complex transformations at test time by inducing low entropy during training.
4 EXPERIMENTAL RESULTS
We evaluate the proposed scheme common CNN architectures for image classification on ImageNet (ResNet-18/34/55 MobileNetV2), and object detection on Pascal VOC dataset (SSD512! (Liu et al., 2016)). The weights were initialized with pre-trained model and quantized with uniform quantization
Our code is hased implementation by Li (2018).
4

---

Under review as a conference paper at ICLR 2020
715
710
20.5
70.0
Accuracy
69.5
Baseline
68.5
CAT bits
Baseline
EBPC- bits
CAT - bits
GF 8 bits
bits
CAT bits
GF 6 bits
CAT 4 bits
665
CAT bits
660
15
3.0
3. Average bits per value
Average bits per value
(a)
Comparison with other methods: EPBC (Cavigelli et al. (2019)) and GF (Gudovskiy et al. (2018)) in (a) ResNet-34 and (b) MobilenetV2. Different marker size refers different activation bitwidths before compression. For GF, compression rate was averaged only over compressed layers.
Figure 2:
Table 1: Results for ResNet-18. 18, ResNet-50, and SSD512. For comparison we include the results obtained by Gudovskiy et al. (2018) for the SSD512 model the same task but with different backbone, for which we obtain better compression rate with a lower accuracy degradation. Compute denotes activation bitwidth used for arithmetic operations. Memory denotes average number of bits for memory transactions (after compression). Compression ratio denotes the reduction in representation size. Weight bitwidth is 8 except the full-precision experiments. Additional experimental results provided in Appendix A.
Architecture
Compute (bits)
Memory (bits)
Compression ratio
Top-1 accuracy (%)
32
32
69.70
ResNet-18, CAT
5
1.5
3.33
69.20
1.51
2.65
68.08
32
32
76.1
ResNet-50, CAT
5
1.60
3.125
74.90
1.78
2.25
74.50
32
32
68.12
SSD512-SqueezeNet (Gudovskiy et al., 2018)
8
64.39
6
2
62.09
32
32
80.72
SSD512-VGG, CAT
2.334
2.57
77.49
1.562
2.56
77.43
using the shadow weights, i.c. applying updates to full precision copy of quantized weights (Hubara al., 2016; Rastegari et al., 2016). The activation were clipped with learnable parameter and then uniform quantized as suggested by Baskin et al. (2018). Similarly to previous works (Zhou al., 2016; Rastegari et al., 2016), we used the trriigh-tthruugh estimator (Bengio al., 2013) to approximate the gradients. We quantize all layers in the network, in contrast to the common practice of leaving first and last layer high-precision (Zhou et al., 2016; Baskin al., 2018). Since the weights quantized only to 8 bit, we have noticed small no difference between the quantized and non-quantized weights.
For optimization, 4 x 10 for up to 30 epochs (usually, 10 - 15 epochs were sufficient for convergence). Our initial choice of temperature was T 10, which performed well. We tried, following the common approach (Jang et al., 2016), to apply exponential scheduling to the temperature, but it did not have any noticeable effect on the results.
use SGD with learning rate of 10-4, momentum 0.9, and weight decay
5

---

Under review as a conference paper at ICLR 2020
715
695
(9)
69.5
65.0
605
68.0
67.
Theoretical Entropy 8 bits
Huttman Coding bits
68.
Theoretical Entropy
555
Theoretical Entropy 66 bits
Huffman Coding
650
Huffman Coding bits
Deseline
645
Baseline
640
1.0
1.2
1.4
15
15
15
2.5
35 Average bits per value
10
4.5
Average bits value
(a)
Figure 3: Tradeoff between rate and accuracy for different values of À (ranged between 0 and 0.3) in (a) ResNet-18 and (b) MobileNetV2. In ResNet-18 18 the activations are quantized to bits, in MobileNet we show results for activation quantized to 6 and 8 bits.
Table Mean and standard deviation over multiple runs ResNet- 18 and ResNet-34.
Architecture Compute, bits
Runs
Accuracy, % (mean-ssd)
Memory, bits (mean-Est))
ResNet-18 ResNet-34 5
5
5
69.122:10.016
1.5150=000087 1.7875
73.025:10.095 4
In Fig. 2 we compare our method with EPBC (Cavigelli et al. (2019)) and GF (Gudovskiy et al. (2018)). EPBC based on lossless compression method that maintains the full precision accuracy while reducing bit rate to approximately 3.5 bits/value in both models. GF, on the other hand, provides strong compression at the expense of larger accuracy degradations. In addition, Gudovskiy et al. (2018) have compressed only part of the layers. Unlike these two methods, CAT allows more flexible tradeoff between compression and accuracy. CAT shows better results in ResNet-34 and show either better accuracy compression for MobileNetV2. We also ResNet-18, 18, ResNet-50, and SSD512 with VGG backbone; the results are listed in Table 1. Even though we can directly compare detection results with Gudovskiy al. (2018), the drop in accuracy is lower in our case.
method additional architectures:
4.1 ABLATION STUDY
Rate-Accuracy Tradeoff The proposed CAT algorithm tries to balance between minimizing the rate and maximizing the accuracy of the network by means of the parameter À in Eq. (2). To check this tradeoff, of À in the range of 0 - 0.3, with results shown in Fig. 3. We show the results of the theoretical entropy and of the Huffman coding, which chosen for its simplicity but more efficient encoders can be combined with our method. For higher rate Huffman coding is close (~3% overhead) to the theoretical entropy, while for lower entropy the difference higher and is bounded by 1 bit per value in the latter case, different lossless coding schemes such arithmetic coding can provide better results.
the same experiment in ResNet- 18 and MobileNetV2 with different values
Robustness For checking the robustness of our method, we performed several runs with the same hyper--prrmeters and different random seed. Statistics reported in Table suggest that the method robust and stable under random intialization.
Soft entropy vs. compressibility loss Replacing the soft entropy with different loss which minimizes the entropy almost did not affect the results, shown in Table 3. We conclude that the desired effect is a result of entropy reduction rather than a particular form of regularization promoting it.

---

Under review as a conference paper at ICLR 2020
Table 3: Performance of soft entropy (8) and compressibility loss (10) on ResNet- 18.
Compute, bits
Loss
Accuracy
Memory, bits
entropy
67.86%
1.43
4
compressability
67.84%
1.50
5
entropy 69.49%
1.79
5
compressability 69.36%
1.73
Batch size We have noticed that training ResNet-50 single GPU mandated the of small batches, leading to performance degradation. Increasing the batch size from to 64 without changing other hyperaarameters increased accuracy by more than 0.5% with entropy increase by less than 0.1 bits/value.
5 DISCUSSION
Quantization of activations reduces memory access costs which are responsible for a significant part of energy footprint in NN accelerators. Conservative quantization approaches, known as post-training quantization, take model trained for full precision and directly quantize it to 8-bit precision. These methods are simple use and allow for quantization with limited data. Unfortunately, post-training quantization below 8 bits usually incurs significant accuracy degradation. Quantization training approaches involve some sort of training either from scratch (Hubara et al., 2016), or as a fine-tuning step from pre-trained floating point model (Han et al., 2016). Training usually compensates significantly for model's accuracy loss due to quantization.
In this work, we take further step and propose a compression-aware training method to aggressively compress activations to as low as average bit/value representations without harming accuracy. Our method optimizes the average-bil bit per value needed to represent activation values by minimizing the entropy. We demonstrate the applicability of our approach on classification tasks using the models MobileNetV2,, ResNet, and object detection task using the model SSD512. Due to the low overhead, the method provides universal improvement for any custom hardware, being especially useful for the accelerators with efficient computations, where memory transfers are significant part of the energy budget. We show that the effect is universal among loss functions and robust to random initialization.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V. Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1141-1151. Curran Associates, Inc., 2017. URL http: /papers nips. cctpppseccppppp/// pdf. (cited on p. 4)
Caglar Aytekin, Francesco Cricri, and Emre Aksu. Compressibility loss for neural network weights. arXiv preprint arXiv: 1905. 00044, 2019. URL http://arxiv. www. www./org/////o/oo//////...t....e 01044 (cited on pp. 3 and
Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Giryes, Avi Mendelson, and Alexander M. Bronstein. NICE: noise injection and clamping estimation for neural network quantization. CoRR, abs/1810.00162, 2018. URL http: htap:://aaxivv www.aawwwww..r..ra/aa/o/// 1810. 111000166. 62. (cited on p. 5)
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL http: http///axvvvv..///// wwtp://g/gabbb///e 3432. (cited on 5)
Lukas Cavigelli, Georg Rutishauser, and Luca Benini. EBPC: Extended bit-plane compression for deep neural network inference and training accelerators. arXiv preprint ar0iv: 1645, 2019, (cited on pp. 3, 5, and 6)
Mahesh Chandra. Data bandwidth reduction in deep neural network soes using history buffer and huffman coding. In 2018 International Conference on Computing, Power and Communication Technologies (GUCON), pp. 1-3. IEEE, 2018. (cited on p. 3)
Brian Chmicl, Chaim Baskin, Ron Banner, Evgenii Zheltonozhskii. Yevgeny Yermolin, Alex Karbachevsky, Alex M. Bronstein, and Avi Mendelson. Feature map transform coding for energy-efficient cnn inference. arXiv preprint arXiv:/905.108 May 2019. URL http: //arxiv.org/ abtp://bb//////// 11..9.....880000 10830. (cited on pp. and 4)
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint arXiv:1807.0696: 2018 8a. URL https: //ax/aavvv...///////aa/////aa wwtp://www/g/bb/////eeeea 1694bb.694.0 06964. (cited on p. 3)
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 06085, 2018b. (cited on p. 3)
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv: 1907.04440 2019. URL https: / http://a.vvvvo//////o//oo org/ abs 1907. 04840. (cited on p.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer, Hawq: Hessian aware quantization of neural networks with mixed-precision. arXiv preprint arXiv: 1005: 090500396, 2019. (cited on p.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint on p. 1)
095744 2019. URL httph: //axiiv.//r.vvvoa/aa org/ abs 20ts/1902. 09574. (cited
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. arXiv preprins arXiv:1908.0550 05033, 2019. URL http: http///xxivv.// org. /as.///.................../..... 05033. (cited p.
8

---

Under review as a conference paper at ICLR 2020
Denis Gudovskiy, Alec Hodgkinson, and Luca Rigazio. Dan feature map compression using learned representation over gf(2). In The European Conference on Computer Vision (ECCV) Workshops, September 2018. (cited pp. 3, 5, and 6)
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016. (cited on pp. 3 and 7)
Patrik O. Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, (NNov):1457-14444 2004. URL hattp://wwe jmlr wwwlr.org//appeeee httg:/apers/v///// hoyer04a. html. (cited p. 4)
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv: 1rXiv:1609.07061 2016. URL https: /tapps:///rxiii /arxiv. org/ /609/1609... 07061. (cited
on pp. 1,5, and 7)
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparaaetterizatioo with gumbel-softmax. arXiv preprint arXiv:161.00114, 2016. URL https::///// arxiv. wwt./orgg/////oo/ooooeeoe abs//bsss1111 01144. (cited on p. 5)
Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, Richard C. Ho, Doug Hogberg, John Hu. Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang. Eric Wilcox. and Doe Hyun Yoon. In-datacenter performance analysis tensor processing unit. In 2017 ACM/IEE 44th Anmual International Symposium Computer Architecture (ISCA), pp. 1-12. IEEE, 2017. (cited p. 2)
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky (ed.), Advances in Neural Information Processing Systems 2, pp. 598-605, Morgan- Morgan-Kaufnnnnn Kaufmann, 1990. URL httt:////// /ttpppappeeee////////e/eeeeessss mips. ce/ pwppr/ optimaliiiommmmmmmo bbainn damage pdf. (cited on pnp.1. 1)
Jun Haeng Lee, Sangwon Ha, Saerom Choi, Won-Jo Lee, and Seungwon Lee. Quantization for rapid deployment of deep neural networks, 2018. (cited 3)
Congcong Li. High quality, fast, modular reference implementation of SSD in PyTorch. https: //github. www. htmm//fffccc/SSD1 2018. (cited on p.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017. htth:
// //openreview.co net/ wwtpe///fouuee forum? id IJgGGTs (cited on p.
Chien Yu Lin and Bo-Cheng Lai. Supporting compressed-sparse activations and weights simd-like accelerator for sparse convolutional neural networks. In 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC) pp. 105-110, Jan 2018. doi: 10.1109/ASPDAC.201.. 82299999...
(cited on p.
Chunlei Liu, Wenrui Ding, Xin Xia, Yuan Hu, Baochang Zhang, Jianzhuang Liu, Bohan Zhuang, and Guodong Guo. RBCN: Rectified binary convolutional networks for enhancing the performance of 1-bit DCNNs. arXiv preprint arXiv:1908.0772 2019. URL https: : / /arxivv. org / abs/ 1908. 07748. 8. (cited p.

---

Under review as a conference paper at ICLR 2020
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 21-37, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46448-0, (cited on p. 4)
Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. Wrpn: Wide reduced- precision networks. International Conference on Learning Representations, 2018. URL
net mettfoorumidd-B22aaeeeeee
(cited on p. 3)
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In The IEEE Conference on Computer Vision and Pattern Recognition June 2019. URL http: http///peaaccccssssceeeesscccsss thecvf. com/ content_cVPR html /Mol wwwpnnooooooootttneeeoo SVPR_2013 _aap9__pper_o_ html. (cited on p. 1)
inntttitttiooniii forrtioooooo__rrraaa__aae N Seeework_Pruning_
Raghid Morcel, Hazem Hajj. Mazen A. R. Saghir, Haitham Akkary, Hassan Artail, Rahul Khanna, and Anil Keshavamurthy. Feathernet: An accelerated convolutional neural network design for resource- constrained fpgas. ACM Trans. Reconfiguruble Technol. Syst, 12(2):6:1-6::2) March 2019. ISSN 1936-7406. doi: 10.1145/3306202. URL http:/ /doi. acm. org/ 10. 110...115//0 1145/3306222 (cited on 2)
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016. (cited on p. 5)
Claude E. Shannon. mathematical theory of communication. The Bell System Technical Journal, 27(3):379-422 423, July 1948. doi: 10. 10020000. 19433--733..1994.60000
(cited on pp. 1 and 3)
Wojciech Szpankowski. Asymptotic average redundancy of huffiman (and shannon-fano) block codes. In 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060), pp. 370-, June 2000. doi: 10.1 109/1SIT:20000000 86600000..666..... (cited on p. 3)
Erwei Wang, James Davis, Peter YK Cheung, and George A Constantinides. Lutnet: Rethinking inference in fpga soft logic. arXiv preprint arXiv: 00938, 2019. (cited on p. 2)
Arie Wahyu Wijayanto, Jun Jin Choong, Kaushalya Madhawa, and Tsuyoshi Murata. Towards robust compressed convolutional neural networks, In 2019 IEEE International Conference on Big Data and Smart Computing (BigComp). pp. 1-8. IEEE, 2019. (cited on 3)
Qingcheng Xiao, Yun Liang, Liqiang Lu, Shengen Yan, and Yu-Win Yu-Wing Tai. Exploring heterogeneous algorithms for accelerating deep convolutional neural networks on fpgas. In Proceedings of the 54th Annual Design Automation Conference 2017, DAC '17, pp. 62:1-62:6. New York, NY, USA, 2017. ACM. ISBN 978- 118-1-4503-4927--, doi: 10. 639.30622444 URI. attp://dcis acm. org 1 45 66666666669...... 3022244,444 (cited p. 2)
Yu Xing. Shuang Liang. Lingzhi Sui, Xijie Jia, Jiantao Qiu, Xin Liu, Yushun Wang, Yi Shan, and Yu Wang. Dnnvm: End-to-end compiler leveraging heterogeneous optimizations on fpga- based enn accelerators. IEEE Transactions Computer-Aided Design of Integrated Circuits and Systems,
pp. 1-1, 2019. doi: 10.1 101//CCD..21999993003.3 (cited on p. 2)
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christo- pher De Sa. Swalp: Stochastic weight averaging in low-precision training. arXiv preprint arXhv:1904.11943 2019a. (cited on p. 3)
Jiwei Yang. Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b. URL http: pestp://openacceee optpa/cccsccsssccooo thecvf. com/ content_ CVPR_ 201/22211///1 0019/html/// /tt/t//m/aaagg../e//eeeaa Quant i i CVPR_2012 _019_paperrr__p__eeeeee_a_a_eeee .htll. (cited on p. 1)
Tien-Ju Yang, Yu-Hsin Chen, Joel Emer, and Vivienne Sze. A method to estimate the energy consumption of deep neural networks. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, pp. 1916-1920. IEFE, 2017. (cited on pp. I and
10

---

Under review as a conference paper at ICLR 2020
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference on Computer Vision (ECCV), 2018. (cited on p.
Yiren Zhao, Xitong Gao, Daniel Bates, Robert Mullins, and Cheng-Zhong Xu. Efficient and effective quantization for sparse dnns, arXiv preprint 1rXiv:1903.0304/: 2019. URL http: /ttpxxivorr////ee...........ttot http:///ggaaaaaaogeeee 03046. (cited on p.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, (621/6606666666 2016. URL pattp::/arxx.. /arxxv...vvv/////////////////a
06160. (cited on p. 5)

---

Under review as a conference paper at ICLR 2020
A ADDITIONAL RESULTS
We list results of the additional experiments we performed Tables A.1 and A.2.
Table A.I: Experimental results for ResNet.
Architecture
Batch size
Ir
A
Compute
Memory
Accuracy, %
0
2.050
68.000
0.05
4
1.540
67.950
0.08
1.430
67.860
0.05
1.790
69,490
0.05
1.750
69.400
0.1
1.410
69.120
0.12
1.361
68.900
0.15
5
1.280
68.914
0.18
1.120
68.160
ResNet-15 18
96
0.001
0.2
1.110
68.300
0.25
1.040
67.800
0.3
0.974
67.700
0
3.100
70.000
0.05
6
1.930
69.710
0.08
1.700
69.500
0.05
7
2.280
69.660
0
5.100
69.900
0.05
8
2.460
69.820
0.08
2.410
69.110
0.05
8
2.750
73.200
ResNet-34
96
0.001
0.05
6
2.000
73.200
0.05
5
1.790
73.100
16
0
2.500
73.700
16
0.05
1.720
73.800
4
64
0.05
1.78
74.5
48
0.08
1.67
74.2
Re
0.0001
16
0
2.950
75.500
16
0.05
1.920
75.460
5
16
0.08
1.700
75.200
16
0.1
1.600
74.900
B SAMPLE SIZE IN SOFT ENTROPY CALCULATION
To check whether the amount of values used to calculate soft entropy is enough, we ran a soft entropy evaluation single tensor and compared it real values. Since the tensors large (hundreds of thousands of elements), even 5% the values already provides reasonable approximation of the real entropy, as shown Fig. B.I.
12

---

Under review as a conference paper at ICLR 2020
Table A.2: Experimental results for MobileNet.
Architecture
Batch size
Ir
Compute
Memory
Accuracy, %
64
0.0001
0
2.200
66.150
64
0.001
0
2.800
66.200
64
0.0001
0.05
2.100
66,900
4
64
0.001
0.05
2.080
66,400
64
0.001
0.08
1.830
66.200
64
0.0001
0.08
1.980
66.450
64
0.001
3.900
69.600
64
0.0001
0
3.700
71,000
96
0.0001
0.05
2.950
71.500
32
0.0001
0.08
2.700
70,930
6
64
0.0001
0.1
2.700
70.950
MobileNetV2
32
0.001
0.15
1.750
64.000
64
0.0001
0.15
2.600
71.200
64
0.0001
0.2
2.450
70.700
64
0.0001
4.750
71.300
64
0.0001
0.05
4.150
71.400
64
0.001
0.08
2.600
70.000
64
0.0001
0.08
4.120
71,600
64
0.001
0.1
2.400
69.600
8
64
0.0001
0.1
4.100
71.250
64
0.001
0.15
2.300
68.500
64
0.001
0.2
2.150
68.000
64
0.0001
0.2
3.500
71.200
32
0.0001
0.3
2.900
70.700
3.675
Real Entropy
Chosen ratio
3.650
Soft Entropy
3.625
Entropy
3.600
3.575
3.550
3.525
3.500
0.025
0.050
0.075
0.100
0.125
0.150
0.175
0.200
Soft entropy data ratio
Figure B.1: Soft entropy with different sample size compared real entropy of single batch. Error bars are for standard deviation over 3 runs.
13


Reviews:
[{"review_id": "rklhGYNC5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The format of the paper does not meet the requirement of ICLR. Due to this, I will give a 3. I suggest the authors to change it as soon as possible.\n\nBesides that, the main idea of the paper is to regularize the training of a neural network to reduce the entropy of its activations. There are extensive experiments in the paper.\n\nThe paper introduce two kinds of method to regularize the entropy. The first method is a soft version of the original entropy, and the second is the compressibility loss. After adding the regularization, the performance drop of the compressed network is reduced. The experiment performance is promising.\n\nI think the method is straightforward and reasonable with only a few questions:\n1. Why do you quantize the weight? Seems it's not necessary because the paper only address activation quantization.\n2. What will happen if the weights are quantized to lower bits? For example, 4bit?\n2. How about adding the regularization to weights?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJghghXPtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a method for training easy-to-quantize models that are quantized after training (post-training quantization). They do so by regularizing by the entropy, thereby forcing the weight distribution to be more compressible. They further compress the weights using entropy coding.\n\nStrengths of the paper:\n- The paper presents strong experimental results on ResNet, SqueezeNet VGG and MobileNet architectures and provides the code, which looks sensible. \n\nWeaknesses of the paper:\n- The authors could have applied CAT to other tasks such as Image Detection, while proving inference times on CPUs. Indeed, it is unclear to me what would be the influence of the entropic decoder which is claimed to be fast for \"efficient implementations\" by the authors.\n- The idea of regularizing by the entropy is not novel (see for instance \"Entropic Regularization\", Grandvalet et a.l),  as well as the idea of further encoding the weights using entropic coders (as in \"Deep Compression\", Han et al.).\n\nJustification of rating:\nThe authors present an intuitive method (yet not novel) for quantizing the weights of a neural network. My main concern would be about the inference time but I consider that the experimental results suggest strong evidence that CAT performs well on a wide variety of architectures. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BklsH3PCcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this article, the authors propose a compression-aware method to achieve efficient compression of feature maps during the inference procedure. When in the training step, the authors introduce a regularization term to the target loss that optimizes the entropy of the activation values. At inference time, an entropy encoding method is applied to compress the activations before writing them into memory. The experimental results indicate that the proposed method achieves better compression than other methods while holding accuracy.\nThere are still some issues as follows:\n1.\tThe authors should carefully check the format of the references in the whole article. For example, in section 2, line 5 from the top and line 8 from the bottom, “Xiao et al. (2017), Xing et al. (2019)” and “(Chmiel et al., 2019)” are in the wrong format.\n2.\tIt is suggested that the authors swap the order of formulation (8) and (9) in section 3.2 so that it will be a good correlation with the formulation (3) and (4).\n3.\tI am interested in learning the time taken by the proposed method during the inference procedure vs other related methods.\n4.\tThe authors studied two differentiable entropy approximation in the paper, and they stated that they calculate soft entropy only on the part of the batch for the reduction of both memory requirements and time complexity in training. I hope the authors will clarify 1) Whether the accuracy will be affected by other differentiable entropy approximations; 2) what is the impact on accuracy if only part of the batch is considered.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work propose a compression-aware training (CAT) method to allows efficient compression of  feature maps during inference. I read the paper myself. The proposed method is quite straightforward and looks incremental compared with existing approaches based on entropy regularization. 


Decision: Reject