Feature Map Transform Coding for Energy-Efficient CNN Inference
Authors: Brian Chmiel
Chaim Baskin
Ron Banner
Evgenii Zheltonozhskii
Yevgeny Yermolin
Alex Karbachevsky
Alex M. Bronstein
Avi Mendelson
Venue:  (2020)
Keywords: compression
efficient inference
quantization
memory bandwidth
entropy

Abstract:     Convolutional neural networks (CNNs) achieve state-of-the-art accuracy in a variety of tasks in  computer vision and beyond. One of the major obstacles hindering the ubiquitous use of CNNs for inference on low-power edge devices is their high computational complexity and memory bandwidth requirements. The latter often dominates the energy footprint on modern hardware. In this paper, we introduce a lossy transform coding approach, inspired by image and video compression, designed to reduce the memory bandwidth due to the storage of intermediate activation calculation results. Our method does not require fine-tuning the network weights and halves the data transfer volumes to the main memory by compressing feature maps, which are highly correlated, with variable length coding. Our method outperform previous approach in term of the number of bits per value with minor accuracy degradation on ResNet-34 and MobileNetV2. We analyze the performance of our approach on a variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40% in the memory energy footprint, compared to quantized network, with negligible impact on accuracy. When allowing accuracy degradation of up to 2%, the reduction of 60% is achieved. A reference implementation}accompanies the paper.

Under review as a conference paper at ICLR 2020
FEATURE MAP TRANSFORM CODING FOR ENERGY- EFFICIENT CNN INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks (CNNs) achieve state-of-the-art accuracy in a variety of tasks in computer vision and beyond. One of the major obstacles hindering the ubiquitous use of CNNs for inference on low-power edge devices is their high computational complexity and memory bandwidth requirements. The latter often dominates the energy footprint on modern hardware. In this paper, we introduce a lossy transform coding approach, inspired by image and video compression, designed to reduce the memory bandwidth due to the storage of intermediate activation calculation results. Our method does not require fine-tuning the network weights and halves the data transfer volumes to the main memory by compressing feature maps, which are highly correlated, with variable length coding. Our method outperform previous approach in term of the number of bits per value with minor accuracy degradation on ResNet-34 and MobileNetV2. We analyze the performance of our approach variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40% the memory energy footprint, compared to quantized network, with negligible impact on accuracy. When allowing accuracy degradation of up 2%, the reduction of 60% is achieved. reference implementation accompanies the paper.
Introduction
Deep neural networks have established themselves as the first-choice tool for wide range of applications. Neural networks have shown phenomenal results in variety of tasks in broad range of fields such as computer vision, computational imaging, and image and language processing. Despite deep neural models impressive performance, the computation and computational requirements are substantial for both training and inference phases. So far, this fact has been a major obstacle for the deployment of deep neural models in applications constrained by memory, computational, and energy resources, such as those running on embedded systems.
To alleviate the energy cost, custom hardware for neural network inference, including FPGAs and ASICs, is actively being developed in recent years. In addition to providing better energy efficiency per arithmetic operation, custom hardware offers more flexibility various strategies to reduce the computational and storage complexity of the model inference, for example by means of quantization (Baskin al., 2018; Hubara et al., 2018; Jacob et al., 2018) and pruning (Han et al., 2016; Louizos et al., 2018 Theis et al., 2018). In particular, quantization to very low precision is especially efficient on custom hardware where arbitrary precision arithmetic operations require proportional resources. To prevent accuracy degradation, many approaches have employed training the model with quantization constraints or modifying the network structure.
A recent study (Yang et al., 2017) has shown that almost 70% of the energy footprint on such hardware due to data movement to and from the off-chip memory. Amounts of data typically need to be transferred to and from the RAM and back during the forward pass through each layer, since the local memory is too small to store all the feature maps. By reducing the number of bits representing these data, existing quantization techniques reduce the memory bandwidth considerably. However, to the best of our knowledge, none of these methods exploit the high amount of interdependence between the feature maps and spatial locations of the compute activations.

---

Under review as a conference paper at ICLR 2020
Contributions. In this paper, we propose novel scheme based transform- quantization of the neural network activations followed by lossless variable length coding. We demonstrate that this approach reduces memory bandwidth by 40% when applied in the post-training regime (i.e., without fine-tuning) with small computational overhead and no accuracy degradation. Relaxing the outperform previous
accuracy requirements increases bandwidth savings to 60%. Moreover, methods in term of number bit per value with minor accuracy degradation. A detailed evaluation of various ingredients and parameters of the proposed method is presented. We also demonstrate a reference hardware implementation that confirms reduction in memory energy consumption during inference.
2 Related work
Quantization. Low-precision representation of the weights and activations is a common means of reducing computational complexity. On appropriate hardware, this typically results in the reduction of the energy footprint as well. It has been demonstrated that in standard architectures quantization down 16 (Gupta et al., 2015) or bits (Jacob al., 2018; Lee et al.. 2018; Yang et al., 2019) per parameter is practically harmless. However, further reduction of bitwidth requires non-trivial techniques (Mishra et al., 2018; Zhang et al., 2018), often with adverse impact on training complexity. Lately, the quantization of weights and activations of neural networks to 2 bits or even 1 (Rastegari et al., 2016, Hubara et al., 2018) has attracted the attention of many researchers. While the performance of binary (i.e.. I-bit) neural networks still lags behind their full-precision counterparts (Ding et al., 2019; Peng & Chen, 2019), existing quantization methods allow 2-4 bit quantization with a negligible impact on accuracy (Choi et al., 2018b;a; Dong et al., 2019).
Quantizing the neural network typically requires introducing the quantizer model at the training stage. However, in many applications the network is already trained in full precision, and there is no access to the training set to configure the quantizer. In such a post-training regime, most quantization methods employ statistical clamping, i.e., the choice of quantization bounds based on the statistics acquired in a small test set. Migacz (2017) proposed using a small calibration set to gather activation statisties and then randomly searching for quantized distribution that minimizes the Kullback- Leibler divergence to the continuous one. Gong et al. (2018), on the other hand, used the Lx norm of the tensor threshold. Lee al. (2018) employed channel-wise quantization and constructed a dataset of parametric probability densities with their respective quantized versions; simple classifier was trained to select the best fitting density. Banner al. (2018) derived an approximation of the optimal threshold under the assumption of Laplacian or Gaussian distribution of the weights, which achieved single-percentage accuracy reduction for 8-bit weights and 4-bit activations. Meller et al. (2019) showed that the equalization of channels and removal of outliers improved quantization quality. Choukroun et al. (2019) used one- one-dimensional line-search to evaluate an optimal quantization threshold, demonstrating state-of- the-art results for 4-bit weight and activation quantization.
Influence of memory access on energy consumption. Yang et al. (2017) studied the breakdown of energy consumption in CNN inference. For example, in GoogLeNet (Szegedy al., 2015) arithmetic operations consume only 10% of the total energy, while feature map transfers to and from an external RAM amount about 68% of the energy footprint. However, due the complexity of real memory systems, not every method that decreases the sheer memory bandwidth will necessarily yield significant improvement in power consumption. In particular, depends on computational part optimization: while memory performance is mainly defined by the external memory chip, better optimization of computations will lead higher relative energy consumption of the memory. For example, while Ansari & Ogunfunmi (2018) 8) reported a 70% bandwidth reduction, the dynamic power consumption decreased by a mere 2%.
Xiao et al. (2017) proposed fusing convolutional layers to reduce the transfer of feature maps. In an extreme case, all layers are fused into a single group. A similar approach was adopted by Xing et al. (2019), who demonstrated hardware design that does not write any intermediate results into the off-chip memory. This approach achieved approximately 15% runtime improvement for ResNet and ofate-of-the-ar throughput. However, the authors did not compare the energy footprint of the design with the baseline. Morcel et al. (2019) demonstrated that using on-chip cache cuts down the memory bandwidth and thus reduces power consumption by an order of magnitude. In addition, Jouppi et al.
2

---

Under review as a conference paper at ICLR 2020
(2017) noted that not only the power consumption but also the speed of DNN accelerators is memory- rather than compute-bout bound. This is confirmed by Wang et al. (2019), who also demonstrated that increasing computation throughput without increasing memory bandwidths barely affects latency.
Network compression. Lossless coding and, in particular, variable length coding (VLC), is way to reduce the memory footprint without compromising performance. In particular, Han et al. (2016) proposed using Huffman coding to compress network weights, alongside quantization and pruning. Wijayanto al. (2019) proposed using the more computationally-demaddde (LZ77 + Huffman) to further improve compression rates. Chandra (2018) used Huffman coding to compress feature maps and thus reduce memory bandwidth. Gudovskiy al. (2018) proposed passing only the
algorithm DEFLATE
feature maps to the memory to reduce the bandwidth. Cavigelli Benini (2019) proposed using R..F-based algorithm to compress sparse network activations.
3 Transform-Domain Compression
In what follows, we briefly review the basics of lossy transform coding. For detailed overview of the subject, we refer the reader Goyal (2001). Let x (.1....,;,)) represent the values of the activations of a NN layer in a block of size n = W x H x C' spanning, respectively, the horizontal and the vertical dimensions and the feature channels. Prior to being sent to memory, the activations, are encoded by first undergoing an affine transform, y = Tx = T(x p)): the transform coefficients are quantized by a scalar quantizer, 2a. whose strength is controlled by the step size A, and subsequently coded by a lossless variable length coder (VLC). We refer to the length in bits of the resulting code, normalized by n as to the average rate, RA. To decode the activation vector, variable length decoder (VLD) is applied first, followed by the inverse quantizer and the inverse transform. The resulting decoded activation, & = T- 2 ②& (Tx)), typically differs from x; the discrepancy is quantified by distortion, Da. The functional relation between the rate and the distortion is controlled by the quantization strength, A, and is called rate-distortion curve.
Classical rate-distortion analysis in information theory assumes the MSE distortion, D) + x|2 While in our case the measure of distortion is the impact on the task-specific loss, we adopt the Euclidean distortion for two reasons: firstly, it is well-studied and leads to simple expressions for the quantizer; and, secondly, computing loss requires access to the training data, which are unavailable in the post-training regime. The derivation of an optimal rate is provided in Appendix D.
A crucial observation justifying transform coding is the fact that significant statistical dependence usually exists between the T; (Cogswell al., 2016). We model this fact by asserting that the activations are jointly Gaussian, ~ (H, Σ). with the covariance matrix Σ, whose diagonal elements are denoted by of. Statistical dependence corresponds to non-zero off-diagonal elements in E. The affinely transformed y T(x 14) is also Gaussian with the covariance matrix Σ' T ET. The distortion (D.2) is minimized over orthonormal matrices by T 1112 diagonalizing the covariance (Goyal, 2001). The latter is usually referred to as the Karhunen-Loeve transform (KLT) or principal component analysis (PCA). The corresponding minimum distortion is D*
me det (m2 2R, Since the covariance matrix symmetric, T is orthonormal, implying T 6 TT.
In Fig. 1, visualization of 2D vector quantization is shown. For correlated channels (Fig. 1a), many 2D quantization bins are not used since they contain no values. Linear transformation (Fig. 1b) provides improved quantization error for correlated channels by getting rid of those empty bins.
3.1 Implementation
In what follows, we describe an implementation of the transform coding scheme at the level of individual CNN layers. The convolutional layer depicted in Fig. 2 comprises a bank of convolutions (denoted by * in the Figure) followed by batch normalization (BN) that is computed on an incoming input stream. The output of BN is a 3D tensor that is subdivided into 3D blocks to which the transform coding is applied. Each such block is sent to an encoder, where it undergoes PCA, scalar quantization and VLC. The bit stream at the VLC output has lower rate than the raw input and is accumulated
3

---

Under review as a conference paper at ICLR 2020
2 bits
2 bits
Dit
bits
bits
2 bits
(b)
(c)
Figure 1: Vector quantization in 2D case. (a) pair of correlated channels scatter plot. All values in a cell are mapped to the center of the cell; hence, small cells induce small quantization noise. Several bins are empty (red cells): (b) Decorrelation improves utilization since the cells are smaller now; (c) Forcing equal bin size along all dimensions further improves utilization. Instead of restricting both channel dynamic range to be divided into same number of bins, we use uniform bin size along all dimensions. VLC allows to further compress the representation since the channels with smaller dynamic range have are mapped mostly to few most probable bins.
in the external memory. Once all the output of the layer has been stored in the memory, it can be streamed as the input to the following layer. For that purpose, the inverse process is performed by the decoder: VLD produces the quantized levels that are scaled back the transform domain, and an inverse PCA is applied to reconstruct each of the activation blocks. The layer non-linearity is then applied, and the activations used input the following layer. While the location of the nonlinearity could also precede the encoder, our experiments show that the described scheme performs better.
from layer F1
to layer 4-
*
BN
PCA
PCA
VLC
VLD
Memory
Figure 2: High-level flow diagram of the encoder-decoder chain. PCA and BN folded into the convolution weights (denoted by +), resulting in single convolution (boxed in grey).
Linear transform. We have explored different sizes blocks for the PCA transform and found 1 1 x1x C' be optimal (the ablation study is shown in Appendix B). Due the choice of 1 x C blocks, the PCA transform essentially becomes Appendix shows depicted in Fig. 2, the convolution bank of the layer, BN and PCA can be folded (Jacob et al., 2018) into a single operation, offering also advantage in the arithmetic complexity.
x tensor convolution kernel (Fig. 1 in the image). This allows further optimization: as
example of its application
The PCA matrix is pre-computed, as its computation requires the expensive eigendecomposition of small batch of (unlabeled) training
the covariance matrix. The covariance matrix estimated or test data and can be updated online. Estimation of the covariance matrix for all layers at once is problematic since quantizing activations in the 1-th layer alters the input to the 7 + 1-st layer, resulting in biased estimation of the covariance matrix in the 1 + 1-st layer. To avoid this, we calculate the covariance matrix layer by layer, gradually applying the quantization. The PCA matrix calculated after quantization of the weights is performed, and is itself quantized to 8 bits.
uniform quantization, where the dynamic range determined according to the channel with the highest variance. Since all channels have an equal quantization step, entropy of the low-variance channels is significantly reduced.
Quantization. For transformed feature maps
4

---

Under review as a conference paper at ICLR 2020
ResNet18
MobileNetV2
72.0
69.5
71.5
Accuracy (70)
68.0
71.0
68.5
68.0
70.0
69.5
Theorctical E Enerapy- 8 Weights
Theoretical Entropy d Weights
67.0
69.0
Huftman 5 be Weitts
Humman Weights
66.5
68.5
2.50
2.75
3,00
325
3.50
3.75
4,00
4.25
4.50
2.50
2.75
3.00
3.25
3.50
3.75
4,00
4,25
4.0
Avg bits per value
Avg bits per value
Figure 3: Rate-distortion curves for ResNet-18 18 and MobileNetV2 architectures with 8- (blue) and 4-bit (red) weight quantization. Distortion is evaluated in terms of top-1 accuracy ImageNet. Dashed lines represent rates obtained by Huffman VLC, while solid lines represent theoretical rates (entropy). More architectures can be found in Fig. A.1 the Appendix.
Variable length coding. The theoretical rate associated with discrete random variable, Y (the output of the quantizer), is given by its entropy H (X) -E X = Σ, p(z,) (((22,ooog log, p(x;). This quantity constitutes the lower bound on the amount of information required for lossless compression of Y. We use Huffman codes, which are a practical variable length coding method (Szpankowski, 2000), achieving the rates bounded by (X) R (X) + (see Fig. 3 for comparison of the theoretical rates to the ones attained by Huffman codes). Fig. F.1 in the Appendix shows an example of Huffman trees constructed directly on the activations and their PCA coefficients.
1 x and grouped convolutions. While for regular 3 x 3 convolutions the computational overhead small, there are two useful cases in which this is not true:
and grouped convolutions. For 1 x convolutions the overhead is higher: the transformation requires much computation as the convolution itself. Nevertheless, it can still be feasible in the case of energy-efficient computations. In the case of grouped convolutions, it impossible to fold the transformation inside the convolution. However, in the common case when the grouped convolution is followed by a regular one, we can change the order of operations: perform BN, activation and transformation before writing the memory. This way, the inverse transformation can be folded inside the following convolution.
4 Experimental Results
We evaluate the proposed framework on common CNN architectures that have achieved high perfor- mance the ImageNet benchmark. The inference contains stages: calibration stage, on which the linear transformation is learned based on a single batch of data, and the test stage.
Full model performance. We evaluted our method on different CNN architectures: ResNet-18 50, and 101 (He et al., 2016); MobileNetV2 (Sandler et al., 2018): and Inception (Szegedy et al., 2016). Specifically, MobileNetV2 is known be unfriendly to activation quantization (Sheng et al., 2018). Performance was evaluated on ImageNet dataset (Russakovsky et al., 2015) on which the networks were pre-trained. The proposed method was applied to the outputs of all convolutional layers, while the weights were quantized to either 4 or 8 bits (two distinct config configurations) using the method proposed by Banner et al. (2018). Rates are reported both in terms the entropy value and the average length of the feature maps compressed using Huffman VLC in Fig. and Fig. A. in the Appendix. We observed that higher compression is achieved for covariance matrices with fast decaying eigenvalues describing low-dimensional data. A full analysis can be found in Appendix G.

---

Under review as a conference paper at ICLR 2020
70.0
Without degracation
Memory Bandwicth Complexity
6.5
5% degradation
69.0
6.0
68.5
scaursy
68.0
67.5
Q 0-> VLC
PCA-2-Q
4.5
66.5
Baseline
4.0
0.75
0.80
0.85
0.90
0.85
1.00
Aug bits value
Computationnl Complexity Ratio
Figure 4: Ablation study of the proposed encoder ResNet-18. Left: rate-distortion curve with different encoder configurations. Theoretical rates are reported; top-1 accuracy is used as the distortion measure. Right: theoretical memory rate bits per value achieved for different levels of PCA truncation for baseline and 0.5% lower than baseline top-1 accuracy.
Table I:
Comparison with EBPC (Cavigelli & Benini, 2019). While EBPC does not affect perfor- mance of the network, our method allows better compression by exploiting rate-distortion tradeoff.
Architecture
Method
Activations
(avg. number of bits per value)
Accuracy
EBPC
3.33
73.3%
ResNet-34
Our method
3.9
72.9%
Our method
3.11
72.1%
EBPC
3.64
71.7%
MobileNetV2
Our method
3.8
71.6%
Our method
3.25
71.4%
Comparison other methods. We compare the proposed method with other post-training quan- tization methods: ACIQ (Banner al., 2018), GEMMLOWP (Jacob & Warden, 2017), and KLD (Migacz, 2017). Note that our method can be applied on top of any of them to further reduce the memory bandwidth. For each method, varied the bitwidth and chose the smallest top-1 accuracy within 0.1% from the baseline and measured the entropy of the activations. Our method reduces, in average, 36% the memory bandwidth relatively the best competing methods; the full comparison can be found in Table A.I in the Appendix.
that attained
As for other memory bandwidth reduction methods, our method shows better performance than Cavigelli & Benini (2019) at the expense of performance degradation (Table 1). The performance difference smaller in MobileNetV2, since mobile architectures tend be less sparse (Park et al., 2018), making RLE less efficient. While the method proposed by Gudovskiy et al. (2018) requires fine-tuning, our method, although introducing computational overhead, can be applied to any network without such limitations. In addition, similarly to (Gudovskiy et al., 2018) it is possible to compress only part of the layers in which the activation size most significant.
Ablation study. An ablation study was performed using ResNet- 18 study the effect of different ingredients of the proposed encoder-decoder chain. The following settings were compared: direct quantization of the activations; quantization of PCA coefficients; direct quantization followed by VLC; and full encoder chain comprising PCA, quantization, and VLC. The resulting rate-distortion curves are compared in Fig. (left). Our results confirm the previous result of Chandra (2018), suggesting that VLC applied to quantized activations can significantly reduce memory bandwidth. They further show that a combination with PCA makes the improvement dramatically bigger. In addition, analyze the effect of truncating the least significant principal components, which reduces the computational overhead of PCA. Fig. 4 (right) shows the tradeoff between the computational and memory complexities, with baseline accuracy and 0.5% below the baseline.

---

Under review as a conference paper at ICLR 2020
Accurscy (8
68-
67-
Q++Vi VIC
VLC
Raseline
150
Total memory power consumption ( mW)
Figure 5: Top-1 accuracy on ResNet18 ImageNet vs, power consumption of our hardware implemen- tation. Each point represents different quantization rate.
5 Hardware Implementation
In order verify the practical impact of the proposed approach of reducing feature map entropy to save total energy consumption, we implemented the basic building blocks of a pre-trained ResNet- 18, with weights and activation quantized ber ISX280LU3F50IVV. From logic utilization and memory energy consumption (exact numbers are shown in Table C.I the Appendix) of convolutional layers we conclude that our method add minor computational overhead in contrast to significant reduction in memory energy consumption. Fig. 5 shows total energy consumption for a single inference of ResNet-18 on ImageNet. In particular, our method is more efficient for higher accuracies, where the redundancy of features is inevitably higher. We further noticed that our approach reached real-time computational inference speed (over 40 fps). The reduction in memory bandwidth
bit, on Intel Stratix- 10 FPGA, part num-
be exploited by using cheaper, slower memory operating at lower clock speeds, which may further reduce its energy footprint. More details appears in the Appendix. Source files for hardware implementation found at reference implementation.
6 Conclusion
This paper presents proof-of-concept of energy optimization in NN inference hardware by lossy compression of activations prior to their offloading to the external memory. Our method uses transform-domait coding, exploiting the correlations between the activation values to improve their compressibility. reducing bandwidth by approximately 25% relative to VLC and approximately 40% relative an 8-bit baseline without accuracy degradation and by 60% relative an 8-bit baseline with less than 2% accuracy degradation. The computational overhead required for additional linear transformation is relatively small and the proposed method can be easily applied on top of any existing quantization method.

---

Under review as a conference paper at ICLR 2020
References
Anaam Ansari and Tokunbo Ogunfunmi. Selective data transfer from drams for cnns. In 2018 IEEE International Workshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2018. (cited on p. 2)
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid deployeet, 2018. (cited pp. 2, 5, 6, 12, and 14)
Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes, Alex M Bronstein, and Avi Mendelson. Uniq: Uniform noise injection for the quantization of neural networks. arXiv preprint arXiv:1804. 10969, 2018. (cited on p. 1)
Zhaowei Cai, Xiaodong Hc, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017. (cited on 14)
Lukas Cavigelli and Luca Benini. Extended bit-plane compression for convolutional neural network accelerators, In 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp. 279-283, IEEE, March 2019. doi: 10. (U99/AICA..009..8777 and 6)
(cited on pp. 3
Mahesh Chandra. Data bandwidth reduction in deep neural network socs using history buffer and huffman coding. In 2018 International Conference Computing, Power and Communication Technologies (GUCON), pp. 1-3. IEEE, 2018. (cited on pp. 3 and 6)
Jungwook Choi, Pierce I-Jen Chuang. Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint arXiv:1807.06900 30. URL https: htt///aaxvvv...///e orw./bi.orr//bb..... (cited on p. 2)
www//rg//ss...ooooo..esesseeeese abs////as//11..../// 1100/11000000 06964.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 06085, 2018b. (cited on pp. 2 and 12)
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference, 2019. (cited on p. 2)
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. BRLhttp: aaaiiiiiii...//.i/ wrg/obbgbo........oooo...ee aaa//b//// //1/11111. 06068.6....8 (cited on 3)
Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Mareulescu. Regularizing activation distribution for training binarized deep networks. arXiv preprint arXiv: 2019. URL https:
wwwp/ogaas.....o..o.... 904. 02823. (cited p.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. arXiv preprint arXiv: pea.i/955009999 2019.
(cited on p. 2)
Jiong Gong, Haihao Shen, Guoming Zhang, Xiaoli Liu, Shane Li. Ge Jin, Niharika Maheshwari, Evarist Fomenko. and Eden Segal. Highly efficient 8-bit low precision inference of convolutional neural networks with intelcaffe. Proceedings of the 1st on Reproducible Quality-FFhe -Ffficieet Systems Tournament Co-designing Pareto-efficient Deep Learning ReQuEST 18, 2018. doi: 10. 145/ 3229762.3229763. URL http: dx. www.ooorg...ii wttp///.///............... 2tt..........1//2222222222222221 322222997772222762232 (cited on p. 2)
Vivek K. Goyal. High-rate transform coding: how high is high, and does matter? In 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060), pp. 207-, June 2000. doi: 10. 00.1109/SST.20000866663
(cited on p. 14)
Vivek K. Goyal. Theoretical foundations of transform coding. IEEE Signal Processing Magazine, 18 (5)-9221, Sep. 2001. ISSN 1053-5888. doi: 10.1 109/79.952802. (cited pp. and 15)
8

---

Under review as a conference paper at ICLR 2020
Denis Gudovskiy, Alec Hodgkinson, and Luca Rigazio. Dan feature map compression using learned representation over gf(2). In The European Conference on Computer Vision (ECCV) Workshops, September 2018. (cited pp. and 6)
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1737-1746, Lille, France, 07-09 Jul 2015. PMLR. URL http: http:e//pccceeeeeeeeeeeee wt..spress/va v3 gupta15 html. (cited on p. 2)
Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016. (cited on pp. I and 3)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR). June 2016. URL httt:://opeaaccessse theevf. wwwe/ecccooo/oeeeeooooooeeeeeeee He Res
ceoon/ooneet_cvvv captt__cvp_20000 1000////0 6/html/
html. (cited on p. 5)
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-ianiv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 2018. (cited pp. 1, 2, and 12)
Benoit Jacob and Pete Warden. gemmlowp: a small self-contained low- oww-precision gemm library, 2017. (cited on pp. and 12)
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018, URL http: hpw./ccsscccccc.cpppppppp thecvf. com/ comm/ntntttt...mmmm ceatteet__ppr____e 2018/ / html/ /oal/JJaa //wacb__auuuu Lrsl__uantiiii iittiiiziion and Trainingg_vvel Bacping______________ 201P__20188 paper html. (cited on pp. 2, and 4)
Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-lue Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, Richard C. Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke. Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of tensor processing unit. 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pp. 1-12. IEEE, 2017. (cited on p. 2)
Jun Haeng Lee, Sangwon Ha, Saerom Choi, Won-Jo Lee, and Seungwon Lee. Quantization for rapid deployment of deep neural networks, 2018. (cited p. 2)
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization deep convolu-
tional networks. In International Conference Machine Learning, pp. 2849-2858, 2016. URL attps://cpenzees/
net. forum? /tt.:ooiiooooooa/aaaaaa j
(cited on p. 14)
Christos Louizos, Max Welling, and Diederik P. Kingmao. Learning sparse neural networks through 10 regularization. ICLR, 2018, (cited on p. 1)
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different- recovering neural network quantization error through weight factorization. arXiv preprint arXiv: 2019. (cited on p.

---

Under review as a conference paper at ICLR 2020
Szymon Migacz. 8-bit inference with tensont, 2017. URL http:///......aa.oooooooaaaaaaa. com/gtc/ httt://0.//prreettttttoo//////// ott.://e.......a.a.aaa..oooaaaoa pdf. (cited on pp. 2, 6, and 12)
Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. Wrpn: Wide reduced-
precision networks. International Conference on Learning Representations, 2018. URL net fitts:://ppereet http://paaraeeeeeeeeeeeoeooooaaa Forum?id=Bl2vaaeeeee
(cited
Raghid Morcel, Hazem Hajj. Mazen A. R. Saghir, Haitham Akkary, Hassan Artail, Rahul Khanna, and Anil Keshavamurthy, Feathernet: An accelerated convolutional neural network design for resource- constrained fpgas. ACM Trans. Reconfigurable Technol. Syst., 12(2):::16::27, March 2019. ISSN 1936-7406. doi: 10.1145/3306202. URL http: /doi. acm. org/ 10. 1.....1444. 1145/ / (cited on p.
Mi Sun Park, Xiaofan Xu, and Cormac Brick. Squantizer: Simultancous learning for both sparse and low-precision neural networks. arXiv preprint arXiv:1812.08301, 2018. (cited on p. 6)
Hanyu Peng and Shifeng Chen. Bdnn: Binary convolution neural networks for fast object de- tection. Pattern Recognition Letters, 2019. ISSN 0167-8655. doi: https///do.oor//0011001 1016/ ippatrec.2019.03.000 URL http: / /wwwww sciencedirect. wttp:/tccccccccccccececeeee htom//cienne/articleee pi1//016786511900000 (cited on p. 2)
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525-542. Springer, 2016. (cited p. 2)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-2522 2015, doi: 10. 123-0015---11
(cited on p. 5)
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and LiangCChiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2018. (cited on p. 5)
Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. quantization-friendly separable convolution for mobilenets, 2018. (cited on 5)
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015. (cited on p. 2)
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. (cited on p. 5)
Wojeiech Szpankowski. Asymptotic average redundancy of huffman (and shannon-fano) block codes. In 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060), pp. 370-, June 2000. doi: 10. 00.1109/SSTT:2000 866668. (cited on p. 5)
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.055 05787, 2018. (cited on 1)
Erwei Wang, James Davis, Peter YK Cheung, and George A Constantinides. Lutnet: Rethinking inference in fpga soft logic. arXiv preprint arXiv: 00938, 2019. (cited on p.
Arie Wahyu Wijayanto, Jun Jin Choong, Kaushalya Madhawa, and Tsuyoshi Murata. Towards robust compressed convolutional neural networks. In 2019 IEEE International Conference on Big Data and Smart Computing (BigComp). pp. 1-8. IEFE, 2019. (cited on p. 3)
10

---

Under review as a conference paper at ICLR 2020
Qingcheng Xiao, Yun Liang, Liqiang Lu, Shengen Yan, and Yu- Wing Tai. Exploring heterogeneous algorithms for accelerating deep convolutional neural networks on fpgas. In Proceedings of the 54th Anmual Design Automation Conference 2017, DAC '17, pp. 62:1-62:6, New York, NY, USA, 2017. ACM. ISBN 978-1 -4783---703-4 doi: 10.1 1114/3061669 639.3022244. URL http://doi acm. org 145 3061 639 3062244, (cited p.
Yu Xing, Shuang Liang, Lingzhi Sui, Xijie Jia, Jiantao Qiu, Xin Liu, Yushun Wang. Yu Wang, and Yi Shan. DNNVM : End-to-end compiler leveraging heterogeneous optimizations on fpga-based CNN accelerators. CoRR, abs/1902.07463, 2019. URL http::t://aaii..trtteteer 67.63. (cited on 2)
902.
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christo- pher De Sa. Swalp: Stochastic weight averaging in low-precision training. arXiv preprint urXiv:/904. 1943, 2019. (cited on 2)
Tien-Ju Yang. Yu-Hsin Chen, Joel Emer, and Vivienne Sze. A method estimate the energy consumption of deep neural networks. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, pp. 1916-1920. IEEE, 2017. (cited on pp. 1 and 2)
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference Computer Vision (ECCV), 2018. (cited on p. 2)

---

Under review as a conference paper at ICLR 2020
A
Additional experimental results
In Fig. A.1 we show Rate-distortion curves on ResNet50, ResNet101 and EnceptionV3 for weights quantized 4 and 8 bits. We show the theoretical entropy value and the real value using Huffman VLC. In Table A. we show the comparison of our method with other post-training quantization method. For fair comparison, we add to all compared methods a VLC and show the minimum amount of information that is required to be transferred to the memory.
Table A.1: Comparison of our method against three known post- post-training quantization methods (i) ACIQ (Banner et al., 2018); (ii) GEMMLOWP (Jacob Warden, 2017); (iii) KLD (Migacz, 2017); We report the smallest bit per value for which degradation is at most 0.1 % of the baseline.
Architecture
Weights (bits)
Method
Activations
(avg number of bits per value)
GEMMLOWP
6.88
ACIQ
6
8
KLD
6.3
Our method
4.15
ResNet-50
GEMMLOWP
6.93
ACIQ
6.2
KLD
6.52
Our method
4.25
8
GEMMLOWP
6.93
ACIQ
6.2
KLD
6.43
Our method
4.3
Inception V3
GEMMLOWP
6.97
ACIQ
6.3
KLD
6.35
Our method
4.6
GEMMLOWP
8.6
ACIQ
7.5
KLD
7.8
Our method
3.8
MobileNetV2
GEMMLOWP
8.8
ACIQ
7.85
KLD
8.1
Our method
4
Layerwise performance. In order to understand the effect of the layer depth on its activation compressibility, we applied the proposed -onsform-domain compression each layer individually. Fig. A.2 the following page reports the obtained rates on ResNet- 18 and 50. High coding gain is noticeable in the first layer, which traditionally more difficult compress, and is consequently quantized to higher precision (Hubara et al., 2018; Banner et al., 2018; Choi et al., 2018b).
A.1 CIFAR-10 results
B Block shape and size
Fig. B.1 shows the rate- rate-distortion curves for blocks of the same size allocated differently to each of the three dimensions: the distortion evaluated both in terms of the MSE and the network classification accuracy. The figure demonstrates that optimal performance for high accuracy is achieved with
12

---

Under review as a conference paper at ICLR 2020
Resliet50
ResWeti11
Accuracy (%)
2.50
2.55
3.00
325
3.50
3.75
4,00
425
4.50
2.50
2.75
3.00
3.25
350
3.75
4,00
4.25
4.0
Inceptionv3
Aug bits per value
77.5
Accuracy (8)
Theoretical Entropy: at Weights
75.0
Theoretical Entropy on We ghtss
Huftraan We
Weeghts.
250
2.55
3,00
3:25
3,50
375
4.50
4.25
450
bits value
Figure A.I: Rate-distortion curves for ResNet50, ResNet101 and Inception V3 architectures with 8- (blue) and 4-bit (red) weight quantization. Distortion is evaluated terms of top-1 accuracy on ImageNet. Dashed lines represent rates obtained by Huffman VLC, while solid lines represent the theoretical rates (entropy).
A00 bit per value
10
2
10
14
10
1.5
Layer indlox
Layer Indexx
Figure A.2: Theoretic average rates for individual layers in ResNet-50 (left) and Resnet-18 18 (right); lower values indicate better compression. Both configurations achieve the full precision baseline top-1 accuracy on ImageNet. The first layer is assigned the lowest bitwidth in both models where we observe high correlations between channels.
1x1x C = n blocks, suggesting that the correlation between the feature maps is higher than that between spatially adjacent activations. For lower accuracy, bigger blocks are even more efficient, but the overhead of times bigger block is too high. Experiments reported later in the paper set the block size values between 64 512 samples.
C Hardware implementation details
We have implemented ResNet-18 using Stratix-10 FPGA by Intel, part number ISX280L.U3F50I2VG. The memory used for energy calculation the Micron 4Gb x16 MT41J256M16. Current consumption of the DDR was taken from the data sheet for read and write operation, and was used calculate the energy required to transfer the feature maps in each layer. The script for calculating the energy consumption accompanies reference implementation.
In our design each convolutional layer of ResNet-18 implemented separately. In each layer we calculate pixel of 1 output feature each clock. For example, the second layer has 64 input and 64 output feature maps, thus takes 64 x (56 x 56) clock cycles to calculate the output before moving the next layer.
13

---

Under review as a conference paper at ICLR 2020
ResNet20 Cifar10
92.5
92.0
Acturary (M
90.5
90.0
Theoratical Entropy Weights
Theoretical Entropy 4 Weights
89.5
Huffman bit Weights
Huffman Weights
89.0
Bascline
4.5
Avg bits per value
Figure A.3: Rate-distortion curves for ResNet-20 with &- (blue) and 4-bit (red) weight quantization. Distortion is evaluated in terms of top-1 accuracy CIFAR-10. Dashed lines represent rates obtained by Huffman VLC, while solid lines represent the theoretical rates (entropy).
0.007
2*2%C
68
0.006
2-24C/4)
11
4-47(C/26)
66
111
Accurayy
64-
0.004
Basaline
2-24C
0.003
1-1CC
2'2M((4)
0.002
|11|//11
1-14C/4)
0.001
Baseline
Avg bits per valuee
Avg bits per value
Figure B.1: The influence of the block shape on the top-1 validation accuracy (left) and MSE (right) of ResNet-11 on ImageNet. Our experiments show that for the same size, most efficient shape is 1 x 1 C, taking advantage of the correlations across different feature maps at the same spatial location. Theoretical rates are shown.
We read the input features only once by caching the pixels and reusing them from internal memory, and only reloading the weights of the filters the current layer.
D Optimal rate derivation
Let assume Gaussian activations, for which there emperical evidence (Lin al., 2016; Cai et al., 2017; Banner et al., 2018), Iy E N ((,,, of), and let R& bits be allocated to each T₁. The optimal l distortion of I; achieved by uniform quantizer constrained by the rate R; can be approximated as D; = ne of?2-2R, 2R, The approximation is accurate until about R, x bit (Goyal, 2000). Unfortunately, B quantizer step A, thus, the latter is computed numerically. From our experiments, we derived the approximation A A 4.2184 2 2- accurate for R2 bits.
general closed-fome expression exits relating R, with the corresponding
Optimal rate allocation consists of minimizing the total distortion given a target average rate R for the entire activation block. Under the previous assumptions, this can be expressed as the constrained optimization problem
min
HE 022-2R
22
28, s.t. (R1 + .... + R.) = R,
(D.1)
Ri R.
6
14

---

Under review as a conference paper at ICLR 2020
Table C.I: Logic utilization and memory energy consumption layers various widths on Intel's Stratix FPGA. Clock frequency was fixed at 160MHz for each design. In LUTSs and DSP we present the % of total resources. In Power and Bandwidth we present the total number (% saving comparison regular quantization)
# channels Method
LUTs
DSPs
Energy (HJ
Bandwidth (Gbps)
Quantization
19K
960
225.93
1.28
64
Q+VLC
19K
960
173.44 (-23%)
0.96 (-25%)
Q+VLC+PCA
19.5K
(+5%)
1056(+10%)
100.6 (-44%)
0.68 (-46.8%)
Quantization
43K
2240
112.96
1.28
128
Q+VLC
43K
2240
148 (-17%)
1.04 (-18.7%)
Q+VLC+PCA
45K(+4.3%)
2366(+5.6%%
117 (-35%)
0.83 (-35.1%)
Quantization
91K
4800
56.5
1.28
256
Q+VLC
91K
4800
46.8 (-17.2%)
1.03 (-19.5%)
Q+VLC+PCA 93K(+4%)
5059(+5.4%)
103 (-42.8%)
0.73 (-42.9%)
Quantization
182K
9600
28.2
1.28
512
Q+VLC
182K
9600
23.9 (-15.6%)
1.05 (-17.9%)
Q+VLC+PCA
186K(+2%)
10051(+4.7%)
91 (-49.5%)
0.66 (-48.4%)
which admits the closed-form solution (Goyal, 2001), R = R + log, 0,
logg kee and R
whenever the latter expression is negative. The corresponding minimum distortion is given by
D* (of
o .......)).2221 2
(D.2)
E Projection into principal components
In Fig. E.1 we show an intuition of the effect of the projection into the principal components, in layers of ResNet18. This projection helps to concentrate most of the data in part channels, and then have the ability to write less data that layers.
F Huffman encoder as VLC
Huffman encoder is a known algorithm for lossless data compression. The ability of compress, i.c achieve the theoretical bound entropy, can seen the balance the huffman tree, means that if the huffman tree is more unbalanced we get better compression. In Fig. F.1 there is an example of huffman tree with and without projection on the principal components, after the projection the huffman tree is more unbalanced.
G Eigenvalues analysis
The eigenvalues of the covariance matrix is measure of the dispersal of the data. If high energy ratio, means the cumulative sum eigenvalues divided by the total sum, can be expressed with small part of the eigenvalues, the data is less dispersal and therefore more compressible. In figure G.1 analyze the covariance energy average ratio in all layers of different architectures. The interesting conclusion is that the ability of compression with the suggested algorithm is correlated with the covariance energy average ratio, means that for new architectures we can look only at the energy ratio of the activation to measure our ability of compression.
15

---

Under review as a conference paper at ICLR 2020
Channel1
Channel2
Channel4
Channel8
Channel16 Channel32 Channel64
PCA projection
Layer
153. 130.1788
303.1672
51611
50. 7651
28.6883
4.8872
6.0076
no projection
20.7648
45,475
60.0975
0.0
35,4096
28,0009
39 39.0129
PCA projection
Layer 2
11
15,4225
20.1357
no projection
23.2001
11.9332
24.2078
Figure E.1: Channels from layers one and two in ResNet18 with and without principal component projection. Using PCA we create new channels where each channel is based on all original channels, ordered by how well the new channels capture the data. Under each channel we can see the energy (in terms of L.2 norm) associated with that channel. Most energy is now concentrated within a few known channels (the first channels), enabling aggressive quantization for the rest the channels. This is unlike the case of no projection where energy is more spread out among the different channels without affording an easy way to prioritize between them. Finally, note that values of the last channels almost identical. For example, all values of channel 64 are mapped to single value which can benefit the entropy encoder by using a short codeword for and thus write less bits to memory for that channel.
16

---

Under review as a conference paper at ICLR 2020
1.0
0.16
0.80
0.0262
0.084
0.0096
0.0185
0.0178
0.0164
0.007
0.006
0.0075
0.0081
0.091
0.085
0.0083
0.0081
0.006
0.001
0.0000086
0.000s
0.000016
0.000008
0,0000066
O. 0.000008
0.000008
0.0000079
6. 0.0000011
0.000002
0.000008
0.0000009
0.000001
0.000001
0.0000004
0,0000005
0.0000006
0,0000004
0.0000003
0.0000001
0. 0.03000008
0.00000022
0.0000004
0.0000004
Figure F.1: Example of Huffman tree with (Right) and without (Left) projection on the principal components. The colored nodes represent the leaves of the tree. We can see that when we project on the principal component the tree is less balanced, means can be better compressed
0.9
0.8
0.7
Energy
0.6
ResNet18
ResNet50
0.5
ResNet101
MobileNetV2
InceptionV3
0.4
0.0
0.1
0.2
0.3
0.4
0.5
Eigen
Figure G.1: Analysis of the eigenvalues ratio that are needed to achieve energy ratio, means cumulative sum of the eigenvalues.
17


Reviews:
[{"review_id": "rkgopztCcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies an important question: how to reduce memory bandwidth requirement in neural network computation and hence reduce the energy footprint. It proposes to use lossy transform coding before sending network output to memory. My concern with the paper is two-fold:\n1) The major technique of transform-domain coding is borrowed from previous work (e.g., Goyal 2001), hence the novelty of the proposed method is in doubt.\n2) The implementation details are not clear. For example, I don't know whether the implementation in section 3.1 is based on CPU or FPGA, and how easily Section 3.1 will be implemented on ASIC. For the experimental results are reported in Section 4, we do not know how much memory and how much cache is used. Will the computation of PCA require a lot of on-device memory? \n\nMore detailed comments:\nSection 1, 2nd paragraph: GPUs are event more popular than FPGAs and ASICs. Can the proposed method be useful for GPU inference?\nSection 1, 3nd paragraph:  The last sentence says \"high interdependence between the feature maps and spatial locations of the compute activations\". However, it is not clear to me how the proposed method takes spatial location into account.\nSection 2: better to review previous work In lossy transform coding\nFigure 1: It seems to me Figure 1 is obvious. What is the novelty?\nSection 4: better to report the details of computing units and memory size. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylZwqW2KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A lossy transform coding approach was proposed to reduce the memory bandwidth of edge devices deploying CNNs. For this purpose, the proposed method compresses highly correlated feature maps using variable length coding. In the experimental analyses, the proposed method outperforms some of the previous work in terms of the compression ratio and accuracy for training ResNet-34 and MobileNetV2. \n\nThe proposed method and initial results are promising. However, the paper and the work should be improved for a clear acceptance:\n\n- Some parts of the method need to be explained more clearly:\n\n– In the statement “Due to the choice of 1 × 1 × C blocks, the PCA transform essentially becomes a 1 × 1 tensor convolution kernel”, what do you mean by “the PCA transform becomes a convolution kernel.”?\n\n- Could you please further explain how you compute PCA using batch data, how you update online and how you employ that in convolution weights together with BN? Please also explain the following in detail:\n\n(I) “To avoid this, we calculate the covariance matrix layer by layer, gradually applying the quantization.” What is the quantization method you applied, and how did you apply it gradually?\n\n(II) “The PCA matrix is calculated after quantization of the weights is performed, and is itself quantized to 8 bits.” How did you quantize the weights, how did you calculate PCA using quantized weights and how did you quantize them to 8 bits?\n\n- Could you please explain the following settings, more precisely: direct quantization of the activations; quantization of PCA coefficients; direct quantization followed by VLC; and full encoder chain comprising PCA, quantization, and VLC? Please note that there are various methods and algorithms which can be used for these quantization steps. Therefore, please explain your proposed or employed quantization methods more clearly and precisely.\n\n–  Please clarify the statement “This projection helps to concentrate most of the data in part of the channels, and then have the ability to write less data that layers.”.\n\n- Did you apply your methods to larger networks such as larger ResNets, VGG like architectures, Inception etc?\n\n- I also suggest you to perform experiments on different smaller and larger datasets, such as Cifar 10/100, face recognition datasets etc., to examine generalization of the proposed methods at least among different datasets.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJgWjNu0KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes to reduce the memory bandwidth (and energy consumption) in CNNs by applying PCA transforms on feature vectors at all spatial locations followed by uniform quantization and variable-length coding.\n\nI appreciate the writing quality: as an outsider to the field of low-power/low-precision deep learning, I found the write-up straightforward and easy to follow. It’s harder for me to precisely assess the significance of the proposed approach, but at a high level it looks reasonable and is backed by convincing empirical evidence.\n\nSmall comment: I don’t believe the submission is following the ICLR 2020 format strictly: the font looks different, and the margins look tighter.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed the use of a lossy transform coding approach to to reduce the memory bandwidth brought by the storage of intermediate activations. It has shown the proposed method can bring good memory usage while maintaining the the accuracy.
The main concern on this paper is the limited novelty. The lossy transform coding is borrowed from other domains and only the use of it on CNN intermediate activation is new, which seems insufficient. 
Decision: Reject