Prune or quantize? Strategy for Pareto-optimally low-cost and accurate CNN

1 INTRODUCTION . Reducing execution cost of deep learning inference is one of the most active research topics for applying superhuman recognition in embedded IoT devices and robots . A typical approach for employing memory- and computation-efficient components is separable convolution , which is a combination of depth-wise and point-wise convolutions ( Iandola et al. , 2016 ; Zoph et al. , 2018 ; Zhang et al. , 2018 ; Howard et al. , 2017 ) , structured/unstructured pruning of connections and activations , and quantizing activation , weight , and their vectors ( Stock et al. , 2019 ; Jegou et al. , 2011 ; Gong et al. , 2014 ) . Among these , separable convolution and structured pruning are similar , in that separable convolution can be viewed as convolutions pruned in a handcrafted manner . From a pruning viewpoint , since the separable convolution structure results from applying aggressive pruning to normal convolution , the result is drastic reductions in memory and computational cost at the expense of greatly decreased accuracy ( Stock et al. , 2019 ) . On the other hand , structured pruning and quantization are seemingly orthogonal approaches that can be naturally combined ( Tung & Mori , 2018 ; Han et al. , 2016 ) . However , their interactions are still not well-studied . For instance , the use of a single-bit representation is being actively explored as an extreme quantization . Since a nonnegligible accuracy drop is inevitable in extreme quantization , some papers have proposed increasing the number of channels to compensate for the lack of expressivity ( Lin et al. , 2017 ) . In other words , a quantization approach can further reduce the number of bits by compromising the increase in number of channels , or the increase in number of computations . This indicates that , con- versely , reducing channels by pruning may limit capability for quantization . This discussion raises a controversial question : which is better , a fat model with smaller bit width or a slim model with larger bit width ? Answering this question requires a metric that fairly measures the effects of both pruning and quantization . One such metric in the literature is the inference speed when the model is executed on specific hardware . This metric is useful or even ideal when the target hardware is known in advance but strongly depends on features of the hardware architecture . Yang et al . ( 2018 ) searched for an optimal architecture using inference time as the optimization objective and found different optimal architectures depending on the target device . For example , if the hardware can not handle extremely low bit-widths ( 1 or 2 bits ) , instead treating them as 8-bit integers with upper bits filled with zeros , we can not exploit the reduction of bit width to improve inference speed . From a theoretical viewpoint , figuring out the extent to which we can reduce the computational complexity of deep neural networks is another important open question . The discussion so far urges us to develop a hardware-agnostic and theoretically reasonable metric for measuring computational costs of neural network architectures . In this paper , we propose the Frobenius norm of the effective value of weight parameters as one such metric . This metric is proportional to the total energy when the model is executed on ideal hardware , where energy consumption for a single multiply-accumulate ( MAC ) computation is proportional to the squared effective amplitude of the individual weight parameter used for the MAC computation . The basic idea of the metric is analogous to a highly efficient class-B amplifier circuit whose energy consumption is determined by the instant signal amplitude ( Sechi , 1976 ) . This metric successfully reflects the effects of both quantization and structured/unstructured pruning in accordance with intuition . Using the proposed metric , we empirically find that a slimmer model can achieve a far better Pareto frontier in a lower computational cost region than can a fatter model after quantization , while a fat model is advantageous for achieving higher accuracy in a larger computational cost region . Finally , we perform experiments under a post-training quantization scenario ( Banner et al. , 2018 ) on ImageNet dataset ( Deng et al. , 2009 ) to verify the validity of our claim , namely that prune-then-quantize is superior to quantize-only or prune-only for achieving a better Pareto frontier . Further , since this metric is relevant to the signal-to-noise ratio ( S/N ) , it is measurable during SGD training , in which the absolute value of weights and the random walk of weight parameters correspond to signal and noise , respectively . We observe that the dependencies of the metric on validation accuracy seem to be correlated between those during training and those applying quantization after training . From this observation , we point out some possibilities for which we could expect robustness of a model for quantization from information obtained during training , we could determine an optimal policy for quantization of that model , and we could develop a novel optimization or regularization scheme . The main contributions of this paper are as follows : • We define a hardware-agnostic metric for measuring the computational cost of pruned and quantized models . • We empirically find that models with fewer parameters achieve far better accuracy in a low computational cost region after quantization . • We show a potential quantitative relation between quantization noise and perturbation of weight parameters during SGD training . And as implications , we hope to exploit our findings for • thorough comparison of various neural network architectures using the proposed hardwareagnostic metric , • development of a method for extracting a quantization policy from information obtained during SGD training , and • development of a training algorithm or regularization scheme for producing robust models based on the relation between quantization noise and perturbation of weight parameters during SGD training . 2 EFFECTIVE SIGNAL NORM . We seek a metric that properly reflects the effects of both quantization and pruning . Conventionally , quantization effectiveness is evaluated according to the number of bits required to achieve a given accuracy , or the accuracy achieved by using certain bit numbers for specific network architectures ( Stock et al. , 2019 ) . We can not use this to compare efficiencies between different architecture models ( e.g. , MobileNet versus ResNet-18 ) . The number of MAC computations or parameters can be used to compare different architectures , but the number of MAC computations does not consider quantization and the number of parameters is not directly related to inference time . Recently , the use of actual or estimated inference speeds as a metric for comparing network architectures has been proposed ( Yang et al. , 2018 ; Wang et al. , 2019a ; Cai et al. , 2019 ) . This metric is very useful when the target hardware is known in advance , and ideal for those who wish to use the model that performs best on that hardware . However , this metric is strongly hardware dependent . Indeed , Yang et al . ( 2018 ) ; Wang et al . ( 2019a ) ; Cai et al . ( 2019 ) found that optimal architectures for different types of target hardware are totally different . Considering interest in , for example , the simplest realizable deep neural network model while achieving a required accuracy , there is a need for a hardware-agnostic metric . The metric for model evaluation should correlate with energy consumed when the model is executed on ideal hardware . We assume that energy consumption by ideal hardware monotonically decreases when the bit width is reduced by quantization and when the number of nonzeros in weight parameters is reduced by pruning . For example , hardware with an 8-bit integer MAC array can not be further accelerated even if the bit width is reduced from 8 to 1 or 2 bits . Thus , the energy consumption measured using such hardware does not satisfy the aforementioned requirement and can not be our metric . Hardware like a CPU , which processes each computation in serial , can naturally exploit the structured or unstructured sparsity of weight parameters by skipping computations with zeroed weights . However , because it is difficult to parallelize computations while maintaining such a strategy , it is generally difficult to benefit from sparsity in GPU-like hardware employing massively parallel MAC units . Hardware dedicated to sparse convolution ( Lu et al. , 2019 ) tends to show better performance only when sparsity is sufficiently high , due to relatively large overheads for encoding and decoding sparse weight parameters in a special format . Therefore , the benefit of sparsity from pruning and low bit width from quantization largely depends on the hardware architecture , so long as we consider only existing hardware . Because we require a hardware-agnostic metric , we assume ideal hardware in which energy consumption is linearly proportional to the number of nonzero weight parameters and monotonically depends on the bit width of weight parameters , as shown in Figure 1 , setting aside the feasibility of such ideal hardware . 2.1 DEFINITION OF EFFECTIVE SIGNAL NORM . We define a metric called the effective signal norm ( ESN ) as ESN = ∑ l ||clf ( Wlint ) ||2F , ( 1 ) with Wlint = ⌊W l/∆l⌋+0.5 , where Wl is the weight tensor and ∆l is the quantization step size of the lth layer ; and cl is a coefficient depending on the layer , in that if cl = 1 , ESN is related to the number of parameters ( cf . memory footprint ) , and if cl is the number of computations per parameter at the lth layer , ESN is related to the number of computations ( cf . FLOP ) . f ( · ) is an element-wise function that determines how the metric responds to the value of each weight parameter . We propose two functions for f ( · ) . The first is f ( Wlint ) = W l int , based on the assumption that energy consumption increases with the square of the value for each weight parameter or for each computation . When cl = 1 , the definition is ESNa = ∑ l ||Wlint||2F . ( 2 ) This assumption is reasonable when we employ an analog ( or in-memory ) MAC computation engine ( Shafiee et al. , 2016 ; Miyashita et al. , 2017 ) , because energy consumption is proportional to the square of the signal amplitude when the signal represents an analog quantity such as voltage or current . Assuming ideal hardware , we adopt a definition where energy consumption varies according to the instant amplitude ( cf . class-B amplifier ) , which is more energy efficient than the case where energy consumption is constant and the value is determined by the maximal amplitude ( cf . class-A amplifier ) ( Sechi , 1976 ) . The second proposed function is f ( Wlint ) = ⌈log2 ( abs ( W l int ) ) + 1⌉ , where log2 ( · ) and abs ( · ) are functions applied to each element of a tensor argument . This is based on the assumption that energy consumption increases with the binary logarithm of the value for each weight parameter . When cl = 1 , the definition is ESNd = ∑ l ||⌈log2 ( abs ( W l int ) ) + 1⌉||2F . ( 3 ) In a digital circuit , a number is represented as a binary digit ( bits ) , so the energy consumption for moving or processing signals is roughly proportional to the number of bits , which is the binary logarithm of the value . It is therefore reasonable to use Equation ( 3 ) for a digital circuit .

Summary: The paper proposes a new metric to evaluate both the amount of pruning and quantization. This metric is agnostic to the hardware architecture and is simply obtained by computing the Frobenius norm of some point-wise transformation of the quantized weights. They first show empirically that this Evaluation metric is correlated with the validation accuracy. Then use this metric to provide some general rules for pruning/quantizing to preserve the highest validation accuracy. Finally, they derive a strategy to perform pruning by monitory the signal to noise ratio during training and show experimentally that such method performs better than competing ones.