Beyond Quantization: Power aware neural networks

1 INTRODUCTION . With the ever increasing popularity of deep neural networks ( DNNs ) for tasks like face detection , voice recognition , and image enhancement , power consumption has become one of the major considerations in the design of DNNs for resource-limited end-devices . Over the last several years , a plethora of approaches have been introduced for achieving power efficiency in DNNs . These range from specialized architectures ( Sandler et al. , 2018 ; Huang et al. , 2019 ; Tan et al. , 2019 ; Radosavovic et al. , 2020 ) , to hardware oriented methods like multiplier-free designs and low-precision arithmetic . Multiplier aware methods attempt to reduce power consumption by avoiding the costly multiplication operations , which dominate the computations in a DNN . Several works replaced multiplications by additions ( Courbariaux et al. , 2015 ; Li et al. , 2016 ; Chen et al. , 2020 ) or by bit shift operations ( Elhoushi et al. , 2019 ) or both ( You et al. , 2020 ) . Others employed efficient matrix multiplication operators ( Tschannen et al. , 2018 ; Lavin & Gray , 2016 ) . However , most methods in this category introduce dedicated architectures , which require training the network from scratch . This poses a severe limitation , as different variants of the network need to be trained for different power constraints . Low-precision DNNs reduce power consumption by using low-precision arithmetic . This is done either via quantization-aware training ( QAT ) or with post-training quantization techniques . The latter avoid the need for retraining the network but often still require access to a small number of calibration samples in order to adapt the network ’ s weights . Such techniques include approaches like re-training , fine-tuning , calibration and optimization ( Banner et al. , 2019 ; Jacob et al. , 2018 ; Nahshan et al. , 2019 ; Li et al. , 2021 ) . All existing methods in this category suffer from a large drop in accuracy with respect to the full-precision version of the network , especially when working at very low bit widths . Moreover , similarly to the multiplier-free approaches , they do not provide a mechanism for traversing the power-accuracy trade-off without actually changing the hardware ( e.g. , replacing an 8-bit multiplier by a 4-bit one ) . In this work , we introduce a power-aware neural network ( PANN ) approach that allows to dramatically cut down the power consumption of any model . Our method can be applied at post-training to improve the power efficiency of a pre-trained model , or in a QAT setting to obtain even improved results . Our approach is based on careful analysis of the power consumed by additions and multiplications , as functions of several factors . We rely on bit toggling activity , which is known to be the main factor affecting dynamic power consumption , and support our theoretical analysis with accurate gate-level simulations on a 5nm process . Our first important observation is that a major portion of the power consumed by a DNN is due to the use of signed integers . We therefore present a simple method for converting any pre-trained model to use unsigned arithmetic . This conversion does not change the functionality of the model and , as can be seen in Fig . 1 , dramatically reduces power consumption on common hardware configurations . Our second observation is that the multiplier ’ s power consumption is dominated by the larger bit width among its two inputs . Therefore , although high accuracy can often be achieved with quite drastic quantization of only the weights , this common practice turns out to be ineffective in terms of power consumption . To be able to take advantage of drastic weight quantization , here we introduce a method that allows removing the multiplier altogether . Our approach can work in combination with any activation quantization method . We show theoretically and experimentally that this method is far advantageous over existing quantization methods at low power budgets , both at post-training and in QAT settings ( see Fig . 1 and Sec . 6 ) . Our method allows working under any power constraint by tuning the number of additions used to approximate each mutltiply-accumulate ( MAC ) operation . This is in contrast to regular quantization methods , which are limited to particular values . We can thus traverse the power-accuracy trade-off without changing the architecture ( e.g. , bit width of the multiplier ) , as required by existing methods . 2 RELATED WORK . Avoiding multiplications In fixed point ( integer ) representation , additions are typically much more power-efficient than multiplications ( Horowitz , 2014b ; a ) . Some works suggested to binarize or ternarize the weights to enable working with additions only ( Courbariaux et al. , 2015 ; Lin et al. , 2015 ; Li et al. , 2016 ) . However , this often severely impairs the network ’ s accuracy . Recent works suggested to replace multiplications by bit shifts ( Elhoushi et al. , 2019 ) or additions ( Chen et al. , 2020 ) or both ( You et al. , 2020 ) . Other methods reduce the number of multiplications by inducing Element toggles Multiplier inputs 0.5b+0.5b Multiplier ’ s internal units 0.5b2 Accumulator input 0.5B Accumulator sum & FF 0.5bacc+0.5bacc Table 1 : Average number of bit flips per signed MAC . The b-bit multiplier inputs are drawn uniformly from [ −2b−1 , 2b−1 ) and its bacc = 2b bit output is summed with the B-bit number in the FF . sparsity ( Venkatesh et al. , 2016 ; Mahmoud et al. , 2020 ) , decomposition into smaller intermediate products ( Kim et al. , 2016 ) , Winograd based convolutions ( Lavin & Gray , 2016 ) , or Strassen ’ s matrix multiplication algorithm ( Tschannen et al. , 2018 ) . Some of these methods require internal changes in the model , a dedicated backpropagation scheme , or other modifications to the training process . Quantization DNN quantization approaches include post-training quantization ( PTQ ) , which is applied to a pre-trained model , and quantization-aware training ( QAT ) , where the network ’ s weights are adapted to the quantization during training ( Gupta et al. , 2015 ; Louizos et al. , 2018 ; Achterhold et al. , 2018 ; Esser et al. , 2019 ) . PTQ methods are more flexible in that they do not require access to the training set . These methods show optimal results for 8-bit quantization , but tend to incur a large drop in accuracy at low bit widths . To battle this effect , some PTQ methods minimize the quantization errors of each layer individually by optimizing the parameters over a calibration set ( Nahshan et al. , 2019 ; Nagel et al. , 2020 ; Hubara et al. , 2020 ) . Others use nonuniform quantization ( Liu et al. , 2021 ; Fang et al. , 2020 ) . Effort is also invested in avoiding the need of any data sample for calibration ( Cai et al. , 2020 ; Shoukai et al. , 2020 ; Nagel et al. , 2019 ; Haroush et al. , 2020 ) . These methods , however , still show a significant drop in accuracy at the lower bit widths , while frequently requiring additional computational resources . Common to all quantization works is that they lack analysis of the power consumed by each arithmetic operation as a function of bit-width , and thus can not strive for optimal power-accuracy trade-offs . 3 POWER CONSUMPTION OF A CONVENTIONAL DNN . The total amount of power consumed by a logic circuit can be attributed to two main sources : a static power component and a dynamic one . The static power is due to a constant leakage current . It does not depend on the circuit ’ s activity and is typically the smaller component among the two . The dynamic power consumed by each node in the circuit is given by P = CV 2fα , where C is the node capacitance , V is the supply voltage , f is the operating frequency , and α is the switching activity factor ( the average number of bit flips per clock ) ( Nasser et al. , 2017 ) . Here we focus on dynamic power , which is a major contributor to the overall power consumption ( see Appendix A.1 and ( Karimi et al. , 2019 ; Kim et al. , 2020 ) ) . Also , this is the only factor affected by the DNN architecture . Most of the computation in a forward pass of a DNN can be attributed to MAC operations . As shown in Fig . 2 , MACs involve a multiplier that accepts two b-bit numbers and outputs a bacc-bit result ( bacc = 2b to account for the largest possible product ) , and an accumulator with a large bit width B to which the multiplier ’ s output is added repeatedly . To understand how much power each of these components consumes , we simulated them in Python . For the multiplier , we used the Booth-encoding architecture , which is considered efficient in terms of bit toggling ( Asif & Kong , 2015 ) . For the accumulator , we simulated a serial adder . Our Python simulation allows measuring the total number of bit flips in each MAC operation , including at the inputs , at the outputs , in the flip-flop ( FF ) register holding the previous sum , and within each of the internal components ( e.g. , the full-adders ) of the multiplier . We also verified our analysis with an accurate physical gate-level simulation on a 5nm process and found good agreement in terms of the dependence of power consumption on the bit widths ( see details in Appendix A.1 ) . Table 1 shows the average number of bit flips per MAC when both inputs to the multiplier are drawn uniformly at random from [ −2b−1 , 2b−1 ) ( Gaussian inputs lead to similar results ; please see Appendix Figs 7-8 ) . As can be seen , the power consumed by the multiplier is given by1 Pmult = 0.5b 2 + b , ( 1 ) where 0.5b2 is due to the bit toggling in the internal units , and 0.5b is contributed by the bit flips in each input . The power consumed by the accumulator is given by Pacc = 0.5B + 2b , ( 2 ) where 0.5B is due to the bit toggling in its input coming from the multiplier , 0.5bacc = b ( recall bacc = 2b ) to the bit flips at the output , and an additional 0.5bacc = b to the bit flips in the FF . These results lead us to our first important observation . Observation 1 . A dominant source of power consumption is the bit toggling at the input of the accumulator ( 0.5B ) . Suppose , for example , we use b = 4 bits for representing the weights and activations and employ a B = 32 bit accumulator , as common in modern architectures ( Kalamkar et al. , 2019 ; Rodriguez et al. , 2018 ) . Then the toggling at the input of the accumulator ( 0.5B = 16 ) is responsible for 44.4 % of the total power consumption ( Pmult + Pacc = 36 ) . At lower bit widths , this percentage is even larger . Unfortunately , existing quantization methods and multiplier-free designs do not battle this source of power consumption . Ni et al . ( 2021 ) have recently shown that the bit-width B of the accumulator can be somewhat reduced by explicitly accounting for overflows . However , this approach requires dedicated training , and degrades the network ’ s classification accuracy at low values of B . As we now show , it is possible to drastically reduce the bit toggles at the input of the accumulator at post-training without changing the model ’ s functionality ( thus retaining the same classification accuracy ) .

Summary: The authors observed that the power consumption is dominated by the bit toggling at the input of the accumulator and decreasing the bit-width of only the weights or only the activations has limited benefit to reduce the the power consumed by the multiplier. The paper proposed PANN, which uses tricks such as unsigned arithmetic in CNN and implement multiplications via additions to achieve a multiplier-free and power-aware neural network. Experiments on both post-training quantization and quantization-aware training showcased better performance compared to other works under the same number of bit-flips. However, the reviewer believes there are fundamental flaws that need to be thoroughly addressed before this paper is ready for ICLR.