Beyond Quantization: Power aware neural networks
Authors: Nurit Spingarn
Elad Hoffer
Ron Banner
Hilla Ben Yaacov
Tomer Michaeli
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep neural networks
weight quantization
model compression
power-accuracy tradeoff
power consumption

Abstract: Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices. Existing approaches for reducing power consumption rely on quite general principles, including avoidance of multiplication operations and aggressive quantization of weights and activations. However, these methods do not take into account the precise power consumed by each module in the network, and are therefore far from optimal. In this paper we develop accurate power consumption models for all arithmetic operations in the DNN, under various working conditions. Surprisingly, we reveal several important factors that have been overlooked to date. Based on our analysis, we present PANN (power-aware neural network), a simple approach for approximating any full-precision network by a low-power fixed-precision variant. Our method can be applied to a pre-trained network, and can also be used during training to achieve improved performance. In contrast to previous approaches, our method incurs only a minor degradation in accuracy w.r.t. the full-precision version of the network, even when working at the power-budget of a 2-bit quantized variant. In addition, our scheme enables to seamlessly traverse the power-accuracy tradeoff at deployment time, which is a major advantage over existing quantization methods that are constrained to specific bit widths.

Under review as a conference paper at ICLR 2022
BEYOND QUANTIZATION: POWER AWARE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Power consumption is major obstacle in the deployment of deep neural networks (DNNss on end devices. Existing approaches for reducing power consumption rely quite general principles, including avoidance of multiplication operations and aggressive quantization of weights and activations. However, these methods do not take into account the precise power consumed by each module in the network, and are therefore far from optimal. In this paper we develop accurate power consumption models for all arithmetic operations in the DNN, under various working conditions. Surprisingly, we reveal several important factors that have been overlooked to date. Based network), simple approach for approximating any full-precision network by a low- power fixed-precision variant. Our method can be applied to a pre-trained network, and can also be used during training to achieve improved performance. In contrast to previous methods, PANN incurs only minor degradation accuracy w.r.t. the full-precision version of the network, even when working at the power-budget of a 2-bit quantized variant. In addition, our scheme enables to seamlessly traverse the power-accuracy trade-off at deployment time, which is major advantage over existing quantization methods that are constrained to specific bit widths.
analysis, we present PANN (power-aware neural
INTRODUCTION
With the ever increasing popularity of deep neural networks (DNNs) for tasks like face detection, voice recognition, and image enhancement, power consumption has become one of the major considerations in the design of DNNs for resource-limites end-devices. Over the last several years, a
plethora of approaches have been introduced for achieving power efficiency in DNNs. These range
from specialized architectures (Sandler et al. 2018 Huang et al. 2019 Tan et al. 2019; Radosavovic
al. 2020), to hardware oriented methods like multiplier-rree designs and low-precision arithmetic.
Multiplier aware methods attempt to reduce power consumption by avoiding the costly multiplication operations, which dominate the computations in DNN. Several works replaced multiplications by additions (Courbariaux et al.) (015 Li et al., [2016] Chen et al., (2020) or by bit shift operations
(Elhoushi et 2019) or both (You et al. 2020). Others employed efficient matrix multiplication operators (T'schannen et al., 2018) Lavin & Gray 2016). However, most methods in this category introduce dedicated architectures, which require training the network from scratch. This poses severe limitation, as different variants of the network need to be trained for different power constraints.
Low-precision DNNs reduce power consumption by using low-precision arithmetic. This is done either via quantization-aware training (QAT) or with post-training quantization techniques. The latter avoid the need for retraining the network but often still require access a small number of calibration samples in order to adapt the network's 's weights. Such techniques include approaches like re- re-training. fine-tuning, calibration and optimization (Banner et al., 2019, Jacob et al., 2018 Nahshan al.) 2019, Li et al.) 2021). All existing methods in this category suffer from large drop in accuracy with respect to the full-precision version of the network, especially when working at very low bit widths. Moreover, similarly to the multiplier-free approaches, they do not provide mechanism for traversing the power-accuracy trade-off without actually changing the hardware (e.g.. replacing an 8-bit multiplier by 4-bit one). In this work, we introduce power-aware neural network (PANN) approach that allows to dramatically cut down the power consumption of any model. Our method can be applied post-training improve the power efficiency of pre-trained model, or in QAT setting

---

Under review as a conference paper at ICLR 2022
80
80
PANN
PANN
PANN
PANN
70
70-
PANN
60
PANN
60-
Unsigned
BRECQ
Unsinel PANN
ZapQ
Unsignad
ZeroQ
Accuracy [%)
Unsgned
BRECQ
Accuracy
50
50-
PANN
BRBOQ
Uniggred
40
40-
Unsigned
Zero
BRECQ
Unsigmed
30-
Regheri--ad
Sourcnetet-V3
Squeczehet-Ve V1
Effic anttee B1 Googleltet
Uniigned
ZaroQ
Goog attet
15
15
0.0
0.2
0.4
0.6
0.8
1.0
14
0.0
0.2
0.6
Power [Bit f riss
Bit flips
(a) Improving upon ZeroQ (4 bits)
(b) Improving upon BRECQ (2 bits)
Figure I: Power-accuracy trade-off at post training. For each pre-trained full-precision model, used (a) ZeroQ (Cai al. 2020) and (b) BRECQ (Li et al. 2021] quantize the weights and activations at post-training. In (a) we quantize to 4 bits and in (b) to 2 bits. Then, we convert the quantized models work with unsigned arithmetic (+), which already cuts down 33% of the power consumption in (a) and 58% in (b) (assuming a 32 bit accumulator). Using our PANN approach to quantize the weights (at post-training! and remove the multiplier, further decreases power consumption and allows achieving higher accuracy for the same power level (1). See more examples in Appendix I
to obtain even improved results. Our approach is based on careful analysis of the power consumed by additions and multiplications, functions of several factors. We rely bit toggling activity, which is known to be the main factor affecting dynamic power consumption, and support our theoretical analysis with accurate gate-level simulations on 5nm process.
Our first important observation is that major portion of the power consumed by a DNN is due to the use of signed integers. We therefore present simple method for converting any pre-trained model to use unsigned arithmetic. This conversion does not change the functionality of the model and, as can be seen in Fig. D. dramatically reduces power consumption on common hardware configurations.
Our second observation is that the multiplier's power consumption is dominated by the larger bit width among its two inputs. Therefore, although high accuracy can often be achieved with quite drastic quantization of only the weights, this common practice turns out to be ineffective in terms of power consumption. To be able take advantage drastic weight quantization, here we introduce a method that allows removing the multiplier altogether. Our approach can work in combination with any activation quantization method. We show thcoretically and experimentally that this method is far advantageous existing quantization methods at low power budgets, both at post-training and in QAT settings (see Fig. [ and Sec. (6)
Our method allows working under any power constraint by tuning the number of additions used to approximate each aactiplly-cccumultee (MAC) operation. This in contrast to regular quantization methods, which are limited to particular values. We thus traverse the power-accuracy trade-off without changing the architecture (e.g., bit width of the multiplier), as required by existing methods.
2 RELATED WORK
Avoiding multiplications In fixed point (integer) representation, additions are typically much more power-efficient than multiplications (Horowitz) 201 a). Some works suggested to binarize or ternarize the weights to enable working with additions only (Courbariaux et al. 2015 Lin et al. 2015 Li et al. 2016). However, this often severely impairs the network's 's accuracy. Recent works suggested to replace multiplications by bit shifts (Elhoushi et al,, 2019) or additions (Chen et al. 2020) both (You al. 2020). Other methods reduce the number of multiplications by inducing
2

---

Under review as a conference paper at ICLR 2022
Multtpplie
Dace
Connerter
B
B
Hlement
Multiplier inputs
0.5b+0.5b
toggles
A ocumulator
Multiplier's 's internal units
0.56²
FF
Accumulator input
0.5
Accumulator sum & FF
0.5b
Figure 2: Multiply-ccummlate. The multiplier accepts two b-bit inputs. The product is then summed with the previous B bit sum, which awaits in the flip-flop (FF) register.
Table I: Average number of bit flips per signed MAC. The b-bit multiplier inputs are drawn uni- formly from [-20-1 22-1) and its bace 26 bit output is summed with the B-bit number in the FF,
sparsity /Venkatesh et al. 2016 Mahmoud et al. 2020). decomposition into smaller intermediate
products (Kim al. 2016). Winograd based convolutions (Lavin & Gray 2016), or Strassen's ''s matrix multiplication algorithm (1 (l'shhnnenn et al. 2018). Some of these methods require internal changes in the model, dedicated backpropagation scheme, or other modifications to the training process.
Quantization DNN quantization approaches include post-training quantization (PTQ), which is applied to a pre-trained model, and quantization-awar training (QAT), where the network's 's weights adapted the quantization during training (Gupta et al. 2013 Louizos et al. 2018 Achterhold et al., 2018:; Esser et al.) 2019). PTQ methods are more flexible in that they do not require access to the training set. These methods show optimal results for 8-bit quantization, but tend to incur large drop in accuracy at low bit widths. To battle this effect, some PTQ methods minimize the quantization errors of each layer individually by optimizing the parameters over a calibration set (Nahshan et al. 22199] Nagel ct al. 2020 Hubara et al. 2020). Others use nonuniform quantization (Liu et al. 202 Fang et al. 2020 Effort is also invested in avoiding the need of any data sample for calibration (Cai et al. 2020 Shoukai et al. 2020 Nagel et al., 2019 Haroush et al. 2020). These methods, however, still show a significant drop in accuracy at the lower bit widths, while frequently requiring additional computational resources. Common to all quantization works is that they lack analysis of the power consumed by each arithmetic operation as function of bit-width, and thus cannot strive for optimal power-accuracy trade-offs.
POWER CONSUMPTION OF CONVENTIONAL DNN
The total amount of power consumed by logic circuit can be attributed to two main sources: static power component and a dynamic one. The static power is due to a constant leakage current. does not depend the circuit's activity and is typically the smaller component among the two. The dynamic power consumed by each node in the circuit is given by P = 2V22 fa, where is the node capacitance, V is the supply voltage, f is the operating frequency. and a is the switching activity factor (the average number of bit flips per clock) (Nasser et al., 2017). Here we focus on dynamic power, which is major contributor to the overall power consumption (see Appendix A. and (Karimi et al. 2019 Kim et al. 2020)). Also, this is the only factor affected by the DNN architecture
Most of the computation in forward pass of a DNN can be attributed to MAC operations. As shown in Fig. 2. MACs involve multiplier that accepts two b-bit numbers and outputs a bace-bit result (bace 2b to account for the largest possible product), and an accumulator with large bit width B to which the multiplier's output is added repeatedly.
To understand how much power each of these components consumes, we simulated them in Python. For the multiplier, we used the Booth-encoding architecture, which is considered efficient in terms of bit toggling (Asif Kong 2015). For the accumulator, we simulated a serial adder. Our Python simulation allows measuring the total number of bit flips in each MAC operation, including at the inputs, at the outputs, in the flip-flop (FF) register holding the previous sum, and within each of the internal components (e.g., the full-adders) of the multiplier. We also verified our analysis with an accurate physical gate-level simulation on a 5nm process and found good agreement in terms of the dependence of power consumption the bit widths (see details in Appendix A.I
Table T shows the average number of bit flips per MAC when both inputs to the multiplier are drawn uniformly at random from 2-2 1, 1-2b- (Gaussian inputs lead to similar results; please
3

---

Under review as a conference paper at ICLR 2022
Appendix Figs 7 8 As can be seen, the power consumed by the multiplier is given by"
Pmult 0.5b² + b,
(1)
where 0.5b² is due the bit toggling in the internal units, and 0.5b is contributed by the bit flips in each input. The power consumed by the accumulator is given by
Pace = 0.5 + 2/,
where 0.5B is due to the bit toggling in its input coming from the multiplier, 0.5 = b (recall Dace 26) the bit flips at the output, and an additional 0.5b to the bit flips in the FF. These results lead us to our first important observation.
Observation 1. dominant source of power consumption is the bit toggling at the input of the accumalator (0.5B).
Suppose, for example, we use b bits for representing the weights and activations and employ a B = 32 bit accumulator, as common in modern architectures (Kalamkar et al. 2019 Rodriguez et al. 2018). Then the toggling at the input of the accumulator (0.5 = 16) responsible for 44.4% the total power consumption (Pmutt Pace = 36). At lower bit widths, this percentage is even larger.
Unfortunately, existing quantization methods and multiplier-free designs do not battle this source of power consumption, Ni et al. (2021) have recently shown that the bit-width B of the accumulator can be somewhat reduced by explicitly accounting for overflows. However, this approach requires dedicated training, and degrades the network's classification accuracy at low values of B. As now show, it is possible to drastically reduce the bit toggles at the input of the accumulator at post-traue training without changing the model's functionality (thus retaining the same classification accuracy).
4 SWITCHING TO UNSIGNED ARITHMETIC
Since the output of the multiplier has only 26 bits, one could expect to experience no more than b bit flips on average at the accumulator's input. Why do we have 0.5/ bit flips instead? The reason is rooted in the use of signed arithmetic. Specifically, negative numbers are represented using two's complement, and thus switching between positive and negative numbers results in flipping of many of the higher bits. For example, when using 32 bit accumulator, if the output of the multiplier switches from +2 to 22, then the bits at the input of the accumulator switch from 00000000000000000000000000000000
Note that this effect dominant only at the accumulator's ' input simply because sign changes at the output are rare.
to 11111111111111111111111111111111
If we could work with unsigned integers, then the higher bits the accumulator's input would always remain zero, which would lead to a substantial reduction in power consumption without any performance degradation. To quantify this, repeated the experiment of Sec. 3 but with the b-bit inputs to the multiplier now drawn uniformly from [0, 20-1) (see Appendix A.2 for details). In this case, the average number of bit flips the input of the accumulator reduced from 0.5 to 0.50 b. Specifically, the average power consumption of an unsigned MAC operation was measured to be
pu mukt 0.5b²
(3)
due to the multiplier and
pu 3b
(4)
due to the accumulator: In 44) 26 bit flips occur at the accumulator's output and the FF, and bit flips
occur at the accumulator's input coming from the multiplier. Thus, although the mutliplier's 's power (3)
turns out to be the same as in the signed setting ( the accumulator's 's power is substantially reduced w.r.t. the signed case (2).
Converting pre-trained network with ReLU activation functions to work with unsigned integers is simple. Specifically, consider a layer performing y = Wa The elements of x are non-negative because of the preceding R Therefore, we can split any such layer into two parallel layers
y W+ + b+,
y
W 6
1 'The amount of power consumed by single bit flip may vary across platforms (e.g., hetween 5nm and 45nm fabrication), but the number of flips per MAC does not change. We therefore report power units of bit-flips, which allows comparing between implementations while disregarding the platform. Batch-norm layers should first he absorbed into the weights and biases.
4

---

Under review as a conference paper at ICLR 2022
where W+ ReLLU(W), bt = (b) W Re), b- = ReLUU-bb,, and compute
y = y
(6)
This way, all MACs are converted to unsigned ones
and only a single subtraction per output element is needed in (6) This one subtraction is negligible w.r.t. the MACs, whose number is usually in the thousands. Please see Fig. [Tbbinn in the Appendix for a schematic illustration.
Figure T shows the effect that this approach has the power consumption of several pretrained networks for ImageNet classification. Figures [(a)aand[[ [ bit and bit quantized networks, respectively. With 32 bit accumulator, merely switching to unsigned arithmetic cuts 33% and 58% of the power consumption of these networks (see Appendix A.3.1 for other accumulator bit widths).
REMOVING THE MULTIPLIER
Having reduced the power consumed by the accumulator, we now turn to treat the multiplier. Quanti- zation methods often use different bit widths for the weights and activations. This flexibility allows achieving good classification accuracy with quite aggressive quantization of one of them (typically the weights), but a finer quantization of the other. An interesting question is whether this approach is beneficial in terms of power consumption.
We repeated the experiment of Sec. 3 this time with the multiplier inputs having different bit widths, by and b... We focused on the standard setting of signed numbers, which we drew uniformly from [-2 21 and [-2* 22= 2 Interestingly, we found that the average number of bit flips in the multiplier's s internal units affected only by the larger among bw and by- Accounting also for the bit flips at the inputs, we obtained that the multiplier's 's total average power is
Pmuut 0.5 (max(bw: )²² + 0.5(b.. + bx).
We found this surprising behavior to be characteristic of both the Booth multiplier and the simple serial multiplier, and verified it also with accurate simulations on a 5nm silicon process gate level synthesis (see Appendix Figs. 9 10). This leads us to our second important observation.
Observation There marginal benefit in the common practice of decreasing the bit width of only the weights or only the activations, at least in terms of the power consumed by the multiplier.
should be noted that in the case of unsigned numbers, where inputs are drawn uniformly from [0, 2 and [0, 12.-1), there exists some power save when reducing one of the bit widths, es- pecially for the serial multiplier (see Appendix Fig. TOP. This highlights again the importance of unsigned arithmetic. However, in all our experiments we do not take this extra benefit of our approach into account when computing power consumption, so that our reports are conservative.
To benefit from the ability to achieve high precision with drastic quantization of only the weights, we now explore a solution that allows removing the multiplier altogether. Unlike other multiplier-free designs, our method allows converting any full-precision pre-trained model into a low-precision power-efficient one without any changes to the architecture.
5.1 POWER AWARE WEIGHT QUANTIZATION
Consider the computation
D W/ 3i:
(8)
which involves MACs. Here, {w,, 4.} are the weights and activations of a convolution or a fully- connected layer. Given {w;, in full precision, our goal is to accurately approximate (8) in power-efficient manner. When quantizing the weights and activations we obtain the approximation
d
Too (wx 7/2 Qx:);)
where the quantizers Q (.) and (.) map R to Z, and Yoe and Yn are their quantization steps To
make the computation 9 power efficient, we propose to implement multiplications via addition..
In quantized models MAC operations always performed integers and rescaling applied the end.
5

---

Under review as a conference paper at ICLR 2022
7-bit
10:
6-bit
8-bit
5-bit
4-bit
3-bit
MSEauo MSE
10a
5
cr
10"
3-
10-
Uniform, Simulation
Uniform, Theary
1 Power of 2-bit
Gaussian, Simulation
width unsigned MAC
10
Rati-1
7
9 10 11
10 11
Activation bit-width
Bit-width of RUQ
(a) Number of additions vs. bit width in PANN
(b) The ratio between MSERUO and MSEPANN
Figure 3: (a) Each color represents the power of an unsigned bbi MAC for some value of b In PANN, we can
constant power curve by modifying the number of additions per element R (vertical axis) on the expense of the activation bit width by (horizontal axis). (b) We plot the ratio between the quantization errors of regular quantizer (RUQ) and a PANN tuned to work at the same power budget. can be seen, PANN outperforms RUQ at the low bit widths (where the MSE ratio is above 1). It should be noted that at the high bit widths, both approaches achieve low errors in absolute terms, but RUQ is relatively better. the Gaussian setting, which is closer the distribution of DNN weights and activations, the range over which PANN outperforms RUQ is a bit larger.
Specifically, assume (1x) is non-negative integer (as in Sec. term (w,) (2,) as
Then we can implement the
(w:) (xr) = (xx) +... (x:),
(10)
(w:) times
so that p is computed as
d Q (wr)
y R3 Yw
@x(x:).
(1I)
This is the basis for our power-aware neural network (PANN) design.
Let w (wi...., wa)T and x (Z1)...,za)) denote the full precision weights and ac- tivations, and denote their quantized versions by w, = (Q.. Qu and æ, (2,, (xx))), respectively. Note that opposed to conventional quantization meth- ods, our approach does not necessitate that the quantized weights confined to any particular range of the form [0, 2dx ). Indeed, what controls our approximation accuracy is not the largest possible entry Wg. but rather the number of additions per input element, which is given by || ll Therefore, given a budget of R additions per input element, we propose to use quantization step of Tw = || 1/ (Rd) in so that
(ww:) = round(w:/<<))
(12)
This quantization ensures that the number of additions per input element is indeed close as possible to the prescribed R. We remark that although we assumed unsigned weights, this quantization procedure can also be used for signed weights (after quantization, the positive and negative weights can be treated separately in order to save power, in Sec.
5.2 POWER CONSUMPTION
We emphasize that in PANN, we would not necessarily want to use the same bit width for the activations as in regular quantization. We therefore denote the activation bit width in PANN by b to distuingish it from the b, bits would use with the regular quantizer. To estimate the power consumed by approach, note that have approximately ||//||] additions of b. bit numbers. On

---

Under review as a conference paper at ICLR 2022
average, each such addition leads 0.5b- bit flips at the accumulator's output and 0.5bg bit flips in the FF register (see Table The imput to the accumulator, however, remains fixed for Q (w:) times when approximating the 7th MAC and therefore changes total of only times throughout the entire computation in [1 each time with 0.5b, bit flips average. Thus, overall, the average power per element consumed by PANN is
| + 0.55ba (R + 0.5)bm.
PAANN
(13)
d
This implies that to comply with prescribed power budget, we can either increase the activation bit width by on the expense of the number of additions R, or vice versa.
Figure 3(a) depicts the combinations of by and R that lead to the same power consumption as that of a b2 bit unsigned MAC, Pa MMc 0.56", + 4b, (see (3).(4)) ), for several values of by (different colors). When we traverse such an equal-power curve, we also change the quantization error. Thus, question whether there exist points along each curve, which lead to lower errors than those obtained with regular quantization at the bit-width corresponding to that curve.
5.3
QUANTIZATION ERROR
Let us compute the quantization error incurred by PANN and compare it to that of a regular uniform quantizer (RUQ). Obviously, the error incurred by the approximation (9) is contributed by both the quantization of the weights and the quantization of the activations. To see how their errors interact, let assume that and æ are statistically independent random vectors, each with iid components. In this case, if the quantization errors Ém = w You [( and E. = x - 7/ ((x)) satisfy E(e,. [ = 0
and [E{x, |ac] 0, then we show (see Appendix A.9 that the mean squared error (MSE) between the full precision operation (8) and its quantized version ( is given by
MSE (w x x- wTx,) = d (o + +0
(14)
Here oi 02 o2 and of denote the second-order moments of the elements of w, x, Ew. and Ex, respectively, and the expression on the right results from neglecting second-order terms.
Consider simplistic scenario where the activations are uniformly distributed in [0, Ma] (recall that activations are non-negative due to the preceding ReLU) and the weights uniformly distributed in [ .MM If we use RUQ with by bits to quantize the activations and a RUQ with by bits to quantize the weights, then we have that
M2
M2
M
M
(15)
22b
12
12 22b
This is because the quantization errors are uniformly distributed as U1-2 (b. (1, 1),2 (b. (bo-1)
and En U11-2-(bw-)) 11b 11) Substituting 15) into 14). we obtain that the MSE of a RUQ is
dM3 M
MSERUQ
(2 2h + 44.2 2hww)
(16)
12a
In PANN, we have that (Ew) ww Ul-
||lس|
|
J. so that E[(E [w]
!wll 12Ra
Therefore,
2Rd
2R
[0[[|00 [[
d2 (0.25
M
02_ 12 Rd)²
(17)
12(Rd)² 192
where we used the fact that wol using PANN together with a by bit RUQ for the activations, we achieve
U0, M.5 Substituting this expression in 14 we find that
MSEPANN dM3 M2 M
2
2b_
(18)
122
4R2
To compare between PANN and RUQ, we need to fix a power budget P. Given such budget, [3 dictates that the number of additions in PANN should be set to R = P/b, 0.5. Substituting this into 19), we obtain that
dM3 M2
62
MSEPANN
2 26
(19)
122
(2P b
7

---

Under review as a conference paper at ICLR 2022
The optimal bit- for the activations can therefore be found numerically by minimizing MSEPANN over by E Et. This typically requires evaluating (19) for small number of candidate bit widths, e.g., by f2. 8}. See Appendix A.8 for thorough analysis.
Figure 3(b) shows the ratio between MSERUQ and MSEANNN (with the optimal by)) function of the bit width of the RUQ, where PANN is tuned to the same power. For the RUQ, we use b2 = ba as its power consumption 7 is anyway dominated by the larger of them. can be seen that for low bit widths, PANN has a significant advantage over RUQ (ratio larger than 1). In the Gaussian setting, which is closer to the distribution of DNN weights and activations, the range over which PANN outperforms RUQ is even larger. As we show in Appendix A.8 this behavior is very similar to that observed in deep networks for image classification.
We emphasize that (99 is valid for uniformly distributed weights and activations, which is often not an accurate enough assumption for DNNs. Thus, in practice the best way to determine the optimal bit width b, is by running the quantized network on a validation set, as summarized in Algorithm
Algorithm I: Determining the optimal parameters for PANN
1: Input: Power budget P
2: Output: Optimal bz. R
3: for each b, [bmin |
do
4: Set R P/bB- 0.5 (Eq. 13
5: Quantize the weights using Eq. 12 with The = (RRd)
6: Quantize the activations to b bits using any quantization method
7: Run the network on a validation set, with multiplications replaced by additions using Eq. (O0
8: Save the accuracy to (bc()
9: end for
10: set by < arg max;, Acc(dx), R R 0.5
6 EXPERIMENTS
We now examine PANN in DNN classification experiments. We start by examining its performance at post training, and then move on to employ it during training. Here we focus only on the effect of removing the multiplier (vertical arrows in Fig. [.. Namely, we assume all models have already been converted to unsigned arithmetic (recall this by itself reduces a lot of the power consumption).
PANN at post training We illustrate PANN's performance in conjunction with a variety of post training quantization methods, including the data free approaches GDFQ (Shoukai al. (0200) and ZeroQ (Cai et al. (2020), the small calibration set method ACIQ (Banner et al. 2211)). and the optimization based approach BRECQ et al. 202I). which is currently the state-of of the for post training quantization at low bit widths. Table 2 reports results with ResNet-50 on ImageNet (See Appendix A. for results with other models). For the baseline methods, we always use equal bit widths for the weights and activations. Each row also shows our PANN variant, which works at the precise same power budget, where we choose the optimal b, and R using Alg. As can be seen, PANN exhibits only a minor degradation w.r.t. the full-precision model, even when working at the power budget of 2 bit networks. This is while all existing methods completely fail in this regime.
employ a straight- through estimator for backpropagation through the quantizers. Table 3 compares our method to LSQ (Esser et al., 2019), which is ofate-of-the-arr QAT approach. where in PANN we use LSQ for quantizing the activations. As can be seen, PANN outperforms LSQ for various models and power budgets. Table we compare our method to the multiplication- ffree approaches AdderNet (Chen) et al. (020) and ShiftAddNet (You et al.) 2020), which are also training- wraiiin-bbased techniques. For each method, we report the addition factor, which is the ratio between its number of additions per layer and regular layer. For example, AdderNet uses multiplications but twice as many additions, so that its addition factor is 2. ShiftAddNet, on the other hand, uses one addition and one shift operation. According to You al. 22200), shift operation costs between 0.2 (on FPGA) and 0.8 (on 45nm ASIC) an addition operation. Therefore ShiftAddNet's addition factor is between 1.2 and 1.8, and for simplicity we regard it as 1.5. In PANN, can choose any addition factor R, and therefore examine
PANN for quantization aware training To use PANN during training,
8

---

Under review as a conference paper at ICLR 2022
POWER (BITS)
ACIQ
ZEROQ
GDFQ
BRECQ
MEM.
LATENCY
BASE
OUR
BASE.
OUR
BASE.
OUR
BASE.
OUR
265 (8)
1x
7.5x
76.02
76.10
75.90
75.77
76.17
76.05
76.10
76.05
217 (6)
1.3x
4.7x
75.41
76.05
73.57
74.65
76.05
76.02
75.86
76.01
134 (5)
2x
3.5x
74.02
75.50
58.62
74.32
71.40
75.96
75.75
75.96
99 (4)
2.3x
2.9x
66.12
75.10
3.53
68.24
50.81
75.20
75.42
75.80
68 (3)
2x
2.2x
7.73
74.16
1.51
68.12
0.24
74.85
68.12
74.62
41 (2)
3x
1.1X
0.20
71.55
0.10
62.96
0.13
74.32
18.80
73.21
Table 2: PTQ: Classification accuracy [%]] of ResNet-50 on ImageNet (FP 76.11 %). The base-
lines (Base.) equal bit widths for weights and activations. This bit width determines the power P. reported in first column in units of Giga bit-flips. The power is calculated as P + Pace (Eqs. (3).44) times the number of MACs in the network. In each row, our variant PANN is tuned work the same power budget, for which we choose the optimal by and using Alg.
Bits (Power), Net
LSQ
PANN
Method
6/6
5/S
4/4
3/3
18 (2), ResNet-18
67.32
70.83
Our (1x)
91.15
91.05
89.93
85.62
30 (3), ResNet-18
69.81
71.12
Our (1.5x)
91.52
91.50
90.05
86.12
41 (2), ResNet-50
71.36
76.65
Our (2x)
91.63
91.61
90.10
86.84
68 (3), ResNet-50
73.54
76.78
ShiftAddNet (1.5x)
87.72
87.61
86.76
85.10
155 (2), VGG-16bn
71.15
73.30
AdderNet (2x)
67.39
65.53
64.31
63.50
Table 3: QAT: Comparison with LSQ. Imagenet classification accuracy [%] of various models. We report the bit width of LSQ and power in Giga bit-flips.
Table 4: QAT: Comparison with multiplier-free meth- ods. Classification accuracy [%] of ResNet-20 on CIFAR- 10. The top row specifies weight/activation bit widths, and the addition factor is specified in parentheses.
our method for R = 1, 2. We can see in the table that PANN outperforms both AdderNet and ShiftAddNet for all bit widths, even when using a smaller addition factor. Please see more QAT comparisons in Appendix A.5.2
Runtime memory footprint and latency of PANN We now analyze the effect of PANN other inference aspects besides power. One important aspect runtime memory footprint. When working with batches of image, the runtime memory consumption is dominated by the activations (Mishra et al. 2017) (see discussion on the memory footprint of the weights Appendix (.7). The optimal number of bits by we use for the activations is typically larger than the bit width b, used in regular quantization. The second column of Table 2 reports the factor b bb, by which the runtime memory of PANN exceeds that of the baseline model. As can be seen, this factor never exceeds 3. In the comparisons with the multiplier-free methods (Table we keep the same bit width for the activations and therefore there is change in the memory footprint. A second important factor is latency. Recall we remove the multiplier and remain only with the accumulator. Since addition is faster than multiplication, one could potentially use higher clock-rate and thus gain speed. However, if conservatively assume the original clock-rate, then the latency is increased by R (each multiplication replaced by R additions). As can be seen Table [] the increase latency is quite small the lower power budgets. For the multiplier-fres methods, we obtain improvement in accuracy even for R 1. In that case, our latency is smaller than that AdderNet (2x) and ShiftAddNet (1.5x). Please refer to Appendix A.7 for more analyses.
CONCLUSION
We presented an approach for reducing the power consumption of DNNs. Our technique relies on detailed analysis of the power consumption of each arithmetic module in the network, and makes use of two key principles: switching to unsigned arithemtic, and employing new weight quantization method that allows removing the multiplier. Our method substantially improves over existing approaches, both at post- post-training and when used during training, and allows achieving a significantly higher accuracy any given power consumption budget.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jan Achterhold, Jan Mathias Koehler, Anke Schmeink, and Tim Genewein. Variational network quantization. In International Conference on Learning Representations, 2018
Shahzad Asif and Yinan Kong. Performance analysis of wallace and radix-4 booth- multipliers. In 2015 Electronic System Level Synthesis Conference (ESLsyn), pp. 17-22. IEEE, 2015.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional In Advances in Neural Information Processing Systems, pp-
networks for rapid-deploymen 7950-7958, 2019.
Oresti Banos, Rafael Garcia, Juan A Holgado-Terriza, Miguel Damas, Hector Pomares, Ignacio Rojas, Alejandro Saez, and Claudia Villalonga. mhealthdroid: a novel framework for agile development of mobile health applications. In International workshop on ambient assisted living, pp. 91-98. Springer, 2014.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13169-13178, 2020.
Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1468-1477, 2020.
Matthieu Courbariaux, Yoshua Bengio, and -ierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. arXiv preprint 15 1.1100333, 2015.
Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, and Joey Yiwei Li. Deepshift: Towards multiplication-les. neural networks. arXiv preprint arXiv:/905. 13298, 2019.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani. Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. In International Conference on Learning Representations, 2019.
Jun Fang. Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piecewise linear quantization for deep neural networks. In European Conference Computer Vision, pp. 69-86. Springer, 2020.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference Machine Learning, pp. 1737-1746. 2015.
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression. In Proceedings of the IEEÉ/CVE Conference on Computer Vision and Pattern Recognition, pp. 8494-8502, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
M Horowitz.
Computing's energy problem (and what we can do about it). 2014 ieee int. Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, 2014a.
Mark Horowitz. Energy table for 45nm process. In Stanford VLSI wiki. 2014b.
Ning-Chi Huang, Huan-Jan Chou, and Kai-Chiang Wu. Efficient systolic array based decompos- able mae for quantized deep neural networks. 2019.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006. 10518, 2020.
10

---

Under review as a conference paper at ICLR 2022
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-aribbmetic-nlly inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint arXiv: 12322, 2019.
Naghmeh Karimi, Thorben Moos, and Amir Moradi. Exploring the effect of device aging on static power analysis attacks. UMBC Faculty Collection, 2019.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang. and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:///axxvv.or../o org/ abs wwws/assl... 06530
Youngbae Kim, Heekyung Kim, Nandakishor Yadav, Shuai Li, and Kyuwon Ken Choi. Low-power rtl code generation for advanced enn algorithms toward object detection in autonomous vehicles. Electronics, 9(3):478, 2020.
Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4013-4021, 2016.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv: MXXiv:6005004111 2016.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi
Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021. URL
few.:errrrriwe..eeoooorai forum?
iddpoWw6hDddxxa
Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. arXiv preprint arXiv:1510.03009 2015.
Xingchao Liu, Mao Ye, Dengyong Zhou, and Qiang Liu. Post-training quantization with multiple points: Mixed precision without mixed precision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8697-8705, 2021.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. arXiv preprint arXiv:/810.01885, 2018.
Mostafa Mahmoud, Isak Edo, Ali Hadi Zadeh, Omar Mohamed Awad, Gennady Pekhimenko, Jorge Albericio, and Andreas Moshovos. Tensordash: Exploiting sparsity accelerate deep neural network training and inference, 2020.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: Wide reduced-preccssoon networks. arXiv preprint arXiv: 1009:0 134, 2017.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. arXiv preprint arXiv:2004.10566 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint arXtw:1911.07199. 2019.
Yehya Nasser. J-C Prévotet, Maryline Hélard, and Jordane Lorandel. Dynamic power estimation based on switching activity propagation. In 2017 27th International Conference on Field Programmable Logic and Applications (FPL), pp. 1-2. 2017.

---

Under review as a conference paper at ICLR 2022
Renkun Ni, Hong- min Chu, Oscar Castaneda Fernandez, Ping- yeh Chiang, Christoph Studer, and Tom Goldstein. Wrapnet: Neural net inference with ultra-low-paa precision arithmetic. In 9th International Conference Learning Representations (ICLR 2021), 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (0428-10436, 2020.
Andres Rodrigue7, Eden Segal, Etay Meiri, Evarist Fomenko, Y Jim Kim, Haihao Shen, and Barukh Ziv. Lower numerical precision deep learning inference and training. Intel White Paper, 3:1-19, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Xu Shoukai, Li Haokun, Zhuang Bohan, Liu Jing, Cao Jiezhang, Liang Chuangrun, and Tan Mingkui. Generative low-bitwidth data free quantization. In The European Conference on Computer Vision, 2020.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform- neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Michael Tschannen, Aran Khanna, and Animashree Anandkumar. Strassennets: Deep learning with multiplication budget. In International Conference on Machine Learning, pp. 4985-4994, PMLR, 2018.
Ganesh Venkatesh, Eriko Nurvitadhi, and Debbie Marr. Accelerating deep convolutional networks using low-precision and sparsity, 2016.
Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspired deep network. Advances in Neural Information Processing Systems, 33, 2020.
A APPENDIX
A.1 POWER SIMULATION ON 5NM PROCESS
Using the Synopsys DesignWarr Ware Library built Verilog RTL (Register Transfer Logic) simula- tion which instantiates signed multipliers and signed adders of 2-8 bit widths. For the multipliers, used Radix-4 Booth-encoder implementation and for the adders, used the Ripple Carry implementation. We synthesized these adders and multipliers, with a 5nm cells library at a clock frequency of 1.6GH
In order to analyze the power consumption of each module individually, we used a hierarchical gate-level synthesis where each module is a different utility that does not share logic with any other module. The synthesis result is the gate-level netlist with the logic gates, which is the actual implementation of the multipliers and adders. Then, using Synopsys PrimeTime Time PX (PTPX), which accurately reflects ASIC power consumption, we ran a simulation with uniformly distributed random inputs on the gate-level netlist and measured the power of multiplication and addition instructions.
It should be noted that, as opposed to papers experimenting with FPGAs, our primary interest is in modern integrated chips (ASICs), like CPUs and GPUs. Gate-level simulations of the type we use here are the only practical (and most accurate) way to estimate ASIC power consumption. This is because even if we had fabricated this netlist as part of real ASIC (which would cost millions of dollars), it would still be impossible to measure the power of a small portion of the chip accurately.
https: wwwp://wwwww.o. www.wwwwww.................. com/
html
12

---

Under review as a conference paper at ICLR 2022
In figures 4(a) and 4(b) we depict the average power consumed by multiplication of two b-bit numbers and by an addition of two b-bit numbers. respectively. We can see that these power measurements agree with our Python simulation, which discuss in the next section. A remark is in place about the slight deviation between the 5nm simulation and our Python simulation seen in Fig. [[a This deviation implies that the advantage of PANN over regular DNN is slightly more substantial than that reported in the paper. Specifically, since our theoretical model slightly underestimates the power consumed by a multiplier at high bit-widths, we actually underestimate the benefit of our method, which avoids multiplications. The configuration files will be published online to enable easy reproduction of the results.
In T we report the amount in [%]] of the dynamic power and the static power as measured using our 5nm simulation.
MEASURED
2-BIT
3-BIT
4-BIT
5-BIT
6-BIT
7-BIT
8-BIT
32-BIT
DYNAMIC POWER (MULTIPLIER)
59
57
55
50
51
STATIC POWER (MULTIPLIER)
41
43
45
49
50
49
49
DYNAMIC POWER (ADDER)
61
60
59
58
58
55
56
60
STATIC POWER (ADDER)
39
40
41
42
42
45
44
40
Table 5: Static power dynamic power in [%]. We can that overall the dynamic power constitutes a major portion of the total power.
A.2 POWER SIMULATION IN PYTHON
Using Python, we implemented simple serial adder and two types of multipliers: a simple serial multiplier and Radix-2 Booth encoding multiplier. A serial multiplier follows the long multiplication concept in which each bit of the multiplicand multiplies the multiplier word. This results in a word of length b 6+11 bits most, called partial product. Going over all bits in the multiplicand results in b partial products that need to be summed. The Booth encoder is more efficient in terms of the number of partial products that need to be summed. It comprises encoder that follows lookup table and directs whether to perform shift, addition or subtraction, basing on consecutive pairs of bits of the multiplicand. For example, suppose we want to multiply number by 15, which is 1111 in binary representation. The serial multiplier performs the computation æ X (2 + 2 + 2 + 29) while the Booth encoder multiplier computes æ x (2* 29), and saves two sums. Those partial products are summed by half and full adders that are the major area and power consumers of multiplier.
We are focusing on the dynamic power, which is a prominent source of power consumption and linearly depends on the switching activity. Therefore, in order to estimate the power, we measured the average number of bit toggles per instruction (e.g., multiplication and addition). We counted the toggles at the inputs of each half or full 1-bit adder component, both for the b-bit multiplier and for the b-bit adder. Please see illustrative example in Fig. In figures 7 we show the average number of toggles per instruction that were counted in the multiplier and in the adder using signed and unsigned numbers, respectively. We and a Gaussian one. We took the uniform distribution to be over the range [-20-1 2--1)) As for the Gaussian, we first drew N full precision numbers from NO, 1). Then, we divided them by their maximum (in absolute value), multiplied by 2--1, and rounded to the closest integer. We clipped the values the range [-2h-1, 20-1) in order to eliminate outliers (specifically the number 20-1). 1). In all our experiments, we took N = 36000. Please see an example histogram for the Gaussian distributed numbers Fig. 5(b) where b 8.
simulation with data drawn from a uniform distribution
A.3 OBSERVATION-1
When working with an accumulator having large bit width B (e.g. B = 32), a dominant source of power consumption is the bit toggling in its inputs, which is 0.5B per instruction comes from the 2's complement representation. Hence, a significant amount of power can be saved when switching unsigned numbers. For example, in the right plot of Fig. 8 we show that the power in the accumulator inputs is reduced from 16 (assuming a 32-bit accumulator) to 0.50 where bace is the bit width at the input to the accumulator.
average. This
13

---

Under review as a conference paper at ICLR 2022
5nm simulation simulatton
Smm smulation Python stmulation
Watt/Bit flpp
Wast/Bit fi
width
Bit
(a) Measuring the power of multiplication operation.
(b) Measuring power of an addition operation.
Figure 4: (a) In red we plot the power consumed by a multiplication operation as measured in our Python simulation. This curve agrees with the theoretical power model in Eq. (3) in the paper, i.e. Pmmlt = 0.36² + (see left side of Fig. 7. In blue, we plot the power measurements on a 5-nm silicon process. In order to ignore the different power units and set both measurements the same axis, scaled the results of the 5nm power simulation so that the curves intersect at bit width of 4. (b) In this experiment,
measured the power consumed by l-bit accumulator without the FF (thus, here baee = b). red we can observe the power measured in our Python simulation, which is very close our theoretical model Pace 0.56 + 0.5b (see right side of Fig. 7 We can see that the power measurements for the 5nm silicon process (blue) nicely agree with our Python simulation. Here we scaled the results of the 5nm simulation using the same factor found in Fig. 4(a)
10
140
Power Ratio unsignedtsigne
0.6
100
60-
40
Smm simulation
Python
00
-100
-50
30
100
Bit width
(a) Switching to unsigned numbers does not affect the multiplier.
(b) An example quantized Gaussian distribution.
Figure 5: (a) Using our Python simulation, We measured the average number of toggles between unsigned multiplication and signed multiplication for bit widths of 4-8. As be seen, obtained an average ratio of 92% (red curve). This observation is aligned with the power measured in our 5nm silicon process (blue curve). (b) Unlike the uniform distribution, values occupies roughly half of the allowed interval and therefore on average, we observe a bit less toggles than with the uniform distribution (here the bit width b 8).
can see that the majority of
As for the multiplier, switching to unsigned values turns out to have a negligible effect terms of power consumption (left plot of Fig. 8). In Fig. 5(a) we show the ratio between the power consumed by multiplication of unsigned numbers and multiplication of signed numbers, as measured in our Python simulation and in the 5nm silicon process. As can be seen, this ratio is close to for all bit
14

---

Under review as a conference paper at ICLR 2022
1
0
0
1-tat Full adiker
1-1at FUll adder
1-hxt full adder
1-hit half adder
1-hat full adier
1-hit full adder
1-hit full adder
1-hit half addler
(a) State 1
(b) State i
Figure 6: Counting toggles in multiplier's internal adders. We depict snapshot of the multiplier's internal components and 0001 and at state we sum 1111 and 0100. In our python simulation, we compare between the bit status of consecutive operations therefore in this example, will count four toggles (two in the input words and two in the internal carry outputs).
consecutive addition instructions. At state i 1 sum 1111
Average number of togg es per mult plication
0.5b24 +-B
500
Average number of toggles per addition
Gaussian
400
0.30xcr
300
Uniform acc sumiFF
Uniform, act inputs sumiFF
Gaussian, acc inputs
100
2
6
Dacc
Figure 7: Python simulation for signed integers. On the left, we plot the power consumed by the multiplier. We counted the toggles the inputs the multiplier (row Table of the paper) as well as the toggles inside its internal units (row 2 in Table 1). We can see that the power measured in our simulations closely agrees with power model in Eq. (1), 0.56² +b. On the right, we plot the power consumed by the accumulator, where the label "acc inputs" refers to the power due to the bit flips at its input (row in Table 1). In this case = 32 and therefore we observe a constant power of 16. The label "acc sum" refers to the power consumed due the toggles at the output of the accumulator (row Table I) which also toggles the bits in the FF. Again, the simulation agrees with our model.
widths. Therefore, adopt the power model for the unsigned multiplier case in the signed setting.
A.3.1 SWITCHING TO UNSIGNED ARITHMETIC
Figure (Iaa) compares the average power consumption of signed MAC to that of an unsigned MAC for a 32 bit accumulator. Specifically, we are dividing P + Pa by Pmult Pace- this setting, it can be seen for example that when working with b 4 bits for the weights and activations, unsigned MACs are 33% cheaper in power. The approach we suggest for switching a linear layer (e.g., convolution, fully connected) to work with unsigned arithmetic illustrated schematically in Fig. (1(b)
Accumulator bit width The bit width of the accumulator is commonly chosen to be 32. One the main reasons for that is this allows flexibility in changing the bit widths of the activations and weights
15

---

Under review as a conference paper at ICLR 2022
Average number toggles per mu tiplication
0,5b Uniform Gaussian
0.50ax
Average number of toggles, per addition
14
Uniform, acc sumnif
Uniform,
Caussan, acc
Gaussaa, acc inputs
400
10
100
10 15 20
Figure 8: Python simulation for unsigned integers. Here repeat the experiment of Fig. but with numbers drawn only from the interval 10,2 for both the uniform and the Gaussian distributions. On the left, we and is aligned with Eq. (3) in the paper. However, on the right we can see that due to the use of unsigned values, the power consumed by the toggling at the accumulator inputs is dramatically reduced (0.56ace instead of 0.5.). The rest of the power contributors did not change much.
that the overall power of the multiplier has not changed much
35
35
Average number toggles per multiplication
b,
b,
30
30
b,
b,
Average toggles
20
25-
15.
20
10-
5
Figure 9: Working with b. < by in Booth encoder multiplier. On the right, we uniformly drew by-bit numbers from [-2h- -1 2hb -b, and -bit numbers from 1-2 2bw-1), for various by br. We can see that the power is affected only by the larger bit width (b.), and remains nearly constant when reducing only b On the left, we repeat the same experiment, however with unsigned values, where the b.-bit input is uniformly drawn from [o, 2 and the b input from [0, 21w Here, there is slight benefit in decreasing only b.. The black dashed curve connects the power measurements for the cases where by by. and follows the parabolic behaviour.
(e.g., from 4-bit to 8-bit and vice versa). Nevertheless, we are not concerned with flexibility, then when quantizing the activations and weights to less than 8 bits, than 32 bits. The required accumulator bit width B can be calculated by
use an accumulator with less
B by + b. + 1 + log Ck²Cin),
(20)
where the convolution kernel size, and Cin is number of input channels. In Table we analyze the case of ResNet networks. We choose the layer with the largest value of k2 which is 3x3x512 (Table 1 in He et al. (2016)). We calculate the required bit width for the accumulator (e.g., B) when the activations and weights quantized to 2-6 bits. In addition, [%] when switching to unsigned arithmetic. As can be seen, even with smaller accumulator bit widths, switching unsigned arithmetic leads to significant saving in power. This is also visually
calculate the power save in
16

---

Under review as a conference paper at ICLR 2022
Average number toggles per muttip ication
number tooples multip
30
30
20-
20-
Dw
Figure 10: Working with by by in simple serial multiplier. On the right, uniformly drew bb- numbers from [-2 11==1) and b bit numbers from [-2 -1 2
such that b I bz- We can that the power affected by the larger bit width (b,)) On the left, we repeat the experiment however with unsigned values, where the bbt input is uniformly drawn from [0, 20.-1) and the -bit input from [0, 20.- Here, there is more benefit in decreasing only b The black dashed
curve connect the power measurements for the cases where b = b. and follows the parabolic behaviour.
70
signed
60
unsigned
W
Power
50
40
W
91%
sex
W
10
Bit-width
(a) Switching to unsigned arithmetic
(b) Unsigned arithmetic layer
power model, show that significant amount of power be saved with minimal effort, by switching to work with unsigned numbers. Here we assume a 32 bit accumulator. (b) Any weight matrix W can be split into its positive and negative parts (See Sec. 4 Assuming the elements of its input & are non-negative (due to the preceding ReLU), this makes all MACs unsigned and thus substantially reduces power consumption.
Figure 11: (a) Based
illustrated in Fig. [2 where we repeat the experiment Fig. T bit with 17 bit accumulator for the 2-bit networks, and with 21 bit accumulator for the 4-bit networks.
A.4 OBSERVATION-2
We now analyze the case where the inputs of the multiplier have different bit widths, by and by- In Fig. we show the average bit toggles in signed and unsigned Booth encoder multiplication, for bw bz. In Fig. [0 show the same analysis for the simple serial multiplier. We observe that when working with signed numbers (the common setting), the power is mostly affected by the larger bit width (b,, in this case).
In the case unsigned numbers, there is some power save when reducing one of the bit widths. In other words, Eq. (7) in the paper is accurate for the popular signed and behaves upper bound for the unsigned case. The difference between Eq. (7) and the actual power consumption in the unsigned setting is more dominant for the simple serial multiplier. Therefore, in certain settings,
17

---

Under review as a conference paper at ICLR 2022
2-BIT
3-BIT
4-BIT
5-BIT
6-BIT
REQUIRED BIT WIDTH B POWER SAVE FOR A B BIT ACCUMULATOR
17
19
21
23
25
39%
28%
21 %
16%
13%
POWER SAVE FOR 32 BIT ACCUMULATOR
58%
44%
33%
25%
19%
Table 6: Required accumulator bit width. Here we compute the bit width required for the accumnu- lator, according the largest linear layer in ResNets (which is 3x3x512). For example, in case of 2 bit width activations and weights, we might use an accumulator with 16 bits and not 32 bits. In the last two rows report the power save in [%] when switching to unsigned arithmetic. Just like in the 32 bit accumulator case, when working with lower bit width accumulators, we can obtain a significant reduction in power by switching to unsigned arithmetic.
80
80
PANN
PANN
PANN
PANN
70
70-
PANN
60
TANN
60
Unsigned
BRECQ
Accuracy [%6]
Unsigned
ZapQ
Unsignal
ZeroQ
Accuracy [%]]
Uniggrei
BRECQ
50-
FANN
50-
PANN
BRBCQ
UnSguett
40-
Unsigned
ZerQ
BRECO
Unsigmed
Rogitet 3226
Reghees 3.20
SavccseNct v1_0
Squecechitt V1.0
Efflcrnttee, B1 Googlette
Unsigned
ZurQ
Goog sttel
15
15
0.0
1.0
0.0
Power [Bit ips]
Bit flips
(a) Improving upon ZeroQ (4 bits)
(b) Improving upon BRECQ (2 bits)
Figure 12: Power-accuracy trade-off at post training with different bit width accumulators. We repeat the experiment of Fig. []] however this time we assume 21 bits accumulator in (a) and 17 bit accumulator in (b). Theretofore, when converting the quantized models to work with unsigned arithmetic (+), it cuts down 21% % of the power consumption in (a) and 39% in (b).
there is an additional benefit of switching to unsigned arithmetic, which we did not report in the experiments in the paper (i.e. the horizontal arrows in Fig. Fig-[shoulde should actually be slightly longer in some cases). Yet, this effect is relatively small compared to the reduction in bit flips in the accumulator.
We validated our observation on the 5-nm silicon process setup. We used measured the power when one of the inputs was drawn uniformly from [0, 2?) and the other from [0, ).. We got 95% of the power that was measured when both inputs were drawn from [0, 27). ). In case of signed values, when one of the inputs was drawn uniformly from [-27 27) and the other from 223 23) we observed 100% of the power that was measured when both inputs were drawn from 2T)
multiplier and
Note that in order to avoid changing the multiplier's architecture, when switching to unsigned numbers, we use only half the range allowed by the bit width &_, i.e. [0, 2h--1). Therefore, we obtain a representation with half the 2d= levels the signed case (note that we also need to replace- - with - + 1). If we permit architectural changes, then a better way to represent unsigned numbers would be to replace the signed multiplier by a (b + 1) x (b + 1) multiplier that support unsigned and signed multiplications but consumes much a by. x by unsigned multiplier that allow representation in the full interval of [0, 2"r) and then will follow Eq. (7) again.
power, to work with
18

---

Under review as a conference paper at ICLR 2022
A.5 ADDITIONAL RESULTS
A.5.1 POST TRAINING QUANTIZATION
Similarly to Table 2 we now examine PANN's performance on additional networks. We use the same methods for activations quantization as Table 2
ACIO Banner al. 2019). Zero CCai
et al. 2020), GDFQ (Shoukai et al. 2020) and BRECQ (Li Ll et al. 2021). We also add Dynamic Quantization, which quantizes the activation and weights on the fly at inference time, according to their dynamic ranges. The results are reported in Tables 7
We first change all networks to work with unsigned arithmetic and measure the classification accuracy, which serves baseline (see left side of each method 'Base.".. As mentioned, this step already saves a significant amount of power, without any change in classification accuracy, compared to signed MAC arithmetic. We use the unsigned MAC power consumption the power budget P and follow Alg. T
We report the classification accuracy on the right side each columns (see 'Our').
POWER (BITS)
DYNAMIC
ACIQ
ZEROQ
GDFQ
BRECQ
BASE.
OUR
BASE,
OUR
BASE.
OUR
BASE.
OUR
BASE
OUR
116 (8)
69.77
69.78
69.61
69.67
69.67
69.68
69.75
69.71
69.73
69.72
95 (6)
66.56
69.50
69.05
69.60
67.51
69.55
69.20
69.35
69.70
69.71
76 (5)
55.52
69.12
67.18
69.53
54.76
69.50
68.60
69.12
69.50
69.56
43 (4)
0.33
68.88
55.00
69.26
26.50
69.10
60.61
69.01
68.69
68.29
30 (3)
0.11
68.28
1.50
68.43
0.23
68.20
19.88
68.52
65.20
67.34
18 (2)
0.09
63.62
0.11
66.68
0.10
66.12
0.12
68.11
43.67
66.73
Table 7: Classification accuracy [%] of ResNet-18 on ImageNet (FP: 69.77%). The baselines (Base.) equal bit widths for weights and activations (leftmost column). The bit width determines the power P, which we specify in units of Giga bit-flips. The power is calculated as P + acc (Eqs. 3 times the number MACs in the network. (1.82 x 109 ResNet-18). In each row, our variant tuned to work at the same power budget, for which we choose the optimal by and R using Alg.
POWER (BITS)
DYNAMIC
ACIQ
ZEROQ
GDFQ
BRECQ
BASE.
OUR
BASE.
OUR
BASE.
OUR
BASE.
OUR
BASE.
OUR
21 (8)
71.82
71.79
69.73
69.71
71.79
71.53
71.88
71.76
71.95
71.85
19 (6)
62.13
64.13
66.16
67.18
69.35
69.58
70.48
70.52
71.36
71.55
15)
13.11
59.55
27.06
61.14
60.49
64.33
65.32
68.31
70.30
70.98
(4)
3.56
51.25
2.32
55.13
13.92
62.14
50.96
66.02
65.12
69.12
(3)
0.05
49.33
0.09
50.23
0.06
61.11
31.19
64.13
55.14
67.85
(2)
0.01
23.26
0.07
35.55
0.03
48.12
1.55
51.12
25.91
61.08
Table 8: Classification accuracy [%] of Mobilenet-V2 V2 on ImageNet (FP: 71.91 %).. The baselines (Base.) use equal bit widths for weights and activations (leftmost column). The bit width determines the power P. which we specify in units of Giga bit-flips. The power is calculated as pu + P (Eqs. 4 times the number of MACs in the network. (0.33 10 in MobileNet-V2) V2). In each row, our variant is tuned to work at the same power budget, for which we choose the optimal b, and R using Alg.
In figures 13 14
we demonstrate PANN post training for different networks under power constrains of 4-bit and bt unsigned MAC. In each figure, we start by running the specified approach to
19

---

Under review as a conference paper at ICLR 2022
POWER (BITS)
DYNAMIC
ACIQ
ZEROQ
GDFQ
BRECQ
BASE.
OUR
BASE.
OUR
BASE.
OUR
BASE.
OUR
BASE
OUR
994 (8)
73.28
73.31
73.24
73.13
73.29
73.30
73.34
73.25
652 (6)
72.15
73.23
73.02
73.05
73.18
73.12
73.31
73.15
505 (5)
64.05
72.88
72.31
73.02
71.11
71.92
72.25
73.02
373 (4)
51.13
72.06
66.20
72.03
64.19
70.19
67.05
71.66
256 (3)
2.15
70.55
31.22
71.18
20.88
69.95
51.16
71.12
155 (2)
0.56
69.95
0.13
71.02
0.18
66.62
3.63
67.96
Table 9: Classification accuracy [%] of VGG-16bn on ImageNet (FP: 73.35%). The baselines (Base.) use equal bit widths for weights and activations (leftmost column). The bit width determines the power P, which we specify in parentheses in units of Giga bit-flips. The power is calculated as pu mult Pa (Eqs. ()) times the number of MACs in the network. (15.53 10 in VGG-16bn) In each row, our variant is tuned to work at the same power budget, for which we choose the optimal by and R using Alg. 1 We failed to run BRECQ due to a CUDA 'out of memory' error.
bits (see caption). We measure the power in bit flips. Specifically, the power of each signed MAC is calculated by Pmalt + Pace (Eqs. ( times the number of MACs the network. Then, we switch to unsigned arithmetic (f-). In this case the power is calculated by Pm + PH (Eqs. (3.4)
quantize the bits and the activation
4h times the number of MACs in the network. For simplicity, let us denote the total power as P As can be seen, this stage save power but does not change the accuracy (points marked "Unsigned)). Constraining to the same power budget pa, we apply PANN (t). Using Alg. we calculate the optimal activation bit width and the corresponding addition factor (points marked as 'PANN'). We can see dramatic improvement in classification accuracy without any change in the power consumption.
PANN
PANN
Unsigned
GDRQ
PAINN
PANN
ACIO
PANN
Unsigned
GDRQ A
60
PANN
Accuracy [%)
PANN
ACIQ
Unsigred
<
GDRQ
Unsigred
Accuracy
CDRQ
Unsigned
ACIQ
Unsigned
ACIQ
RagNel-3.2C
Regtees 2G Sqpaeeshsoorr. Mriantee-2 Googlshee
Goog clte
15
15
1.1
0.2
1.1
Power [Bit liips]
Power [Bit flips]
(a) Improving upon ACIQ (4- MAC)
(b) Improving upon GDFQ (4-bit MAC)
Figure 13: Power-accuracy trade-off at post training. For each pre-trained full full-precision model, we used (a) ACIQ (Banner al. 2019) and (b) GDFQ (Shoukai et al. (020) to quantize the weights and activations to 4 bits at post-training. Converting the quantized models to work with unsigned arithmetic (+)) already cuts down 33% of the power consumption (assuming 32 bit accumulator). Using our PANN approach to quantize the weights (at cost-training) and remove the multiplier (t), further improves model accuracy for the same power level.
A.5.2 QUANTIZATION AWARE TRAINING
In Table 10 we report the classification accuracy of different networks on ImageNet, when we use PANN during training. Each row defines specific power budget, corresponding to 2-bit, 3-bit and
20

---

Under review as a conference paper at ICLR 2022
80
Reghett 6O0M
70-
70-
PANN
60
60-
PANN
PAINN
PANN
50-
PANN
Accuracy (%)
50-
PANN
Accuracy
40
PAIN
40
PANN
30
30
15-
ZaroO Unsigned
Unsigeed
GDR
Unsigned
Unainne
Zevo
ZeroQ
Onaggnd
CDIQ
GDFQ
Bit flips
X12
Bit flips
x10
(a) Improving upon ZeroQ (2- MAC)
(b) Improving upon GDFQ (2-bit MAC)
Figure 14: Power-accuracy trade-off at post training. For each pre-trained full-precision model, used (a) ZeroQ (Cai al., 2020) and (b) GDFQ (Shoukai al. 2020) quantize the weights and activations to 2 bits at post-training Converting the quantized models to work with unsigned arithmetic (+), already cuts down 58% of the power consumption (assuming 32 bit accumulator). Using our PANN approach to quantize the weights (at post-training) and remove the multiplier (t), further improves model accuracy for the same power level.
4-bit unsigned MACs (weights and activations have equal bit widths). We compare our results to LSQ (Esser et al. 2019), whose accuracy reported in parentheses. Note that the total number of bit flips differs between the networks, because each has different number of MACS in its forward pass. Therefore, instead of specifying results as a function the total number of bit flips, we report results function of the bit-width Each bit- width defines power budget for which tune PANN (Alg. D. where we use LSQ to quantize the activations. We can see that PANN outperforms LSQ at all power budgets.
Power (bit-width)
ResNet-18
ResNet-34
ResNet-50
ResNet-101
VGG-16bn
FP
70.13
73.88
76.87
77.55
73.33
2
70.03 (67.32)
72.54 (71.21)
76.65 (71.36)
77.13 (75.21)
73.30 (71.15)
3
70.12 (69.81)
73.87 (72.88)
76.78 (73.54)
77.24 (76.62)
73.31 (73.26)
70.10 (70.13)
73.96 (73.90)
76.81 (76.89)
77.33 (77.52)
73.46 (73.51)
Table 10: PANN for QAT. Here we report more results PANN for quantization aware training. In parentheses report the classification accuracy [%] of LSQ Imagenet, where both activations and weights are quantized to 2,3 or 4 bits. As for PANN, we follow Alg. to calculate the optimal activation bit width and addition factor.
A.5.3 ADDITIONAL COMPARISONS TO MULTIPLICATIION-FER METHODS
In tables II 12
we report additional comparisons with the recent multiplication free methods Shif- tAddNet (You al. 2020) and AdderNet (Chen et al. 2020), this time on the CIFARION and ing methods.
| et al. (2014) datasets. Here again PANN is used during training, like the compet-
A.6 HYPER PARAMETERS FOR PANN IN QAT
A.6.1 LSQ
In all experiments we used the SGD optimizer with momentum of 0.9 and weight decay of 10 10-4. We used the softmax cross entropy loss. Unlike the original paper, we started the training from
21

---

Under review as a conference paper at ICLR 2022
METHOD
6/6
5/5
4/4
3/3
OUR (1x)
66.
64.50
62.80
55.51
OUR (1.5x)
66.23
65.8
63.58
56.85
OUR (2x)
66.90
66.50
63.99
57.51
SHIFTADDNET (1.5x)
64.08
64.05
63.23
61.31
ADDERNET (2x)
41.57
35.20
29.19
21.50
Table 11: QAT: Comparison with multiplier-free methods.
Classification accuracy [%] of
multiplier- methods on CIFAR-100. The top row specifies weight/activation bit widths, and the addition factor is specified in parentheses.
METHOD
6/6
5/5
4/4
3/3
OUR (1x)
95.01
84.13
65.36
59,9
OUR (1.5x)
95.05
85.91
68.96
62.32
OUR (2x)
95.34
87.36
70.82
62.51
SHIFTADDNET YOU ET AL.
(220
(1.5x)
85.61
63.34
35.77
18.19
ADDERNET CHEN ET AL. 2020) (2x)
89.31
68.21
26.77
10.56
Table 12: QAT: Comparison with multiplier-free methods. Classification accuracy [%] of multiplier-free methods on the MHEALTH dataset [Banos et al. (2014). The top row specifies weight/activation bit widths, and the addition factor specified in parentheses.
pre-trained networks and with smaller initial learning rate of 10-3 Please refer to Table 13 for PANN details, scheduling and number of epochs used for the training.
ARCH.
QAT (b./ba)
P
b
R
LR SCHEDULE
B,
EPOCHS
RESNET- 18
LSQ (2/2)
18
3
2.83
x EVERY 25 EPOCHS
128
75
RESNET-18
LSQ (3/3)
30
6
2.5
x0.1 EVERY 25 EPOCHS
128
75
RESNET-18
LSQ (4/4)
43
6
3.5
x0.1 EVERY 25 EPOCHS
128
75
RESNET-34
LSQ (2/2)
2.83
x EVERY 20 EPOCHS
60
RESNET-34
LSQ (3/3)
61
6
2.5
x 1 EVERY 20 EPOCHS
64
60
RESNET-34
LSQ (4/4)
88
6
3.5
0. EVERY 20 EPOCHS
64
60
RESNET-50
(2/2)
41
3
2.83
X0.1 EVERY 20 EPOCHS
64
60
RESNET-50
LSQ
68
6
2.5
0 EVERY 20 EPOCHS
60
RESNET-50
LSQ (4/4)
99
6
3.5
0 EVERY 20 EPOCHS
60
RESNET-101
(2/2)
78
3
2.83
0 EVERY 20 EPOCHS
64
60
RESNET-101
LSQ (3/3)
128
6
2.5
X0.1 EVERY 20 EPOCHS
64
60
RESNET-101
LSQ
187
6
3.5
X0.1 EVERY 20 EPOCHS
64
60
VGG-16BN
LSQ (2/2)
2.83
X EVERY EPOCHS
60
VGG- 16BN
LSQ (3/3)
279
6
2.5
X EVERY 20 EPOCHS
60
VGG-16BN
LSQ (4/4)
372
6
3.5
x EVERY 20 EPOCHS
64
60
Table 13:
used in LSQ. When using pure LSQ as the bascline approach, quantize both the weights and the activations to the same bit width as specified in the second column (b./b.)). Then, when applying PANN, we keep the exact training regime and the quantized activations, and only change the quantized weights to be calculated by PANN. Here we report the optimal bit width for the activations and the corresponding addition factor (Alg I
A.6.2 MULTIPLIER FREE APPROACHES
In all experiments followed the training regime described in You et al. 2020). Specifically, for CIFARIO or CIFAR100 we used batch size of 256, and 160 epochs. The inttil learning rate was 0.1 and then divided by 10 the 80-th and the 120-th epoch. We used the SGD optimizer with momentum of 0.9 and weight decay of 4. For the MHEALTH dataset, we used only 40 epochs to train. The initial learning rate 0.01 and then divided by 10 at the 20-th and the 30-th epochs.
22

---

Under review as a conference paper at ICLR 2022
Similarly to the CIFAR experiments, used SGD optimizer with momentum of 0.9 and weight decay of le
A.7 HARDWARE-ACCURAC TRADE-OFF
When operating on single image at inference time (rather than a large batch), the memory footprint of the weights is not negligible anymore (Mishra et al. 2017). Therefore, we need to also account for the bit-width required for storing the quantized weights. In [4fle report the optimal activation bit width and addition factor for each power constraint use ZeroQ to quantize the activations of pre-trained full-precision ResNet-50. We then measure the maximal addition factor per neuron, which defines the bit width bR required to store the weights. We observe that overall, the increase in the runtime memory footprint of the weights is relatively low, especially in the low power regimes.
certain setting. Specifically, we
Up to now, we have only shown results with the bit width by (and corresponding additions factor R) that is optimal in terms of classification accuracy. However, for given power budget P. choosing by (and R) can be done while also accounting for other factors, like latency and memory footprint. We illustrate this in Table 15 for the case power constraing corresponding to 2-bit MAC. Here, we report results for all options for by and that conform to that power budget. While b = and R 1.16 is optimal in terms of classification accuracy, the user choose other options, e,g. according to latency or memory constraints.
POWER (br/br))
be
LATENCY(= R)
bR
ACTIVATIONS MEMORY
WEIGHTS MEMORY
2/2
1.16x
3x
1.5x
3/3
6
2.25x
2x
1x
4/4
2.9x
1.75x
0.75
5/5
3.5x
1.6x
0.8x
6/6
4.75x
1.33x
0.83
7/7
8
6.06x
1.14x
0.714x
8/8
7.5x
1x
0.625
Table 14: Runtime memory footprint of PANN. We report the increase in the memory required to store the weights and activations, when using PANN. Each row specifies power budget, corre- sponding to by bit width unsigned MAC. We follow Alg. I to find the optimal bit width b, and the additions factor R, which is equal to the latency increase. We measure the maximal value of additions per neuron which defines the required number of bits for storing the weights (bn). Then, we calculate the increase weights memory footprint as bz/ bg-
b
LATENCY(= R)
bR
ACTIVATIONS MEMORY
WEIGHTS MEMORY
ACCURACY [%]
2
4.5x
5
1x
2.5x
0.9
3
2.83x
3
1.5x
1x
6.55
2.0x
3
2x
0.75x
61.15
5
1.5x
2
2,5x
0.4x
65.69
6
1.16x
2
3x
0.33x
71.55
7 0.92x
2
3.5x
0.28x
70.18
8
0.75x
4x
0.25x
60.01
Table 15: Hardware-accuracy trade-off. Here we analyze the run-time memory footprint and latency increase for all different values of i, and R that lead to the same power of a 2-bit unsigned MAC (blue curve in Figure 3(a)). For each setting we measure the classification accuracy of ResNet- 50 on ImageNet. Here we use ACIQ (Banner et al. (219)) for quantizing the activations. The baseline (pure ACIQ) accuracy is 0.20% (Table third column, last row).
23

---

Under review as a conference paper at ICLR 2022
Unifomm
Sinclatinn Theerttiall
Serpater Thouetial
ones
0055
Simslation Thooetical
Semulation
Thep volicl
0023
0022
9.02
6003
Bit Widto
WitWi
Bir Width
Bitt Width
Gaussian
20202522 2.2.2
122
22.
MS2
MS2
BiL Width
Bii Wich
Bil Widlh
Bit Widh
ImrgeeNer
100
Ems (96]
R R 2.2.2
WitWidd
Bit Width
Bit Widhh
Bitt Wodth
Power unsigned MAC
Power umsigned
Power unsigned 4-hit MAC
Power unsigned 5-bot MAC
3-bit MAC
Figure 15: Optimal bit width analysis for PANN. The first two rows depict the MSE as a function of the activation bit width 6y for the cases where the weights and the activations are uniformly distributed and for the setting in which they are Gaussian (the activations are further subjected to ReLU function in this case). The third row shows the classification error of a ResNet18 model on ImageNet. Although the precise value of the optimal by is bit different than the first two rows, the qualitative behavior is similar. For both for the Gaussian simulation and the ImageNet results, we used ACIQ Banner et al. (2019) to quantize the activations.
A.8 QUANTIZATION ERROR ANALYSIS
Figure 1.5 (first row) depiets MSEPANN (Eq. 19)) as a function of d, for several power budgets P. First, it can be seen that our theoretical analysis agrees well with simulations. Second, it shows that the optimal by (where the minimum MSE is attained) increases with the power budget. This implies that at higher power budgets, is preferable to increase the bit width of the activations on the expense of reducing the number of additions R. The second row of the figure illustrates that the qualitative conclusions drawn from our uniform distribution analysis also hold when the weights are Gaussian and the activations are Gaussian numbers after ReI.U function (here we used the ACIQ quantizer (Banner al. 2019)). In the third row, we show that similar behaviors characterise the error rates of ResNet-18 18 model for ImageNet classification when using PANN to quantize its weights and ACIQ to quantize the activations.
24

---

Under review as a conference paper at ICLR 2022
A.9 PROOF OF EQ. (14)
Let w and a be statistically independent random vectors, each with iid components. Recall that Wi and 2ig are obtained by applying scalar function each of the elements of and æ; respectively. Therefore w. and B, also have iid components. We assume that w = wy + Ew and æ = By + Ex. where E [w] 0 and E [z] 0. Then have that
u T x w Fig) "]
MSE E
(w x (w + Ew) EE)))
E
(w" x (w twTT x + w we + + x + T )
E (wo 6n + If + t )
(21)
and therefore,
(w Ex) + E ( + (ET)ax)
MSE
+ 2TR
[w0
T o [c) + 2F ww" ege [ + 2T [E
(22)
We now turn to show that the last three terms equal zero. For the first those terms, we have
E T Ex E
E (ww" "xxx" sw]
E [ [wo" exax" ew w. w.
E [uo" T [ un Ew] Ew
E [w"E [ Ew]
0,
(23)
where in the second line we used the law total expectations, the fourth line we used the fact that the pair [w, E } is statistically independent of the pair Eæ), and in the fifth line used the fact
that [E(==="") x="x"=>>" x=]] [[E[= x x""] = 0 because of our assumption that Elez xx) = 0.
For the second among the last three terms in we have that
E /ww Ex
= Ex
F ws
c we
Ex
E [=]]] [wee"
Eæ
6x
E [=]]]
[we?" T'
0,
(24)
where we used the fact that the pair (w, Em independent of ens and [E[we]]] E[E[we]] wo] E [Ew [w]"] 0 because of our assumption that Ejew w/ = 0.
For the the last term in 22 we have that
E
xe
En
xE
E
E le De E Ew
E [s T
T T
Ew
Ew
[ E
[xe]] 6w
0,
(25)
25

---

Under review as a conference paper at ICLR 2022
where we used the fact that the pair {æ, E is independent of ww, and E[xxa]] 0, in thus remain only with the first three terms (22)
23 We
so that
MSE =
(w Ex)
2
+
E (
+ E (<'
E w E 20
+ E
x w - + E En
E E 1 Ex
[
+ E E x Eww x|]
+ [ e Ew
Jur E [= |
[) w]
+ R LnFR [Ewe
I
m m]
[
[e [ Em
= E [w"'E [= 7"
w] + [="" [Eme
T x
+ E [ [ Ew]
E [w=" (o 1) w] + E [= (o 1)
ad]
+
[e" (o² I [
d oo to o2.
(26)
where d o² o² and o² denote the second-order moments of the elements of w, æ, ews and & respectively, and I is the d identity matrix. Here, in the fifth equality used the fact that w independent of Ex, used the fact that ε and Ew iid vectors with zero mean, since E(E=] [[=[(=]<=]] = 0 and E[sw] = E E|E E w/] = 0. This completes the proof of Eq. (14) in the main text.
independent of ww. and Ew is independent of &x. In the sixth equality
26


Reviews:
[{"review_id": "mxLAt4VXXl3", "reviewer": "Reviewer_GEgt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weakness:\n\n* The authors rely on a 2000 publication to support the argument that dynamic power dominates. That's hardly definitive today, especially the power is simulated in the 5nm regime. Stronger support and/or better methodology is needed.\n* The authors didn't consider the memory energy, which usually dominates the total energy, as opposed to MAC energy. In general, the experimental methodology is too simplistic, essentially a python script counting bit flips.There are just way too many other things that affect the energy. At least try some CACTI models; better yet, use a memory compiler.", "summary_of_the_paper": "The paper proposes to remove signed arithmetics and removing multiplication. The power model is limited to dynamic power, and just the MAC power.", "summary_of_the_review": "The idea is of intellectual merit, but the evaluation ignores too much important aspect of real energy/power modeling. The results are far from convincing in that regard.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Soqp7nIjRIA", "reviewer": "Reviewer_CvJm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n* The proposed approach (PANN) is original and unlike any previous work in the field of quantization.\n* The paper highlights unique new insights which are valuable for future hardware considerations.\n* The proposed approach shows strong results for a given power budget for which it clearly outperforms conventional uniform quantization (under their power model assumption).\n* The authors are somewhat upfront with some limitations and added the memory and latency overhead to table 2 and discussed them.\n* Overall the paper is well and clearly written.\n\nWeak points:\n* The paper only considers the power consumption of the MAC array. However, the paper ignores the power consumption (and latency) required for the memory movement, which is non neglectable. The proposed approaches, both the switching to unsigned tensors (eq 5 and 6) and the switching to additions (eq 11), can further increase the memory transfer costs making this an even more important part to consider for the power model (and latency considerations).\n* It is unclear whether the provided power simulations generalize to MAC operations in neural networks, and specifically of the proposed PANN approach. The simulation uses uniformly distributed random inputs (see appendix A.1), however in neural networks weights are roughly a normal distributed and similar activations (or a clipped normal distribution in the case of ReLU activations). It is unclear whether the proposed power model would also be a good approximation for such a different expected input distribution. With the proposed splitting of the positive and negative parts of the weight tensor (eq 5 and 6), the weight distribution will be even more different from the assumed uniform distribution. It is important to validate that the power model is valid under the expected distributions as most claims of the paper are based on observations from this power model.\n* While the proposed approach shows good accuracy improvement for a power budget, it also has significant limitations. Namely it leads to memory overhead, latency overhead and requires dedicated hardware implementations to be utilized. For example, the 3 bit power budget model, corresponds roughly in memory and latency to a 6 bit uniform quantization model, and in this case the accuracy of the uniform model is significantly higher. Thus there is a clear trade-off between power and memory/latency. \n* The assumption about uniform distributed weights and activations in section 5.3 is fairly unrealistic. Would the outcome (significantly) change under the assumption of a normal distribution?\n\n\nNotes:\n* Observation 1: I agree that the accumulator can have significant impact on the power usage of the MAC array. Though the required bit width for the accumulator scales with the bit width of the input tensors. The common use of 32 bit accumulator is often based on 8 bit quantization (see Rodriguez et al.), thus the example provided might overemphasis this effect.\n* It is quite common to use asymmetric quantization (unsigned int + zero offset, see Jacob et al. 2018), thus unsigned integers are already frequently used. What is the advantage of the proposed splitting approach compared to standard asymmetric uniform quantization?\n* Sometimes bits and power budget can be confused, especially by a less careful reader (e.g. table 2, 3). It would improve clarity if the tables and text clearly use power and only state comparable bit width where needed in brackets (now it is frequently the other way around, which on a first look seems like N bit quantization, but does not highlight the power aspect).\n\nQuestions:\n* In figure 4a) (Appendix, validation of power model), the results are scaled independently for multiplication and additions. On the other side, in the paper we trade off multiplications with additions. How is this scaling exactly down and can this scaling actually have an influence on the trade-off made between multiplications and additions?\n* How does PANN work during QAT? It is a bit unclear from the paper (e.g. how is the quantization simulated, is the budget set before or after training, how are ranges determined, etc)\n", "summary_of_the_paper": "The paper argues that power consumption is a major obstacle in deploying DNNs to end devices and that current quantization approaches do not take power consumption directly into account and therefore are not optimal in reducing it. Using an approximate power model based on the average number of bit flips, they make two observation that are frequently overlooked by existing quantization approaches: \n1) A significant portion of the power consumption of the MAC operation is due to the usage of signed integers and using unsigned integers instead can significantly reduce the power consumption.\n2) The multipliers power consumption is dominated by the larger bit widths (weight or activation) and therefore using lower bit for one of the two (e.g. weights) is not power efficient.\n\nBased on this the authors introduce a new weight quantization approach (PANN) which removes the multiply operation and replaces it with additions. This allows PANN to efficiently reduces the power consumption. For the same power budget, PANN achieves significantly higher accuracy (or effective bit width), but comes at the cost of higher latency and memory usage.\n", "summary_of_the_review": "The authors propose a novel new approach to reduce directly the power consumption for inference. The originality and good results are the main strength of the paper while the weak points are mostly centered around some assumption and justifications of the power model and the limitations of the proposed approach. Overall both side are fairly in balance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EtCH_RDOKV", "reviewer": "Reviewer_NEeA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths\n* Power-aware DNNs are interesting and important that merits exploration.\n* The mean squared quantization error of a regular uniform quantizer and the proposed PANN was theoretically analyzed given first-order approximation, and they are further compared under the same fixed power budget.\n* Low-bit networks in post-training quantization with PANN exhibits only a minor degradation in accuracy compared to floating-point counterparts.\n\n### Weaknesses\n* Although bit-flips can be considered an important source of power consumption, the proposed approach of counting bit-flips and minimizing such counts per image inference is fundamentally flawed. The design of the multiply-accumulate provides an incomplete and oversimplified description of the actual hardware realization of a DNN. For instance, the summation of $N$ values can be implemented as an adder tree instead of working on a single accumulator, and would result in a reduction of the number of additions to approximately $\\log(N)$ and thus reducing the number of bit-flips. In addition, how the circuit should parallelize and/or time-share computations would have great impact on the number of bit-flips. Considering only a long multiply-accumulate chain for each layer is unrealistic and not faithful to the actual hardware designs.\n* The proposed method in Section 5 to replace multipliers with repeated additions by accumulation is equal or inferior to multiplication in terms of bit-flips. Intuitively, counting bit operations, repeated addition consumes $min(O(2^N M), O(2^M N))$ for an $(N,M)$-bit multiplication, whereas multiplication would use $O(MN)$.\n* Some important experimental settings (bit-level details, training epochs, optimizer, etc.) are missing. It would be better to provide such details in appendix to for reproducibility as code is not available in this submission.\n\n### Minor\n* “Our approach can can work in combination with any activation quantization method.” Duplicate “can”.", "summary_of_the_paper": "The authors observed that the power consumption is dominated by the bit toggling at the input of the accumulator and decreasing the bit-width of only the weights or only the activations has limited benefit to reduce the the power consumed by the multiplier. The paper proposed PANN, which uses tricks such as unsigned arithmetic in CNN and implement multiplications via additions to achieve a multiplier-free and power-aware neural network. Experiments on both post-training quantization and quantization-aware training showcased better performance compared to other works under the same number of bit-flips. However, the reviewer believes there are fundamental flaws that need to be thoroughly addressed before this paper is ready for ICLR.\n", "summary_of_the_review": "Although power-aware DNNs are interesting and this topic is an important research direction, this paper has some fundamental flaws that prevented the reviewer from recommending acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "59dFBEprtw", "reviewer": "Reviewer_yuqG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strength **\n1. The idea that quantized the model using the power budget is very important to real-world applications.\n 2. The paper achieved superior accuracy than the previously developed quantized methods.\n** Weakness ** \n1. The experimental application is too narrow. Since the authors mentioned that the proposed method can be applied to \"any\" model, a wide variety of applications (BERT for QnA task, etc..) should be included. \n2. If each layer has a different quantization bit-width, the hardware implementation cost could be increased when compared to the fixed bit-width-based hardware.\n\n** Minor Comments **\n- Please fix me if I am misunderstood. The proposed method determines the quantization bit-width using the power budget of the entire network, so each layer or neuron has a different bit-width. If my understanding is right, mentioning this explicitly might help to understand the paper.\n- The phrase \"to quantize the weights and activations at post-training\" in the caption of Figure 1 is a bit confusing. In the current explanation, the readers can be confused why removing the multiplier increases accuracy.\n- In the case of batchnorm, the authors mentioned that \"Batch-norm layers should first be absorbed into the weights and biases\". How can it be handled when using the layernorm?\n- The state-of-the-art models employed activations with negative values. How to convert to unsigned integers when using gelu or elu?\n- The paper can be enhanced if the authors provide how many bits are allocated for each layer for 2 bits (power) in Table 2.\n- There is a typo on page 2, \"Our approach can work in combination\" -> \"Our approach can work in combination\"", "summary_of_the_paper": "This paper proposed a method for reducing the power consumption of deep neural networks called a power-aware neural network (PANN).  The proposed method exploits the entire power consumption of the model to determine the quantization bit widths, so it could achieve higher accuracy than some previously developed methods when the power budget is set to ultra-low bits such as 2-bit or 3-bit.", "summary_of_the_review": "The idea of finding the optimal quantized model in terms of the power budget is an important issue for real-world applications. Although some part of the paper is ambiguous to me, I think this paper could be a bridge to reduce the gap between the model researchers and hardware engineers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper argues that the existing approaches for reducing power consumption do not model the precise power usage of each model. To remedy this an approximate power usage model is proposed using bit flips and a simple approach called PANN is introduced that relies on tricks such as unsigned arithmetics and implementation of multiplications with addition. The reviewers have found the overall direction of this paper in modeling power consumption important and have acknowledged the clarity of presentation. However, they have also raised serious concerns regarding (i) the efficacy of modeling power consumption with bit flips and ignoring memory power, (ii) its relevance to modern hardware, and (ii) the efficacy of replacing multipliers with repeated additions. Unfortunately, the paper in it is current form does not provide a compelling answer to these concerns. Given these criticisms, we don't believe that the paper is ready for publication at ICLR.
Decision: Reject