Year: 2021

Abstract:
Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices. Existing approaches for reducing power consumption rely on quite general principles, including avoidance of multiplication operations and aggressive quantization of weights and activations. However, these methods do not take into account the precise power consumed by each module in the network, and are therefore far from optimal. In this paper we develop accurate power consumption models for all arithmetic operations in the DNN, under various working conditions. Surprisingly, we reveal several important factors that have been overlooked to date. Based on our analysis, we present PANN (power-aware neural network), a simple approach for approximating any full-precision network by a lowpower fixed-precision variant. Our method can be applied to a pre-trained network, and can also be used during training to achieve improved performance. In contrast to previous methods, PANN incurs only a minor degradation in accuracy w.r.t. the full-precision version of the network, even when working at the power-budget of a 2-bit quantized variant. In addition, our scheme enables to seamlessly traverse the power-accuracy trade-off at deployment time, which is a major advantage over existing quantization methods that are constrained to specific bit widths.

heading: 1 INTRODUCTION
text: With the ever increasing popularity of deep neural networks (DNNs) for tasks like face detection, voice recognition, and image enhancement, power consumption has become one of the major considerations in the design of DNNs for resource-limited end-devices. Over the last several years, a plethora of approaches have been introduced for achieving power efficiency in DNNs. These range from specialized architectures (Sandler et al., 2018; Huang et al., 2019; Tan et al., 2019; Radosavovic et al., 2020), to hardware oriented methods like multiplier-free designs and low-precision arithmetic.
Multiplier aware methods attempt to reduce power consumption by avoiding the costly multiplication operations, which dominate the computations in a DNN. Several works replaced multiplications by additions (Courbariaux et al., 2015; Li et al., 2016; Chen et al., 2020) or by bit shift operations (Elhoushi et al., 2019) or both (You et al., 2020). Others employed efficient matrix multiplication operators (Tschannen et al., 2018; Lavin & Gray, 2016). However, most methods in this category introduce dedicated architectures, which require training the network from scratch. This poses a severe limitation, as different variants of the network need to be trained for different power constraints.
Low-precision DNNs reduce power consumption by using low-precision arithmetic. This is done either via quantization-aware training (QAT) or with post-training quantization techniques. The latter avoid the need for retraining the network but often still require access to a small number of calibration samples in order to adapt the network’s weights. Such techniques include approaches like re-training, fine-tuning, calibration and optimization (Banner et al., 2019; Jacob et al., 2018; Nahshan et al., 2019; Li et al., 2021). All existing methods in this category suffer from a large drop in accuracy with respect to the full-precision version of the network, especially when working at very low bit widths. Moreover, similarly to the multiplier-free approaches, they do not provide a mechanism for traversing the power-accuracy trade-off without actually changing the hardware (e.g., replacing an 8-bit multiplier by a 4-bit one). In this work, we introduce a power-aware neural network (PANN) approach that allows to dramatically cut down the power consumption of any model. Our method can be applied at post-training to improve the power efficiency of a pre-trained model, or in a QAT setting
to obtain even improved results. Our approach is based on careful analysis of the power consumed by additions and multiplications, as functions of several factors. We rely on bit toggling activity, which is known to be the main factor affecting dynamic power consumption, and support our theoretical analysis with accurate gate-level simulations on a 5nm process.
Our first important observation is that a major portion of the power consumed by a DNN is due to the use of signed integers. We therefore present a simple method for converting any pre-trained model to use unsigned arithmetic. This conversion does not change the functionality of the model and, as can be seen in Fig. 1, dramatically reduces power consumption on common hardware configurations.
Our second observation is that the multiplier’s power consumption is dominated by the larger bit width among its two inputs. Therefore, although high accuracy can often be achieved with quite drastic quantization of only the weights, this common practice turns out to be ineffective in terms of power consumption. To be able to take advantage of drastic weight quantization, here we introduce a method that allows removing the multiplier altogether. Our approach can work in combination with any activation quantization method. We show theoretically and experimentally that this method is far advantageous over existing quantization methods at low power budgets, both at post-training and in QAT settings (see Fig. 1 and Sec. 6).
Our method allows working under any power constraint by tuning the number of additions used to approximate each mutltiply-accumulate (MAC) operation. This is in contrast to regular quantization methods, which are limited to particular values. We can thus traverse the power-accuracy trade-off without changing the architecture (e.g., bit width of the multiplier), as required by existing methods.

heading: 2 RELATED WORK
text: Avoiding multiplications In fixed point (integer) representation, additions are typically much more power-efficient than multiplications (Horowitz, 2014b;a). Some works suggested to binarize or ternarize the weights to enable working with additions only (Courbariaux et al., 2015; Lin et al., 2015; Li et al., 2016). However, this often severely impairs the network’s accuracy. Recent works suggested to replace multiplications by bit shifts (Elhoushi et al., 2019) or additions (Chen et al., 2020) or both (You et al., 2020). Other methods reduce the number of multiplications by inducing
Element toggles
Multiplier inputs 0.5b+0.5b Multiplier’s internal units 0.5b2 Accumulator input 0.5B Accumulator sum & FF 0.5bacc+0.5bacc
Table 1: Average number of bit flips per signed MAC. The b-bit multiplier inputs are drawn uniformly from [−2b−1, 2b−1) and its bacc = 2b bit output is summed with the B-bit number in the FF.
sparsity (Venkatesh et al., 2016; Mahmoud et al., 2020), decomposition into smaller intermediate products (Kim et al., 2016), Winograd based convolutions (Lavin & Gray, 2016), or Strassen’s matrix multiplication algorithm (Tschannen et al., 2018). Some of these methods require internal changes in the model, a dedicated backpropagation scheme, or other modifications to the training process. Quantization DNN quantization approaches include post-training quantization (PTQ), which is applied to a pre-trained model, and quantization-aware training (QAT), where the network’s weights are adapted to the quantization during training (Gupta et al., 2015; Louizos et al., 2018; Achterhold et al., 2018; Esser et al., 2019). PTQ methods are more flexible in that they do not require access to the training set. These methods show optimal results for 8-bit quantization, but tend to incur a large drop in accuracy at low bit widths. To battle this effect, some PTQ methods minimize the quantization errors of each layer individually by optimizing the parameters over a calibration set (Nahshan et al., 2019; Nagel et al., 2020; Hubara et al., 2020). Others use nonuniform quantization (Liu et al., 2021; Fang et al., 2020). Effort is also invested in avoiding the need of any data sample for calibration (Cai et al., 2020; Shoukai et al., 2020; Nagel et al., 2019; Haroush et al., 2020). These methods, however, still show a significant drop in accuracy at the lower bit widths, while frequently requiring additional computational resources. Common to all quantization works is that they lack analysis of the power consumed by each arithmetic operation as a function of bit-width, and thus cannot strive for optimal power-accuracy trade-offs.

heading: 3 POWER CONSUMPTION OF A CONVENTIONAL DNN
text: The total amount of power consumed by a logic circuit can be attributed to two main sources: a static power component and a dynamic one. The static power is due to a constant leakage current. It does not depend on the circuit’s activity and is typically the smaller component among the two. The dynamic power consumed by each node in the circuit is given by P = CV 2fα, where C is the node capacitance, V is the supply voltage, f is the operating frequency, and α is the switching activity factor (the average number of bit flips per clock) (Nasser et al., 2017). Here we focus on dynamic power, which is a major contributor to the overall power consumption (see Appendix A.1 and (Karimi et al., 2019; Kim et al., 2020)). Also, this is the only factor affected by the DNN architecture.
Most of the computation in a forward pass of a DNN can be attributed to MAC operations. As shown in Fig. 2, MACs involve a multiplier that accepts two b-bit numbers and outputs a bacc-bit result (bacc = 2b to account for the largest possible product), and an accumulator with a large bit width B to which the multiplier’s output is added repeatedly.
To understand how much power each of these components consumes, we simulated them in Python. For the multiplier, we used the Booth-encoding architecture, which is considered efficient in terms of bit toggling (Asif & Kong, 2015). For the accumulator, we simulated a serial adder. Our Python simulation allows measuring the total number of bit flips in each MAC operation, including at the inputs, at the outputs, in the flip-flop (FF) register holding the previous sum, and within each of the internal components (e.g., the full-adders) of the multiplier. We also verified our analysis with an accurate physical gate-level simulation on a 5nm process and found good agreement in terms of the dependence of power consumption on the bit widths (see details in Appendix A.1).
Table 1 shows the average number of bit flips per MAC when both inputs to the multiplier are drawn uniformly at random from [−2b−1, 2b−1) (Gaussian inputs lead to similar results; please see
Appendix Figs 7-8). As can be seen, the power consumed by the multiplier is given by1
Pmult = 0.5b 2 + b, (1)
where 0.5b2 is due to the bit toggling in the internal units, and 0.5b is contributed by the bit flips in each input. The power consumed by the accumulator is given by
Pacc = 0.5B + 2b, (2)
where 0.5B is due to the bit toggling in its input coming from the multiplier, 0.5bacc = b (recall bacc = 2b) to the bit flips at the output, and an additional 0.5bacc = b to the bit flips in the FF. These results lead us to our first important observation. Observation 1. A dominant source of power consumption is the bit toggling at the input of the accumulator (0.5B).
Suppose, for example, we use b = 4 bits for representing the weights and activations and employ a B = 32 bit accumulator, as common in modern architectures (Kalamkar et al., 2019; Rodriguez et al., 2018). Then the toggling at the input of the accumulator (0.5B = 16) is responsible for 44.4% of the total power consumption (Pmult + Pacc = 36). At lower bit widths, this percentage is even larger.
Unfortunately, existing quantization methods and multiplier-free designs do not battle this source of power consumption. Ni et al. (2021) have recently shown that the bit-width B of the accumulator can be somewhat reduced by explicitly accounting for overflows. However, this approach requires dedicated training, and degrades the network’s classification accuracy at low values of B. As we now show, it is possible to drastically reduce the bit toggles at the input of the accumulator at post-training without changing the model’s functionality (thus retaining the same classification accuracy).

heading: 4 SWITCHING TO UNSIGNED ARITHMETIC
text: Since the output of the multiplier has only 2b bits, one could expect to experience no more than b bit flips on average at the accumulator’s input. Why do we have 0.5B bit flips instead? The reason is rooted in the use of signed arithmetic. Specifically, negative numbers are represented using two’s complement, and thus switching between positive and negative numbers results in flipping of many of the higher bits. For example, when using a 32 bit accumulator, if the output of the multiplier switches from +2 to −2, then the bits at the input of the accumulator switch from 00000000000000000000000000000010 to 11111111111111111111111111111110. Note that this effect is dominant only at the accumulator’s input simply because sign changes at the output are rare.
If we could work with unsigned integers, then the higher bits at the accumulator’s input would always remain zero, which would lead to a substantial reduction in power consumption without any performance degradation. To quantify this, we repeated the experiment of Sec. 3, but with the b-bit inputs to the multiplier now drawn uniformly from [0, 2b−1) (see Appendix A.2 for details). In this case, the average number of bit flips at the input of the accumulator reduced from 0.5B to 0.5bacc = b. Specifically, the average power consumption of an unsigned MAC operation was measured to be
P umult = 0.5b 2 + b (3)
due to the multiplier and P uacc = 3b (4) due to the accumulator. In (4), 2b bit flips occur at the accumulator’s output and the FF, and b bit flips occur at the accumulator’s input coming from the multiplier. Thus, although the mutliplier’s power (3) turns out to be the same as in the signed setting (1), the accumulator’s power (4) is substantially reduced w.r.t. the signed case (2).
Converting a pre-trained network with ReLU activation functions to work with unsigned integers is simple. Specifically, consider a layer performing y =Wx+ b. The elements of x are non-negative because of the preceding ReLU2. Therefore, we can split any such layer into two parallel layers as
y+ =W+x+ b+, y− =W−x+ b−, (5)
1The amount of power consumed by a single bit flip may vary across platforms (e.g., between a 5nm and a 45nm fabrication), but the number of bit flips per MAC does not change. We therefore report power in units of bit-flips, which allows comparing between implementations while disregarding the platform.
2Batch-norm layers should first be absorbed into the weights and biases.
where W+ = ReLU(W ), b+ = ReLU(b), W− = ReLU(−W ), b− = ReLU(−b), and compute y = y+ − y−. (6)
This way, all MACs are converted to unsigned ones in (5), and only a single subtraction per output element is needed in (6). This one subtraction is negligible w.r.t. the MACs, whose number is usually in the thousands. Please see Fig. 11(b) in the Appendix for a schematic illustration.
Figure 1 shows the effect that this approach has on the power consumption of several pretrained networks for ImageNet classification. Figures 1(a) and 1(b) show 4 bit and 2 bit quantized networks, respectively. With a 32 bit accumulator, merely switching to unsigned arithmetic cuts 33% and 58% of the power consumption of these networks (see Appendix A.3.1 for other accumulator bit widths).

heading: 5 REMOVING THE MULTIPLIER
text: Having reduced the power consumed by the accumulator, we now turn to treat the multiplier. Quantization methods often use different bit widths for the weights and activations. This flexibility allows achieving good classification accuracy with quite aggressive quantization of one of them (typically the weights), but a finer quantization of the other. An interesting question is whether this approach is beneficial in terms of power consumption.
We repeated the experiment of Sec. 3, this time with the multiplier inputs having different bit widths, bw and bx. We focused on the standard setting of signed numbers, which we drew uniformly from [−2bw , 2bw−1) and [−2bx , 2bx−1). Interestingly, we found that the average number of bit flips in the multiplier’s internal units is affected only by the larger among bw and bx. Accounting also for the bit flips at the inputs, we obtained that the multiplier’s total average power is
Pmult = 0.5 (max{bw, bx})2 + 0.5(bw + bx). (7) We found this surprising behavior to be characteristic of both the Booth multiplier and the simple serial multiplier, and verified it also with accurate simulations on a 5nm silicon process gate level synthesis (see Appendix Figs. 9,10). This leads us to our second important observation. Observation 2. There is marginal benefit in the common practice of decreasing the bit width of only the weights or only the activations, at least in terms of the power consumed by the multiplier.
It should be noted that in the case of unsigned numbers, where inputs are drawn uniformly from [0, 2bw−1) and [0, 2bx−1), there exists some power save when reducing one of the bit widths, especially for the serial multiplier (see Appendix Fig. 10). This highlights again the importance of unsigned arithmetic. However, in all our experiments we do not take this extra benefit of our approach into account when computing power consumption, so that our reports are conservative.
To benefit from the ability to achieve high precision with drastic quantization of only the weights, we now explore a solution that allows removing the multiplier altogether. Unlike other multiplier-free designs, our method allows converting any full-precision pre-trained model into a low-precision power-efficient one without any changes to the architecture.

heading: 5.1 POWER AWARE WEIGHT QUANTIZATION
text: Consider the computation
y = d∑ i=1 wi · xi, (8)
which involves d MACs. Here, {wi, xi} are the weights and activations of a convolution or a fullyconnected layer. Given {wi, xi} in full precision, our goal is to accurately approximate (8) in a power-efficient manner. When quantizing the weights and activations we obtain the approximation
y ≈ d∑
i=1
γwQw(wi) · γxQx(xi), (9)
where the quantizers Qw(·) and Qx(·) map R to Z, and γw and γx are their quantization steps3. To make the computation (9) power efficient, we propose to implement multiplications via additions.
3In quantized models MAC operations are always performed on integers and rescaling is applied at the end.
Specifically, assume Qw(wi) is a non-negative integer (as in Sec. 4). Then we can implement the term Qw(wi) · Qx(xi) as
Qw(wi) · Qx(xi) = Qx(xi) + · · ·+Qx(xi)︸ ︷︷ ︸ Qw(wi) times , (10)
so that (9) is computed as y ≈ γwγx d∑ i=1 Qw(wi)∑ j=1 Qx(xi). (11)
This is the basis for our power-aware neural network (PANN) design.
Let w = (w1, . . . , wd)T and x = (x1, . . . , xd)T denote the full precision weights and activations, and denote their quantized versions by wq = (Qw(w1), . . . ,Qw(wd))T and xq = (Qx(x1), . . . ,Qx(xd))T , respectively. Note that as opposed to conventional quantization methods, our approach does not necessitate that the quantized weights be confined to any particular range of the form [0, 2bw). Indeed, what controls our approximation accuracy is not the largest possible entry in wq, but rather the number of additions per input element, which is given by ‖wq‖1/d. Therefore, given a budget of R additions per input element, we propose to use a quantization step of γw = ‖w‖1/(Rd) in (9), so that
Q(wi) = round(wi/γw). (12) This quantization ensures that the number of additions per input element is indeed as close as possible to the prescribed R. We remark that although we assumed unsigned weights, this quantization procedure can also be used for signed weights (after quantization, the positive and negative weights can be treated separately in order to save power, as in Sec. 4).

heading: 5.2 POWER CONSUMPTION
text: We emphasize that in PANN, we would not necessarily want to use the same bit width for the activations as in regular quantization. We therefore denote the activation bit width in PANN by b̃x to distuingish it from the bx bits we would use with the regular quantizer. To estimate the power consumed by our approach, note that we have approximately ‖w‖1 additions of b̃x bit numbers. On
average, each such addition leads to 0.5b̃x bit flips at the accumulator’s output and 0.5b̃x bit flips in the FF register (see Table 1). The input to the accumulator, however, remains fixed for Qw(wi) times when approximating the ith MAC and therefore changes a total of only d times throughout the entire computation in (11), each time with 0.5b̃x bit flips on average. Thus, overall, the average power per element consumed by PANN is
PPANN = ‖w‖1b̃x + 0.5b̃xd
d = (R+ 0.5)b̃x. (13)
This implies that to comply with a prescribed power budget, we can either increase the activation bit width b̃x on the expense of the number of additions R, or vice versa.
Figure 3(a) depicts the combinations of b̃x and R that lead to the same power consumption as that of a bx bit unsigned MAC, P uMAC = 0.5b 2 x + 4bx (see (3),(4)), for several values of bx (different colors). When we traverse such an equal-power curve, we also change the quantization error. Thus, the question is whether there exist points along each curve, which lead to lower errors than those obtained with regular quantization at the bit-width corresponding to that curve.

heading: 5.3 QUANTIZATION ERROR
text: Let us compute the quantization error incurred by PANN and compare it to that of a regular uniform quantizer (RUQ). Obviously, the error incurred by the approximation (9) is contributed by both the quantization of the weights and the quantization of the activations. To see how their errors interact, let us assume that w and x are statistically independent random vectors, each with iid components. In this case, if the quantization errors εw = w− γwQ(w) and εx = x− γxQ(x) satisfy E[εw|w] = 0 and E[εx|x] = 0, then we can show (see Appendix A.9) that the mean squared error (MSE) between the full precision operation (8) and its quantized version (9) is given by
MSE = E [( wTx−wTq xq )2] ≈ d (σ2wσ2εx + σ2xσ2εw) . (14) Here σ2w, σ 2 x, σ 2 εw , and σ 2 εx denote the second-order moments of the elements of w, x, εw, and εx, respectively, and the expression on the right results from neglecting second-order terms.
Consider a simplistic scenario where the activations are uniformly distributed in [0,Ma] (recall that activations are non-negative due to the preceding ReLU) and the weights are uniformly distributed in [− 12Mw, 1 2Mw] . If we use a RUQ with bx bits to quantize the activations and a RUQ with bw bits to quantize the weights, then we have that
σ2x = M2x 3 , σ2εx = M2x 12 · 22bx , σ2w = M2w 12 , σ2εw = M2w 12 · 22bw . (15)
This is because the quantization errors are uniformly distributed as εx ∼ U [−2−(bx−1), 2−(bx−1)] and εw ∼ U [−2−(bw−1), 2−(bw−1)]. Substituting (15) into (14), we obtain that the MSE of a RUQ is
MSERUQ = dM2xM 2 w 122 ( 2−2bx + 4 · 2−2bw ) . (16)
In PANN, we have that (εw)i|w ∼ U [−‖w‖12Rd , ‖w‖1 2Rd ], so that E[(εw) 2 i |w] = ‖w‖21 12Rd . Therefore,
σ2εw ≈ E[‖w‖21] 12(Rd)2 = d2(0.25Mw) 2 12(Rd)2 = M2w 192R2 , (17)
where we used the fact that |wi| ∼ U [0, 0.5Mw]. Substituting this expression in (14), we find that using PANN together with a b̃x bit RUQ for the activations, we achieve
MSEPANN = dM2xM 2 w
122
( 2−2b̃x + 1
4R2
) . (18)
To compare between PANN and RUQ, we need to fix a power budget P . Given such a budget, (13) dictates that the number of additions in PANN should be set to R = P/b̃x − 0.5. Substituting this into (19), we obtain that
MSEPANN = dM2xM 2 w
122
( 2−2b̃x +
b̃2x
(2P − b̃x)2
) . (19)
The optimal bit-width for the activations can therefore be found numerically by minimizing MSEPANN over b̃x ∈ Z+. This typically requires evaluating (19) for a small number of candidate bit widths, e.g., b̃x ∈ {2, . . . , 8}. See Appendix A.8 for a thorough analysis.
Figure 3(b) shows the ratio between MSERUQ and MSEPANN (with the optimal b̃x) as a function of the bit width of the RUQ, where PANN is tuned to the same power. For the RUQ, we use bx = bw as its power consumption (7) is anyway dominated by the larger of them. It can be seen that for low bit widths, PANN has a significant advantage over RUQ (ratio larger than 1). In the Gaussian setting, which is closer to the distribution of DNN weights and activations, the range over which PANN outperforms RUQ is even larger. As we show in Appendix A.8, this behavior is very similar to that observed in deep networks for image classification.
We emphasize that (19) is valid for uniformly distributed weights and activations, which is often not an accurate enough assumption for DNNs. Thus, in practice the best way to determine the optimal bit width b̃x is by running the quantized network on a validation set, as summarized in Algorithm 1.
Algorithm 1: Determining the optimal parameters for PANN
1: Input: Power budget P 2: Output: Optimal b̃x, R 3: for each b̃x ∈ [b̃minx , b̃maxx ] do 4: Set R = P/b̃x − 0.5 (Eq. (13)) 5: Quantize the weights using Eq. (12) with γw = ‖w‖/(Rd) 6: Quantize the activations to b̃x bits using any quantization method 7: Run the network on a validation set, with multiplications replaced by additions using Eq. (10) 8: Save the accuracy to Acc(b̃x). 9: end for
10: set b̃x ← argmaxb̃x Acc(b̃x), R← P/b̃x − 0.5

heading: 6 EXPERIMENTS
text: We now examine PANN in DNN classification experiments. We start by examining its performance at post training, and then move on to employ it during training. Here we focus only on the effect of removing the multiplier (vertical arrows in Fig. 1). Namely, we assume all models have already been converted to unsigned arithmetic (recall this by itself reduces a lot of the power consumption).
PANN at post training We illustrate PANN’s performance in conjunction with a variety of post training quantization methods, including the data free approaches GDFQ (Shoukai et al., 2020) and ZeroQ (Cai et al., 2020), the small calibration set method ACIQ (Banner et al., 2019), and the optimization based approach BRECQ (Li et al., 2021), which is currently the state-of-the-art for post training quantization at low bit widths. Table 2 reports results with ResNet-50 on ImageNet (See Appendix A.5.1 for results with other models). For the baseline methods, we always use equal bit widths for the weights and activations. Each row also shows our PANN variant, which works at the precise same power budget, where we choose the optimal b̃x and R using Alg. 1. As can be seen, PANN exhibits only a minor degradation w.r.t. the full-precision model, even when working at the power budget of 2 bit networks. This is while all existing methods completely fail in this regime.
PANN for quantization aware training To use PANN during training, we employ a straightthrough estimator for backpropagation through the quantizers. Table 3 compares our method to LSQ (Esser et al., 2019), which is a state-of-the-art QAT approach, where in PANN we use LSQ for quantizing the activations. As can be seen, PANN outperforms LSQ for various models and power budgets. In Table 4 we compare our method to the multiplication-free approaches AdderNet (Chen et al., 2020) and ShiftAddNet (You et al., 2020), which are also training-based techniques. For each method, we report the addition factor, which is the ratio between its number of additions per layer and a regular layer. For example, AdderNet uses no multiplications but twice as many additions, so that its addition factor is 2. ShiftAddNet, on the other hand, uses one addition and one shift operation. According to You et al. (2020), a shift operation costs between 0.2 (on FPGA) and 0.8 (on a 45nm ASIC) an addition operation. Therefore ShiftAddNet’s addition factor is between 1.2 and 1.8, and for simplicity we regard it as 1.5. In PANN, we can choose any addition factor R, and therefore examine
our method for R = 1, 1.5, 2. We can see in the table that PANN outperforms both AdderNet and ShiftAddNet for all bit widths, even when using a smaller addition factor. Please see more QAT comparisons in Appendix A.5.2
Runtime memory footprint and latency of PANN We now analyze the effect of PANN on other inference aspects besides power. One important aspect is runtime memory footprint. When working with batches of image, the runtime memory consumption is dominated by the activations (Mishra et al., 2017) (see discussion on the memory footprint of the weights in Appendix A.7). The optimal number of bits b̃x we use for the activations is typically larger than the bit width bx used in regular quantization. The second column of Table 2 reports the factor b̃x/bx by which the runtime memory of PANN exceeds that of the baseline model. As can be seen, this factor never exceeds 3. In the comparisons with the multiplier-free methods (Table 4), we keep the same bit width for the activations and therefore there is no change in the memory footprint. A second important factor is latency. Recall we remove the multiplier and remain only with the accumulator. Since addition is faster than multiplication, one could potentially use a higher clock-rate and thus gain speed. However, if we conservatively assume the original clock-rate, then the latency is increased by R (each multiplication is replaced by R additions). As can be seen in Table 2, the increase in latency is quite small at the lower power budgets. For the multiplier-free methods, we obtain improvement in accuracy even for R = 1. In that case, our latency is smaller than that of AdderNet (2×) and ShiftAddNet (1.5×). Please refer to Appendix A.7 for more analyses.

heading: 7 CONCLUSION
text: We presented an approach for reducing the power consumption of DNNs. Our technique relies on a detailed analysis of the power consumption of each arithmetic module in the network, and makes use of two key principles: switching to unsigned arithemtic, and employing a new weight quantization method that allows removing the multiplier. Our method substantially improves over existing approaches, both at post-training and when used during training, and allows achieving a significantly higher accuracy at any given power consumption budget.

heading: A APPENDIX
text: A.1 POWER SIMULATION ON 5NM PROCESS
Using the Synopsys DesignWare Library4, we built a Verilog RTL (Register Transfer Logic) simulation which instantiates signed multipliers and signed adders of 2-8 bit widths. For the multipliers, we used a Radix-4 Booth-encoder implementation and for the adders, we used the Ripple Carry implementation. We synthesized these adders and multipliers, with a 5nm cells library at a clock frequency of 1.6GHz.
In order to analyze the power consumption of each module individually, we used a hierarchical gate-level synthesis where each module is a different utility that does not share logic with any other module. The synthesis result is the gate-level netlist with the logic gates, which is the actual implementation of the multipliers and adders. Then, using Synopsys PrimeTime PX5 (PTPX), which accurately reflects ASIC power consumption, we ran a simulation with uniformly distributed random inputs on the gate-level netlist and measured the power of multiplication and addition instructions.
It should be noted that, as opposed to papers experimenting with FPGAs, our primary interest is in modern integrated chips (ASICs), like CPUs and GPUs. Gate-level simulations of the type we use here are the only practical (and most accurate) way to estimate ASIC power consumption. This is because even if we had fabricated this netlist as part of a real ASIC (which would cost millions of dollars), it would still be impossible to measure the power of a small portion of the chip accurately.
4https://www.synopsys.com/silicon-design.html 5https://www.synopsys.com/support/training/signoff/primetimepx-fcd.html
In figures 4(a) and 4(b) we depict the average power consumed by a multiplication of two b-bit numbers and by an addition of two b-bit numbers, respectively. We can see that these power measurements agree with our Python simulation, which we discuss in the next section. A remark is in place about the slight deviation between the 5nm simulation and our Python simulation seen in Fig. 4(a). This deviation implies that the advantage of PANN over a regular DNN is slightly more substantial than that reported in the paper. Specifically, since our theoretical model slightly underestimates the power consumed by a multiplier at high bit-widths, we actually underestimate the benefit of our method, which avoids multiplications. The configuration files will be published online to enable easy reproduction of the results.
In Table 5, we report the amount in [%] of the dynamic power and the static power as measured using our 5nm simulation.

heading: A.2 POWER SIMULATION IN PYTHON
text: Using Python, we implemented a simple serial adder and two types of multipliers: a simple serial multiplier and a Radix-2 Booth encoding multiplier. A serial multiplier follows the long multiplication concept in which each bit of the multiplicand multiplies the multiplier word. This results in a word of length b+ 1 bits at most, called a partial product. Going over all bits in the multiplicand results in b partial products that need to be summed. The Booth encoder is more efficient in terms of the number of partial products that need to be summed. It comprises an encoder that follows a lookup table and directs whether to perform shift, addition or subtraction, basing on consecutive pairs of bits of the multiplicand. For example, suppose we want to multiply a number x by 15, which is 1111 in binary representation. The serial multiplier performs the computation x× ( 23 + 22 + 21 + 20 ) while the
Booth encoder multiplier computes x× ( 24 − 20 ) , and saves two sums. Those partial products are summed by half and full adders that are the major area and power consumers of a multiplier.
We are focusing on the dynamic power, which is a prominent source of power consumption and linearly depends on the switching activity. Therefore, in order to estimate the power, we measured the average number of bit toggles per instruction (e.g., multiplication and addition). We counted the toggles at the inputs of each half or full 1-bit adder component, both for the b-bit multiplier and for the b-bit adder. Please see an illustrative example in Fig. 6. In figures 7,8 we show the average number of toggles per instruction that were counted in the multiplier and in the adder using signed and unsigned numbers, respectively. We ran our simulation with data drawn from a uniform distribution and a Gaussian one. We took the uniform distribution to be over the range [−2b−1, 2b−1). As for the Gaussian, we first drew N full precision numbers from N(0, 1). Then, we divided them by their maximum (in absolute value), multiplied by 2b−1, and rounded to the closest integer. We clipped the values to the range [−2b−1, 2b−1) in order to eliminate outliers (specifically the number 2b−1). In all our experiments, we took N = 36000. Please see an example histogram for the Gaussian distributed numbers in Fig. 5(b), where b = 8.

heading: A.3 OBSERVATION-1
text: When working with an accumulator having a large bit width B (e.g. B = 32), a dominant source of power consumption is the bit toggling in its inputs, which is 0.5B per instruction on average. This comes from the 2’s complement representation. Hence, a significant amount of power can be saved when switching to unsigned numbers. For example, in the right plot of Fig. 8 we show that the power in the accumulator inputs is reduced from 16 (assuming a 32-bit accumulator) to 0.5bacc where bacc is the bit width at the input to the accumulator.
As for the multiplier, switching to unsigned values turns out to have a negligible effect in terms of power consumption (left plot of Fig. 8). In Fig. 5(a) we show the ratio between the power consumed by multiplication of unsigned numbers and multiplication of signed numbers, as measured in our Python simulation and in the 5nm silicon process. As can be seen, this ratio is close to 1 for all bit
widths. Therefore, we adopt the same power model for the unsigned multiplier case as in the signed setting.

heading: A.3.1 SWITCHING TO UNSIGNED ARITHMETIC
text: Figure 11(a) compares the average power consumption of a signed MAC to that of an unsigned MAC for a 32 bit accumulator. Specifically, we are dividing P umult + P u acc by Pmult + Pacc. In this setting, it can be seen for example that when working with b = 4 bits for the weights and activations, unsigned MACs are 33% cheaper in power. The approach we suggest for switching a linear layer (e.g., convolution, fully connected) to work with unsigned arithmetic is illustrated schematically in Fig. 11(b).
Accumulator bit width The bit width of the accumulator is commonly chosen to be 32. One of the main reasons for that is this allows flexibility in changing the bit widths of the activations and weights
(e.g., from 4-bit to 8-bit and vice versa). Nevertheless, if we are not concerned with flexibility, then when quantizing the activations and weights to less than 8 bits, we can use an accumulator with less than 32 bits. The required accumulator bit width B can be calculated by
B = bx + bw + 1 + log ( k2Cin ) , (20)
where k is the convolution kernel size, and Cin is the number of input channels. In Table 6 we analyze the case of ResNet networks. We choose the layer with the largest value of k2Cin, which is 3x3x512 (Table 1 in He et al. (2016)). We calculate the required bit width for the accumulator (e.g., B) when the activations and weights are quantized to 2-6 bits. In addition, we calculate the power save in [%] when switching to unsigned arithmetic. As can be seen, even with smaller accumulator bit widths, switching to unsigned arithmetic leads to a significant saving in power. This is also visually
illustrated in Fig. 12, where we repeat the experiment of Fig. 1 bit with a 17 bit accumulator for the 2-bit networks, and with a 21 bit accumulator for the 4-bit networks.

heading: A.4 OBSERVATION-2
text: We now analyze the case where the inputs of the multiplier have different bit widths, bx and bw. In Fig. 9 we show the average bit toggles in signed and unsigned Booth encoder multiplication, for bw ≤ bx. In Fig. 10 we show the same analysis for the simple serial multiplier. We observe that when working with signed numbers (the common setting), the power is mostly affected by the larger bit width (bx in this case).
In the case of unsigned numbers, there is some power save when reducing one of the bit widths. In other words, Eq. (7) in the paper is accurate for the popular signed case and behaves as an upper bound for the unsigned case. The difference between Eq. (7) and the actual power consumption in the unsigned setting is more dominant for the simple serial multiplier. Therefore, in certain settings,
there is an additional benefit of switching to unsigned arithmetic, which we did not report in the experiments in the paper (i.e. the horizontal arrows in Fig. 1 should actually be slightly longer in some cases). Yet, this effect is relatively small compared to the reduction in bit flips in the accumulator.
We validated our observation on the 5-nm silicon process setup. We used 8 × 8 multiplier and measured the power when one of the inputs was drawn uniformly from [0, 27) and the other from [0, 23). We got 95% of the power that was measured when both inputs were drawn from [0, 27). In case of signed values, when one of the inputs was drawn uniformly from [−27, 27) and the other from [−23, 23) we observed 100% of the power that was measured when both inputs were drawn from [−27, 27). Note that in order to avoid changing the multiplier’s architecture, when switching to unsigned numbers, we use only half the range allowed by the bit width bx, i.e. [0, 2bx−1). Therefore, we obtain a representation with half of the 2bx levels of the signed case (note that we also need to replace −2bx−1 with −2bx−1 + 1). If we permit architectural changes, then a better way to represent unsigned numbers would be to replace the signed multiplier by a (bx + 1)× (bx + 1) multiplier that can support unsigned and signed multiplications but consumes much more power, or to work with a bx × bx unsigned multiplier that allow representation in the full interval of [0, 2bx) and then will follow Eq. (7) again.

heading: A.5 ADDITIONAL RESULTS

heading: A.5.1 POST TRAINING QUANTIZATION
text: Similarly to Table 2, we now examine PANN’s performance on additional networks. We use the same methods for activations quantization as in Table 2: ACIQ6 (Banner et al., 2019), ZeroQ7 (Cai et al., 2020), GDFQ8 (Shoukai et al., 2020) and BRECQ9 (Li et al., 2021). We also add Dynamic Quantization, which quantizes the activation and weights on the fly at inference time, according to their dynamic ranges. The results are reported in Tables 7-9.
We first change all networks to work with unsigned arithmetic and measure the classification accuracy, which serves as a baseline (see left side of each method ‘Base.’). As mentioned, this step already saves a significant amount of power, without any change in classification accuracy, compared to signed MAC arithmetic. We use the unsigned MAC power consumption as the power budget P and follow Alg. 1. We report the classification accuracy on the right side of each columns (see ‘Our’).

heading: 116 (8) 69.77 69.78 69.61 69.67 69.67 69.68 69.75 69.71 69.73 69.72
text: In figures 13-14, we demonstrate PANN at post training for different networks under power constrains of 4-bit and 2-bit unsigned MAC. In each figure, we start by running the specified approach to
6https://github.com/submission2019/cnn-quantization 7https://github.com/amirgholami/ZeroQ 8https://github.com/xushoukai/GDFQ 9https://github.com/yhhhli/BRECQ
quantize the bits and the activation to 4 or 2 bits (see caption). We measure the power in bit flips. Specifically, the power of each signed MAC is calculated by Pmult + Pacc (Eqs. (1),(2)) times the number of MACs in the network. Then, we switch to unsigned arithmetic (←). In this case the power is calculated by P umult + P u acc (Eqs. (3),(4)) times the number of MACs in the network. For simplicity, let us denote the total power as P u. As can be seen, this stage save power but does not change the accuracy (points marked as ‘Unsigned’). Constraining to the same power budget P u, we apply PANN (↑). Using Alg. 1, we calculate the optimal activation bit width and the corresponding addition factor (points marked as ‘PANN’). We can see a dramatic improvement in classification accuracy without any change in the power consumption.

heading: A.5.2 QUANTIZATION AWARE TRAINING
text: In Table 10, we report the classification accuracy of different networks on ImageNet, when we use PANN during training. Each row defines a specific power budget, corresponding to 2-bit, 3-bit and
4-bit unsigned MACs (weights and activations have equal bit widths). We compare our results to LSQ (Esser et al., 2019), whose accuracy is reported in parentheses. Note that the total number of bit flips differs between the networks, because each has a different number of MACS in its forward pass. Therefore, instead of specifying results as a function of the total number of bit flips, we report results as a function of the bit-width. Each bit-width defines a power budget for which we tune PANN (Alg. 1), where we use LSQ to quantize the activations. We can see that PANN outperforms LSQ at all power budgets.

heading: A.5.3 ADDITIONAL COMPARISONS TO MULTIPLICATION-FREE METHODS
text: In tables 11-12 we report additional comparisons with the recent multiplication free methods ShiftAddNet (You et al., 2020) and AdderNet (Chen et al., 2020), this time on the CIFAR100 and MHEALTH Banos et al. (2014) datasets. Here again PANN is used during training, like the competing methods.

heading: A.6 HYPER PARAMETERS FOR PANN IN QAT
text: A.6.1 LSQ
In all experiments we used the SGD optimizer with momentum of 0.9 and weight decay of 10−4. We used the softmax cross entropy loss. Unlike the original paper, we started the training from
pre-trained networks and with a smaller initial learning rate of 10−3. Please refer to Table 13 for PANN details, scheduling and number of epochs used for the training.

heading: A.6.2 MULTIPLIER FREE APPROACHES
text: In all experiments we followed the training regime described in (You et al., 2020). Specifically, for CIFAR10 or CIFAR100 we used a batch size of 256, and 160 epochs. The initial learning rate was 0.1 and then divided by 10 at the 80-th and the 120-th epoch. We used the SGD optimizer with momentum of 0.9 and weight decay of 1e− 4. For the MHEALTH dataset, we used only 40 epochs to train. The initial learning rate was 0.01 and then divided by 10 at the 20-th and the 30-th epochs.
Similarly to the CIFAR experiments, we used SGD optimizer with momentum of 0.9 and weight decay of 1e− 4.

heading: A.7 HARDWARE-ACCURACY TRADE-OFF
text: When operating on a single image at inference time (rather than on a large batch), the memory footprint of the weights is not negligible anymore (Mishra et al., 2017). Therefore, we need to also account for the bit-width required for storing the quantized weights. In Table 14 we report the optimal activation bit width and addition factor for each power constraint in a certain setting. Specifically, we use ZeroQ to quantize the activations of a pre-trained full-precision ResNet-50. We then measure the maximal addition factor per neuron, which defines the bit width bR required to store the weights. We can observe that overall, the increase in the runtime memory footprint of the weights is relatively low, especially in the low power regimes.
Up to now, we have only shown results with the bit width b̃x (and corresponding additions factor R) that is optimal in terms of classification accuracy. However, for a given power budget P , choosing b̃x (and R) can be done while also accounting for other factors, like latency and memory footprint. We illustrate this in Table 15 for the case of a power constraing corresponding to 2-bit MAC. Here, we report results for all options for b̃x and R that conform to that power budget. While b̃x = 6 and R = 1.16 is optimal in terms of classification accuracy, the user can choose other options, e.g., according to latency or memory constraints.

heading: A.8 QUANTIZATION ERROR ANALYSIS
text: Figure 15 (first row) depicts MSEPANN (Eq. (19)) as a function of b̃x for several power budgets P . First, it can be seen that our theoretical analysis agrees well with simulations. Second, it shows that the optimal b̃x (where the minimum MSE is attained) increases with the power budget. This implies that at higher power budgets, it is preferable to increase the bit width of the activations on the expense of reducing the number of additions R. The second row of the figure illustrates that the qualitative conclusions drawn from our uniform distribution analysis also hold when the weights are Gaussian and the activations are Gaussian numbers after a ReLU function (here we used the ACIQ quantizer (Banner et al., 2019)). In the third row, we show that similar behaviors characterise the error rates of a ResNet-18 model for ImageNet classification when using PANN to quantize its weights and ACIQ to quantize the activations.
A.9 PROOF OF EQ. (14)
Let w and x be statistically independent random vectors, each with iid components. Recall that wq and xq are obtained by applying a scalar function on each of the elements of w and x, respectively. Therefore wq and xq also have iid components. We assume that w = wq + εw and x = xq + εx, where E[εw|w] = 0 and E[εx|x] = 0. Then we have that
MSE = E [( wTx−wTq xq )2] = E [( wTx− (w + εw)T (x+ εx)
)2] = E [( wTx− (wTx+wTεx + εTwx+ εTwεx)
)2] = E [( wTεx + ε T wx+ ε T wεx )2] (21)
and therefore,
MSE = E [( wTεx )2] + E [( εTwx )2] + E [( εTwεx )2] + 2E [ wTεxε T wx ] + 2E [ wT εxε T wεx ] + 2E [ εTwxε T wεx ] . (22)
We now turn to show that the last three terms equal zero. For the first of those terms, we have
E [ wT εxε T wx ] = E [ wT εxx T εw ]
= E [ E [ wT εxx T εw |w, εw ]]
= E [ wTE [ εxx T |w, εw ] εw ]
= E [ wTE [ εxx T ] εw ]
= 0, (23)
where in the second line we used the law of total expectations, in the fourth line we used the fact that the pair {w, εw} is statistically independent of the pair {x, εx}, and in the fifth line we used the fact that E[εxxT ] = E[E[εxxT |x]] = E[E[εx|x]xT ] = 0 because of our assumption that E[εx|x] = 0. For the second among the last three terms in (22), we have that
E [ wT εxε T wεx ] = E [ εTxwε T wεx ] = E [[ εTxwε T wεx | εx
]] = E [ εTx [ wεTw | εx ] εx ]
= E [ εTx [ wεTw ] εx ]
= 0, (24)
where we used the fact that the pair {w, εw} is independent of εx, and E[wεTw] = E[E[wεTw|w]] = E[wE[εw|w]T ] = 0 because of our assumption that E[εw|w] = 0. For the the last term in (22), we have that
E [ εTwxε T wεx ] = E [ εTwxε T xεw ] = E [ E [ εTwxε T xεw | εw
]] = E [ εTwE [ xεTx | εw ] εw ]
= E [ εTwE [ xεTx ] εw ]
= 0, (25)
where we used the fact that the pair {x, εx} is independent of εw, and E[xεTx ] = 0, as in (23). We thus remain only with the first three terms of (22), so that
MSE = E [( wTεx )2] + E [( εTwx )2] + E [( εTwεx )2] = E [ wTεxε T xw ] + E [ xTεwε T wx ] + E [ εTwεxε T xεw
] = E [ E [ wTεxε T xw |w ]] + E [ E [ xTεwε T wx |x ]] + E [ E [ εTwεxε T xεw | εw
]] = E [ wTE [ εxε T x |w ] w ] + E [ xTE [ εwε T w |x ] x ] + E [ εTwE [ εxε T x | εw ] εw ]
= E [ wTE [ εxε T x ] w ] + E [ xTE [ εwε T w ] x ] + E [ εTwE [ εxε T x ] εw ]
= E [ wT ( σ2εxI ) w ] + E [ xT ( σ2εwI ) x ] + E [ εTw ( σ2εxI ) εw ]
= d ( σ2wσ 2 εx + σ 2 xσ 2 εw + σ 2 εxσ 2 εw ) , (26)
where σ2w, σ 2 x, σ 2 εw , and σ 2 εx denote the second-order moments of the elements of w, x, εw, and εx, respectively, and I is the d × d identity matrix. Here, in the fifth equality we used the fact that w is independent of εx, x is independent of εw, and εw is independent of εx. In the sixth equality we used the fact that εx and εw are iid vectors with zero mean, since E[εx] = E[E[εx|x]] = 0 and E[εw] = E[E[εw|w]] = 0. This completes the proof of Eq. (14) in the main text.

Review summaries:
The authors observed that the power consumption is dominated by the bit toggling at the input of the accumulator and decreasing the bit-width of only the weights or only the activations has limited benefit to reduce the the power consumed by the multiplier. The paper proposed PANN, which uses tricks such as unsigned arithmetic in CNN and implement multiplications via additions to achieve a multiplier-free and power-aware neural network. Experiments on both post-training quantization and quantization-aware training showcased better performance compared to other works under the same number of bit-flips. However, the reviewer believes there are fundamental flaws that need to be thoroughly addressed before this paper is ready for ICLR. 

The paper argues that power consumption is a major obstacle in deploying DNNs to end devices and that current quantization approaches do not take power consumption directly into account and therefore are not optimal in reducing it. Using an approximate power model based on the average number of bit flips, they make two observation that are frequently overlooked by existing quantization approaches:  1) A significant portion of the power consumption of the MAC operation is due to the usage of signed integers and using unsigned integers instead can significantly reduce the power consumption. 2) The multipliers power consumption is dominated by the larger bit widths (weight or activation) and therefore using lower bit for one of the two (e.g. weights) is not power efficient.  Based on this the authors introduce a new weight quantization approach (PANN) which removes the multiply operation and replaces it with additions. This allows PANN to efficiently reduces the power consumption. For the same power budget, PANN achieves significantly higher accuracy (or effective bit width), but comes at the cost of higher latency and memory usage.