WrapNet:  Neural Net Inference with Ultra-Low-Precision Arithmetic

1 INTRODUCTION . Significant progress has been made in quantizing ( or even binarizing ) neural networks , and numerous methods have been proposed that reduce the precision of weights , activations , and even gradients while retaining high accuracy ( Courbariaux et al. , 2016 ; Hubara et al. , 2016 ; Li et al. , 2016 ; Lin et al. , 2017 ; Rastegari et al. , 2016 ; Zhu et al. , 2016 ; Dong et al. , 2017 ; Zhu et al. , 2018 ; Choi et al. , 2018a ; Zhou et al. , 2016 ; Li et al. , 2017 ; Wang et al. , 2019 ; Jung et al. , 2019 ; Choi et al. , 2018b ; Gong et al. , 2019 ) . Such quantization strategies make neural networks more hardware-friendly by leveraging fast , integer-only arithmetic , replacing multiplications with simple bit-wise operations , and reducing memory requirements and bandwidth . Unfortunately , the gains from quantization are limited because quantized networks still require high-precision arithmetic . Even if weights and activations are represented with just one bit , deep feature computation requires the summation of hundreds or even thousands of products . Performing these summations with low-precision registers results in integer overflow , contaminating downstream computations and destroying accuracy . Moreover , as multiplication costs are slashed by quantization , high-precision accumulation starts to dominate the arithmetic cost . Indeed , our own hardware implementations show that an 8-bit× 8-bit multiplier consumes comparable power and silicon area to a 32-bit accumulator . When reducing the precision to a 3-bit× 1-bit multiplier , a 32-bit accumulator consumes more than 10× higher power and area ; see Section 4.5 . Evidently , low-precision accumulators are the key to further accelerating quantized nets . In custom hardware , low-precision accumulators reduce area and power requirements while boosting throughput . On general-purpose processors , where registers have fixed size , low-precision accumulators are exploited through bit-packing , i.e. , by representing multiple low-precision integers side-by-side within a single high-precision register ( Pedersoli et al. , 2018 ; Rastegari et al. , 2016 ; Bulat & Tzimiropoulos , 2019 ) . Then , a single vector instruction is used to perform the same operation across all of the packed numbers . For example , a 64-bit register can be used to execute eight parallel 8-bit additions , thus increasing the throughput of software implementations . Hence , the use of low-precision accumulators is advantageous for both hardware and software implementations , provided that integer overflow does not contaminate results . We propose WrapNet , a network architecture with extremely low-precision accumulators . WrapNet exploits the fact that integer computer arithmetic is cyclic , i.e , numbers are accumulated until they reach the maximum representable integer and then “ wrap around ” to the smallest representable integer . To deal with such integer overflows , we place a differentiable cyclic ( periodic ) activation function immediately after the convolution ( or linear ) operation , with period equal to the difference between the maximum and minimum representable integer . This strategy makes neural networks resilient to overflow as the activations of neurons are unaffected by overflows during convolution . We explore several directions with WrapNet . On the software side , we consider the use of bitpacking for processors with or without dedicated vector instructions . In the absence of vector instructions , overflows in one packed integer may produce a carry bit that contaminates its neighboring value . We propose training regularizers that minimize the effects of such contamination artifacts , resulting in networks that leverage bit-packed computation with very little impact on final accuracy . For processors with vector instructions , we modify the Gemmlowp library ( Jacob et al. , 2016 ) to operate with 8-bit accumulators . Our implementation achieves up to 2.4× speed-up compared to a 32-bit accumulator implementation , even when lacking specialized instructions for 8-bit multiplyaccumulate . We also demonstrate the efficacy of WrapNet in terms of cycle time , area , and energy efficiency when considering custom hardware designs in a commercial 28 nm CMOS technology . 2 RELATED WORK AND BACKGROUND . 2.1 NETWORK QUANTIZATION . Network quantization aims at accelerating inference by using low-precision arithmetic . In its most extreme form , weights and activations are both quantized using binary or ternary quantizers . The binary quantizer Qb corresponds to the sign function , whereas the ternary quantizer Qt maps some values to zero . Multiplications in binarized or ternarized networks ( Hubara et al. , 2016 ; Courbariaux et al. , 2015 ; Lin et al. , 2017 ; Rastegari et al. , 2016 ; Zhu et al. , 2016 ) can be implemented using bitwise logic , leading to impressive acceleration . However , training such networks is challenging since fewer than 2 bits are used to represent activations and weights , resulting in a dramatic impact on accuracy compared to full-precision models . Binary and ternary networks are generalized to higher precision via uniform quantization , which has been shown to result in efficient hardware ( Jacob et al. , 2018 ) . The multi-bit uniform quantizer Qu is given by : Qu ( x ) = round ( x/∆x ) ∆x , where ∆x denotes the quantization step-size . The output of the quantizer is a floating-point number x that can be expressed as x = ∆xxq , where xq is the fixed-point representation of x . The fixed-point number xq has a “ precision ” or “ bitwidth , ” which is the number of bits used to represent it . Note that the range of floating-point numbers representable by the uniform quantizer Qu depends on both the quantization step-size ∆x and the quantization precision . Nonetheless , the number of different values that can be represented by the same quantizer depends only on the precision . Applying uniform quantization to both weights w = ∆wwq and activations x = ∆xxq simplifies computations , as an inner-product simply becomes z = ∑ i wixi = ∑ i ( ∆w ( wq ) i ) ( ∆x ( xq ) i ) = ( ∆w∆x ) ∑ i ( wq ) i ( xq ) i = ∆zzq . ( 1 ) The key advantage of uniform quantization is that the core computation ∑ i ( wq ) i ( xq ) i can be carried out using fixed-point ( i.e. , integer ) arithmetic only . Results in ( Gong et al. , 2019 ; Choi et al. , 2018b ; Jung et al. , 2019 ; Wang et al. , 2019 ; Mishra et al. , 2017 ; Mishra & Marr , 2017 ) have shown that high classification accuracy is attainable with low-bitwidth uniform quantization , such as 2 or 3 bits . Although ( wq ) i , ( xq ) i , and their product may have extremely low-precision , the accumulated result zq of many of these products has very high dynamic range . As a result , high-precision accumulators are typically required to avoid overflows , which is the bottleneck for further arithmetic speedups . 2.2 LOW-PRECISION ACCUMULATION . Several approaches have been proposed that use accumulators with fewer bits to obtain speed-ups . For example , reference ( Khudia et al. , 2021 ) splits the weights into two separate matrices , one with small- and another with large-magnitude entries . If the latter matrix is sparse , acceleration is attained as most computations rely on fast , low-precision operations . However , to significantly reduce the accumulator ’ s precision , one would need to severely decrease the magnitude of the entries of the first matrix , which would , in turn , prevent the second matrix from being sufficiently sparse to achieve acceleration . Recently , ( de Bruin et al. , 2020 ) proposed using layer-dependent quantization parameters to avoid overflowing accumulators with fixed precision . Fine-tuning is then used to improve performance . However , if the accumulator precision is too low ( e.g. , 8 bits or less ) , the optimized precision of activations and weights is too coarse to attain satisfactory performance . Another line of work ( Sakr et al. , 2019 ; Micikevicius et al. , 2017 ; Wang et al. , 2018 ) uses 16-bit floating-point accumulators for training and inference—such approaches typically require higher complexity than methods based on fixed-point arithmetic . 2.3 THE IMPACT OF INTEGER OVERFLOW . Overflow is a major problem , especially in highly quantized networks . Table 1 demonstrates that overflows occur in around 11 % of the neurons in a network with 3-bit activations ( A ) and binary weights ( W ) that is using 8-bit accumulators for inference after being trained on CIFAR-10 with standard precision . Clearly , overflow has a significant negative impact on accuracy . Table 1 shows that if we use an 8-bit ( instead of a 32-bit ) accumulator , then the accuracy of a binary-weight network with 2-bit activations drops by more than 40 % , even when only 1.72 % neurons overflow . If we repeat the experiment with 3-bit activations and binary weights , the accuracy is only marginally better than a random guess . Therefore , existing methods try to avoid integer overflow by using accumulators with relatively high precision , and pay a correspondingly high price when doing arithmetic . 3 WRAPNET : DEALING WITH INTEGER OVERFLOWS . We now introduce WrapNet , which includes a cyclic activation function and an overflow penalty , enabling neural networks to use low-precision accumulators . We also present a modified quantization step-size selection strategy for activations , which retains high classification accuracy . Finally , we show how further speed-ups can be achieved on processors with or without specialized vector instructions . We propose training a network with layers that emulate integer overflows on the fixed-point preactivations zq to maintain high accuracy . However , directly training a quantized network with an overflowing accumulator diverges ( see Table 2 ) due to the discontinuity of the modulo operation . To facilitate training , we insert a cyclic “ smooth modulo ” activation immediately after every linear/convolutional layer , which not only captures the wrap-around behavior of overflows , but also ensures that the activation is continuous everywhere . The proposed smooth modulo activation c is a composite function of a modulo function m and a basis function f that ensures continuity . Specifically , given a b-bit accumulator , our smooth-modulo c for fixed-point inputs is as follows : f ( m ) = m , for − kk+12 b−1 ≤ m ≤ kk+12 b−1 −k2b−1 − km , for m < − kk+12 b−1 k2b−1 − km , for m > kk+12 b−1 c ( zq ) = f ( mod ( zq + 2b−1 , 2b ) − 2b−1 ) , where k is a hyper-parameter that controls the slope of the transition . Note that we apply constant shifts to keep the input of f in [ −2b−1 , 2b−1 ) . Figure 1a illustrates the smooth modulo function with two different slopes k = 1 , 4 . As k increases , the cyclic activation becomes more similar to the modulo operator and has a greater range , but the transition becomes more abrupt . Since our cyclic activation is continuous and differentiable almost everywhere , standard gradient-based learning can be applied easily . A convolutional block with cyclic activation layer is shown in Figure 1b . After the convolution result goes into the cyclic activation , the result is multiplied by ∆z to compute a floating-point number , which is then processed through BatchNorm and ReLU . A fixed per-layer quantization step-size is then used to convert the floating-point output of the ReLU into a fixed-point input for the next layer . We detail the procedure to find this step-size in Section 3.2 .

Summary: This paper explores to solve an often ignored issue in quantization: accumulation precision. As the bit-width of input scales down, the area/energy cost of the accumulator starts to dominate.  The cyclic method proposed by the authors at the first glance is not intuitive. However, it's surprising that the surveyed models could be tuned to live with significant overflows---as long as it can be tuned, which is enabled by the "differentiable overflow" brought by the cyclic method.  There are several issues to be addressed before the paper can be accepted: