WrapNet:  Neural Net Inference with Ultra-Low-Precision Arithmetic

1 INTRODUCTION . Significant progress has been made in quantizing ( or even binarizing ) neural networks , and numerous methods have been proposed that reduce the precision of weights , activations , and even gradients while retaining high accuracy ( Courbariaux et al. , 2016 ; Hubara et al. , 2016 ; Li et al. , 2016 ; Lin et al. , 2017 ; Rastegari et al. , 2016 ; Zhu et al. , 2016 ; Dong et al. , 2017 ; Zhu et al. , 2018 ; Choi et al. , 2018a ; Zhou et al. , 2016 ; Li et al. , 2017 ; Wang et al. , 2019 ; Jung et al. , 2019 ; Choi et al. , 2018b ; Gong et al. , 2019 ) . Such quantization strategies make neural networks more hardware-friendly by leveraging fast , integer-only arithmetic , replacing multiplications with simple bit-wise operations , and reducing memory requirements and bandwidth . Unfortunately , the gains from quantization are limited because quantized networks still require high-precision arithmetic . Even if weights and activations are represented with just one bit , deep feature computation requires the summation of hundreds or even thousands of products . Performing these summations with low-precision registers results in integer overflow , contaminating downstream computations and destroying accuracy . Moreover , as multiplication costs are slashed by quantization , high-precision accumulation starts to dominate the arithmetic cost . Indeed , our own hardware implementations show that an 8-bit× 8-bit multiplier consumes comparable power and silicon area to a 32-bit accumulator . When reducing the precision to a 3-bit× 1-bit multiplier , a 32-bit accumulator consumes more than 10× higher power and area ; see Section 4.5 . Evidently , low-precision accumulators are the key to further accelerating quantized nets . In custom hardware , low-precision accumulators reduce area and power requirements while boosting throughput . On general-purpose processors , where registers have fixed size , low-precision accumulators are exploited through bit-packing , i.e. , by representing multiple low-precision integers side-by-side within a single high-precision register ( Pedersoli et al. , 2018 ; Rastegari et al. , 2016 ; Bulat & Tzimiropoulos , 2019 ) . Then , a single vector instruction is used to perform the same operation across all of the packed numbers . For example , a 64-bit register can be used to execute eight parallel 8-bit additions , thus increasing the throughput of software implementations . Hence , the use of low-precision accumulators is advantageous for both hardware and software implementations , provided that integer overflow does not contaminate results . We propose WrapNet , a network architecture with extremely low-precision accumulators . WrapNet exploits the fact that integer computer arithmetic is cyclic , i.e , numbers are accumulated until they reach the maximum representable integer and then “ wrap around ” to the smallest representable integer . To deal with such integer overflows , we place a differentiable cyclic ( periodic ) activation function immediately after the convolution ( or linear ) operation , with period equal to the difference between the maximum and minimum representable integer . This strategy makes neural networks resilient to overflow as the activations of neurons are unaffected by overflows during convolution . We explore several directions with WrapNet . On the software side , we consider the use of bitpacking for processors with or without dedicated vector instructions . In the absence of vector instructions , overflows in one packed integer may produce a carry bit that contaminates its neighboring value . We propose training regularizers that minimize the effects of such contamination artifacts , resulting in networks that leverage bit-packed computation with very little impact on final accuracy . For processors with vector instructions , we modify the Gemmlowp library ( Jacob et al. , 2016 ) to operate with 8-bit accumulators . Our implementation achieves up to 2.4× speed-up compared to a 32-bit accumulator implementation , even when lacking specialized instructions for 8-bit multiplyaccumulate . We also demonstrate the efficacy of WrapNet in terms of cycle time , area , and energy efficiency when considering custom hardware designs in a commercial 28 nm CMOS technology . 2 RELATED WORK AND BACKGROUND . 2.1 NETWORK QUANTIZATION . Network quantization aims at accelerating inference by using low-precision arithmetic . In its most extreme form , weights and activations are both quantized using binary or ternary quantizers . The binary quantizer Qb corresponds to the sign function , whereas the ternary quantizer Qt maps some values to zero . Multiplications in binarized or ternarized networks ( Hubara et al. , 2016 ; Courbariaux et al. , 2015 ; Lin et al. , 2017 ; Rastegari et al. , 2016 ; Zhu et al. , 2016 ) can be implemented using bitwise logic , leading to impressive acceleration . However , training such networks is challenging since fewer than 2 bits are used to represent activations and weights , resulting in a dramatic impact on accuracy compared to full-precision models . Binary and ternary networks are generalized to higher precision via uniform quantization , which has been shown to result in efficient hardware ( Jacob et al. , 2018 ) . The multi-bit uniform quantizer Qu is given by : Qu ( x ) = round ( x/∆x ) ∆x , where ∆x denotes the quantization step-size . The output of the quantizer is a floating-point number x that can be expressed as x = ∆xxq , where xq is the fixed-point representation of x . The fixed-point number xq has a “ precision ” or “ bitwidth , ” which is the number of bits used to represent it . Note that the range of floating-point numbers representable by the uniform quantizer Qu depends on both the quantization step-size ∆x and the quantization precision . Nonetheless , the number of different values that can be represented by the same quantizer depends only on the precision . Applying uniform quantization to both weights w = ∆wwq and activations x = ∆xxq simplifies computations , as an inner-product simply becomes z = ∑ i wixi = ∑ i ( ∆w ( wq ) i ) ( ∆x ( xq ) i ) = ( ∆w∆x ) ∑ i ( wq ) i ( xq ) i = ∆zzq . ( 1 ) The key advantage of uniform quantization is that the core computation ∑ i ( wq ) i ( xq ) i can be carried out using fixed-point ( i.e. , integer ) arithmetic only . Results in ( Gong et al. , 2019 ; Choi et al. , 2018b ; Jung et al. , 2019 ; Wang et al. , 2019 ; Mishra et al. , 2017 ; Mishra & Marr , 2017 ) have shown that high classification accuracy is attainable with low-bitwidth uniform quantization , such as 2 or 3 bits . Although ( wq ) i , ( xq ) i , and their product may have extremely low-precision , the accumulated result zq of many of these products has very high dynamic range . As a result , high-precision accumulators are typically required to avoid overflows , which is the bottleneck for further arithmetic speedups . 2.2 LOW-PRECISION ACCUMULATION . Several approaches have been proposed that use accumulators with fewer bits to obtain speed-ups . For example , reference ( Khudia et al. , 2021 ) splits the weights into two separate matrices , one with small- and another with large-magnitude entries . If the latter matrix is sparse , acceleration is attained as most computations rely on fast , low-precision operations . However , to significantly reduce the accumulator ’ s precision , one would need to severely decrease the magnitude of the entries of the first matrix , which would , in turn , prevent the second matrix from being sufficiently sparse to achieve acceleration . Recently , ( de Bruin et al. , 2020 ) proposed using layer-dependent quantization parameters to avoid overflowing accumulators with fixed precision . Fine-tuning is then used to improve performance . However , if the accumulator precision is too low ( e.g. , 8 bits or less ) , the optimized precision of activations and weights is too coarse to attain satisfactory performance . Another line of work ( Sakr et al. , 2019 ; Micikevicius et al. , 2017 ; Wang et al. , 2018 ) uses 16-bit floating-point accumulators for training and inference—such approaches typically require higher complexity than methods based on fixed-point arithmetic . 2.3 THE IMPACT OF INTEGER OVERFLOW . Overflow is a major problem , especially in highly quantized networks . Table 1 demonstrates that overflows occur in around 11 % of the neurons in a network with 3-bit activations ( A ) and binary weights ( W ) that is using 8-bit accumulators for inference after being trained on CIFAR-10 with standard precision . Clearly , overflow has a significant negative impact on accuracy . Table 1 shows that if we use an 8-bit ( instead of a 32-bit ) accumulator , then the accuracy of a binary-weight network with 2-bit activations drops by more than 40 % , even when only 1.72 % neurons overflow . If we repeat the experiment with 3-bit activations and binary weights , the accuracy is only marginally better than a random guess . Therefore , existing methods try to avoid integer overflow by using accumulators with relatively high precision , and pay a correspondingly high price when doing arithmetic . 3 WRAPNET : DEALING WITH INTEGER OVERFLOWS . We now introduce WrapNet , which includes a cyclic activation function and an overflow penalty , enabling neural networks to use low-precision accumulators . We also present a modified quantization step-size selection strategy for activations , which retains high classification accuracy . Finally , we show how further speed-ups can be achieved on processors with or without specialized vector instructions . We propose training a network with layers that emulate integer overflows on the fixed-point preactivations zq to maintain high accuracy . However , directly training a quantized network with an overflowing accumulator diverges ( see Table 2 ) due to the discontinuity of the modulo operation . To facilitate training , we insert a cyclic “ smooth modulo ” activation immediately after every linear/convolutional layer , which not only captures the wrap-around behavior of overflows , but also ensures that the activation is continuous everywhere . The proposed smooth modulo activation c is a composite function of a modulo function m and a basis function f that ensures continuity . Specifically , given a b-bit accumulator , our smooth-modulo c for fixed-point inputs is as follows : f ( m ) = m , for − kk+12 b−1 ≤ m ≤ kk+12 b−1 −k2b−1 − km , for m < − kk+12 b−1 k2b−1 − km , for m > kk+12 b−1 c ( zq ) = f ( mod ( zq + 2b−1 , 2b ) − 2b−1 ) , where k is a hyper-parameter that controls the slope of the transition . Note that we apply constant shifts to keep the input of f in [ −2b−1 , 2b−1 ) . Figure 1a illustrates the smooth modulo function with two different slopes k = 1 , 4 . As k increases , the cyclic activation becomes more similar to the modulo operator and has a greater range , but the transition becomes more abrupt . Since our cyclic activation is continuous and differentiable almost everywhere , standard gradient-based learning can be applied easily . A convolutional block with cyclic activation layer is shown in Figure 1b . After the convolution result goes into the cyclic activation , the result is multiplied by ∆z to compute a floating-point number , which is then processed through BatchNorm and ReLU . A fixed per-layer quantization step-size is then used to convert the floating-point output of the ReLU into a fixed-point input for the next layer . We detail the procedure to find this step-size in Section 3.2 .

Summary: The paper introduces a new neural network layer that enables training NNs with quantized activations using reduced bit-width accumulators. The cyclic activation layer makes overflows smooth, instead of being discontinuous and this enables achieving better accuracy on quantized networks on reduced bit-width accumulators. They also introduce overflow and carry penalties to dissuade the training regime from reaching overflow states.