WrapNet:  Neural Net Inference with Ultra-Low-Precision Arithmetic
Authors: Renkun Ni
Hong-min Chu
Oscar Castaneda
Ping-yeh Chiang
Christoph Studer
Tom Goldstein
Venue: ICLR 2021 Poster (2021)
Keywords: quantization
efficient inference

Abstract: Low-precision neural networks represent both weights and activations with few bits, drastically reducing the cost of multiplications. Meanwhile, these products are accumulated using high-precision (typically 32-bit) additions.  Additions dominate the arithmetic complexity of inference in quantized (e.g., binary) nets, and high precision is needed to avoid overflow. To further optimize inference, we propose WrapNet, an architecture that adapts neural networks to use low-precision (8-bit) additions while achieving classification accuracy comparable to their 32-bit counterparts. We achieve resilience to low-precision accumulation by inserting a cyclic activation layer that makes results invariant to overflow. We demonstrate the efficacy of our approach using both software and hardware platforms.

Published as a conference paper at ICLR 2021
WRAPNET: NEURAL NET INFERENCE WITH ULTRA-LOW-PRECISION ARITHMETIC
Renkun Ni University of Maryland rn9zm@ cs. umd. edu
Hong-min Chu University Maryland hmchuêcs, umd. edu
Oscar Castañeda
ETH Zurich caoscar@ethz.com
Ping-yeh Chiang University of Maryland pchiangécs. umd. edu
Christoph Studer ETH Zurich studereethz.ch
Tom Goldstein
University of Maryland tomg@cs und. edu
ABSTRACT
Low-precision neural networks represent both weights and activations with few bits, drastically reducing the cost of multiplications. Meanwhile, these products are accumulated using high-precision (typically 32-bit) additions. Additions dom- inate the arithmetic complexity of inference quantized (e.g., binary) nets, and high precision is needed to avoid overflow. To further optimize inference, we pro- pose WrapNet, an architecture that adapts neural networks to use low-precision (8-bit) additions while achieving classification accuracy comparable to their 32- bit counterparts. We achieve resilience to low-precision accumulation by inserting cyclic activation layer that makes results invariant overflow. We demonstrate the efficacy of our approach using both software and hardware platforms.
INTRODUCTION
Significant progress has been made in quantizing (or even binarizing) neural networks, and numer- ous methods have been proposed that reduce the precision of weights, activations, and even gradi- ents while retaining high accuracy (Courbariaux et al.) (Hbaraa et al. 2016
al.) [016]] Lin et al. 2017 Rastegari et al. 2016 Zhu et al. Dong et al. 2017 Zhu et al. 2018 Choi et al.) 2018a| Zhou et al. 2016 Li al. 2017 Wang et al., 2019 Jung al.) 2019 Choi et al. 2018b Gong et al. 2019). Such quantization strategies make neural networks more hardware-friendl by leveraging fast, integer-only arithmetic, replacing multiplications with simple bit-wise operations, and reducing memory requirements and bandwidth.
Unfortunately, gains from quantization are limited because quantized networks still require arithmetic. Even if weights and activations are represented with just bit, deep feature computation requires the summation of hundreds or even thousands of products. Perform- ing these summations with low-precision registers results in integer overflow, contaminating down- stream computations and destroying accuracy. Moreover, as multiplication costs are slashed by quantization, high-precision accumulation starts dominate the arithmetic cost. Indeed, our own hardware implementations show that an 8-bit x 8-bit multiplier consumes comparable power and silicon area to a 32-bit accumulator. When reducing the precision 32-bit accumulator consumes more than 10x higher power and area; see Section 4.5 Evidently, low-precision accumulators are the key to further accelerating quantized nets.
3-bit 1-bit multiplier,
In custom hardware, low- low-precision accumulators reduce area and power requirements while boost- ing throughput. On general- purpose processors, where registers have fixed size, low-precision ac- cumulators are exploited through bit-packing, i.e., by representing multiple low-precision integers side-by-side within a single high-precision register (Pedersoli et al. 2018 Rastegari et al. 2016 Bulat & Tzimiropoulos 2019). Then, a single vector instruction is used to perform the same oper- ation across all of the packed numbers. For example, a 64-bit register can be used to execute eight parallel 8-bit additions, thus increasing the throughput of software implementations, Hence, the of wow-precision accumulators advantageous for both hardware and software implementations, provided that integer overflow does not contaminate results.

---

Published as a conference paper at ICLR 2021
We propose WrapNet, network architecture with extremely low-precision accumulators. WrapNet exploits the fact that integer computer arithmetic is cyclic, i.e, numbers are accumulated until they reach the maximum representable integer and then "wrap around" to the smallest representable integer. To deal with such integer overflows, we place a differentiable cyclic (periodic) activation function immediately after the convolution (or linear) operation, with period equal the difference between the maximum and minimum representable integer. This strategy makes neural networks resilient to overflow as the activations of neurons are unaffected by overflows during convolution.
We explore several directions with WrapNet. On the software side, we consider the use of bit- packing for processors with or without dedicated vector instructions. In the absence of vector in- structions, overflows in one packed integer may produce carry bit that contaminates its neighboring value. We propose training regularizers that minimize the effects of such contamination artifacts, resulting in networks that leverage bit-packed computation with very little impact on final accuracy. For processors with vector instructions, we modify the Gemmlowp library (Jacob et al. 2016) to operate with 8-bit accumulators. Our implementation achieves up to 2.4x speed-up compared to 32-bit accumulator implementation, even when lacking specialized instructions for 8-bit multiply- accumulate. We also demonstrate the efficacy of WrapNet in terms of cycle time, area, and energy efficiency when considering custom hardware designs in a commercial 28 nm CMOS technology.
2
RELATED WORK AND BACKGROUND
2.1 NETWORK QUANTIZATION
Network quantization aims at accelerating inference by using low-precision arithmetic. In its most extreme form, weights and activations both quantized using binary or ternary quantizers. The binary quantizer Qo corresponds to the sign function, whereas the ternary quantizer Q. maps some values zero. Multiplications in binarized or ternarized networks (Hubara et al. 2016 Courbariaux et al. 2015 Lin et al., [017]] Rastegari et al. 2016 Zhu et al. 2016) can be implemented using bit- wise logic, leading impressive acceleration. However, training such networks is challenging since fewer than 2 bits are used to represent activations and weights, resulting in dramatic impact on accuracy compared to full-precision models.
Binary and ternary networks are generalized to higher precision via uniform quantization, which has been shown to result in efficient hardware (Jacob et al. 2018). The multi-bit uniform quantizer Qu is given by: (x) round z A.., where A. denotes the quantization step-size. The output of the quantizer is a floating-
number x that can be expressed as æ = AzTT. where 2g is the fixed- -oonttpont representation of æz The fixed-point number 2ig has "precision" or "bitwidth," which is the number of bits used to represent it. Note that the range of by the uniform quantizer Qu depends both the quantization step-size Ar and the quantization precision. Nonetheless, the number of different values that can be represented by the same quantizer depends only on the precision.
point numbers representable
Applying uniform quantization to both weights w = Aw and activations x = Agtt simplifies computations, as an inner-product simply becomes
was, Σ ((Au(
(A.Ae) [(wg)((ar) A.
The key advantage of uniform quantization is that the core computation Σ (wrg ) (Irg) can be carried out using fixed-point (i.e., integer) arithmetic only. Results in (Gong et al. 2019 Choi et al. 2018b) Jung et al. 2019 Wang et al. 2019 Mishra et al.) 2017 MMshra & Marr 2017) have shown that high classification accuracy is attainable with low-bitwidth uniform quantization, such as 2 or 3 bits. Although )s (Ig)ss and their product may have extremely low www-precision, the accumulated result 2y of many of these products has very high dynamic range. As result, high-precision accumulators are typically required to avoid overflows, which is the bottleneck for further arithmetic speedups.
2.2 LOW-PRECISION ACCUMULATION
Several approaches have been proposed that use accumulators with fewer bits to obtain speed-ups. For example, reference (Khudia al. 202T) splits the weights into two separate matrices, one with
2

---

Published as a conference paper at ICLR 2021
Table 1: Average overflow rate (in 8 bits) of each layer for low-precision network and correspond- ing test accuracy using either 32-bit or 8-bit accumulators during inference on CI.AR10.
Bit (A/W)
Overflow rate (8-bit)
Accuracy (32-bit)
Accuracy (8-bit)
full precision
92.45%
3/1
10.84%
91.08%
10.06% 44.04%
2/1
1.72%
88.46%
small- and another with large-magnine entries. If the latter matrix is sparse, acceleration attained most computations rely fast, low-precision operations. However, to significantly reduce the accumulator's precision, one would need to severely decrease the magnitude of the entries of the first matrix, which would, in turn, prevent the second matrix from being sufficiently sparse achieve acceleration. Recently, (de Bruin et al. 2020) proposed using layer-dependent quantization param- eters to avoid overflowing accumulators with fixed precision. Fine-tuning is then used improve performance. However, if the accumulator precision is too low (e.g., 8 bits or less), the optimized precision of activations and weights too coarse to attain satisfactory performance. Another line of work (Sakr et al. 2019] Micikevicius et al.) 2017; Wang et al.. (0018) uses 16-bit floating-pon ppint accumulators for training and inference such approaches typically require higher complexity than methods based on fixed-point arithmetic.
2.3 THE IMPACT OF INTEGER OVERFLOW
Overflow is major problem, especially in highly quantized networks. Table demonstrates that overflows occur in around 11% of the neurons in a network with 3-bit activations (A) and binary weights (W) that is using 8-bit accumulators for inference after being trained on CIFAR-I6 10 with standard precision. Clearly, overflow has significant negative impact on accuracy. Table [ shows that if we use an 8-bit (instead of a 32-bit) accumulator, then the accuracy of a binary-weight network with 2-bit activations drops by more than 40%, even when only 1.72% neurons overflow. we repeat the experiment with 3-bit activations and binary weights, the accuracy is only marginally better than a random guess. Therefore, existing methods try to avoid integer overflow by using accumulators with relatively high precision, and pay correspondingly high price when doing arithmetic.
WRAPNET: DEALING WITH INTEGER OVERFLOWS
We now introduce WrapNet, which includes cyclic activation function and an overflow penalty, enabling neural networks to use low-precision accumulators. We also present modified quantiza- tion step-size selection strategy for activations, which retains high classification accuracy. Finally, we show how further speed-ups can be achieved on processors with or without specialized vector instructions.
We propose training a network with layers that emulate integer overflows on the fixed-point pre- activations 2, to maintain high accuracy. However, directly training a quantized network with an overflowing accumulator diverges (see Table ( due to the discontinuity of the modulo operation. To facilitate training. we insert a cyclic "smooth modulo" activation immediately after every lin- car/convolutional layer, which not only captures the wrap-around behavior of overflows, but also ensures that the activation is continuous everywhere. The proposed smooth modulo activation c is a composite function of modulo function and a basis function that ensures continuity. Specifi- cally, given a b-bit accumulator, our smooth-modulo e for fixed-point inputs is as follows:
for 8+12" 2b-2-11 < m k 2b
m. --2b-1 1 km, for m < k+1
k+1
f(m) =
20-1
k2b-1 km,, for > k 20-1
ktI
c(22) = (mod(zz + 2 1 2 2 2
where k is a shifts to keep the input of in 2-2h-1, 20-1). Figure la ialltusteeees the smooth modulo function with
that controls the slope of the transition. Note that we apply constant
3

---

Published as a conference paper at ICLR 2021
Origial Cyclic km Cyclie lend
BatchWorm
Conv
Cycle
ReLU
16
(a)
(b)
Figure 1: (a) Example of the proposed cyclic activation with different slopes k and the original modulo operator for a 4-bit accumulator. (b) Convolutional block with proposed eyclic activation.
two different slopes 1, As increases, the cyclic activation becomes more similar to the modulo operator and has a greater range, but the transition becomes more abrupt. Since our cyclic activation is continuous and differentiable almost everywhere, standard gradient-basee learning can be applied easily. A convolutional block with cyclic activation layer is shown in Figure [b After the convolution result goes into the cyclic activation, the result is multiplied by A, to compute a paatiigg-pintt number, which is then processed through BatchNorm and ReLU. A fixed per-layer quantization step-size is then used to convert the floating-por point output of the ReLU into a fixed-point
input for the next layer. We detail the procedure to find this step-size in Section 3.2
3.1 OVERFLOW PENALTY
An alternative way to adapt quantized networks to low-precision accumulators is to directly reduce the amount of overflows. To achieve this, we propose a regularizer which penalizes outputs that exceed the bitwidth of the accumulation register. Concretely, for a b-bit accumulator, we define an overflow penalty for the I-th layer of the network as follows: Rt = (1/N) Σ, max [|x{|=/] 2 2 Here, 24 is the fixed-poine result 1 for the i-th neuron of the I-th layer, and N is the total number of neurons in the 1-th layer. The overflow penalty is imposed after every quantized linear layer and before the cyclic activation. All these penalties combined into one regularizer Ro E, RÇ.
3.2 SELECTION OF ACTIVATION QUANTIZATION STEP-SIZE SIZE
To keep multiplication simple, the soating-point output of ReLU must be quantized before it is fed into the following layer. However, shown in Table [ significant number of overflows occur even with 3-bit activations. From our experiments (see Table 33. we have observed that overflow occurs too frequently (i.e., on more than 10% of the neurons), then WrapNet starts to suffer significant accuracy degradation. However, if we reduce the activation precision so that no overflows happen all, several layers will have 1-bit activations (see Table 3 thereby increasing quantization errors and degrading accuracy. To balance accumulation and quantization errors, we adjust the quantization step-size A, each layer based on the overflow rate, i.e., the percentage p% of neurons that overflow in the network. If the overflow rate p% is too large, then we increase A, to reduce the overflow rate p%. The selected quantization step-size then fixed for further fine-tuning.
3.3 ADAPTING TO BIT- BIT-PACKING
Most modern processors provide vector instructions that enable parallel operation on multiple 8- bit numbers. For instance, the AVX2 (NEON) instruction set on x86 (ARM) processors provides parallel processing with 32 (16) 8-bit numbers. Vector instructions provide a clean implementation of bit-packing, which WrapNet can leverage to attain significant speed-ups. While some embed- ded processors and legacy chips do not provide vector instructions, bit-packing Without vector instructions for multiplication, binary/ternary weights must be used to replace mul- tiplication with bit- logic (Bulat & Tzimiropoulos/ 2019 Pedersoli et al. 2018). Furthermore, bit-pace packinckig of additions is more delicate: Each integer overflow not only results in wrap-around behavior, but also generates carry bit that contaminates the adjacent number--seecalizees vector
still be applied.
4

---

Published as a conference paper at ICLR 2021
instructions avoid such contamination. We propose the following strategies to minimize the impact of carry propagation.
Reducing variance in the number of carries. The number of carries generated during convo- lution operation can be large. Nevertheless, if we can keep the number of carries approximately the same for all the neurons among batch of images, the estimated number of carries can be sub- tracted from the result to correct the outputs of a bit-packed convolution operation. To achieve this, during training, we calculate the number of carries for each neuron and impose regularizer, It" to keep the variance of the number of carries small. The detailed formulation of Re can be found in Appendix A.I] Using buffer bit. Alternatively, since each addition can generate most one carry bit, we can place buffer bit between every low-bit number in the bit-packing. For example, instead of packing eight 8-bit representations into a 64-bit number, we pack eight 7-bit numbers with one buffer bit between each of them. These buffer bits absorb the carry bits, and are cleared using bit-wise logic after each addition. Buffering makes representations 1-bit smaller, which potentially degrades accuracy. A hybrid approach. To get the benefits from both strategies, use variance penalty on layers that have small standard deviation to begin with, and equip the remaining layers with buffer bit.
EXPERIMENTS
We compare the accuracy and efficiency of WrapNet to networks with full-precision accumulators using the CIFAR-I0 and ImageNet datasets. Most experiments use binary or ternary weights for WrapNet as AVX2 lacks 8-bit multiplication instructions, but supports 8-bit additions and logic operations needed for binary/ternary convolutions.
4.1 TRAINING PIPELINE
We first pre-train a network with quantized weights and no cyclic layers, while keeping full- free-iiniisoon activations. Then, select the quantization step-sizes of the activations (see Section 3.2) such that each layer has an overflow rate around p% (a syper-parameter with respect to the desired accumulator bitwidth. Given the selected quantization step-size for each layer and the pre-trained network, we insert our proposed cyclic activation layer. We then warm-up our WrapNet by fine- tuning with full-precision activation for several epochs. Finally further fine-tune the network with both activations and weights quantized. Both overflow and carry variance regularizers are only applied in the final fine-tuning step, except when training ResNet for ImageNet, where the regularizers are also included during warm-up.
4.2 ADAPTING TO LOW-PRECISION ACCUMULATORS
We conduct ablation studies the following factors: the type of cyclic function, the initial overflow rate for quantization step-size and precision selection, and the coefficient of the overflow penalty regularizer. These experiments are conducted VGG-7 (Li al. 2016), which is commonly used in the quantization literature for CIFAR-10. We binarize the weights as in (Rastegari et al. 2016), and we train WrapNet to adapt an 8-bit accumulator. As our default setting, we use k as the transition slope, 5% as the initial overflow rate, and 0 as the coefficient of the regularizer.
Cyclic activation function. We compare the performance of various transition slopes I of our cyclic function è in Table 2 and we achieve the best performance when A = 2. If l is too small, then the accuracy decreases due narrower effective bitwidth (only half of the bitwidth used when &- = 1). Meanwhile, the abrupt transition for large l hurts the performance as well. In the extreme case where the cyclic function degenerates to modulo (k- + oo). WrapNet diverges to random guessing. which highlights the importance of training with "smooth" cyclic non-line to assimilate integer overflow. We also find that placing a ReLU after batch norm yields the best performance, though the cyclic function is already non linear. More experimental results can be found in Appendix I
Quantization step-size. As described in Section 3.2 the quantization step-sizes are selected to balance the rounding error of the activations and accumulation errors due to overflow. We compare the classification performance when choose different step-sizes to control the overflow rate as in
5

---

Published as a conference paper at ICLR 2021
Table 2: Results for different transition slopes for cyclic function; denotes divergence.
k
2
4
10
Accuracy 90.24%
90.52%
90.25%
89.16%
Table 3:
Results for different quantization step-sizes based on overflow rate p()). denotes divergence.
Table 4: Results for fine-tuning with the overflow penalty (R").
p
Bits
Accuracy
p
Bits
Accuracy
Ro
p%
Accuracy
Difference
0
90.07%
20
4
88.25%
20
88.25%
2
3
90.51%
30
5
85.30%
5
90.52%
2.27%
90.52%
40
5
36.11%
0.01
90.05%
10
4
89.92%
50
5
0.01
5
90.81 %
0.76%
Table 3 If the initial overflow rate is large, then the quantization step-size will be finer, but training less stable. We obtain the best performance when the initial overflow rate is around 5%. The median bitwidths of the activations across layers are also reported in Table 3 Note that if we want to suppress all overflows, we can only use 1-bit activations. We also observe that WrapNet can attain reasonable accuracy (85%) even with large overflow rate (around 30%), which demonstrates that our proposed cyclic activations provides resilience against integer overflows.
Overflow penalty. The overflow penalty regularizer improves stability to step-size selection. More specifically, in T the difference in accuracy between two step-size selections decreases from 2.27% to 0.76% after adding the regularizer. The overflow penalty also complements cyclic activation, as we achieve the best performance when using both of them together during the fine- tuning stage. Moreover, in Appendix [B.2] we compare our results fine-tuning the pre-trained network using the overflow regularizer only. In the absence of cyclic layer, neural networks still suffer from low accuracy (as in Section 2.3 unless very strong penalty imposed.
4.3 ADAPTING TO BIT- BIT-PACKING
We now show the efficacy of WrapNet for bit-packing without vector operations. We use the same architecture, binary weights, 8-bit accumulators, and hyper-parametes parameters in Section 4.2 The train- ing details can be found in Appendix A.2 We consider CIFAR-10. and we compare with the best result of WrapNet from the previous section as baseline. Without specific vector instructions, accuracy degenerates to random guess because of undesired carry contamination during inference.
Surprisingly, with the carry variance regularizer, WrapNet works well even with abundant carry con- tamination during inference (for each neuron, 384 on average over all the dataset). The regularizer drops the standard deviation the per-neuron carry contamination by 90%. When we use the hybrid approach, the accuracy is further improved (89.43%) and close to the best result (90.81) %) we can achieve with vector instructions that do not propagate carries across different numbers (see Table 5)
Table 5: Results for adaptation to bit-packing with 8-bit accumulator. (v) denotes no carry contami- nation in a vector instruction; (c) denotes carry propagation between different numbers.
Method
Accuracy (v)
Accuracy (c)
Carry
Carry Std
Baseline
90.81%
10.03%
254.91
159.55
Buffer Bit
88.22%
R Hybrid
87.86%
384.42
17.91
89.43%
482.4
16.18

---

Published as a conference paper at ICLR 2021
4.4 BENCHMARK RESULTS
In this section, we compare our WrapNet when there is carry contamination, with the following 32-bit accumulator baselines: a full-precision network (FP), network trained with binary/ternary weights but with full full-precision activations (BWN/TWN), and network where both weights and activations are quantized to the same precision as our WrapNet (BWNTWN-QA) We benchmark results both CIFAR-10 and ImageNet. We use VGG7 and ResNet20 for CIFAR-10 experiments, and we use AlexNet (Krizhevsky et al. 2012 [Smon et al.) 2016), ResNet18 and ResNet50 (He al. 2016) for our ImageNet experiments. Details of training can be found in Appendix B.3
For CIFAR-10, even with 8-bit accumulator, results comparable to both BWN and TWN. When adapting to a 12-hit accumulator, we further achieve performance on-par with TWN and better than BWN (see Table 6). For ImageNet, our WrapNet can achieve accuracy as good BWN when adapting to a 12-bit accumulator where we can use binary weights and roughly 7-bit quantized activations. However, in the extreme low cow-precision case (8-bit), the accuracy of our binary WrapNet drops around 8% due to the limited bitwidth we can use for activations. As reported in Table O the median activation bitwidth roughly 3-bit, and for some layers in AlexNet, we can only use 1-bit activations. Despite the gap from BWN, observe that our model can achieve comparable performance as BWN-QA where the same precision is used for activations. When using ternary weights and an 8-bit accumulator, our WrapNet only drops by 3% and 2% from TWN for ResNet18 and ResNet50, respectively.
addition, in the case of adapting to a 12-bit accumulator, our ternary WrapNet with roughly 7-bit activations is slightly better than TWN for ResNet50. Note that, without cyclic activation function, all the results for networks using 8-bit accumulator are as poor as random guessing which is consistent with Table
Table 6: Top-1 test accuracy for both CIFAR-10 and ImageNet with different architectures. Here, "Ace" represents accumulator, and "QA" represents quantized activation.
Bits
CIFAR-I0
ImageNet
Activation
Weight
Acc
VGG7
ResNet20
AlexNet
ResNet18
ResNet50
FP
32
32
32
92.45%
91.78%
60.61% %
69.59%
76.1%%
BWN
32
32
91.55%
90.03%
56.56%
63.55%
72.88%
BWN-QA
~ 3
32
91.30%
89.86%
46.30%
57.54%
66.85%
WrapNet
8
90.81%
89.78%
44.88%
55.60%
64.30%
WrapNet
91.59% 12
90.17%
56.62%
63. 13%
72.37% %
TWN
32
2
32
91.56%
90.36%
57.57%
65.70%
73.31
TWN-QA
4
2
32
91.49%
90.12%
55.84%
63.67%
72.50%
WrapNet
2
8
91.14%
89.56%
52.24%
62.13%
71.62%
WrapNet
~ 7
2
91.53% 12
90.88%
57.60%
63.84%
73.93%
4.5 EFFICIENCY ANALYSIS
We conduct an efficiency analysis of parallelization by bit-packing, both with and without vector operations, on an Intel i7-7700HQ CPU operating 2.80 GHz. We also conduct detailed study of improvements that can be obtained using custom hardware.
AVX2 instruction efficiency analysis. We study the empirical efficiency of WrapNet when vec- tor operations are available. We extended Gemmlowp (Jacob et al.] 2016) to implement matrix multiplications using 8-bit accumulators with AVX2 instructions. To demonstrate the efficiency of low-precision accumulators, we compare our implementation with the AVX2 version of Gemmlowp, which uses 32-bit accumulators. We report the execution speed of both on various convolution ker- nels of ResNet18 in Table 7 From Table 7 we observe significant speed-ups ranging from 2x to 2.4x among different blocks. Besides, we compare the entire inference time (ms) of ResNet18 for WrapNet (234.74) with abbaaccumulator quantized network (31 (312.42), which gains 33% speed-up. The result provides solid evidence for the efficiency advantage of using low wow-precision accumulators. We remark that in average, the time cost for cyclic activation only around 10% of the time cost

---

Published as a conference paper at ICLR 2021
Table Time cost (ms) for typical 3 3 con- volution kernels in ResNet using different ac- cumulator bitwidths.
Table 8: Time cost (ms) for 3 3 convolution ker- nels in ResNet with no vector instructions using bit packing.
Input size
Output
8-bit
32-bit
Input size
Output
bit packing
naï'ee
64x56x56
64
3.467
8.339
64x56x56
64
29.80
83.705
128x28x28
128
2.956
6.785
128x28x28
128
23.86
80.557
256x 14x14
256
2.499
5.498
256x 1x6x14x14
256
21.71
86.753
512x7x7
512
2.710
5.520
512x7x7
512
20.41
87.671
for the GEMM kernel. We also remark that AVX2 lacks single instruction that performs both mul- tiplication and accumulation for 8-bit data, but it does have such instruction for 32-bit data. Thus, further acceleration can be achieved systems like ARM where such combined instructions for 8-bit data are available.
Bit-packing results without vector operations. We implement naïve for-loop based matrix multi- plication, which uses buffer bit and logical operations introduced in Section 3..3 to form the baseline. We then pack four 8-bit integers into 32 bits, and report the execution speed of both implementa- tions on various convolution kernels of ResNet18 in Table The results show significant speed-ups ranging from 2.8x to 4.3x. Such observations demonstrate our proposed approach handle extra carry bits makes bit-packing viable and efficient, even when vector instructions are not available.
Hardware analysis. To illustrate the potential benefits of WrapNet for custom hardware accel- erators, we have implemented a multiply-accumulat accumulate (MAC) unit in commercial 28nm CMOS technology. The MAC unit consists of (i) a multiplier with an output register, (ii) an accumula- tor with its corresponding register, and (iii) auxiliary circuitry. Please refer to Appendix [[ the details. We have considered 8-bit 8-bit and 3-bit x 1-bit multipliers, as well 32-bit and 8-bit accumulators, where the latter option is enabled by our WrapNet approach and its cyclic activation function. We consider a slope & 2 for the cyclic activation. Figure 2/shsss our post-layout results.
Figure 2a shows that reducing the multiplier bitwidth decreases the cycle time by 7%; reducing the accumulator precision from 32-bit to 8-bit further the cycle time by 16%. Figures Figures[bJande higghiiltt the importance of reducing the accumulator's precision. When using an 8-bit x 8-bit multiplier, the 32-bit accumulator already constitutes more than 40% of the area and energy Once the multiplier's precision reduces, the accumulator dominates area- and energy-efficiency. Thanks to WrapNet, we can reduce the accumulator precision from 32-bit to 8-bit, thus reducing the accumulator's 's and energy-effciiecy by more than 5x and 4x, respectively. WrapNet requires the implementation of the cyclic activation, which has an area- and energy mnfryyeffficiency comparable (although lower) to that of the accumulator. In spite of this overhead, WrapNet is still able to reduce the total MAC unit's area- and energy -erryyefficiency by up to 3x and 2x, respectively. While our hardware implementation only uses one adder per inner-product, we note that WrapNet can also be applied to spatial architectures, such systolic arrays, which use several adders per inner-product. For such spatial architectures, WrapNet avoids an increase in the adders' bitwidth, normalizing all
MAC unit.
0.35
200
450
0.51
Winglet
175
400
0.24
0.24
190
350
500
WropNet
Whaphett
3435
0.35
50
10%
15%
0.25
25
41%
41%
2514
4890
3ba
Horth
Sbclb
Stallb
3be1b
Iba
Wbotb
3bx2bb
Mbbibb
1b
12b
37h
12h
Rbb
32b
32t
abbasc
(c)
Figure 2: (a) Cycle time, (b) area and (c) energy efficiency for different MAC units implemented in 28mm CMOS. We consider 8-bit 8-bit or 3-bit 1-bit multipliers with 32- 8-tit accumulators.
8

---

Published as a conference paper at ICLR 2021
adders to the same low bitwidth. Moreover, the of several adders per inner- amortizes the overhead from the cyclic activation, which only one is needed per inner-product. Finally, note that this analysis only considers the computation part of a hardware accelerator this is where WrapNet has a significant impact-the memory sub-system will remain virtually the same, as existing methods already quantize the output activations to low-bit before storing them in memory.
CONCLUSION
We have proposed WrapNet, a novel method to render neural networks resilient to integer overflow, which enables the use of low-precision accumulators. We have demonstrated the effectiveness of our adaptation on both CIFAR-I0 and ImageNet. In addition, our custom GEMM kernel achieves 2.4x acceleration over its standard library version, and our hardware exploration shows significant improvements in area- and Energy-efficiency Our hope is that hardware-awaad aware architectures will en- able deep learning applications future innovations in GPU and data center technologies, we hope that WrapNet can provide further speed-ups by enabling inference using suarter-precisions step forward in terms of performance from the currently available half- feff-precision standard available on emerging GPUs.
wide range of platforms and mobile devices. Furthermore, with
ACKNOWLEDGEMENT
The university of Maryland team was supported by the ONR MURI program, AFOSR MURI pro- gram, and the National Science Foundation DMS division. Addition support was provided by DARPA GARD, DARPA QED4RMI, and DARPA YFA.
REFERENCES
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks. arXiv preprint arXiv:1909, 13863, 2019.
Jungwook Choi, Pierce 1-Jen Chuang. Zhuo Wang. Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). arXiv preprint vrXiv:1807.0696 2018a.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 1805,06085, 2018b.
Matthieu Courbariaux, Yoshua Bengio, and Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv: 162.00800 2016.
Barry de Bruin, Zoran Zivkovie, and Henk Corporaal. Quantization of deep neural networks for accumulator-coonstrannee processors. Microprocessors and Microsystems, 72: 72:102872, 2020.
Yinpeng Dong, Renkun Ni, Jianguo Li, Yurong Chen, Jun Zhu, and Hang Su. Learning accurate low-bit deep neural networks with stochastic quantization. arXiv preprint arXiv:170..01001, 2017.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4852 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.

---

Published as a conference paper at ICLR 2021
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in neural information processing systems, pp. 4107-4115, 2016.
Benoit Jacob, Pete Warden, Miao Wang, David Andersen, Maciek Chociej, Justine Tunney, Mark Matthews, Marie White, Suharsh Sivakumar, Sagi Marcovich, Sarah Knepper, Mourad Gouicem, Richard Winterton, David Mansell, Andreas Gal, Alexey Frunze, and Alexey Frunze. Google gemmlowp. 2016. URL pttps://githubb http://giibbboommggggeeemmoo
wlg.ogglleeemmmmmmm owp
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- integer-arithmette arithmetic-on) inference. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2704-2713, 2018.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Daya Khudia, Jianyu Huang, Protonu Basu, Summer Deng, Haixin Liu, Jongsoo Park, and Mikhail Smelyanskiy. Fbgemm: Enabling high peeeorerffrm low-precision deep learning inference. arXiv preprint ar 2rX0v:2101.05615, 2021.
Alex Krizhevsky, lla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Fengfu Li. Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXi::1605.04711, 2016.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In Advances Neural Information Processing Systems, pp. 511558111800 1-811, 2017.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. arXiv preprint arXiv: 2017.
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint 17 2017.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: wide rrducionnrecision
networks. arXiv preprint arXiv: 170.00 2017.
Fabrizio Pedersoli, George Tzanetakis, and Andrea Tagliasacchi. Espresso: Efficient forward prop- agation for binary deep neural networks. 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Charbel Sakr, Naigang Wang, Chia- Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, and Kailash Gopalakrishnan. Accumulation bit-width scaling for ultra-low precision training of deep networks. arXiv preprint wrXiv:1901.06588 2019.
Marcel Simon, Erik Rodner, and Joachim Denzler. Imagenet pre-trained models with batch normal- ization. arXiv preprint 1rXiv:1612.01452. 2016.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-awarr automated quan- tization with mixed precision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
10

---

Published as a conference paper at ICLR 2021
Naigang Wang. Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances neural information processing systems, pp. 7675-7684, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06116. 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternrry quantization. arXiv preprint arXiv:16/2.01064 2016.
Xiaotian Zhu, Wengang Zhou, and Houqiang Li. Adaptive layerwise quantization for deep neural network compression. 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2018.

---

Published as a conference paper at ICLR 2021
A DETAILS OF CARRY VARIANCE REDUCTION REGULARIZER
A.1 CARRY VARIANCE CALCULATION
With two's complement representations for signed integers, carry bit is generated in the following three cases: (i) addition of two negative numbers, (ii) addition of two positive numbers whose result exceeds the representation range, thus provoking integer overflow, and (iii) addition of positive and a negative number whose result is positive number. Dealing with these cases individually complicated, but the calculation can be simplified by first reinterpreting the two's complement representation as unsigned integer. Carry bits resulting from accumulation of unsigned integers easier to calculate as they can only happen in case (ii) as described above.
Since we only consider binary/ternary weights for bit-packing, carry bits can only be generated during accumulation, and not by multiplication. To produce a single output from a convolution,
we must perform the accumulation Σ V, of all entries of the vector v E R This is done by batching computations inside b-bit register as follows. First, we bit-pack groups of numbers v, into several high-resolution registers. For example. let us consider the use of 32-bit registers to pack four 6 8-bit numbers; then, we need to use [ 32-bit registers to represent all entries of v. In the absence of vector instructions, the addition of these high-resolution registers will generate carry bits that will contaminate the adjacent bit bit-packed numbers. After all [L/4] additions take place, we add the 4 bit- ttt-packed numbers together to get final result.
When one output feature calculated by bit- backiag as described above, the effect of carry bits is easy to simulate; accumulations can be done without accounting for carry bits during convolution, and then the earry bits can be added into the the final result after convolution takes place. If the total number of carries is large. this final correction Algorithm D to compute the total number of carry bits that are generated an accumulation. The first equation simply reinterprets the signed representation to its unsigned counterpart Then, compute the amount of carry bits Cs as well as the result r; remaining within the b-bit accumulator. Due to carry contamination, the carry bits o; will be added the result 7;, which may generate new carry bits Ci+1. We keep on adding the new carry bits to the accumulator until no new carry bits generated. Note that, in real hardware at inference time, the most signifiant carry bit produced inside register will be thrown away. For simplicity, our simulations during training accumulate all carry bits, including the most significant. We find that dropping the most significant carry during inference does not significantly impact testing.
in turn produce new carry bits. Hence, we
Algorithm 1: Carry Amount Calculation
initialization v, by
w = -, ((ign( v.) + (
+ ((-sign(v.)) + 1)/2) (v; + 22);
= u, = - 0;
while q # do
C++1 (c: + r:)/2*1:
ri+1 (c: rr) mod 2%:
== c c C411
G: = C7 = Ti+1
end return c
Given the number of carry bits calculated during the inner product, the variance of the carry among batch (b.) of images calculated follows:
mR, (nz
1
i.l
nik
b.
kml
h,
vars, (
(n - mm, (re(d)))
where ri is the carry bit for the i-th neuron 1-th layer (assuming all the feature maps are vec- torized). The estimated mean among all the images are learned by moving average based on the
12

---

Published as a conference paper at ICLR 2021
mean of batches equation However, the sign and rounding function may have gradient al- most everywhere. To make all the operations differentiable, we replace the sign function with a tanh function and
straight through estimator for rounding during the backward pass (gradient is identity). Then finally, our regularizer Rt will be the mean variance among all the neurons.
A.2 TRAINING WITH CARRY VARIANCE REDUCTION REGULARIZER
Due to the large amount and high variance of carry-bit occurrences, it is hard to fine-tune Wrap- Net even when using the carry variance reduction regularizer. The generated carry bits will be ac- cumulated, which increases the overflow rate dramatically. In addition, the accumulation error will contaminate downstream computations and destroy accuracy. As a result, we fine-tune WrapNet with simulated carry bits layer by layer, starting from the layer which has the least carry variance. For the hybrid approach, we stop simulating the carry bit when we notice significant accuracy drop; the remaining layers are trained using a buffer bit instead.
B EXPERIMENTAL DETAILS
B.1 MORE CYCLIC FUNCTIONS
We compare two more "smooth" cyclic functiins with our proposed cyclic activation function in Section 3 Specifically, transition slope k as alternative cyclic activations. Figure illustratee the compared functions. We compare the results with and without ReLU activation after batch normalization well. Table 9 shows that retaining the ReLU activation after the batch normalization layer always achieves a better result, and that proposed cyclic activation outperforms the other two choices.
consider cyclic absolute value function, and a ReLU-like function with
/66
16
Cyclic ReLU-iike Absolute Value
-8
Figure 3: Example of the compared cyelic functions for 4-bit accumulator.
Table 9: Results for different types of cyclic activation
Cyclic Function
ReLU
slope &
Accuracy( %)
Proposed
2
90.52
Proposed
2
89,28
ReLU-like
1
90.25
ReLU-like
2
90.31
ReLU-like
3
90.15
ReLU-like
88.62 1
ReLU-like
2 89,01
ReLU-like
3
88.53
Absolute
90.17
Absolute
89.19
B.2 FULL OVERFLOW PENALTY RESULTS
Table shows the results for fine-tuning our WrapNet with different coefficients for the overflow penalty. When applying the overflow penalty, the overflow rate decreases and achieve
13

---

Published as a conference paper at ICLR 2021
higher accuracy. In addition, when apply the regularizer network with low-resolution accu- mulators that does not use our cyclic activation, the network still suffers from performance degra- dation unless a large coefficient is used. However, strong penalty kills almost all of the overflow, which may limit the performance deep neural network.
Comparison for fine-tuning network without cyclic activation and our WrapNet, with overflow penalty Rº
Table 10:
Cyclic
R
Overflow rate (%)
Accuracy(%)
6.29
90.52
0.001
1.88
90.33
0.01
1.24
90.81
0.1
1.04
89.52
0.01
5.91
64.69
0.1
0.35
88.94
0.06
90.26
0.03
90.20
B.3 TRAINING DETAILS FOR BENCHMARK RESULTS
For fair comparison, all our baselines (BWN/TWN, BWN-/TWN-QA) are fine-tuned from pre- trained full- puel-precision network. We leave the first and last layer at full-precision as in (Rastegari al. 2016; Zhou et al. 2016). training pipeline, where we first warm-up our WrapNet with full-precision activations, and then fine-tune the network for quantized activations. We set the transition slope 2, and the initial overflow rate p 5%. The overflow penalty coefficients for CIFAR-10 and ImageNet are 0.01 and 0.001, respectively.
obtain the benchmark results our WrapNet, we follow
For the CIFAR-10 10 results, both warm-up and fine-tuning stages, we rum 200 epochs, and the learning rate divided by 10 every 60 epochs. For all the ImageNet results, we SGD with momentum 0.9, weight decay our optimizer. We run 60 epochs for both warm-up and fine-tuning stages, where the initial learning rate is 0.01, which is divided by 10 at (20, 40, 50) epochs. We note that, due to depth of ResNet, we select a fixed quantization step-size for all the layers, where the average initial overflow rate is around 5%. As result, the overflow penalty also imposed during the warm-up stage for ResNet experiments.
ADAM
optimizer with initial learning rate of 0.001. For
10- as
C HARDWARE ANALYSIS
(w
Zq
c(.)
C
(Zgg)
(I (Ig)
Figure 4: aultiply-accumulate (MAC) unit, together with cyclic activation function c(.), imple-
mented for hardware analysis.
Figure 4 shows the multiply-accumulate (MAC) unit implemented in TSMC 28nm CMOS. The MAC unit multiplies two scalars and accumulates these products using an adder. To perform this functionality, the MAC unit is composed of multiplication, accumulation, and auxiliary circuitry, colored in Figure 4 with blue, orange, and gray, respectively. Clock distribution circuitry is not shown, but included our results as part of the auxiliary circuitry. Furthermore, we have imple- mented the cyclic activation function in hardware, colored in Figure 4 with yellow, which is only
14

---

Published as a conference paper at ICLR 2021
used together low- low-bitwidth accumulators. To achieve lower cycle times (i.e., faster operation fre- quencies), as well as to separate the multiplier's and accumulator's s critical paths, we introduced a pipeline register between the multiplier and accumulator. For implementation results, we con- sider this pipeline register as part of the multiplication circuitry.
We implemented the circuit in Figure 4 using different bitwidths for the multiplier (8-bit 8-bit 3- b I-bit) and the accumulator (32-bit or 8-bit). When using the 8-bit x 8-bit multiplier with the 32- bit accumulator, we 16 bits for the multiplier's output register represent all possible products. When using the 8-bit x 8-bit multiplier with the 8-bit accumulator, we use 8 bits for the multiplier's output register, since the accumulator does not support larger bitwidths. When using the 3-bit x 1-bit multiplier, we use 4 bits for the multiplier's output register, regardless of the accumulator's bitwidth. The cyclic activation is only implemented when using the 8-bit accumulator, for both multiplier's bitwidth. We implemented the eyclic activation for slopes of k = 2 and k
The four different MAC units were synthesized using Synopsys Design Compiler (DC), and au- tomatically placed-and-routed using Cadence Innovus. Power analysis was done using Cadence Innovus with stimuli-based post-layout simulations at 0.9V and 25 in the typical-typical corner. For the stimuli, we used weights and activations extracted from layer of the ResNet-18 network. Tables [1 12 and 13 show the implementation results from Figure in tabular form. Note that throughput computed 2/cycle time, the MAC unit completes two operations (multiplication and accumulation) in a single clock cycle. However, in Figure 2 that, for all metries presented (cycle time, a better performance. Note that circuits with a higher throughput (which corresponds, in this case, lower cycle time) often result in higher area and power consumption. As a matter of fact, dy- namic power consumption is directly proportional to operation frequency (i.e., 1/cycle time). Thus, to perform a fair comparison, we have normalized the area and power reported in Table by the throughput achieved, resulting in the area- and energy mergg-effficieniie reported in Tables 12 and 13 respectively.
we decided to report cycle time so and energy-efficienee
lower value corresponds to
Table 11: Hardware implementation results for one multiply-accumulate (MAC) unit in 28nm CMOS
Bits
Cyclic act.
Cycle time
Throughput
Cell area
Power
Act.
Weight
Acc.
slope k
(ns)
(Gops)
(çm²²
(mW)
8
8
32
0.31
6.5
1 198
2.78
3
32
0.29
7.0
732
1.90
8
8
8
2
0.24
8.3
521
1.60
8
8
0.25
8.1
523
1.64
3
1
8
2
0.24
8.3
290
0.93
3
0.24
8.3
285
0.87
Table 12: Area breakdown of one multiply-aceumulat (MAC) unit in 28nm CMOS
Bits
Cyclic act.
Cell area efficiency /Gops)
Act.
Weight
Acc.
slope
Multiplier
Accumulator
Cyclic act.
Auxiliary
Total
8
8
32
96 (48%)
91 (46%)
12 (6%)
199
3
32
3 (2%)
93 (89%)
9 (9%)
105
8
8
8
2
31 (49%)
12 (19%)
10 (16%)
10 (16%)
63
8
8
33 (50%)
12 (19%)
(13%)
12 (18%)
65
3
8
2
2 (5%)
17 (46%)
15 (41%)
3 (8%)
37
3
8
2 (5%)
18 (52%)
12 (35%)
(8%)
35
D USING MORE WEIGHT BITS
Since ARM provides arithmetic operations that handle multiplication between various 8-bit num- bers in parallel, we further conduct experiments in which more bits used for weight quantization.
15

---

Published as a conference paper at ICLR 2021
Table 13: Energy breakdown of one muliiply-aceumulate (MAC) unit 28nm CMOS
Bits
Cyclic act.
Energy efficiency (fJ/op)
Act.
Weight
Acc.
slope
Multiplier
Accumulator
Cyclic act.
Auxiliary
Total
8
32
144 (34%)
173 (40%)
111 (26%)
428
3
32
10 (4%)
197 (73%)
64 (23%)
271
8
48 (25%)
29 (15%)
17 (9%)
98 (51%)
192
8
8
8
53 (26%)
28 (14%)
17 (8%)
105 (52%)
203
3
8
(7%)
42 (37%)
23 (20%)
42 (36%)
115
3
8
(6%)
42 (40%)
24 (23%)
33 (31%)
105
Table displays the classification accuracy, as well as the overflow rate of the final models, Sur- prisingly, in some cases, we may have lower overflow rate even when using more bits for the weight quantization. We also collect the accuracy degradation from the full precision network. Our results show that the best performance is achieved when we 4-bit weights, which is close the full-precision result (around 0.7% degradation).
Table 14: Results for WrapNet with more bits for weight quantization, where we use ternary weights for 2-bit.
Bits
Overflow Rate
Accuracy
Degradation
1.24%
90.81%
1.64%
2
0.12%
91.14%
1.31% %
3
0.02%
91.55%
0.90%
4
0.04%
91.73%
0.72%
5
0,4%
91.20%
1.25%
16


Reviews:
[{"review_id": "p37xqcUEhDD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method (WrapNet) for the problem of efficient low-precision inference. The main contribution is to allow efficient low-bit (e.g., 8 bit) accumulators via the use of a novel cyclic activation function that constrains the value space of the layer outputs, while still allowing good accuracy.\n\nPros:\n\nThe paper is clearly written, although some training details would be nice to see in the main paper instead of the appendix.\n\nCyclic activation functions have been used in other works, but this work is to my knowledge among the first to use a cyclic ReLU and most likely the first to use it in order to reduce the output value space in order to do more efficient accumulation in inference hardware.\n\nThe significance of this work lays in the fact that this is the first paper that allows lower precision accumulators while still demonstrating quite good performance, even in the case of demanding Imagenet classification. The low-power and low-latency inference are of key importance for more widespread use of neural networks in e.g., low-power devices. Actually the performance is almost as good as with other low-precision schemes, which use high-precision accumulators. In addition, the paper has quite extensive study and analysis of performance in various hardware architectures.\n\nCons:\n\nThe method introduces various additional hyperparameters, e.g., regularizer coefficient, transition slope, etc. Although these can be tuned with a validation set.\n\nIn results Table 6, why we are not given results that would be comparable to BWN-QA and TWN-QA. For instance, WrapNet with 3 bits/4bits activation and e.g., 12-16 bits accumulator.  \n\nSome instability of training seems to happen with different initial overflow rates.\n\nThe training setup is relatively complicated  (layer-by-layer training) when using carry variance reduction regularizer.\n\nTypo maybe in page 2 in \"reference (fbg, 2018)\"\n\nEdit: I am happy with the author responses and have raised the score accordingly to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7DDHXt5COf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a scheme named WrapNet to further reduce the bit width of accumulation operations in the deep neural network inference process. Through a novel cyclic activation and regularizer penalizing overflow, the proposed method can be implemented in processors with or without specialized vector instructions using less time, area, and energy cost. \n\n\nStrength:\n\n--The manuscript is presented well and quite easy to follow.\n\n--The overall idea makes sense and the proposed method can reduce computational resources and speed up computing greatly in the accumulation operations. \n\n\nWeakness:\n\n--The step-size selection strategy is more like an empirical parameter adjustment, lack of a uniform selection strategy. \n\n-- The results are not so convincing due to the lack of experiments on large-scale models like ResNet50 on ImageNet.\n\n\nComments:\n\n(1) The step-size selection strategy seems trying to balance the quantization error and the overflow rate. However, it is more like an empirical adjustment. When the DNN model and the dataset changes or there is a new task, how can it be used in new scenarios because the overflow rate may differ a lot? I think there is a lack of a uniform selection strategy.\n\n(2) The batch normalization layer may change the distributions of weights and activations. In fact, there are some works which have successfully remove batch normalization with comparable performance. And removing batch normalization can better accelerate the DNN computing. In a network without batch normalization, the overflow rate may differ from DNNs with it. Can you do some experiments for comparison and give a detailed explanation of the proposed method used in the DNN model without batch normalization.\n\n(3) The model is pre-trained with quantized weights and full-precision activations. Can the proposed method be applied into the model from scratch given the estimated step size? And how it performs?\n\n(4) It’s better to give the performance of the proposed method on ResNet50, ImageNet to make the results more convincing. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EWkBW2PUhIN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores to solve an often ignored issue in quantization: accumulation precision. As the bit-width of input scales down, the area/energy cost of the accumulator starts to dominate.  The cyclic method proposed by the authors at the first glance is not intuitive. However, it's surprising that the surveyed models could be tuned to live with significant overflows---as long as it can be tuned, which is enabled by the \"differentiable overflow\" brought by the cyclic method.  There are several issues to be addressed before the paper can be accepted:\n\n(1) In the equations of page3,   the boundary in the 2nd line of f(m) has a 'c(zq)',  is that a typo?\n\n(2) Since the paper only covers CIFAR10 and some ImageNet works that are on the easier side to quantize, such as ResNet18 and VGG, the cyclic method could meet its limitation on ResNet50 and Mobilenet. The authors didn't discuss an important concept: accumulation length. As the accumulation length increases, the events of overflow could rise sharply, and the training could fail without room for the cyclic method to optimize the slope k.\n\n(3) Some comments on the relation between the accumulation length and bit-packing would also be helpful. For example, if the accumulator with 8-way bit-packing is working on the same GEMM, the accumulation length would be reduced by 8---that would be desirable. Although, a higher level reduction would be required then.\n\nIn general, the paper is well written and brings attention to the important topic of accumulation for reduced precision inference. The paper attempts to solve the overflow problem, although not perfectly, with a differentiable \"failure\" approach. The paper provides great hardware insights for hardware/software co-design. Therefore, I recommend the paper to be accepted on the condition that the authors could address my comments fairly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6V-XGwj70Zh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\n\nThe paper introduces a new neural network layer that enables training NNs with quantized activations using reduced bit-width accumulators. The cyclic activation layer makes overflows smooth, instead of being discontinuous and this enables achieving better accuracy on quantized networks on reduced bit-width accumulators. They also introduce overflow and carry penalties to dissuade the training regime from reaching overflow states.\n\n## Strengths:\n* New NN layer (operator), cyclic activation layer that makes quantized accumulated outputs smooth when overflow is present. The authors show that even at large overflow rates, the network accuracy does not degrade that much.\n* Introduction of the overflow penalty to dissuade overflow and together with the cyclic activation layer achieves higher accuracy even at high overflow probabilities.\n* Evaluation using both vector instructions and carry penalty when vector instructions are not present; they show that carry penalty mitigates carry contamination leading to a closer accuracy compared to their vector implementation.\n* Hardware analysis of possible cost savings for reduced accumulator implementations in a MAC unit.\n\n## Weaknesses and Questions for authors:\n\n* Intuitive explanation as to how $f$ was designed is missing.\n* Isn’t computing $f$ need to be done in full precision or in higher precision than the accumulator? For instance say $k = 8$, then computing $k \\times 2^{b-1}$ needs $b+2$ bits to compute. What is the overhead of computing $f$ in higher precision? Based on the results from section 4.5, this overhead may not be that much, but still needs to be quantified. \n* Also, the overhead of computing $f$ varies with $k$. The same goes for hardware analysis. A quantification is needed to understand the tradeoffs.\n* I assume that loss function with regularizers is computed in full precision. Correct me if I am wrong.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers agree that this paper presents an interesting idea. Practically implementing a BNN that gains real world speedup is challenging, and as past work [1] showed, the bottleneck could shift into other layers(besides the accumulation). The paper would benefit from a thorough discussion about the practical impact in implementing the proposed method and relation to past work. 

The meta-reviewer decided to accept the paper given the positive aspects, and encourages the author to further improve the paper per review comments.

Thank you for submitting the paper to ICLR. 

[1] Riptide: Fast End-to-End Binarized Neural Networks

Decision: Accept (Poster)