Winning Both the Accuracy of Floating Point Activation and the Simplicity of Integer Arithmetic
Authors: Yulhwa Kim
Jaeyong Jang
Jehun Lee
Jihoon Park
Jeonghoon Kim
Byeongwook Kim
Baeseong park
Se Jung Kwon
Dongsoo Lee
jae-joon kim
Venue: ICLR 2023 poster (2023)

Abstract: Even though floating point (FP) numbers have been adopted as a de facto standard data format for deep learning computing, the complexity of FP arithmetic impedes a broader deployment of Deep Neural Networks (DNNs). Recent works such as quantization have attempted to replace the FP matrix multiplication (MatMul) of DNNs with simple integer MatMul by transforming the datatypes of both weights and activations into integers. Unfortunately, unlike weight values that are static, it is challenging to represent dynamic activations with integers. In this paper, to simultaneously achieve the accuracy of FP activation and the simplicity of integer arithmetic, we present a method for replacing FP arithmetic with integer one without changing FP activations in the storage format while weights are quantized. The proposed method pre-aligns the significands of FP activations just ahead of the MatMul on-the-fly so that the aligned significands (integers) can be used for the computation. Inspired by an observation that conventional FP arithmetic does not produce precise results due to rounding, we demonstrate that our proposed integer arithmetic-based scheme can produce the same level of errors as that of the FP arithmetic in case DNNs use FP activations and quantized weights. Experimental results show that the hardware based on the proposed scheme shows significant improvement over FP arithmetic-based designs in terms of energy efficiency and throughput-per-area while maintaining a similar level of accuracy.

Published as a conference paper at ICLR 2023
WINNING BOTH THE ACCURACY OF FLOATING POINT ACTIVATION AND THE SIMPLICITY OF INTEGER ARITHMETIC
Yulhwa Kim', Jacyong Jang', Jehun Lec', Jihoon Park2, Jeonghoon Kim", Byeongwook Kim2, Baeseong park2, Se Jung Kwon', Dongsoo Jac-Joon Kim! Seoul National University, NAVER Cloud fyulhwakim, jaeyongjaaan jehun. lee, ji jihoonpark, kimjae joon)ennu fesu aa..nuuac..... (jeonghoon. samuel, byeonguk. kim, baeseong park, sejung kwon, dongsoo. 1ee _ www.eeeaaveeo.oooooooooo com
ABSTRACT
Even though floating point (FP) numbers have been adopted as de facto standard data format for deep learning computing. the complexity of FP arithmetic impedes a broader deployment of Deep Neural Networks (DNNs). Recent works such quantization have attempted to replace the FP matrix multiplication (MatMul) DNNs with simple integer MatMul by transforming the datatypes of both weights and activations into integers. Unfortunately, unlike weight values that are static, it is challenging to represent dynamic activations with integers. In this paper. simultaneously achieve the accuracy of FP activation and the simplicity inte- ger arithmetic, we present method for replacing FP arithmetic with integer without changing FP activations in the storage format while weights are quantized. The proposed method pre-aligns the significands of FP activations just ahead of the MatMul on-the- fly so that the aligned significands (integers) can be used for the computation. Inspired by an observation that conventional not produce precise results due to rounding, we demonstrate that our proposed integer arithmetic-based scheme can produce the same level of errors as that of the FP arithmetic in case DNNs use FP activations and quantized weights. Ex- perimental results show that the hardware based on the proposed scheme shows significant improvement over FP crithmetic-basse designs in terms of energy effi- per-area ciency and throughput Beahhputt-e--anee area while maintaining a similar level accuracy.
arithmetic does
INTRODUCTION
Deep Neural Networks (DNNs) usually use Floating-Poin (FP) number systems to represent wide range of weight and activation values. Such a comprehensive representation, however, demands high computational complexity and cost for FP matrix multiplication (MatMul) (Sze et al. 2017). On the other hand, integer (a.k.a fixed-point) arithmetic logic is much simpler while consuming less energy compared to FP counterpart (Jouppi al. 2021). As such, the computational efficiency of DNNs can be enhanced by replacing FP arithmetic with integer one. Accordingly, quantization has been actively studied as promising technique to support DNN computations with integer arithmetic, as maps the input values of a (virtually) continuous domain (FP numbers) to the output values of a discrete set (integers) (Jacob al., 2018). Note that even though several studies have success- fully quantized weights and activations of some target DNNs with low-precision integer values (Li al. 202T Wu et al. (022)) quantization still challenging for numerous DNNs. In particular, activation values are known to be more difficult to be quantized than the weight parameters because activations are dynamically generated during inference while the distribution of weights static. The uncertainty of the distribution of dynamic activation values limits the ability to estimate proper quantization range (Choi et al.) 2018). Such issues on activation quantization become even more serious when DNNs involve highly non-linear activation functions (c.g., GeLU) or modules that increase the variance of the activations (c.g.. softmax and normalization layers) (Jeon et al.) 2020). As result, while the weight parameters be successfully quantized even for generative mod-

---

Published as a conference paper at ICLR 2023
precise
summattion
1.001 x
1,101 x
1.01X
(a)
Source of Overhead
align 5
+
0.01100
1.101
1,001
1.00
101
x31
x
5
0.01011
100
1,011
1.100
x32
X2
x23
(b)
1,011
1.001
1.101
1.01000
X21
X X2
pre-align trunctation) (6-2)
Source of
Error
Source of
normalize + 1.0111
add/sub G 0.11111
x2"
x2"
+ 1,11011 X2"
1.11011 X2"
0011100
0.00101
0.01101
x2"
X2
integer sum int2fp
Errop
rounding $ 1.100
X2"
S 1+ I.II
X2.3
+
1.110
Figure 1:
An example of FP summation with (a) conventional FP computation and (b) proposed
method. The precise summation is described in the box on top.
2022 Kwon et al. 2022 Frantar et al. 2022) and extra-large models such as GPT-NeoX-2000 (Chung et al. 2020 Yao el al. 2022).
els (Xu et al. 2018 Bai al., 2019 Jeon al. 2022 Park et al.
activation quantization usually relies on intensive quantization quantization-aware aware training or sophisticated vestigation algorithms such as dynamic min/max searching (Tao et al. 2022). Note that activation quantization mandatory if integer arithmetic logic is involved for MatMul operations. Thus, to avoid such significant efforts to quantize complex DNNs (mainly due to activation quantization), recent neural processing units tend to employ FP arithmetic units even for inference process at the cost of increased energy and area (Jouppi et al. 2021).
To address the challenges discussed above, we propose scheme that can achieve both the accuracy of FP activations and the simplicity of integer arithmetic. Our motivation stems from an observation that most multiplications can be removed once weights are quantized to be binary-coded (Jeon et al,, 2020). Then, consecutive FP additions are mainly required perform MatMul. and hence, find conventional FP units can be much simplified. To be more specific, when processing the MatMul of DNNs, our proposed method first pre-aligns the significands of FP activations to be added. Corre- spondingly, FP activations can be reformatted into integer values and FP arithmetic units (FPUs) can be replaced with integer units during MatMul operations. A naive pre-alignment for accurate com- putation requires very high-resolution integer units for the computation, which negates the benefits of using integer units. Inspired by an observation that conventional FP arithmetic does not guarantee the exact results due to rounding errors (Wilkinson, 1994), we show that the same level of compu- tational error can be obtained even when the pre-al pre-aligned significands are aggressively truncated. We then implement an integer-based FP arithmetic unit (iFPU) hardware for MatMul computation based on the proposed scheme. A comprehensive evaluation of the iFPU on various DNNs shows that the iFPU significantly improves energy efficiency and srouughutt-pe-arree over the conventional FPU-based MatMul engine while maintaining the neural network accuracy.
2 BACKGROUND
2.1 FLOATING-PPINT ARITHMETIC AND ROUNDING ERROR
FP format represents number as (- 11) x (m) x (ee-bias) which consists of sign (s), exponent (e), and significand (or mantissa, m) Muller et al. 2018). Float32 assigns bit for s and 8 bits for e. Precision (p), the effective bit count of the significand, is 24 bits (among which 23 bits explicitly stored). Bfloat16, which has been gaining popularity in the field of deep learning, intensely cuts down stored significand bits (compared to 23 in float32) to lower the total number of bits per value, and thereby reduces memory footprint (Wang & Kanwar 2019). The bias of the exponent term is usually set to half of the exponent maximum.
FP format can cover a wide range of numbers by separating the significant digits and the scale of the number. Note that because of the precision limits, there is a gap between two consecutive FP numbers. Such a gap is called unit of least precision (ulp) whose value is represented by the least significant digit. Hence, it hard to represent real numbers precisely with FP format even the numbers are in the dynamic range of the FP format, and rounding is required for converting real numbers into FP numbers. FP arithmetic typically normalizes significands for each computation, and the rounding operation is followed by the normalization to convert the computation result into FP number. Round-to -onn-tt--eearest is the most frequently chosen as a rounding mode where the difference between the real value and the round-off value can be large as half of ulp, and its relative error is bounded by e = Julp = 2 P, which is referred to as machine epsilon. Both ulp and e are widely used to evaluate the accuracy of numeric calculations (Goldberg) 199T).
2

---

Published as a conference paper at ICLR 2023
As every FP operation includes the rounding stage, rounding enror is unavoidable in FP arithmetic. Although the error of a single FP arithmetic operation may be small enough to be ignored, the er- ror can be substantial if series of multiple FP arithmetic results are accumulated. For example, an inner product of MatMul involves multiple FP additions in a row and the FP summation piles up the rounding error of each FP adder (Figure ( Accordingly, numerous solutions have been introduced to compensate for the eттоr of the FP summation (Muller et al. 2018)) Such error com- pensations cause an additional computation burden for tracking and fixing the error. Since the effect of the rounding errors on DNN accuracy is negligible, popular deep learning frameworks such as PyTorch and CuDNN (Paszke et al., 2019; Chetlur et al.) 2014) allow the rounding errors (without the compensation algorithms) in favor of simple computation. Note that as the level of rounding error depends on the precision p (only bits for b the error becomes noticeable for bfloat 6. Therefore, summation of bfloat16 values uses float32 adders (instead of bfloat 6 adders) to preserve Henry et al. 2019).
the accuracy of accumulated results (Wang & Kanwar 2019 Intel 2018
2.2 RELATED WORKS
Block Floating Point (BFP) has been proposed as a compromise between FP and integer formats. It assigns a single shared exponent to a group of FP values while maintaining individual signifi- cands (Wilkinson 1994). The BFP has drawn attention as flexible low-bit numeric format for quantization because the shared exponent can represent the dynamic range of values with little over- head. Hence, BFP can achieve a higher compression ratio than integer formats (Zhang et al., 2022a). In addition, since the individual significand values are integer, the BFP formats enable simpler com- putation than FP formats (Köster et al., 2017). Note that critical limitation in previous works based on BFP formats is that the same level of accuracy as that of conventional FP computations cannot be guaranteed (even theoretically). Previous works tend to find the optimal BFP formats with the least memory/compputtion density by evaluating DNN accuracy for various bit resolution and group sizes (Song et al. 2018: Lian et al. 2019 Rouhani et al. 2020). Another drawback in some previous works on BFP is that DNNs with BFP format need to be fine-tuned usually by quantization-awve aware training to improve the accuracy (Zhang et al. 2022a Rouhani et al.) 2020). Since a quantized neu- ral network allows only one fixed block size that is optimized for target hardware during training, neural network needs to be retrained for different hardware choices if a block size differs.
Truncated binary multipliers with error compensation schemes have been proposed to reduce the number of outputs in integer multiplications (Petra et al. 2009). While both the truncated mul- tipliers and our proposed work use the truncations to improve computational efficiency, there critical differences between them. In the truncated binary integer multipliers, the amount of the truncated bits is fixed while it varies in FP additions cases which our work focuses on. In addition, (Petra et al. 2009) presents truncation error correction function utilizing the fact that some of the truncated partial products share the same inputs with the remaining partial products, so they have correlations with the remaining partial product values. Unfortunately, in FP addition cases, the trun- cated significands do not have any correlation with the remaining bits so it hard to devise similar error correction schemes. Hence, there is strong need to develop alternative ways to control the truncation errors in FP operations.
3 RECONSTRUCTION OF FP-BASED MATMUL WITH INTEGER ARITHMETIC
3.1 OVERVIEW OF THE PROPOSED MATMUL RECONSTRUCTION AND COMPUTATION
In this section, we propose a methodology to reconstruct FP MatMul with integer arithmetic for efficient DNN computation, focusing on FP activations and quantized weights. In most cases, the weight matrix with m-bit quantization can be expressed where as is a scaling factor and B, is a binary weight matrix of each bitplane. Here, as can be a power of 2 for uniform quantization or can be an FP value for non-uniform quantization. MatMul is composed of multiple dot products, and a dot product between activations and weights is defined as Ek=1 (ag wx) (a: activation, w: weight, fan-in of the layer). If we apply binary-coded weights and properly change the order of the operations, we can rewrite the dot product as follows:
binary-coded matrix: Σm as BB
m
(as x Bo.k), Bo. E [-1,+11
(1)
b
Am
3

---

Published as a conference paper at ICLR 2023
2.1
-0.7
3.1 3.1
(Eq:1)
2.1 ADB
-0.7
1.1
+
3
003 2 Astivation
0.9
31
Weight
[STEP2]
[STEP1] Pre-Alignment (FP
Integer)
Integer Addition
[STEP3] Pormalarrll (Integer + FP)
1.1
123
Figure 2: Overview of the proposed MatMul computing scheme for DNNs with FP activations.
Bir-- Tine nning
Storage
BFP Computation
Proposed Computation
Sterage
22p
9
On the-flyyyyy car version
inttpp
experenn:
ataned
Sigiliaand
of The Musi
g=4
signifoand-
expareent
"shared
Previous Approach (e.g., MSFP)
Proposed Approach
Figure 3: Comparison of a previous approach (e.g., MSFP (Rouhani et al. 2020)) and the proposed
approach for applying block floating point (BFP) to DNN computation. In the case of MSFP, the original network needs to be retrained for the MatMul engines with different block sizes, but in the proposed scheme, the original network can be fed into the engines with any block sizes.
For each bitplane, MatMul of weights and activations is reconfigured as the addition/ssbbraationn of activation values except for a few at multiplications that are necessary to merge the outputs from each bitplane. Because FP mullully-accumulate operations require more hardware resources than FP additions, even such a reconfiguration of matrix multiplication to remove most multiplications can improve the efficiency of DNN computations significantly (Jeon et al. 2020). Even so, be- cause FP additions are still computationally more expensive than integer additions, replacing FP additions with integer additions can save even more energy and area. Therefore, we propose to reconstruct FP-based MatMul (Eq. D using integer additions (Figure - One of the key compo- nents of the proposed method is the pre-aligmment, which reformats the FP activation values into integer values on-the- on-the-fly by sharing the exponent value among the activations that are fed to dot product of the MatMul at time. The pre-alignment finds the maximum of the exponents among the activations and aligns corresponding significands simultaneously based on the difference of each exponent and the maximum exponent. As a result, unlike conventional FP arithmetic that performs the alignment for each addition, our proposed computing methodology aligns the activation values once per MatMul, and thus, reduces the overall cost the alignment process significantly. Note that as opposed to previous works that share the exponent among a block of inputs in the storage format (e.g.. MSFP (Rouhani et al. 2020), our design performs the exponent sharing during the compu- tation. Since different exponents representation power of the conventional FP format (Figure Because pre-aligned activations can be represented by the aligned significands which are integer values, an FP addition of the MatMul can be replaced by an integer addition. After the whole summation process, the proposed method reformats the summation results back to FP values by normalizing the results with the maximum exponent found in the pleganigmennt stage. Then, the computation results from each weight bitplane are multiplied by a and merged finish the target MatMul operation.
allowed in the storage format in our scheme, we keep the
As the exponent of float32 (or bíloat16) is 8-bit, the maximum amount of the significand shifting 255 and the resolution of the aligned activation becomes 279 (or 263) bits. Note that such a large bit width might negate the benefits of using integer units. For example, while 32-bit integer addition consumes 10.3% energy of float32 addition, 279-bit integer requires a level of energy per addition comparable to that of float32 addition (Appendix B.I.. To avoid the large design overhead, we propose to use only the top t(= + 5) bits of the aligned activation when 8 indicates the number of extra significand bits for reducing truncation error. Since the conventional FP addition also experiences errors due to truncation of significand, relatively small extra bits for the proposed method can derive a level of errors similar that of conventional FP addition (as described in Figure
3.2 COMPUTATION ERROR AFTER SIGNIFICAND TRUNCATION
To study the characteristics of errors in the proposed method with truncated significands, we first analyze the computation error with single addition/subbrractonn between two FP values x and y-
4

---

Published as a conference paper at ICLR 2023
We assume 2 y/ 2 O, Zg.21 of generality, because only the difference between the exponents decides the amount of shifting and truncation. Here, and denote the binary value of i-th significand bit, and the leading bit 2n is for & when æ > 0. When either k or is O, there is no need for significand shifting and truncation, and hence, integer-based FP arithmetic guarantee the precise computation without any extra bit (i.e., 8 == 0).
...-11 and y yo-y/1 1.11.---p-111 2 (k: 0) without loss
When & > ment, should be shifted to right by k, so can be rewritten as = 0.0 0yf -.. V where b is equal to As only the top t(= p + 6) bits of the significand remain after the trun- cation, the truncated result becomes 0.0 --- y/ ... yt-1. When 8 2 k, the difference between and ÿ Otherwise, the difference between and is bounded as follows:
need shift and truncate the significand of for the computation. For the align-
y 0.0 -oy- -h 2 2------11/-- 2 (k
The relative error of the addition with the truncated significand is defined follows:
(xx + y)
(x 5)
Eodd
x + y
x +
By applying both (xr y/
|r| and Eq. 2 to Eq.
we can obtain
Cadd < 2 (p+o 1) 2 6)) s 22+----))
(4)
Because the machine epsilon given as = P Cods S e when 1 and eadd I fee when ò
For subtraction, the relative error is defined similarly follows:
Ix y) (x Ø)
yy
Caub
(5)
vl
y
When 8 2 k, ly ø] is 0 so that Czub is 0. The minimum of x is 1, and y has the maximum value when all 1 are Correspondingly, |x - yl is bounded as follows:
0.11
P-
for k 1
x- vl
(6)
0.0 ... 01 ... 2 2 +2 2- +
.... .............. (k-1) for 1e 2 2
When & is and 5 is 0, get 1 5l < 2- from Eq. 2 For such case, according to Eq. and Eq. [] we have sub < 1. The worst case happens when x = 1 and y = 0.111 ... 1. When & by applying Eq. 2nd Eq. to Eq.5 we get caub I for d = and caub 1/2c for 5 2. As result, regardless of FP formats, the proposed method has the error level as summarized in the following Remark 1.
P
and
Remark The integer-based additionnsuubraacionn has the same level of error as that of the conventional FP addition/subbraction with I extra bit, and the error becomes half with 2 extra bits.
Note that the error of FP summation is the same as the accumulated value of errors from each addition Muller et al., 2018). The reconstructed MatMul, however, induces an additional stage of
FPU Proposed (6:
Proposed (6: 1)
Proposed (8: 2)
(a)
18
(b)
10-
Relative Error (10-)
Relative Error
12
102
6
10-3
10
128
256
512
1024
2048
4096
8192
128
256
512
1024
2048
4099
8192
Fan
In
Fan In
Figure 4: (a) Average and (b) maximum FP summation emrors of conventional FP computation and the proposed method with extra bits 1,2) against the accurate FP summations with Schewchuk algorithm (Shewchuk) 1997).

---

Published as a conference paper at ICLR 2023
converting integer summation results to FP values, and thus, additional rounding error during the FP formatting (Figure B(a)). For example, to sum 128 FP values, a conventional FP-based MatMul has 127 error sources with bound while the reconstructed MatMul with extra bit has 128 error sources with bound e such that the reconstructed MatMul might experience a slightly larger error than conventional FP-hased MatMul. Therefore, to guarantee the same error level as that of the conventional FP arithmetic, 2 extra bits are used for pre-alignment. Then, reconstructed MatMul has 127 error sources with bound 0.5e and an additional error source with bound e.
To verify the computation error of the proposed method, we randomly sample float32 values and compare the computation error of FP summation between conventional FP computation and the proposed method. To explore wide range of float32 values, we sample e, and m values indepen- dently assuming a uniform distribution, and then concatenate those values. We vary the fan-in (i.e., the number of values to be accumulated) from 1 128 to 8 8192, and sample 50,000 sets of FP numbers for each fan-in selection. The Schewchuk algorithm is employed to obtain accurate FP summation baseline data for error measurement (Shewchuk 1997).
As shown in Figure the proposed method produces a similar level errors to that of the conven- tional FP arithmetic for various fan-in values when 6=2. Because larger errors are more likely to be accumulated with larger fan-in, we see that both average and maximum errors tend to grow as the fan-in increases (Figure 4.. Nonetheless, the average error (12.3 x 10-?) and the maximum error (2.4 x 10-2 or 2.4%) are relatively small even with 8192 fan-in, which justifies the current practice of implementing conventional FP additions without error correction for DNN inference. Corre- spondingly, the proposed method can support as precise numerical computation as conventional FP arithmetic does.
4 EXPERIMENT
4.1 IFPU: A MATMUL ENGINE FOR THE PROPOSED METHOD
Overall Architecture. To evaluate the proposed method with real hardware implementation, we first design a MatMul engine called iFPU. Figure the overview of systolic iFPU architecture which adopts the design principle of Google's s TPU [Jouppi
weight
"Pre-alinment Unit
Pre-alignment Unit
PE
PE
PE
lg2nl
Unified Buffer
achivnno
Exponent
Find Max
PE
PE
fo2nt
pertial
al.] 2017). iFPU performs FP MatMul in the form of a set of FP summation (Eq. ( that is physi- cally implemented as integer summation for high ef- ficiency. After the computation, the iFPU converts integer results into FP values through the int2fp con- verter at the end of the Processing Element (PE) ar- rays. Then, scale & accumulator is used to multiply ag and add summation results of each weight bit- plane to finish the MatMul (Eq. []]. The size of MatMul that can be processed in the iFPU at a time is bounded by the number of PFs, and as practical design, we evaluate the iFPU with 32x 64 64, or 128x 128 PEs for the experiment. When fan-in of the DNN layer exceeds the row count of PEs, activations of the layer are tiled fit the row-count limit, and each tile fed into the iFPU at a time and processed with integer adders in the PEs. To complete the entire MatMul, the computing results for different tiles should be merged, and for this, float32 adders (accumulator) are used again.
sum
fo2int
PE
PE
PE
Int2fp Converter
shifter
Scale & Accumulator
Shift aamcont
*PE: Processing Element (Inlegar Adder)
Figure 5: A block diagram of iFPU
Precision of Integer Adder. As the PE array of the iFPU accumulates the pre-aligned and truncated significands, the size of the integer adder in each PE depends on f, which is determined by the precision of the given FP format (p) and extra bits (d) attached to control truncation error. Based on the theoretical analysis given in Section 3.2 the iFPU for float32 activations conducts 26-bit integer addition with 5 = 2. Though the iFPU introduces additional FP accumulations due to the MatMul tiling, the error level integer-based FP addition with = is half of the conventional FP addition according to Remark [ Therefore, the iFPU with d = can still preserve the same level of computing error as that of conventional FP MatMul (Figure G(a)). Furthermore, the iFPU for bfloat16 activations
be designed to be even smaller and more energy efficient by using smaller precision integer adders thanks to the reduced bit precision for significands. Interestingly, conventional 6 accumulation still uses float32 adders preserve the accuracy of accumulated

---

Published as a conference paper at ICLR 2023
Relative Error (10-1
FPU IPU (O:
IFPU 16:
IFPU
128
512
2048
8192
120
512
1024
2043
4056
8182
120
2048
8182
32x32 PEs
64x64 PEs
128x128 PEs
(a) Evaluation with float32 activation
Relative Error (10-))
FPU
IFPU
(6:
IPU 26
(F:3
126
512
1024
2348
8192
120
512
2048
4086
8182
120
512 1024
2048
4098
8192
32x32 PEs
64x64 PEs
128x128 PEs
(b) Evaluation with 6 activation
Figure 6: Numerical computation errors of MatMul for DNNs with FP activation. We measure the computation error of conventional FPU-based engine and the proposed iFPU against the accurate FP computation with Schewchuk algorithm (Shewchuk) 1997). The number of PEs and fan-in annotated along the horizontal axis.
Cosine distane
12 x10:
12
10
x10s
mean
6
0
5
10
12
10
11
13
9
10
12
32x32 PEs
64x84 PES
128x120 PEs
(a) Evaluation with float32 activation
Cosine distance
x10-
25
x10
X10=
std
minimax
15
8 PEs
10
11
12
6 64x64
10 11 18
8 9
10 12
32x32
PES
128x120 PEs
(b) Evaluation with bfloat16 activation
Figure 7: Cosine distance between MatMul results of BERT-base (task: MRPC) extracted from inference results using conventional FPU-based engine (NVIDIA RTX3090) and the proposed iFPU. The last feed-forward layer in each encoder block (1-12th layers) and pooler (13th layer) is used for the evaluation. The number of PEs and layer indices are annotated along the horizontal axis.
results (Wang& & Kanwar 2019 Intel 2018 Henry al. 2019). However, the accumulated results are converted back to bfloat1 6, is possible to maintain the accuracy of bfloat 16 accumulation with less accurate adders than float32 adders. Figure (b)) shows that the proposed 66 iFPU with = 3 (which uses bit adders) provides comparable accuracy to that of conventional 6 adders.
4.2 ANALYSIS OF THE DNN COMPUTATION ACCURACY
MatMuls of DNN with iF vs FPU. In the previous section, we compared the accuracy of the proposed integer-based FP MatMul with precise results. Since our goal is replace the FPU with the proposed iFPU, it is also important to compare the computational difference between the conven- tional error-prone prone FPU-based engine and the iFPU. For an in-depth understanding of DNN inference with the iFPU, we first compare the inference output of each layer in the BERT-base model (Devlin) (et al., 2018) computed with an FPU-based engine (NVIDIA RTX3090) and the proposed iFPU. BERT-base uses 4-bit weight values and the target task is MRPC. In iFPU, MatMuls between weights and activations are processed with the proposed integer- approach, but other opera- tions such softmax
processed by using conventional FPU. We employ cosine distance the metric to measure the difference in layer outputs. Note that the cosine distance is for two identical vectors and 2 for entirely opposite vectors. In this experiment, the last feed-forward layer in each

---

Published as a conference paper at ICLR 2023
Table I: Accuracy of DNNs inference with conventional FPU-based engine (NVIDIA RTX3090) and proposed
PE arrays). The numbers parentheses represent accuracy difference between FPU & iFPU.
float32 activation
bfloat16 activation
VGG-9
ResNet-18
OPT-1.3B
VGG-9
ResNet-18
OPT-1.3B
FPU
92.91
70.27
12.96
92.91
70.28
12.96
iFPU-32
92.91 (+0.00)
70.27 (+0.00)
12.96 (+0.00)
92.91 (+0.00)
70.26 (-0.02)
12.96 (+0.00)
iFPU-64
92.91 (+0.00)
70.27 (+0.00)
12.96 (+0.00)
92.90 (40.01)
70.27 (-0.01)
12.97 (+0.01)
iFPU-128
92.91 (+0.00)
70.27 (+0.00)
12.96 (+0.00)
92.92 (+0.01)
70.26 (-0.02)
12.98 (+0.02)
ResNet-50
RegNet
MnasNet
ResNet-50
RegNet
MnasNet
FPU
76.32
78.18
75.99
76.33
78.17
75.96
iFPU-32
76.31 (-0.01)
78.18 (=0.00)
75.99 (+0.00)
76.38 (+0.05)
78.18 (=0.01)
75.97 (+0.01)
iFPU-64
76.31 (40.01)
78.18 (-0.00)
75.99 (+0.00)
76.38 (+0.05)
78.18 18.18 (+0.01)
75.96 (+0.00)
iFPU-128
76.31 (4.001)
78.18 (+0.00)
75.99 (+0.00)
76.40 (+0.07)
78.18 (+0.01)
75.97 (+0.01)
BERT- BASH w/ float32 activation
CoLAA
MRPC
SST-2
STS-B
QQP
MNI.Imm/mm
QNLI
RTE
Avg.
FPU
56.36
89.05
91.51
87.52
83.73
81.95/82.56
89.00
70.04
81.28
iPPU-32
56.36
89.05
91.51
87.52
83.73
81,95/82.56
89.00
70.04
81.28 (+0.00)
iPPU-64
56.36
89.05
91.51
87.52
83.73
81.95/82.56
89.00
70.04
81.28 (+0.00)
iFPU-128
56.36
89.05
91.51
87.52
83.73
81.95/82.56
89.00
70.04
81.28 (+0.00)
BERT BASE 6 activation
CoLA
MRPC
SST-2
STS-B
QQP
MNLI-m/mm
QNLI
RTE
Avg.
FPU
56.08
89.05
91.51
87.52
83.74
81.97/82.57
89.00
70.04
81.30
iFPU-32
56.10
89.05
91.51
87.52
83.72
81.94/82.55
89.05
70.04
81.28 (-0.02)
iFPU-64
56.36
89.05
91.63
87.52
83.72
81.9 56
89.00
70.04
81.31 (+0.01)
1FPU-128
56.10
88.83
91.63
87.52
83.72
81.96/82.54
89.05
70.04
81.27 (40.03)
encoder block and pooler is chosen for evaluation. Figure shows that the FPU and the iFPU pro- duce almost identical outputs for each layer. The averages of the distance are less than 1.2 x 10-6 and 2.5 x 10-1 for float32 and 6 activations, respectively. Moreover, the distance between layer outputs from the two engines remains close throughout the forward path. As a result, we can expect that the proposed iFPU can support DNN inference with almost the same accuracy as that of conventional FPU.
DNN Inference Accuracy. We select 7 types of DNN models to compare DNN model accuracy between the FPU and iFPU: BERT-base, VGG-9, ResNet-18, ResNet-50, RegNet-3.2GB MnasNet- 2.0, and O.T-1.3B. The accuracy of BERT-base is evaluated on the General Language Understand- ing Evaluation (GLUE) benchmark (Wang et al. 2019). VGG-9 (Simonyan & Zisserman 2014) is
evaluated on CIFAR-10 10 (Krizhevsky al. 2009) ResNet-18. ResNet-50 (He et al. 2016). RegNet- 3.2GF (Radosavovic et al. (0220). and MnasNet-2.0 (Tan et al.) 2019) measure top-1 accuracy on al.) 2015). OPT-1.3B (Zhang al. 2022b is an open-sourced NLP model provided by Meta Al roughly matching the performance and sizes of the GPT-3 class of mod- els and is evaluated by estimating the perplexity on WikiText-2 dataset (Merity et al., 2016). All DNN models use 4-bit weight values that are quantized by a binary-coding quantization scheme. Note that no modifications to DNN structures are needed to deploy the weight-quantized DNNs to various iFPUs because 1) activations are FP values and 2) iFPUs are designed to process any Mat- Mul for DNNs as long as weights are quantized. Table [ summarizes the DNN inference results. Because the iFPU can produce almost identical MatMul results FPU, the proposed iFPUs preserve the DNN accuracy for both float32 and 6 activations as we expected.
ImageNet (Russakovsky
4.3 ANALYSIS OF COMPUTATION EFFICIENCY
Setup. To evaluate the efficiency of proposed iFPUs, synthesize the proposed hardware in 28mm CMOS technology. For a fair evaluation of the impact of replacing FP MatMul with integer- based MatMul, we also design two 'baseline" engines for the conventional FP-based MatMul (Fig-
8

---

Published as a conference paper at ICLR 2023
ure As the first baseline (FP-MAC), Figure aaa) is designed with FP MAC units to process FP MatMul as a naive approach. In addition, as the second baseline (FP-ADD), Figure 8(b) is designed with FP adders to process FP MatMul reconfigured as Eq. [ Because bitplanes of weight values decomposed for FP-ADD and iFPU, binary weights are processed in a bit-paralle manner in FP- ADD and iFPU, while FP-MAC processes the whole weight values in each MAC unit. Compared to those two baseline engines, iFPU exhibits the lighter PEs along with additional units such as the pre-alignment unit and int2Ip converter. Lastly, an int8 MatMul engine (INT8) also implemented for the comparison between the proposed iFPU MatMul and integer MatMul.
Results. Simulation results using the synthesized hard- ware demonstrate that the proposed iFPUs can improve both energy and area compared the baselines, as the FP units of the baseline engines are replaced with the more area/energy efficient integer units (Figure 99) For float32
Unifiedd Buffer
FP
Uniffed Buffer
MAC
MAC
Add
Add
FP
FP
MAC
MAC
Add
Add
activations, the proposed iFPU improves throughput-per area (TOPS/mmm²) by up to 7.9x and energy efficiency (TOPS/W) by up to 6.4x compared to the FP-MAC base- line. For bfloat16 activations, the proposed iFPU achieves even larger improvements because the size of the corresponding integer-based unit is reduced the bit resolution of the aligned-truncated significands is reduced by 15 bits compared to float32 activa- tion cases. The prooughput-per-reee of the iFPU is improved by up 9.9x and energy efficiency is enhanced by up to 11.9x compared to the FP-MAC baseline. The improvement over the baseline becomes larger as the number of PEs increases because the overhead of additional logic such as pre-alignment units in the proposed scheme can be amortized (detailed in Appendix C.2). We also compare the iFPUs with the INT8 engine. While bfloat16 activations close the gap between the FP-MAC baseline and the INT8 engine significantly in terms of throughput-per-are 6 activations) achieves even higher energy efficiency than the INT8 engine in some cases (Figure 9b.
(a)
Accumulator
(b)
Accumulator
Scale
Figure 8: Baseline MatMul engines (a)
FP-MAC and (b) FP-ADD
iFPU (with
12
FP-MAC
FP-ADD
Norm. TOPS/mm²
Norm. TOPSWW
IPPU
88.
INT8
6.2
7.4
8.9
7.9
5.0
7
1.0
3.6
3.6
3.6
4,1
2.88
2.8
11.5
0.0
Noa132 blicarté int
foet32 blooarte ims
float32 blloaté6 ni8
1oa62 bloar16 int8
foaat3 bloal16 into
flcat32 66cca16 Int
32x32 PEs
64x64 PEs
128x128 PEs
32x32 PEs
64x64 PEs
126x128 PEs
Figure 9: Normalized energy efficiency (TOPS/W) (left) and throughput-pers cichhput-per-aree per-area (TOPS/mm²) (right) of MatMul Engines: baselines and iFPUs for FP MatMul; INT8 for int8 MatMul. The number of PEs and target activation types are annotated along the horizontal axis.
CONCLUSION
The need to accomplish computing MatMul by using FP activations and quantized weights is in- creasing due to the growing usage of complex non-linear activation functions DNN models such as Transformers. Conventional computing platforms such as CPU, GPU, and NPU, however, inefficient in performing such computations. In this paper, we propose new MatMul comput- ing scheme dedicated DNNs with FP activations and binary-coding weight quantization. The proposed method accelerates the FP MatMul of DNNs using the shared exponent and the integer arithmetic improve computational efficiency. Previous works which also used the block floating point number with shared exponent often claim the validity of their design by presenting comparable DNN accuracy without verifying the robustness of MatMul results in a rigorous manner. We theo- retically prove that the proposed scheme can produce the same error level as that of conventional FP arithmetic. To evaluate the computational efficiency of the proposed method, we design and syn- thesize a MatMul engine, iFPU, following the principle of integer-based operations. Experimental results support our claim that, compared to the conventional FPU-based design, the iFPUs accelerate the wnlg quantized DNNs with 6.4x and 7.9x higher energy efficiency and throughput- per- area for float32 activations, respectively. In addition, the iFPUs yield 1 11.9x x and 9.9 x higher energy efficiency and throug peraaara,, respectively, when associated with bfloat 6 activations.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University) (10%). and 200-202----,068, Artificial Intelligence Innovation Hub (10%)).
REFERENCES
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. International Conference on Learning Representations, 2019.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint
2018.
Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, and Dongsoo Lee. Extremely low bit transformer quantization for on-device neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020,
pp. 4812 4826, 2020.
Jacob Devlin, Ming-Wey Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732. 2022.
David Goldberg. What every computer scientist should know about floating-pon arithmetic. ACM computing surveys (CSUR), 23(1):5-48, 1991.
Mark Harris. Mixed-precisson programming with cuda 8, 2016. URL htttpp:://deeeopp:::
nvicie www.///llg/miieeeecccoocooooeoce pttp://wwwieedeiiiieeeeeeeee pwwcisionpprorarmaei.....n potnnnproorrmmmmmee ng cuda 8/]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Greg Henry, Ping Tak Peter Tang, and Alexander Heinecke. Leveraging the 6 artificial intel- ligence datatype for higher-precising computations. 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp. 69-76. IEEE, 2019.
Mark Horowitz. Computing's energy problem (and what we can do about it). In 2014 IEEE Inter- national Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, IEEE,
2014.
Intel. B
hardware numerics definition, 2018.
URL
https: wwww.
intel.comm/ cwm http://onnnaaete//ea//aaaallllea /occummens//
bfl6 www.://wwwee-eeeeeee-e-eeaaaaaa. 07.
pdf
Accessed: 2022-09-
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-rrihmmtiioonny inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018
Yongkweon Jeon, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Jeongin Yun, and Dongsoo Lee. Biqgemm: matrix multiplication with lookup table for sinary-coding-bare quantized dnns. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14. IEFE, 2020.
10

---

Published as a conference paper at ICLR 2023
Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. Mr.biq: Post-training non-uniform quantization based on minimizing the reconstruction error. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12329-12338, 2022.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, pp. 1-12, 2017.
Norman P Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, al. Ten lessons from three genera- tions shaped google's tpuv4i: Industrial product. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 1-14. IEEE, 2021.
Schoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer- only bert quantization. In International Conference on Machine Learning, pp. 5506-5518 PMLR, 2021.
Urs Köster, Tristan Webb, Xin Wang. Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Hornff, et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. Advances in neural information processing systems, 30, 2017.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Wop Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization- aware gurameter-efficient adaptation of large-scale pre-tra pre-trained language models. arXiv preprint arXiv:2210.03885, 2022.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In Interna- fional Conference on Learning Representations, 2021.
Xiaocong Lian, Zhenyu Liu, Zhourui Song, Jiwu Dai, Wei Zhou, and Xiangyang Ji. High- performance ipga-based enn accelerator with block-floatingg soock--foating-oing arithmetic. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 178))1874-1185, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1 600000843,, 2016.
Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. Wrpn: Wide reduced-preeission networks. In International Conference on Learning Representations, 2018
Jean-Michel Muller, Nicolas Brisebarre, Florent De Dinechin, Claude-Pierre Jeannerod, Vincent Lefevre, Guillaume Melquiond, Nathalie Revol, Damien Stehlé, Serge Torres, et al. Handbook of dloating-poin arithmetic. Springer, 2018
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.095:: 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, pighhpeeformance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
Nicola Petra, Davide De Caro. Valeria Garofalo, Ettore Napoli, and Antonio GM Strollo. Truncated binary multipliers with variable correction and minimum mean square error. IEEE Transactions Circuits and Systems Regular Papers, 1716) 2009.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10428-10433, 2020.

---

Published as a conference paper at ICLR 2023
Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of narrow pre- cision inferencing at cloud scale with microsoft floating point. Advances in neural information processing systems, 33: 10271-10281, 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-25,, 2015.
Jonathan Richard Shewchuk. Adaptive precision floating-point arithmetic and fast robust geometric predicates. Discrete Computational Geometry, 18(3):305-363, 1997.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv:4099 1556, 2014.
Zhourui Song, Zhenyu Liu, and Dongsheng Wang. Computation error analysis of block floating point arithmetic oriented convolution neural network accelerator design. In Proceedings of the AAAA Conference on Artificial Intelligence, volume 32, 2018.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295-2 -2229, 2017.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 2820-2828, 2019.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. arXiv preprint arXiv:2203. 10705, 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations, 2019.
Shibo Wang and Pankaj Kanwar. Bffoat16: The secret to high performance on cloud tpus. Google Cloud Blog, 4, 2019.
James Hardy Wilkinson. Rounding errors algebraic processes. Courier Corporation, 1994.
Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, and Yuxiong He, Extreme compression for pre-trained transformers made simple and efficient. arXiv preprint arXiv:2206.00 01182, 2022.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. In International Conference Learning Representations, 2018.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for scale transformers. arXiv preprint arXiv:2206.0 861, 2022.
Sai Qian Zhang, Bradley McDanel, and HT Kung. Fast: Dnn training under variable precision block floating point with stochastic rounding. In 2022 IEEE International Symposium High- Performance Computer Architecture (HPCA), pp. 860. IEEE, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2005.01068, 2022b.
12

---

Published as a conference paper at ICLR 2023
A COMPUTATIONAL COST OF FP ARITHMETIC VS. INTEGER ARITHEMETIC
Table 2: Energy of computing units synthesized in 28mm tech node (MAC: multiply-accumullte))
MAC
Multiply
Add
float32
int8
float32
int32
int8
float32
int32
Energy per Operation Normalized Energy
1.51 pJ
0.08 pJ
1.23 pJ
0,94
0.06 pJ
0.28 pJ 9.3x
0.03 pJ
x 1.0x
20.5x
15.7x
1.0x
1.0x
To cover wide range of numbers, FP format does not fix the location of the radix point (Goldberg) 1991). Hence, FP arith- metic needs handle input and output values with different scal- ing factors, and the FP arithmetic units need to align and normal- ize significands before and after each computation, respectively. The alignment and normalization logics consist of barrel shifters that can shift data word by a specified amount, and the cost of the barrel shifter far exceeds the cost of other arithmetic logics in terms of both energy and area, increasing the cost of FP computa- tion (Horowitz) 2014). Hence, in general, integer arithmetic logic much smaller and consumes less energy than FP counterpart.
Floating-Point Multiply- Accumulator
Floating- Point Adder
Integer Adder
32-bit
It is well known that 8-bit integer achieve up to 4x through- put improvement compared to IEEE-754 single-precision format (float32) in widely used GPUs (Kim et al. 2021), as the throughput of 8-bit operations is generally 4x that of 32-bit operations (Har-
Figure 10: Area comparison of computing units (layouts synthesized in 28nm node).
ris 2016). The advantage of integer ASIC (Mishra et al. (2018). For in-depth understanding, we synthesize computing units for FP and integer 28nm tech node. As shown Table 2 multppicattonnacccuulaaa
be magnified when the hardware platform moves to
(MAC) for float32 consumes 18.9x more energy than 8-bit integer (int8), widely used integer format for quantized DNNs. Please note that float32 MAC consists of a float32 multiplication and a float32 addition while an int8 MAC consists of an int8 multiplication and an int32 addition. The bit resolution of the adder the int8 MAC higher than that of the multiplier, because int8 multiplication results in 16-bit values and the bit resolution of MAC values increases the number of accumulated values increases for integer format. addition, the area cost of the integer unit is also much smaller than FP units as shown in Figure I0, Therefore, many studies have attempted activation quantization despite the various difficulties the quantization process because both weight parameters and activations should be quantized to replace FP arithmetic with integer arithmetic.
B SUPPLEMENT FOR PROPOSED SIGNIFICAND TRUNCATION
B.1 ENERGY IMPROVEMENT WITH SIGNIFICAND TRUNCATION
TOL1O101
X2"
1011010100
float32
0.281
1.00 N010011
2
1.000100100
2
int279
0.281
11011101
2
0.000000011
int32
0.029
Align
Truncate
0.0
0.1
0.2
0.3
100001001001 10000
2
(p 8, 2-2)
Energy per Addition (pJ)
0. 0.00000001 1O
Figure 11: Energy of adders synthesized in 28nm tech node (tested 0.9V).
Figure 12:
Example of the significand trunca- tion followed by the pre-alignment.
With naive pre-alignment of float32 activations, the maximum amount of the significand shifting is 255 and the resolution of the aligned activation becomes 279 bits. As shown in Figure [T] while 32-bit integer consumes 0.029 pl per addition, both float32 and 279-bit integer consumes 0.281 pl
13

---

Published as a conference paper at ICLR 2023
per addition. To avoid the large design overhead, we truncate the pre-aligned significands as shown in Figure 12. The aggressive truncation still did not cause accuracy degradation in FP additions as described in the Section 3.2
B.2 TRUNCATED BINARY MULTIPLIERS Vs. PROPOSED SIGNIFICAND TRUNCATION
(a)
XI
X2
X3
X4
(b)
X1
X2
Xa
X4
x x
X
Y1
y2
Y3
Y4
Y1
Y2
ya
Y4
K1yı
Xiya
Xiya
K
X1
X2
X3
X
X2y1
X:ya
Xay
x
>
Y
Y2
Ya
Y4
X:Yı
X:Y2
Xsy3
X
(b a) bit shifting
X-Y1
Xayz
X4y:
84ya
inmcctton bound
truncation bound
Figure 13:
Comparison of the truncation scheme in the (a) truncated binary multiplier for integer multiplication and (b) proposed method for FP addition/ssbbraction..
Truncated binary multipliers (Petra et al. 2009) also discuss the truncation to improve computa- tional efficiency, but there are critical differences between truncated binary multipliers and the proposed work as summarized in Figure 13 First of all, truncated binary multipliers deals with integer multiplications while the proposed work focuses FP additions/subiractions Due the differences in the number format (integer vs. FP) and arithmetic operations (multiplications vs. aoditiions/subbacctions)) the two works present completely different error analysis models and error reduction schemes.
The error analysis models between truncated binary multipliers and our case are different, because the amount of truncation is fixed in the truncated binary integer multipliers and the amount of trun- cation varies our work as the amount significand shift varies depending on the input data. Moreover, in truncated binary multipliers, the bit resolution of truncated output is defined by the application requirement. On the other hand, as we proposed to truncate the pre-aligned values to adopt lower-bit integers and improve computational efficiency, the proper bit resolution of truncated values should be found to meet the accuracy requirement in our case.
In addition, in integer multiplication case, some of the truncated partial products share the same inputs with the remaining partial products, so they have correlations with the remaining partial product values. (Petra al. 2009) proposed an error minimization scheme which exploits such characteristics. On the other hand, in the FP addition/subtractin case, the truncated significands do not have any correlation with the remaining bits schemes. Instead, we focused on the fact that conventional FP operation is also not precise due to the rounding of output significands so that we only need match the error level of the proposed scheme to the conventional FP operations. Based the facts, we showed theoretical analysis such additionnsubbtaaction can have the similar error level as that of
hard devise similar error correction
that the proposed integer-based the conventional FP additiinn/subraacionn when small number (1-2) of extra bits are attached to the shifted significands. With this finding, we can design an efficient integer-based FP addition logic without having complex error correction function estimated based on the truncated bits.
C IN-DEPTH HARDWARE ANALYSIS
C. DETAILED HARDWARE DESCRIPTION OF THE PROPOSED IFPU
Figure 14 describes the proposed iFPU in detail. The proposed iFPU bit-flexible accelerator which can handle variable bitwidth of weight values. The iFPU processes weights in bit-parallel manner by processing each weight bitplane in different columns of the PE array. For example, 4-bit weights use 4 PE columns for the computation, and 8-bit weights use 8 PE columns for the computa- tion. After the integer- based summations are done each column of the PE array, the integer results converted into FP values and multiplied by scaling factors which represent the significance of each bitplane. Then, computing results of each bitplane are merged in the accumulator (FP adder) to finish the MatMul. As the output resolution of FP accumulation remains the same regardless of the
14

---

Published as a conference paper at ICLR 2023
weight
Pre-alignment Unit
Processing Element (PE)
PE
PE
PE
Pre-alignment Unit
f2ni
Unified Buffer
activation
(t log:R ) bit Integer Adder
Exponent Find Max
PE
PE
PE
fp 2int
*Ri HPE vi row dim
181?
1832
partial
Int2fp Converter
sum
2int
PE
PE
PE
sign-magnitude representation
int2fp Converter
B2R
Cmax
Bnex
Exponent
Nomallzatoin
Gmax
shifter
Adjust
Rounding
Scale (32-bit FP Multiplier)
shift amount
Ol
sign
exponent
mantissa
Accumulator (32-bit FP Adder)
6max
32-bift
Figure 14: A detailed block diagram of iFPU. The iFPU processes weights in bit-parallel manner by processing each bitplane of the weights in each column of the PE array. (Bo, binary weights in Eq.
size of the accumulation thanks to the characteristics of the FP format, the size of the accumulator does not need increase for the increased weight bit width.
C.2 AREA/ENERGY BREAKDOWN OF PROPOSED IFPU
PE arrays
Pre-Alignment Unit
In12fp
Scale Accumulator
Others
IFPU-bfloat16
IFPU-bfloat16
IFPU-float32
iFPU-Hloat32
FP-ADD
FP-ADD
FP-MAC
FP-MAC
2
5
0.1
0.2
0.6
Area (mm?)
Power (W)
Figure 15: Area (mm²) (left) and power (W) (right) of MatMul Engines: baselines and iFPUs for FP MatMul with 32x32 PEs.
arryys
Pre-Alignment Unit
Int2fp
Accumulator
100%
100%
Power Breakdown
Area Breakdown
50%
50%
0%
0%
32x32 PEs
64x64 PEs
126x128 PEs
32x32 PEs
64x64 PEs
128x120 PEs
Figure 16: Area breakdown (left) and power breakdown (right) of proposed iFPUs with 32x32, 64x64, and 128x128 PEs.
In this section, the area and power of the MatMul engines designed in Section 4.3 more detail for deeper understanding of the proposed scheme. First, a breakdown of the area/power of various MatMul engines with 32x32 PEs is shown in Figure FP-ADD reconstructs FP-MAC with a series of FP additions by separately processing each weight bitplane (Eq. [ so to match the effective throughput of FP-ADD with that of FP-MAC in case of 4-bit weights, 4 FP-ADD
analyzed in
15

---

Published as a conference paper at ICLR 2023
operations used for the evaluation. Hence, though the area/energy of single float32 adder is lower than that float32 MAC unit (Table 2). FP-ADD requires slightly larger area and power than FP-MAC. On the other hand, though iFPU also introduces times more operations than FP- MAC, iFPUs achieve large area and power reduction as the area/energy cost PE arrays become significantly lower by replacing FP adders with integer adders. The area/power reduction is even larger in 6 cases because smaller integer umits can be used. As the area/power cost of PE arrays iFPUs decreases, the relative portion of area/power of supporting logic (such as scale & accumulator) in the total area/power increases. Hence, the supporting logic accounts for more than half of the total area/power of iFPUs with 32x32 PEs. Meanwhile, the overhead of the supporting logic decreases the size of PE arrays increases. We report the area/power breakdown of iFPUs with various number of PEs in Figure T6 The experimental results show that, as the size of arrays increase, the supporting logic is shared among more PEs and the overhead can be amortized.
C.3 IMPACT OF THE WEIGHT BITWIDTH THE PROPOSED 1FPU
64
Weight bits:
12
14
15
Norm. TOPS/W
float32
bfloat16
float32
bfloat16
float32
bfloat16
32x32 PEs
64x64 PEs
128x128 PEs
(a) Normalized energy efffciency (TOPS/W) of iFPUs
Weight
Norm. IOPs/mm²
16
float32
bllaa116
float32
bfloat16
float32
blloat16
32x32 PEs
64x64 PEs
128x128 PEs
(b) Normalized throughput-per-area (TOPS/mmm²) of iFPUs
Computational efficiency iFPUs normalized with that the baseline FP MatMul engine (FP-MAC). Y-axis is the normalized value against FP-MAC and the iFPUs show higher effi- ciency than FP-MAC even for high- nigh-precision weight bits. The number of PEs and target activation types are annotated along the horizontal axis.
Figure 17:
This section analyzes impact of weight bitwidth on the efficiency improvement achievable with the proposed iFPU. The experimental setup is the same as Section 4.3 except the weight bits. While only 4-bit weight cases are evaluated in Section 4.3 this section evaluates weights with to 16 bits. Because the proposed scheme processes each bitplane of the weights the bit-parallel manner, higher-bit weights require more operations with PE, scale, and accumulators. Hence, as shown in Figure [77 the benefits of the iFPUs diminish as the number of weight bits increases. Nevertheless, even for 8-bit weight case, iFPUs achieve better computational efficiency compared to the FP-MAC baseline.
C.4 COMPARISON OF THE PROPOSED IFPU WITH INT4 MATMUL ENGINE
In Figure [8] an int4 MatMul engine (INT4) is evaluated and compared with the other MatMul engines analyzed in Section 4.3 INT4 MatMul shows high energy efficiency and throughput-per-
16

---

Published as conference paper at ICLR 2023
14
FP-ADD
1223
11..9
Norm. S0Pm/
2022
26.9
Norm. TOPS/W
IFPU
INTS
INT4
17.9
7
16.0
1022 0716 nnt kd44
152 br16 mis n4
1p92 br16 in int4 128x126 128 PEs
1p32
bife In19 mn44
1p32 bife nil n14
1052 BF16 infs ml/4
32x32 PEs
64x64 PEs
32x32 PEs
64x64 PEs
128x1 28 PEs
Figure 18: Normalized energy efficiency (TOPS/W) (left) and throughput-pers
(TOPS/mm²)
(right) of MatMul Engines: baselines and iFPUs for FP MatMul; INT8/INT4 for int8/int4 MatMul. The number of PEs and target activation types annotated along the horizontal axis.
area. However, to take advantage INT4 MatMul, both weight and activation should be quantized to bits, which may not provide desired accuracy in many cases.
C.5 HARDWARE EVALUATION WITH MEMORY ACCESS
BERT-base
<PP-1133
Normalized Energy
1.0
x1.7
x1.7
Mathiul Engine
x360
x2.7
x4.7
X40
x3.1
x6.6
On-Chip SRAM
0.5
DRAM
0.0
MAC ADD IFPJ
FP.
FP. FP- MAC ADD IFPU bfloat16
MAC ADD IFPU float32
FP. MAC ADD FPU
FP.
float32
bfloat16
Figure 19: Normalized energy consumption of MatMul engines (FP-MAC, FP-ADD, and iFPU) with memory system. The inference energy is measured for BERT-based and OPT-1.3B with 4-bit weights and float32/ffloat16 activations.
Setup. To understand the effectiveness of the proposed method in the real computing scenario, the baselines (FP-MAC and FP-ADD) and the proposed iFPU with 128x 128 PEs are further evalu- ated including memory access. For off-chip memory, we scaled down the bandwidth of HBM2 in TPU (Jouppi et al. 202T) considering the ratio of the number of PEs that make up Matrix Multiply Unit (MXU). which 1:4 and adopted energy per bit of HBM2 from Table 2 in (Touppi al. 2021); we used the bandwidth of 153.5 GB/s and the energy per bit of 3,9 pJ/bit. We also scaled the size of the unified buffer (on-chip SRAM buffer) in (ouppi et al. 2021) by dividing by 4. The unified buffer size in our design was 32MB. For SRAMs, we used the 28nm CMOS memory compiler and the energy per bit of 0.155 pJ/bit was used. To overlap memory access with computation, double buffering scheme was adopted in the unified buffer.
Results. We evaluate single batch inference of BERT-base and OPT-1.3B. We set the sequence length of BERT-base and O.T- as 128 and 1024 respectively. As double buffering hides the memory access latency, the proposed iFPU with memory model can achieve the same amount of throughput-per-te improvement as that of the baseline for the case in which memory access is not considered. the other hand, the gain in energy efficiency slightly changes after considering memory access. As shown in Figure [9] the dram access energy accounts for relatively small portion of total energy consumption in the baselines, because the data intensively reused in the MatMul computation. As the proposed iFPU reduces the energy cost of computation, memory access energy becomes relatively significant the proposed system. Thus, when considering the cost of memory access, the amount of improvement in the energy efficiency slightly decreases. Nevertheless, the iFPU with memory access still can improve the energy efficiency by up to 6.6x compared to FP-MAC baseline.
17

---

Published as a conference paper at ICLR 2023
D FINE-TUNING CONDITION FOR BERT-BASE TRAINING
Table 3: papem-parameterss for fine-tuning BERT-base on GLUE benchmark. The fine-tuning use
A optimizer and the number of training epochs is 10. The learning rates decay linearly and the weight decay is set to 0.01.
GLUE
Configuration
CoLA
MRPC
SST-2
STS-B
QQP
MNLI
QNLI
RTE
Batch size
16 le-4
32
32
32
32
16
16
16
Learning rate
le-4
le-4
2c-4
le-4
5e-5
5e-5
5e-5
lc-4
18


Reviews:
[{"review_id": "tDh3aUdy6lm", "reviewer": "Reviewer_7bGi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use integer arithmetic on truncated floating-point operands. The idea is that FP arithmetic anyway suffers from numerical errors associated with rounding. Therefore, the author propose that FP arithmetic can be replaced altogether by pre-aligned integer arithmetic. Essentially, the representation is kept in FP, which preserves accuracy, but compute is done in integer, which is more efficient.", "summary_of_the_review": "The paper introduces a new approximate way of computing FP MatMuls using integer arithmetic. The idea is novel and interesting. I have a few questions I hope the authors can answer.", "strength_and_weaknesses": "The proposed idea is very interesting. However, I have several questions to the authors:\n\n1) The proposed work focuses much on the compute efficiency. I wonder if this is really critical, as the memory access cost is known to usually be a bottleneck for DNN accelerator architecture. Can the authors comment on how significant reducing the cost of compute is when memory access is taken into account?\n\n2) The authors claim that FP is needed to maintain accuracy. This is usually true when quantization methods do not have a good handle on the range vs resolution trade-off in the data. However, for a properly clipped quantizer, integer quantization can be shown to be as accurate as FP [1]. In such a case, integer arithmetic can naturally be used. Can the authors compare their work to such an fully-integer baseline?\n[1] Sakr, Charbel, et al. \"Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training.\" International Conference on Machine Learning. PMLR, 2022.\n\n3) I am curious as to how the DNN accuracy number were obtained in the experimental section. Specifically, how is the FP to INT MatMul emulation performed? The implementation of CUDA kernels, which are invoked by deep learning frameworks, are fast because of the reduction operations that can eliminate the need to store intermediate results in memory. To emulate the proposed method, the authors need to perform an element-wise multiplication, followed by truncation, and then successive additions and truncations. This is very memory and time consuming. Do the authors have a trick to bypass these challenges in the emulation? Or are the experiments simply very slow?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h1PoitVlUjJ", "reviewer": "Reviewer_yB16", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an efficient matrix multiplication hardware. Concretely, the paper performs MatMuls with quantized weights and floating-point activations operants and outputs a numerical approximation of the floating-point result.", "summary_of_the_review": "Overall interesting approach with high potential real-world usage due to only weights required to be quantized.", "strength_and_weaknesses": "# Strength \n- The paper introduces an interesting approach for inferencing neural networks faster.\n- The method only requires the weights to be quantized. This puts much less stress on the stability of the numerics and training compared to a fully quantized network where both the weights and the activations are quantized.\n- Extensive experimental evaluation on NLP and CV tasks with comparison to fp32 and bf16 computations.\n\n# Weaknesses\nThe paper misses an evaluation of important computer vision architectures. Particularly, the paper evaluates VGG9, which is undoubtedly outdated and not relevant anymore (only 92% on CIFAR, where 96% can be easily obtained with a ResNet and CUTOUT augmentation).\nMoreover, the ResNet variant for ImageNet is rather small (ResNet18), whereas the \"gold standard\" for CV evaluations is the ResNet50. I believe larger models or new models (EfficientNet, ConvNext) would make the evaluation of the CV part more relevant for ICLR 2023.\n\nThe method seems to rely heavily on weight quantization, i.e., requiring the assumption that the network still performs well if the weights are expressed by low-bit integers.\nIt would be, therefore, relevant for the paper to provide results on how much the performance of a network suffers from the weight quantization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G50wibMXc-3", "reviewer": "Reviewer_zxxg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new method for the hardware implementation of deep neural networks (DNNs) which are quantized but use floating-point (FP) activations. Prior art has attempted to solve this problem by devising new data formats e.g., bfloat16 and Block Floating Point number for weights and activations, but they have limitations (high overhead to overcome rounding errors and lack of configurability for given hardware platform respectively). \n\nThe paper presents an alternative approach which optimizes the matrix multiplication (MatMul) operations using a special design block, iFPU. It involves decomposing quantized weights into bit-planes (taking out common factors and reducing weights into a series of +1/-1 values). Activation values (FP/bfloat16 format) are similarly reduced by normalizing them to the largest common exponent and converting the mantissa bits to integers. Thereafter, in this reduced space the multiply ops are converted to a series of integer adds (subs) which helps reduce the Matmul complexity.\n\n Further, the authors further simplify the add/sub ops by truncating the least significant bits of the integer values added upto the precision of the output plus a few bits. This truncation helps reduce the complexity of integer add/sub operations significantly. Once the add operations are done, the resulting values are converted back to FP format followed by scaling with values in each bit-plane (removed from weights) and a floating point add.\n\nThe authors demonstrate through detailed exploration of FP add operations that their proposed truncation technique doesn’t cause any error higher than the rounding error of existing FP add operations. Additionally, they also demonstrate that this continues to hold when testing on real networks by showing a strong match between the activation values calculated (in network layers) and comparing the output accuracy generated by FP and iFPU HW implementation. \n\nLastly, authors compare their proposed iFPU implementation with two other HW implementations, FP (default) and FP-ADD, which does similar decompose and add/sub in bit-plane space steps as iFPU without any truncation. The results show that iFPU achieves higher energy efficiency and area efficiency than the other implementations.\n", "summary_of_the_review": "The paper presents strong results showing the accuracy and energy/area efficiency of the proposed HW implementation of FP activation and 4-bit integer quantized weights Matmul operations. But its not obvious that the benefits observed will scale for other quantization bit-widths. Further, as shown in Figure 9 a significant part of the energy/area efficiency benefits are derived due to truncation of FP add operations which relies on similar experiments as another paper referenced above. If that is removed from the results, the improvement in energy/area efficiency due to the other ideas presented in the paper are smaller. ", "strength_and_weaknesses": "The paper presents a good overview of the target application by highlighting the limitations of the floating-point format and prior approaches. The diagrams presented are clear and help explain the proposed ideas. The authors derive the conclusion in Remark1 with detail which helps establish the important point that iFPU maintains the same error at the output despite lowering complexity of the operations performed compared to FP. The presented experiments are useful and highlight the capabilities of the proposed hardware implementation, iFPU as opposed to prior art. \n\nWhile the authors present their ideas on truncation for FP multiplication, the idea of truncation in binary multipliers is not new, e.g., N. Petra, et. al., \"Truncated Binary Multipliers with Variable Correction and Minimum Mean Square Error,\" in IEEE Transactions on Circuits and Systems I: Regular Papers, 2010, presents similar ideas which discuss truncating the bits from the partial products derived after binary multiplication and their impact on the output accuracy. In fact, the authors in Petra et. al., paper define an additional correction function estimated based on the truncated bits which further helps reduce the error measured. I request the authors to cite this paper and elaborate how their contributions are unique from those in that paper. \n\nAn important assumption in the proposed technique is the ability to simplify the quantized weight values to respective bit-planes, which allow simplifying the multiply ops to add/subs. For the paper the authors assume the use of 4-bit quantized weights which might favor the simplification of weights into +1/-1 easily. But often it is harder to achieve the same level of accuracy for a given ML task with a DNN with lower quantization bit sizes. As the quantization bit size increases to 8/16bit integer values, how do the benefits of the proposed iFPU scale? Since for larger bit size the number of bit-planes will increase which will increase the cost of the Scale and Accumulator block shown in Figure 5. \n\nFP-Add is a useful baseline added by the authors which highlights the individual contribution of the bit-plan decomposition and truncation ideas used. I am curious why authors select Int8 Matmul engine considering that the iFPU assumes 4-bit quantized weights, it would be helpful to add an Int4 Matmul engine data as well cause that would be the baseline of the energy efficiency achievable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster