Efficient Systolic Array Based on Decomposable MAC for Quantized Deep Neural Networks
Authors: Ning-Chi Huang
Huan-Jan Chou
Kai-Chiang Wu
Venue:  (2020)

Abstract: Deep Neural Networks (DNNs) have achieved high accuracy in various machine learning applications in recent years. As the recognition accuracy of deep learning applications increases, reducing the complexity of these neural networks and performing the DNN computation on embedded systems or mobile devices become an emerging and crucial challenge. Quantization has been presented to reduce the utilization of computational resources by compressing the input data and weights from floating-point numbers to integers with shorter bit-width. For practical power reduction, it is necessary to operate these DNNs with quantized parameters on appropriate hardware. Therefore, systolic arrays are adopted to be the major computation units for matrix multiplication in DNN accelerators. To obtain a better tradeoff between the precision/accuracy and power consumption, using parameters with various bit-widths among different layers within a DNN is an advanced quantization method. In this paper, we propose a novel decomposition strategy to construct a low-power decomposable multiplier-accumulator (MAC) for the energy efficiency of quantized DNNs. In the experiments, when 65% multiplication operations of VGG-16 are operated in shorter bit-width with at most 1% accuracy loss on the CIFAR-10 dataset, our decomposable MAC has 50% energy reduction compared with a non-decomposable MAC.

Under review as a conference paper at ICLR 2020
EFFICIENT SYSTOLIC ARRAY BASED ON DECOMPOS- ABLE MAC FOR QUANTIZED DEEP NEURAL NET- WORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep Neural Networks (DNNs) have achieved high accuracy various machine learning applications in recent years. As the recognition accuracy of deep learn- ing applications increases, reducing the complexity of these neural networks and performing the DNN computation come an emerging and crucial challenge. Quantization has been presented to reduce the utilization of computational resources by compressing the input data and weights from floating- point numbers to integers with shorter bit-width. For practical power reduction, it is necessary to operate these DNNs with quantized parameters on appropriate hardware. Therefore, systolic arrays are adopted to be the major computation units for matrix multiplication in DNN accelerators. To obtain a better tradeoff between the precision/accuracy and power consumption, using parameters with various bit-widths among different layers within a DNN is an advanced quantization method. In this paper, we propose novel decom- position strategy to construct low-power decomposable multiplier-accumulatr (MAC) for the energy efficiency of quantized DNNs. In the experiments, when 65% multiplication operations of VGG-16 are operated in shorter bit-width with at most 1% accuracy loss the CIFAR-10 10 dataset, our decomposable MAC has 50% energy reduction compared with a non-decomposable MAC.
embedded systems mobile devices be-
INTRODUCTION
In the past decade, deep neural networks (DNNs) have achieved high accuracy in various ma- chine learning applications, such as image classification, object detection, and speech recognition. AlexNet (Krizhvssky al., 2012) applied 8 layers and 60 million parameters to achieve top-5 error rate of 15.3%, which won the championship in the ILSVRC2012 competition, and thus has driven research trends DNNs and advanced machine learning applications. In the following years, SZzgedy et al. (2015); Simonyan & Zisserman (2014): He et al. (2016); Huang et al. (2017) presented to apply deeper neural networks with layers and parameters for higher accuracy.
As the accuracy of deep learning applications increases, optimizing these complex networks to be simpler, more computationall-ffiieni becomes an emerging and crucial challenge. Therefore, many methods have been proposed to reduce the complexity of neural networks. For example, pruning redundant connections within the model (Han et al. 2015 Zhu & Gupta 2017). reducing the kernels/filters within layers (Liu et al. 2017 Luo al. 2017| Howard al.) (017). removing several layers (Wen al.) 2016
even suitable for embedded systems and mobile devices
Li et al. 2016), or
using fewer bits (also known as quantization) during the computation (Lin et al.
2016
Zhou et al.
2017
Jouppi al. 2017 Hubara et al.) 2017 Gysse et al. 2018 Mishra al. 2017 Jacob et al.
2018
have been presented to streamline DNNs with negligible accuracy loss.
According to the accuracy requirements of an application and its associated DNN structure, the bit-width of the setting of parameters for the DNN may be different from one layer to another. In addition, the activations and weights in some DNNs can be squeezed into 8 bits or less without significant accuracy loss (Gysel et al. 2018 Mishra al. 2017). Fig. [a reveals the accuracy (normalized to that of the original DNN operated with bit floating-pott activations and weights) on CIFAR- 10 dataset of DNNs with the activations and weights being quantized into uniform bit-

---

Under review as a conference paper at ICLR 2020
100%
Normalized Accuracy
VGG-16
601s
VGG-19
11.05
52.05
205
DenseNet-121
21.5%
45.9%
0%
VGG 16
VGG-19 19
DemsNNr- 121
DemoNet- 169
MobiloNet
DenseNet-169
MobileNet
6b 8b W 88b A, W A W #bb Ab 4b W
8b A. 8b W
8b A, W
= 4b 4b
(a) Normalized accuracy of quantized DNNs with uniform bit-width (i.e.. uniform quantization)
(b) Breakdown of layer-wise quantization with at most 1% accuracy loss
Figure I: Tradeoff between accuracy and bit-width of quantized activations (A) and weights (W)
width, 8 bits or 4 bits, across all layers within the specific neural networks. The accuracy loss can be less than 1% when 8-bit activations and 8-bit weights are utilized these DNNs. However, directly compressing all activations and/or weights into 4 bits might devastate the accuracy. Instead of quantizing all the parameters within a neural network to uniform bit-width, adjusting the bit-width of quantized parameters from layer to layer (i.c., layer-wise quantization) between required computing resources and resulting accuracy (Mishra et al. 2017 Jacob et al.] 2018). Based the quantization schemes presented by [in al. (201 6); Jacob et al. (2018), Fig. lb shows the breakdown of layer-wise quantization, in terms of the percentage of multiplication- accumulations operations, within the quantized DNNs by trading with most 1% accuracy loss.
obtain a better tradeoff
For practical reduction in power consumption, it is necessary to run these DNNs with quantized parameters on appropriate hardware. Fig. 2a illustrates simple DNN with multiple hidden layers. Within each layer, activations from the previous layer are multiplied by kernels/filters to obtain the outputs of the current layer; throughout all layer DNN, numerous multiplications and accu- mulations are involved. To accelerate DNN operations, Доорр et al. (2017); Zhang et al. (2018); (MACs)
Gupta al. (2015); [hen et al.] (2017) employ systolic array of muluplier-accumulaar
(see Fig. 2b as their major computation units for matrix multiplication, which perform 8-bit integer multiplication and 16-bit fixed-point multiplication, respectively.
To obtain a better tradeoff between accuracy and power consumption by quantizing parameters with various bit-width among different layers (i.e., layer-wiss wise quantization), the processing elements for DNNs should support multiplication operations with variable bit-widths. Configurable multipli- ers (Sharma et al. 2018, Pfänder et al., 2004 Bermak et al.) 1997; Haynes & Cheung 1998) are presented to adjust the bit-width of operating MACs for power saving, when shorter bit- is suf- ficient to satisfy the accuracy requirement; when longer bit-width used, they generate the product of multiplication by combining the outputs from neighbor small MACs. However, the power/energy overhead for the combination of outputs from MAC neighbors may be significant and should not be ignored. In other words, layer-wise guantization needs supported by customized hardware; other- wise the benefit in terms of power/energy consumption will be slashed (cut down significantly).
In this paper, we propose a novel decomposition strategy for low-power decomposable iitiiplicattonacccmlllla
Based on the strategy, multiplication with longer bit-width can be departed and composed by its partial products with less computational quantity. addition, accord-
Weight Memory
Activation Memory
MAC
Input Layer
Hidden
Output
Layers
Layer
(a) DNN architecture
(b) Systolic array of MACs
Figure 2: DNN and systolic array
2

---

Under review as a conference paper at ICLR 2020
ing to the bit-width requirement of the current computation, decomposable MAC based strategy can dynamically adjust the bit-width of its final multiplication result. In addition, accord- ing to the bit- wittwidth requirement of the current computation. decomposable MAC based strategy can dynamically adjust the bit-width of its final multiplication result. Instead of generating multiplication results with required bit-width directly by connecting signals among neighbor MACs for composition as 2004 Bermak et al., 1997 Haynes & Cheung [998), our strategy reduces the selection signals for the operation bit-width of multiplication units. Therefore, our strategy can obtain significant energy efficiency from multiplication with variable bit-widths.
configurable MACs (Sharma et al. [2018] Pfänder et al.,
BASIC IDEA OF OUR PROPOSED DECOMPOSITION STRATEGY
2.1
EXAMPLE OF OUR DECOMPOSABLE MAC OPERATION
17 6]x +
21
18
18
10
R
9001
V918
6186
1
(7n)
(12))
0211000
(32c)
0
e10eL1,
019011,
(a)
(b)
(c)
(d)
Figure 3: Example of matrix multiplication
To reduce the power overhead of a decomposable MAC, we first observe the behavior of matrix multiplication. Fig. 3 illustrates example of matrix multiplication with unsigned integers for simplicity (the extension of signed numbers will be discussed in Section [A.]). Suppose that the inputs are matrices A [7 6] and B
2)", and the result matrix of A x B will be [7 x 1 6 x 2] = [19]. If A and are represented with 4-bit and 2-bit binaries, respectively, and only bbitx2-bitt multipliers
provided, the matrix multiplication should be decomposed and composed to obtain the correct result. As shown in Fig. Bal each 4-bitx 2-bit multiplication is decomposed into two 2-bitx2-bii multiplications. Therefore, this computation needs four 2-bitx2-bbe multiplications and two them have to be shifted left with 2 bits (see Fig. 3b). If we follow a straightforward method presented by [Sharma et al. (2018) computation, i.e., obtain the result of 7 x and then the result of 6 x 2. four multiplications, two shift operations and three additions are necessary, as shown in Fig. 3c This indicates that four 2-bitx 2-bit multipliers, two 4-bit adders and one 6-bit adder will be utilized for the computation.
obtain "meaningful" intermediate values during the
Fig. 3d shows the decomposition and composition flow based on our idea: if we only care the final result but not the intermediate values, the partial results which have the same shift distances can be composed together before the shift operation. Afterwards, this matrix multiplication needs four multiplications, three additions and only one shift operation. Thus, four 2-bit 2-bit multipliers and three bit adders will be utilized.
2.2 CONTRIBUTIONS OF OUR PROPOSED DECOMPOSITION STRATEGY
The contributions and advantages of this proposed work are twofold:
We propose decomposition strategy for low-power decomposable multiplication- accumulation operations. Based on our decomposition strategy, longer bit-width multi- plication can be departed and composed with the results of shorter bit-width MACs. There- fore, compared with non-decomposable multiplier/MAC, when a DNN has its activations and weights being quantized into variable bit-widths among different layers, our decom- posable MAC can generate the matrix multiplication results with less energy consumption.
According to our proposed decomposition strategy, the partial results which have the same shift distance composed together before shifted and added with other partial results.
3

---

Under review as a conference paper at ICLR 2020
2m
(a)
(b)
2n- Figure 4: 2m-bitx 2m-bit multiplication
Instead of designing decomposable multiplier to construct systolic array by straight- forward decomposition methods, we efficiently utilize the shorter bit-width MACs within systolic arry. Hence, our decomposable MAC has less power overhead of connecting signals among multiplication blocks/modules physically.
3 DETAILS OF OUR PROPOSED DECOMPOSABLE MAC
After given an illustrative example for the major idea of our decomposition strategy. we will intro- duce the detailed concept and architecture of our proposed decomposable MAC in this section.
3.1 CONCEPT OF OUR DECOMPOSITION STRATEGY
Our decomposition strategy can be extended to any multiplication with variable bit-widths, As illus- trated in Fig. 4a if the inputs of a multiplication are represented 2m-bit integers and m-bitx m-bit multipliers are provided, the multiplication can be departed into four m-bitxm-B m-bi multiplications. Fig. 4b shows that when the straightforward method presented by /Sharma al. (2018) is applied, it needs two kinds of shifters (one is for left-shift with 2m bits and the other is for left-shift with m bits) and three shift operations for the computation. With our decomposition strategy, needs one shifter and two shift operations shown in Fig. 4c For both Fig, 4b and 4c two 2mn-bit adders and one 3mn-bit adder will used. If an m-bitxm-bll multiplier is provided and the inputs of multiplication are x m-bit integers a and b, the multiplication result can be obtained by:
21-2
a
28
2imb)) = Σ
(2
aayb-s)
1=0
1=0
1=0
1=0
where a; and b, are m-bit integers expressing and by a E 2: a, and b == Ei=d 24 respectively. With our decomposition strategy, the multiplication only needs m multi- plications, 21 2 shifts and totally (4f2 51 + 1) x m 1-bit full adders (FAs), which are much less FAs with straightforward decomposition
than 12 shifts and at least (12 x logg I 1/2) method.
Furthermore, when decomposition strategy is applied on matrix multiplication, the number of FAs for the composition partial results can be reduced significantly. Suppose that the inputs of matrix multiplication are matrices A and B, and the result is matrix C, then every element in C can be calculated by: Gy = Em= agbg where ag and by are the kes element in row of matrix A and the ke, element in column j of matrix B, respectively. If the elements in matrices A and B are 2mt integers and the given multiplier can perform m-bit multiplication, the element oj can be obtained by departing and composing:
Gj
akI td
aRR)(bbi 2m buo)
(2)
N
N
N
m
(akiba) + 2m Σ (apibro + abobbr) +
Σ
akobso kml
k=
4

---

Under review as a conference paper at ICLR 2020
4m
MU
20
MU
MU
MAC
MU
(a) A decomposable MAC based our proposed decomposition strategy
(b) A configurable multiplier based on the method
presented by (Sharma et al. (2018)
Figure 5: Decomposable MAC/multiplier for amn-iix2mm multiplication
where ag1, 0ko, bb1- and 0uo are numbers expressed in bits and their combination. d&1 x 2 + ago and 6b1 x 2t + bko. represent the value of elements ag and og. respectively. In addition, the result of multiplication with power of 2 can be obtained by left-shift operation. Furthermore, the shift distance can be constant with the value being m if Equation 2is rewritten as:
N
N
= 2 12 Σ (anIba)) +
Σ (aaabo + akobaa) b) +
OkoORo
When the elements in matrices A and B are represented as x m-bit integers, the elements in result matrix C' can be obtained by replacing the expressions of dg and by in Equation 2 as appropriate polynomials in form of Σ 2hm where da is m-bit integer. In this condition, straightfor- ward method needs N12 2 N shifts, but decomposition strategy still needs only 27 2 2 shifts. To perform the matrix multiplication with appropriate hardware, we describe the detailed architecture the following subsection.
3.2 ARCHITECTURE OF OUR PROPOSED DECOMPOSABLE MAC
According to our decomposition strategy, a decomposable MAC can be constructed by connecting the output of an m-bitxm-bb MAC multiplexer (MUX) for the selection between shifted and non-shifted results illustrated in Fig. 5a
Table lists the number of FAs and 1-bit 2-to-1 MUXs of two multipliers and one MAC, including multiplier, a straightforward decomposable multiplier and the
non-decomposable 2m-bitx MAC based on our decomposition strategy. Suppose that these multipliers use array multipliers for the multiplication and the decomposable ones ripple-carty adders for the partial product combination (the applied multipliers and adders can be any kind of existing multipliers and adders, and we choose array multipliers and ripple-carry adders here for simplicity). In this condition, the non-decomposable multiplier contains 2m((2m - 1) 4m 2m FAs.
Based straightforward method, decomposable multiplier requires enough multiplication units in their design, e.g., four mibit multipliers are necessary for a 2m--it 2m--it multiplication. In addition, the decomposable multiplier needs select the required combination of the results (four m-bit two x two 2m-bitxm-bir or one 2m-bitx22-bbl multiplication
Table Comparisons of required FAs and MUXs
#FAs for multiplication #FAs for composition #MUXs
Non-decomposition
(one 2m x 2m multiplier)
4m 2m
0
0
Straightforward decomposition method (four m multipliers)
4m, 4m
7m
42m
Our decomposition strategy
(one m X m MAC)
m m
4m
4m
5

---

Under review as a conference paper at ICLR 2020
Weight Memory
Activation Memory
Zamiag
Activations
Teee Reerfeeee feeed Plecee
logh
Sunvarton
MAC
MAC
MAL
Ascumalation Unil
and Stragy thotall Seeuts
Asc
Asc
Aoc
M
Figure 6: Architecture of our systolic array of decomposable MACs
results), and then add them together. Therefore, 7m FAs and 42m MUXs are required for the composition of straightforward decomposable multiplier.
However, the decomposable MAC based on our decomposition strategy can use a single m-bitxm- bit multiplication unit and 4m-bit adder to achieve scalability, as illustrated in Fig. [ The number of required MUXs can be 4m because our decomposable MAC only needs to select the result with correct shift distance from the accumulation result. Thus, the number of required FAs and MUXs for composition based on our decomposition strategy can be less than those required by the straight- forward decomposable multiplier. Because the longest path our decomposable MAC is shorter, the delay of our decomposable MAC can be less than both delays of the non-decomposable multi- plier and the straightforward decomposable multiplier. Accordingly, the energy consumption of our decomposable MAC can also be less.
Our decomposition strategy can be applied on a systolic array (Kung & Leiserson 1982) of MACs to construct more energy-efficient MAC for DNN computation. Fig. 6 iluuttarss the block diagram our decomposable MAC, which consists of a systolic array an N grid of shorter bit-width MACs and N accumulation units. Each MAC unit perform m-bit m-bit mul- tiplication and (2m - log2 N) -bi accumulation, where m the basic length for the decomposable multiplication and 2m + logz N is the minimum length for representing the maximum accumulation result of N MACs within the same column the systolic array. In addition, each accumulation unit contains M-bit accumulator and MUX for the selection between shifted and non-shifted partial results, where M is the minimum length for the representation of elements in result matrix.
1979 Kung
In our proposed decomposable MAC, a 2-dimensional systolic array of shorter bit-width MACs is employed to calculate the partial results of matrix multiplication. By connecting multiple MACs as the processing elements in the systolic array, data can be passed and computed through more processing elements after I/O operation. The weights are pre-loaded to the registers of corresponding MACs before the matrix multiplication execution. As shown in Fig. 6 the activations of each layer is propagated from left to right and the partial results of matrix multiplication are accumulated from top to bottom cycle by cycle.
After all the partial results with common shift distance are accumulated together, these pieces should be shifted and composed. As illustrated in Fig. 6 we connect the outputs of the systolic array to MUXs for selection between shifted and non-shifted partial results. Therefore, according to decomposition strategy, reordering the addition sequence of partial results in matrix multiplication with common shift distances not only reduce the number of shift operations and the required MUXs significantly, but also reduce the connecting signals among multiplication blocks/modules physi- cally.
EXPERIMENTAL RESULTS
In this section, we compare a systolic array of MAC's based on our decomposition strategy to systolic arrays of non-decomposable MACs and straightforward decomposable MACs. In the experiments,

---

Under review as a conference paper at ICLR 2020
implement the systolic arrays of MACs in Verilog and the MACs synthesized by Synopsys Design Compiler (DC) with Synopsys SAED EDK 28nm standard cell library. Moreover, the basic arithmetic components, including adders and multipliers, provided by DesignWare.
4.1
BASIC COMPARISONS WITH DIFFERENT SYSTOLIC ARRAY LENGTH
Table 2: Latency, area and energy consumption of systolic array of MACS
Latency(ns)
Area(um²)
Energy(p.])
Non Non-decomposabee Straightforward decomposable
2.00
311679.56
54.144
2.72
601602.42
110,989
Our decomposable
1.16
516512.40
37.764
To evaluate the latency, area and energy consumption of non-decomposable MAC, a straight- forward decomposable MAC and our decomposable MAC, we implement these systolic arrays in Verilog. Three different size of systolic arrays are implemented, including a 16x 16 grid of MACs, a 32x32 grid of MACs and 64 64 grid of MACs. To align the capability of performing matrix mul- tiplication in each cycle as the systolic array of straightforward decomposable MACS does, the size of our systolic array is adjusted to be 64, 32x 128, and 646256. Within these systolic arrays, the width of accumulators/adders of MACs have been adjusted by considering the corresponding maximum length of accumulation in the systolic array. In addition, all of these three systolie arrays of MACs have ability to compute correct result of multiplication with its input bit-width being 4-bit 4-tit, 8-bit 4-bit, and 8-bit x 8-bit.
Table 2 shows the synthesized results of three systolic arrays with a 16x 16 grid of non- decomposable MACs, 16x 16 grid of straightforward decomposable MACs, and 16x64 grid of MACs based on our proposed decomposition strategy. The first column is the type of MACs utilized within the systolic array. The second column shows the latency of each type of MAC. Columns three and four show the area and energy consumption of the whole systolic array of MACs. To accelerate the operations of matrix multiplication, a straightforward decomposable MAC is designed to per- form as one MAC with an 8-bit 8-bit multiplier, two parallel MACs with 8-bitx4-bit multipliers, or four parallel MACs with 4-bit 4-bit multipliers. Hence, additional components are necessary for the construction of a straightforward decomposable MAC (as mentioned in Section 3.2) and thus the whole area and energy consumption the systolic array of straightforward decomposable MACs are larger than those of the systolic array of non-decomposable MACs. To reduce the energy consumption of matrix multiplication, our decomposition strategy reuses the connections within the systolic array compose partial results of multiplication for the final correct results. Therefore, the input bit-width of our MAC unit just needs to be 4-bitx4-bit, which significantly reduce the area and energy consumption of decomposable MACs.
In Fig. 7 to compare the synthesized results more clearly, we record their energy consumption and performance in three different operation conditions, including multiplication with all the bit- width of activations and weights being 4x4, 8x4, and 8x8, respectively. As shown in Fig. [7] the energy consumitiion of the systolic array with our proposed decomposition strategy is much less than that applied straightforward decomposition method. In Fig. 7b the performance of both
Avcrage Consunction
Avseage Ieformenca
(nammalaa))
16x64
Our 3b.188
Our 32x128
Our 646256
Straighffoward
Straighhfooward 18x16
Strightforward
Straightfonoord
Straghtfooward 54x54
Straightfonwadd G4X54
4x4
2x4
Bud
404
Boll
Bod
Mulipiictton
Bith
(a) Normalized average energy consumption
(b) Normalized performance
Figure 7: Normalized average energy consumption and performance of the systolic arrays of de- composable MACs (normalized to those of the systolic array of non-decomposable MACs)
7

---

Under review as a conference paper at ICLR 2020
systolic arrays of decomposable MACs are higher than that of non-decomposable MACs when op- erating multiplication with all the bit-width being 4x4 and 8x4. This is because both arrays of decomposable MACs generate four times 4-bitx4-bit multiplication results and double 8 xx4 multiplication results compared with an array of non-decomposable MACs. However, when oper- ating 8-bitx8-bit multiplication, the performance of the systolic array of straightforward decom- posable MACs is lower than the non-decomposable one. Due to the connecting signals among the multiplication units of straightforward decomposable multiplier, its latency will be longer than non-decomposable multiplier and cause lower throughput when operating as one MAC with an 8-bitx 8- bit multiplier. The latency of the MAC based on our proposed decomposition strategy shorter than the non non-decomposable one, and thus the performance of our systolic array can still be higher than the non-decomposable one when operating 8-bitx8-bi multiplication.
4.2 ENERGY EFFICIENCY WHEN OPERATING MATRIX MULTIPLICATION OF DEEP NEURAL NETWORKS
VGG-16
VGG-19
DenseNet-121 DenseNet-169
MobileNet
Energy Consumption
-30%
-15%
-39.765
-15.10%
-75X
Figure 8: Energy reduction of the systolic array of MACs applied our decomposition strategy (nor- malized to the energy consumption of the systolic array of non-decomposable MACs)
To evaluate the energy efficiency of the systolic array of MACs applying our decomposition strategy, we perform matrix multiplication in deep learning applications according to the bit-width require- ment mentioned Section [ with the synthesized systolic arrays shown in Table 2
Fig. B shows the energy consumption of the systolic array of MACs based our decomposition strategy compared with a systolic array of nonoeecoeeo MACs. When DNN has its acti- vations and weights being quantized into shorter bit-width, the systolic array of non-decomposable MACs still operate 8-bitx 8-bit multiplication within its multiplication units, and thus cannot ef- ficiently obtain the benefit energy saving from the quantization. However, due to applying shorter bit-width MACs for the decomposition and MUXs for the selection between shifted and non-shifted accumulation results for the composition, our decomposable MAC has significant energy reduction when the required bit-widtt multiplication shorter.
5 CONCLUSION
In this paper, we propose decomposition strategy for the energy efficiency of quantized DNN computation. According to our proposed strategy, longer bit-width multiplication can be departed and composed with partial results of shorter bit-width MACs. Therefore, compared with non- decomposable MAC, our decomposable MAC can generate matrix multiplication results with less energy consumption, when DNN has its activations and weights being quantized into variable bit- widths among different layers. In addition, we efficiently utilize the MACs within a systolic array for the composition of partial results. Hence. systolic array of MACs based on decomposition strategy has less power overhead than systolic array of straightforward decomposable MACs.
REFERENCES
Charles R Baugh and Bruce A Wooley. A two's complement parallel array multiplication algorithm.
IEEE Transactions on computers, 100512):045--147, 1973.
A Bermak, Martinez, and J-L Noullet. High-density 16/8/4-bit configurable multiplier. JEE Praceedings-Circuti Devices and Systems, 144(5):272-276, 1997.
8

---

Under review as a conference paper at ICLR 2020
Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. Eyeriss: An energy- configurable accelerator for deep convolutional neural networks. IEEE Journal of solid-State Circuits, 52(1):127-138, 2017.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Philipp Gysel, Jon Pimentel, Mohammad Motamedi, and Soheil Ghiasi. Ristretto: A framework for empirical study of resource-efficient inference in convolutional neural networks. IEEE Transac- tions on Neural Networks and Learning Systems, (99):166, 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1 1135-1143, 2015.
Simon D Haynes and Peter YK Cheung. Configurable multiplier blocks for embedding in fpgas. Electronics Letters, 34(7):638-639, 1998.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 770..4444.444. 2017
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1700-4708, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1):6869-66) 6898, 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1704-2711 2018.
efficient
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et In-datacenter performance analysis of tensor processing unit. In 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pp. 1-12. IEEE, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton, Imagenet classification with deep convo- lutional neural networks. In Proceedings of the 25th International Conference on Neural Infor- mation Processing Systems- Volume 1, NIPS' 12, pp. 1097-1105, USA, 2012. Curran Associates Inc. BtRLbttp://dl //ll cm. org http/iiaaaiiiooo cfmeefmmmm.e..e wtt..ccmmmmmm.mmm.eee 2204229999777
Hsiang-Tsaa Tsung Kung. Why systolic architectures? IEEE computer, 15(1):37-46, 1982.
HT Kung and Charles E Leiserson. Systolic arrays (for vlsi). In Sparse Matrix Proceedings 1978, volume 1, pp. 256-282. Society for industrial and applied mathematics, 1979.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 08710, 2016.
Daryy Lin, Sachin Talathi, and Sreckanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-2858, 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference Computer Vision, pp. 2736-2744, 2017.

---

Under review as a conference paper at ICLR 2020
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 10.85058-000 2017.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: wide reduced- precciconr networks. arXiv preprini urXiv: 134, 2017.
Oliver A Pfänder, Roland Hacker, and Hans-Jörg Pfleiderer. multiplexer- concept for configurable multiplier arrays. In International Conference on Field Programmable Logic and Applications, pp. 938-942. Springer, 2004.
Hardik Sharma, Jongse Park. Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural networks. In Proceedings of the 45th Annual International Symposium on Computer Ar- chitecture, pp. 764-775. IEEE Press, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Jeff Zhang, Kartheek Rangineni, Zahra Ghodsi, and Siddharth Garg. Thundervolt: enabling ag- gressive voltage underscaling and timing error resilience for energy efficient deep learning ac- celerators. In Proceedings of the 55th Annual Design Automation Conference, DAC 2018, San Francisco, CA, USA, June 24-29, 2018, pp. 19:1-19:6, 2018. doi: 10.1145/3195990331966223 11451 1145/11955911 3195970.0090. 3196129
URL https:://oi.oog://///
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044 2017.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878. 2017.
A APPENDIX
A.1 EXTENSION TO SIGNED MULTIPLICATION
Our proposed decomposition strategy be applied signed multiplication by replacing the bitxm-bit unsigned multiplier into a multiplier which can perform combined unsigned/signed mul- tiplication. For example, according to the two's complement multiplication developed by |Baugh & Wooley (173)), a 4-bitx signed multiplication can illustrated as Fig. 9a The correct result of signed multiplication to the left of the first partial product term. To support decomposable signed multiplication. flag signals S, for the determination of the most significant bit (MSB) of signed integer a, S, for the
be obtained by inverting some of the product terms and inserting
8a
saby
0-b
S,b.
S.0. 00
(ouh)
0g
0yby
dedy
(b) A 2-bitx2-bit multiplier for combined un- signed/signed multiplication
(a) 4-bitx4-bit signed multiplication
Figure 9: A signed multiplication with decomposable multipliers
10

---

Under review as a conference paper at ICLR 2020
determination of the MSB of signed integer b, and Lg for the determination of the least significant bit of should be provided in each 2-bitx2-bel multiplier. When the partial product terms of a 2- 2-bb2 multiplier which replaces the unsigned multiplier is the same as those shown in Fig. 9b the decomposable multiplier can support signed multiplication.


Reviews:
[{"review_id": "SyeLKScX5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a decomposable MAC unit for low-power computation for matrix-multiplication operation for neural networks.\n\nAlthough I believe I understood the idea of this technique, I strongly believe this paper should be submitted to an architecture of design automation conference instead of ICLR. I am also not in the position to assess the experiments, which were conducted with synthesis of Synopsis 28nm library. The paper discussed systolic array, MAC in detail, without too many algorithmic elements inside, therefore may be a paper not toward the audience of this conference.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skg4roPYcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes a novel decomposition strategy for matrix multiplication to aim for less energy consumption. They demonstrate that energy consumption is reduced on several CNNs. It is an interesting work but it is not sure that there is a difference in contribution when compared to previous work.\n\nThe main argument to impact to the score:\n\n    1. A similar idea is addressed in [1]. It is highly recommended to show the difference in contribution.\n    2. In Figure 7,  the performance and energy consumption of systolic arrays of decomposable MAC  is shown. However, the only energy consumption is shown in section 4.2 when the method is applied to DNNs. It is better to show both performance and energy consumption.\n    3. It is inappropriate to quantize VGG16, VGG19, DenseNet-121, DenseNet-169 and MobileNet based on CIFAR10 dataset since all models are designed for ImageNet dataset. It is appropriate to quantize these models on ImageNet dataset.\n\n\n[1]     Sungju Ryu, Hyungjun Kim, Wooseok Yi and Jae-Joon Kim. BitBlade: Area and Energy-Efficient Precision-Scalable Neural Network Accelerator with Bitwise Summation. DAC. 2019.\n\nMinor comments not to impact the score: \n    1. \"Very Deep Convolutional Networks for Large-Scale Image Recognition\" has been accepted by ICLR 2014. It is better not to use arXiv preprint on citations unless there is a reason.\n    2. It is recommended to put citations on CIFAR-10 dataset.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJxryQQy5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to shorten the shift-addition operations in the straightforward configurable MACs (Sharma et al., 2018), to an addition-shift style. The authors claim that the new design is able to lower the energy consumption in the matrix multiplication. In the experimental analysis, the authors demonstrate the effectiveness of the proposed method.\n\nThis paper should be rejected since it proposes exactly the same architecture with the following published work:\nBitBlade: Area and Energy-Efficient Precision-Scalable Neural Network Accelerator with Bitwise Summation\n\nAlso, the authors do not provide a valid approach for the auto-selection of quantization bits, which is more significant in my opinion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an energy-efficient architecture for quantized deep neural networks based on decomposable multiplication using MACs. Although the proposed approach is shown to be somehow effective, two reviewers pointed out that the very similar idea was already proposed in the previous work, BitBlade [1]. As the authors did not submit a rebuttal to defend this critical point, I’d like to recommend rejection. I recommend authors to discuss and clarify the difference from [1] in the future version of the paper. 

[1] Sungju Ryu, Hyungjun Kim, Wooseok Yi, Jae-Joon Kim. BitBlade: Area and Energy-Efficient Precision-Scalable Neural Network Accelerator with Bitwise Summation. DAC'2019

Decision: Reject