Efficient Systolic Array Based on Decomposable MAC for Quantized Deep Neural Networks

1 INTRODUCTION . In the past decade , deep neural networks ( DNNs ) have achieved high accuracy in various machine learning applications , such as image classification , object detection , and speech recognition . AlexNet ( Krizhevsky et al. , 2012 ) applied 8 layers and 60 million parameters to achieve a top-5 error rate of 15.3 % , which won the championship in the ILSVRC2012 competition , and thus has driven research trends on DNNs and advanced machine learning applications . In the following years , Szegedy et al . ( 2015 ) ; Simonyan & Zisserman ( 2014 ) ; He et al . ( 2016 ) ; Huang et al . ( 2017 ) presented to apply deeper neural networks with more layers and parameters for higher accuracy . As the accuracy of deep learning applications increases , optimizing these complex networks to be simpler , more computational-efficient , or even suitable for embedded systems and mobile devices becomes an emerging and crucial challenge . Therefore , many methods have been proposed to reduce the complexity of neural networks . For example , pruning redundant connections within the model ( Han et al. , 2015 ; Zhu & Gupta , 2017 ) , reducing the kernels/filters within layers ( Liu et al. , 2017 ; Luo et al. , 2017 ; Howard et al. , 2017 ) , removing several layers ( Wen et al. , 2016 ; Li et al. , 2016 ) , or using fewer bits ( also known as quantization ) during the computation ( Lin et al. , 2016 ; Zhou et al. , 2017 ; Jouppi et al. , 2017 ; Hubara et al. , 2017 ; Gysel et al. , 2018 ; Mishra et al. , 2017 ; Jacob et al. , 2018 ) have been presented to streamline DNNs with negligible accuracy loss . According to the accuracy requirements of an application and its associated DNN structure , the bit-width of the setting of parameters for the DNN may be different from one layer to another . In addition , the activations and weights in some DNNs can be squeezed into 8 bits or less without significant accuracy loss ( Gysel et al. , 2018 ; Mishra et al. , 2017 ) . Fig . 1a reveals the accuracy ( normalized to that of the original DNN operated with 32-bit floating-point activations and weights ) on CIFAR-10 dataset of DNNs with the activations and weights being quantized into uniform bit- width , 8 bits or 4 bits , across all layers within the specific neural networks . The accuracy loss can be less than 1 % when 8-bit activations and 8-bit weights are utilized in these DNNs . However , directly compressing all activations and/or weights into 4 bits might devastate the accuracy . Instead of quantizing all the parameters within a neural network to uniform bit-width , adjusting the bit-width of quantized parameters from layer to layer ( i.e. , layer-wise quantization ) can obtain a better tradeoff between required computing resources and resulting accuracy ( Mishra et al. , 2017 ; Jacob et al. , 2018 ) . Based on the quantization schemes presented by Lin et al . ( 2016 ) ; Jacob et al . ( 2018 ) , Fig . 1b shows the breakdown of layer-wise quantization , in terms of the percentage of multiplicationaccumulations operations , within the quantized DNNs by trading with at most 1 % accuracy loss . For practical reduction in power consumption , it is necessary to run these DNNs with quantized parameters on appropriate hardware . Fig . 2a illustrates a simple DNN with multiple hidden layers . Within each layer , activations from the previous layer are multiplied by kernels/filters to obtain the outputs of the current layer ; throughout all layer in a DNN , numerous multiplications and accumulations are involved . To accelerate DNN operations , Jouppi et al . ( 2017 ) ; Zhang et al . ( 2018 ) ; Gupta et al . ( 2015 ) ; Chen et al . ( 2017 ) employ a systolic array of multiplier-accumulators ( MACs ) ( see Fig . 2b ) as their major computation units for matrix multiplication , which perform 8-bit integer multiplication and 16-bit fixed-point multiplication , respectively . To obtain a better tradeoff between accuracy and power consumption by quantizing parameters with various bit-width among different layers ( i.e. , layer-wise quantization ) , the processing elements for DNNs should support multiplication operations with variable bit-widths . Configurable multipliers ( Sharma et al. , 2018 ; Pfänder et al. , 2004 ; Bermak et al. , 1997 ; Haynes & Cheung , 1998 ) are presented to adjust the bit-width of operating MACs for power saving , when shorter bit-width is sufficient to satisfy the accuracy requirement ; when longer bit-width is used , they generate the product of multiplication by combining the outputs from neighbor small MACs . However , the power/energy overhead for the combination of outputs from MAC neighbors may be significant and should not be ignored . In other words , layer-wise quantization needs to supported by customized hardware ; otherwise the benefit in terms of power/energy consumption will be slashed ( cut down significantly ) . In this paper , we propose a novel decomposition strategy for low-power decomposable multiplication-accumulation . Based on the strategy , multiplication with longer bit-width can be departed and composed by its partial products with less computational quantity . In addition , accord- ing to the bit-width requirement of the current computation , a decomposable MAC based on our strategy can dynamically adjust the bit-width of its final multiplication result . In addition , according to the bit-width requirement of the current computation , a decomposable MAC based on our strategy can dynamically adjust the bit-width of its final multiplication result . Instead of generating multiplication results with required bit-width directly by connecting signals among neighbor MACs for composition as straightforward/intuitive configurable MACs ( Sharma et al. , 2018 ; Pfänder et al. , 2004 ; Bermak et al. , 1997 ; Haynes & Cheung , 1998 ) , our strategy reduces the selection signals for the operation bit-width of multiplication units . Therefore , our strategy can obtain significant energy efficiency from multiplication with variable bit-widths . 2 BASIC IDEA OF OUR PROPOSED DECOMPOSITION STRATEGY . 2.1 EXAMPLE OF OUR DECOMPOSABLE MAC OPERATION . To reduce the power overhead of a decomposable MAC , we first observe the behavior of matrix multiplication . Fig . 3 illustrates an example of matrix multiplication with unsigned integers for simplicity ( the extension of signed numbers will be discussed in Section A.1 ) . Suppose that the inputs are matrices A = [ 7 6 ] and B = [ 1 2 ] T , and the result matrix of A × B will be [ 7 × 1 + 6 × 2 ] = [ 19 ] . If A and B are represented with 4-bit and 2-bit binaries , respectively , and only 2-bit×2-bit multipliers are provided , the matrix multiplication should be decomposed and composed to obtain the correct result . As shown in Fig . 3a , each 4-bit×2-bit multiplication is decomposed into two 2-bit×2-bit multiplications . Therefore , this computation needs four 2-bit×2-bit multiplications and two of them have to be shifted left with 2 bits ( see Fig . 3b ) . If we follow a straightforward method presented by Sharma et al . ( 2018 ) to obtain “ meaningful ” intermediate values during the computation , i.e. , obtain the result of 7 × 1 and then the result of 6 × 2 , four multiplications , two shift operations and three additions are necessary , as shown in Fig . 3c . This indicates that four 2-bit×2-bit multipliers , two 4-bit adders and one 6-bit adder will be utilized for the computation . Fig . 3d shows the decomposition and composition flow based on our idea : if we only care the final result but not the intermediate values , the partial results which have the same shift distances can be composed together before the shift operation . Afterwards , this matrix multiplication needs four multiplications , three additions and only one shift operation . Thus , four 2-bit×2-bit multipliers and three 4-bit adders will be utilized . 2.2 CONTRIBUTIONS OF OUR PROPOSED DECOMPOSITION STRATEGY . The contributions and advantages of this proposed work are twofold : • We propose a decomposition strategy for low-power decomposable multiplicationaccumulation operations . Based on our decomposition strategy , longer bit-width multiplication can be departed and composed with the results of shorter bit-width MACs . Therefore , compared with a non-decomposable multiplier/MAC , when a DNN has its activations and weights being quantized into variable bit-widths among different layers , our decomposable MAC can generate the matrix multiplication results with less energy consumption . • According to our proposed decomposition strategy , the partial results which have the same shift distance are composed together before shifted and added with other partial results . Instead of designing a decomposable multiplier to construct a systolic array by straightforward decomposition methods , we efficiently utilize the shorter bit-width MACs within a systolic array . Hence , our decomposable MAC has less power overhead of connecting signals among multiplication blocks/modules physically . 3 DETAILS OF OUR PROPOSED DECOMPOSABLE MAC . After given an illustrative example for the major idea of our decomposition strategy , we will introduce the detailed concept and architecture of our proposed decomposable MAC in this section . 3.1 CONCEPT OF OUR DECOMPOSITION STRATEGY . Our decomposition strategy can be extended to any multiplication with variable bit-widths . As illustrated in Fig . 4a , if the inputs of a multiplication are represented as 2m-bit integers andm-bit×m-bit multipliers are provided , the multiplication can be departed into four m-bit×m-bit multiplications . Fig . 4b shows that when the straightforward method presented by Sharma et al . ( 2018 ) is applied , it needs two kinds of shifters ( one is for left-shift with 2m bits and the other is for left-shift with m bits ) and three shift operations for the computation . With our decomposition strategy , it needs one shifter and two shift operations as shown in Fig . 4c . For both Fig . 4b and 4c , two 2m-bit adders and one 3m-bit adder will be used . If an m-bit×m-bit multiplier is provided and the inputs of a multiplication are l ×m-bit integers a and b , the multiplication result can be obtained by : c = a× b = ( l−1∑ i=0 2i·mai ) ( l−1∑ i=0 2i·mbi ) = 2l−2∑ i=0 ( 2i·m i∑ j=0 ajbi−j ) ( 1 ) where ai and bi are m-bit integers expressing a and b by a = ∑l−1 i=0 2 i·mai and b = ∑l−1 i=0 2 i·mbi , respectively . With our decomposition strategy , the multiplication only needs l2m-bit×m-bit multiplications , 2l − 2 shifts and totally ( 4l2 − 5l + 1 ) ×m 1-bit full adders ( FAs ) , which are much less than l2 − 1 shifts and at least ( l2 × log2 l + l/2 ) × m FAs with a straightforward decomposition method . Furthermore , when our decomposition strategy is applied on matrix multiplication , the number of FAs for the composition of partial results can be reduced significantly . Suppose that the inputs of a matrix multiplication are matrices A and B , and the result is matrix C , then every element in C can be calculated by : cij = ∑N k=1 akbk where ak and bk are the kth element in row i of matrix A and the kth element in column j of matrix B , respectively . If the elements in matrices A and B are 2m-bit integers and the given multiplier can perform m-bit×m-bit multiplication , the element cij can be obtained by departing and composing : cij = N∑ k=1 ( ak1 × 2m + ak0 ) ( bk1 × 2m + bk0 ) = 22m N∑ k=1 ( ak1bk1 ) + 2 m N∑ k=1 ( ak1bk0 + ak0bk1 ) + N∑ k=1 ak0bk0 ( 2 ) where ak1 , ak0 , bk1 , and bk0 are numbers expressed inm bits and their combination , ak1×2m+ak0 and bk1 × 2m + bk0 , represent the value of elements ak and bk , respectively . In addition , the result of multiplication with power of 2 can be obtained by left-shift operation . Furthermore , the shift distance can be a constant with the value being m if Equation 2 is rewritten as : cij = 2 m [ 2m N∑ k=1 ( ak1bk1 ) + N∑ k=1 ( ak1bk0 + ak0bk1 ) ] + N∑ k=1 ak0bk0 ( 3 ) When the elements in matrices A and B are represented as l×m-bit integers , the elements in result matrix C can be obtained by replacing the expressions of ak and bk in Equation 2 as appropriate polynomials in form of ∑l−1 k=0 dk2 km , where dk is a m-bit integer . In this condition , a straightforward method needs Nl2−N shifts , but our decomposition strategy still needs only 2l− 2 shifts . To perform the matrix multiplication with appropriate hardware , we describe the detailed architecture in the following subsection .

Summary: This paper proposes to shorten the shift-addition operations in the straightforward configurable MACs (Sharma et al., 2018), to an addition-shift style. The authors claim that the new design is able to lower the energy consumption in the matrix multiplication. In the experimental analysis, the authors demonstrate the effectiveness of the proposed method.