Improving the accuracy of neural networks in analog computing-in-memory systems by a generalized quantization method

1 INTRODUCTION . Deep neural networks ( DNNs ) have been widely used in a variety of fields , such as computer vision ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , speech recognition ( Graves et al. , 2013 ; Hinton et al. , 2012 ; Graves & Jaitly , 2014 ) , natural language processing ( Kim , 2014 ; Yang et al. , 2016 ; Lai et al. , 2015 ) and so on ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . However , the high complexity of DNN models makes them hard to be applied on edge devices ( mobile phones , onboard computers , smart sensors , wearable devices , etc . ) , which can only provide limited computing speed and power ( Sze et al. , 2017 ) . Crossbar-enabled analog computing-in-memory ( CACIM ) systems is a promising approach to facilitate the applications of DNN on edge devices ( Yang et al. , 2013 ) . It can carry out some typical operations in situ , exactly where the data are located ( Ielmini & Wong , 2018 ) . Such as the multiplyaccumulate operation ( MAC ) , which is the most frequently performed operation in DNNs . The cost of data transferring for doing the operations can be reduced . Both the computation speed and energy efficiency can be improved significantly ( Yao et al. , 2020 ) . The footstone of CACIM systems for DNN is the crossbar array of the computational memory units ( Hu et al. , 2012 ) . As shown in Figure 1 , taking the memristor device as an example , each weight ( Wij ) of the connection in one layer of a neural network is stored as the conductance state ( Gij ) of a memristor . The input data are represented as the voltage ( Vi ) . After applying the voltage ( Vi ) to each row , the current ( Ij ) collected at each column is exactly the MAC result according to Kirchhoff ’ s law and Ohm ’ s law , Ij = ∑ i ViGij . Before applying a DNN in CACIM systems , an essential step is writing the weights of DNN into the memory units , which is usually called as mapping . However , the mapping overhead is directly related to the precision of weights.Therefore , a weight quantization method that compresses highprecision weights in DNN is important for an efficient implementation of DNN in CACIM systems . The most important criterion of a weight quantization method is the accuracy loss , which has a strong correlation with the quantization error . The quantization error is determined by the quantizer used in the quantization method . As far as our knowledge is concerned , the generalized quantizer has not been used to quantize DNN weights in CACIM systems . The previous work used either uniform quantizers ( Jung et al. , 2019 ; Yang et al. , 2019 ) or non-uniform quantizers ( Tang et al. , 2017 ; Sun et al. , 2018 ) . However , an empirical study has shown that the weights in one layer of DNN usually follow a bell-shape and long-tailed distribution ( Han et al. , 2016 ) . Meanwhile , we found that weights in the last fully-connected layer of DNN for the classification tasks usually follow an asymmetric distribution . The generalized density-aware quantizer ( GDQ ) , which means the quantization results are determined all depends on the data without any constraints , can obtain less quantization error than either uniform or non-uniform quantizer ( Figure 2 ) . Since the weights are stored and operated as analog quantities in CACIM systems , using GDQ to quantize the weights won ’ t produce extra cost in the inference phase . In CACIM systems , the noise of analog weights is inevitable and can not be ignored . The perturbations of weights will degrade the performance of networks severely . It is better to quantize the weights and improve the robustness to noise in the training process together . In this work , we introduced a generalized density-aware quantization method and noise-aware training scheme ( NATS ) for DNN in CACIM systems and achieved no degradation of performance by using 8-level weights on a series of computer vision tasks . Under the same weight level , our proposed method performed better than others . 2 PRELIMINARY . A quantization method for DNNs consists of two parts , one is the quantizer , the other is the quantization algorithm which describes how to use the quantizer in a neural network . 2.1 QUANTIZER . Formulation : A quantizer is a function , f : R → q , where q = { qi ∈ R|i = 1 , 2 , · · · , v } . x = { xi ∈ R|i = 1 , 2 , · · · , d } is the data to be quantized . Each qi has a corresponding domain Qi ⊂ R that f ( x ) = qi , if x ∈ Qi , ( 1 ) where ⋃v i=1Qi = R , and Qi ∩Qj = ∅ when i 6= j . In most cases , { Qi|i = 1 , 2 , · · · , v } are v intervals separated by v − 1 endpoints e = { ei ∈ R|i = 1 , 2 , · · · , v − 1 } on the real axis . Without loss of generality , we assume that q1 < q2 < · · · < qv and e1 < e2 < · · · < ev−1 , that is , Q1 = { x : −∞ < x ≤ e1 } , Q2 = { x : e1 < x ≤ e2 } , ... Qv−1 = { x : ev−2 < x ≤ ev−1 } , Qv = { x : ev−1 < x < ∞ } . ( 2 ) We use Θ = { q , e } to denote a quantizer , and call v = |q| the precision of the quantizer . The quantization error of a data point z ( xi ) is defined as z ( xi ) = f ( xi ) − xi = qα − xi , if xi ∈ Qα . ( 3 ) A quantizer Θ = { q , e } is uniform if q is an arithmetic progression . The quantization method using uniform quantizer is referred as a uniform quantization method , such as the BinaryConnect ( Courbariaux et al. , 2015 ) , binary weight network ( BWN ) ( Rastegari et al. , 2016 ) , which have two levels , the ternary weight networks ( TWN ) ( Li et al. , 2016 ) and the trained ternary quantization ( TTQ ) ( Zhu et al. , 2016 ) , which have three levels , and some other methods that have more levels ( He & Fan , 2019 ; Jung et al. , 2019 ; Yang et al. , 2019 ; Shin et al. , 2017 ; Esser et al. , 2019 ) . The q in a non-uniform quantizer is constrained to be a kind of non-uniform distribution . Such as q is a geometric sequence ( Li et al. , 2020 ; Miyashita et al. , 2016 ; Zhou et al. , 2017 ) . These quantization methods work best when the data to be quantized follows the corresponding exponential distribution . The work ( Choi et al. , 2016 ) adopts a k-means like algorithm to quantized the weights . Beyond the uniform quantization and non-uniform quantization , the generalized quantization methods do not constrain the distribution of q. q is determined based on the distribution of the data to be quantized , which is robust to all kinds of data distribution . 2.2 QUANTIZATION ALGORITHM . To accelerate the inference process of a neural network , the quantizer is usually applied to both the weights and feature maps . In most of CACIM systems , the activations are still implemented by digital circuits , which is significantly different from the analog weights . So in this work , we focused on the weight quantization problem . There are two main strategies for weight quantization . The first one directly quantizes the weights in a trained neural network without fine-tuning or retraining ( Zhao et al. , 2019 ; Banner et al. , 2019 ) . This strategy is fast and convenient , but its accuracy loss is usually greater than the other one , which will repeat the training and quantization iteratively until the performance is better enough ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ; Li et al. , 2016 ; Zhu et al. , 2016 ) . In the iterative quantization scheme , starts from the pre-trained neural network is more likely to get better performance than starts from scratch ( Yang et al. , 2019 ) . 3 GENERAL QUANTIZATION ALGORITHMS FOR DNNS IN CACIM SYSTEMS . 3.1 LLOYD ’ S QUANTIZER . We used Lloyd ( 1982 ) ’ s quantizer to quantize the weights of DNN in this work . The Θ = { q , e } is a quantizer as defined in Section 2.1 . Quantization distortion E of a quantizer is defined as E = ∫ +∞ −∞ z2 ( x ) dF ( x ) , ( 4 ) = v∑ α=1 ∫ Qα ( qα − x ) 2 dF ( x ) , ( 5 ) where F ( x ) is the cumulative probability distribution function of x . To minimize E , the quantizer iteratively optimizes the q and e until the relative error of E of two consecutive iterations is smaller than a given threshold . 3.2 NOISE-AWARE TRAINING SCHEME . We used the noise-aware training scheme ( Murray & Edwards , 1994 ) to improve the robustness to weight noise in this work . A Gaussian noise with zero mean and σ standard deviation was added to each weight when doing the forward calculation . The σ is determined by the production of the maximum of quantized weights ( |W̄ | ) and a constant ratio δ. W̃ = N ( 0 , δ · |W̄ |max ) ( 6 ) 3.3 TRAIN A WEIGHT QUANTIZED NEURAL NETWORK . Algorithm 1 Training a L-layers quantization network : Input : A mini-batch of inputs and targets ( I , Y ) , the pretrained full precision weights W , v distinguish quantized levels , learning rate η. Initialize : quantizer Θ by Lloyd ( 1982 ) , projection thresh T for i = 1 to L do quantized weight W̄i is calculated by Θi noised weight Ŵi = W̄i + W̃ i end for compute model output : Ŷ = forward ( I , Ŵ ) compute loss L for i = L to 1 do compute weight gradient ∂L ∂Wi = ∂L ∂Ŵi update the full presicion weight Wi according to ∂L ∂Wi and η if | ‖ W̄i ⊙ Wi ‖1 ‖ W̄i ⊙ W̄i ‖1 − 1| > T then re-initialize Θ by Lloyd ( 1982 ) end if end for The training algorithm with quantization and noise awareness is shown in Algorithm 1 . ‖ X ‖p= ( ∑ i ∑ j | xij |p ) 1 p is its p−norm , X ⊙ Y denotes the element-wise multiplication . There is one quantizer for each layer in the neural network . As some previous work do ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ) , the quantized weights are used in the forward and backward propagations , but the update is applied on the full-precision weights . The Straight-Through-Estimator ( STE ) method ( Bengio et al. , 2013 ) are used during backward propagations . In order to reduce the frequency of optimizing the quantizer , we used the projection of the quantized weight vector on the full-precision weight vector to determine whether the quantizer need to be updated after each iteration . When the projection exceeds a certain range , the quantizer will be updated . It is found that , if we start from a pre-trained neural network , the distribution of weights won ’ t have too much change . This means the projection won ’ t exceed the proper range during the whole training phase and the quantizer will be optimized only once at the beginning .

Summary: This paper proposes a method to train weight-quantized neural networks.  The authors propose to directly calculate the endpoints that minimize the quantization error according to the weight distribution of each layer. Empirical results on image classification tasks and object detection tasks show that the proposed method outperforms other compared weight quantization methods under the same number of bits.