Improving the accuracy of neural networks in analog computing-in-memory systems by a generalized quantization method
Authors: Lingjun Dai
Qingtian Zhang
Huaqiang Wu
Venue:  (2021)
Keywords: analog computing-in-memory
quantization algorithm
deep neural networks

Abstract: Crossbar-enabled analog computing-in-memory (CACIM) systems can significantly improve the computation speed and energy efficiency of deep neural networks (DNNs). However, the transition of DNN from the digital systems to CACIM systems usually reduces its accuracy. The major issue is that the weights of DNN are stored and calculated directly on analog quantities in CACIM systems. The variation and programming overhead of the analog weight limit the precision.
Therefore, a suitable quantization algorithm is important when deploying a DNN into CACIM systems to obtain less accuracy loss. The analog weight has its unique advantages when doing quantization. Because there is no encoding and decoding process, the set of quanta will not affect the computing process. Therefore, a generalized quantization method that does not constrain the range of quanta and can obtain less quantization error will be effective in CACIM systems. For the first time, we introduced a generalized quantization method into CACIM systems and showed superior performance on a series of computer vision tasks, such as image classification, object detection, and semantic segmentation. Using the generalized quantization method, the DNN with 8-level analog weights can outperform the 32-bit networks. With fewer levels, the generalized quantization method can obtain less accuracy loss than other uniform quantization methods.

Under review as a conference paper at ICLR 2021
IMPROVING THE ACCURACY OF NEURAL NETWORKS IN ANALOG COMPUTING-IN-MEMOON SYSTEMS BY A GENERALIZED QUANTIZATION METHOD
Anonymous authors Paper under double-blind review
ABSTRACT
Crossbar-enabled analog computing srntting-ii-memnnn in-mmoory (CACIM) systems can signifi- cantly improve the computation speed and energy efficiency of deep neural net- works (DNNs). However, the transition of DNN from the digital systems to CACIM systems usually reduces accuracy. The major issue that the weights of DNN are stored and calculated directly on analog quantities in CACIM sys- tems. The variation and programming overhead the analog weight limit the precision. Therefore, suitable quantization algorithm is important when deploy- ing a DNN into CACIM systems to obtain less accuracy loss. The analog weight has its unique advantages when doing quantization. Because there is encod- ing and decoding process, the set of quanta will not affect the computing process. Therefore, a generalized quantization method that does not constrain the range of quanta and can obtain less quantization error will be effective in CACIM systems. For the first time, we introduced generalized guantization method into CACIM systems and showed superior performance on series of computer vision tasks, such as image classification, object detection, and semantic segmentation. Using the generalized quantization method, the DNN with 8-level analog weights can outperform the 32-bit networks. With fewer levels, the generalized quantization method can obtain less accuracy loss than other uniform quantization methods.
INTRODUCTION
Deep neural networks (DNNs) have been widely used variety of fields, such as computer vision (Krizhevsky et al. 2012, Simonyan & Zisserman 2015 He et al. (006)), speech recognition (Graves)
ct al. 2013; Hinton et al. 2012: Graves & Jaitly 2014), natural language processing (Kim] 2014 Yang et al. 2016; Lai et al. 2015 and so on (Mnih et al. 2015 Silver et al. 2016). However, the high complexity of DNN models makes them hard to be applied on edge devices (mobile phones, on- board computers, smart sensors, wearable devices, etc.), which only provide limited computing speed and power (Sze et al., (017).
Crossbar-enabled analog computing-in-memoryy (CACIM) systems is promising approach fa- cilitate the applications of DNN on edge devices (Yang et al. 2013). It can carry out some typical operations in situ, exactly where the data are located (Ielmini & Wong 2018). Such as the multiply- accumulate operation (MAC), which is the most frequently performed operation in DNNs. The cost of data transferring for doing the operations can be reduced. Both the computation speed and energy efficiency can be improved significantly Yao et al. 2020).
The footstone of CACIM systems for DNN is the crossbar array of the computational memory units (Hu et al. 2012). As shown in Figure [ taking the memristor device as an example, each weight (W,;) of the connection in one layer of neural network is stored as the conductance state (Gt,) of memristor. The input data are represented as the voltage (V)). After applying the voltage (V)) to each row, the current (4₁) collected at each column exactly the MAC result according to Kirchhoff's law and Ohm's law, I, > V,Gag.
Before applying a DNN in CACIM systems, an essential step is writing the weights of DNN into the memory units, which is usually called as mapping. However, the mapping overhead is directly

---

Under review as a conference paper at ICLR 2021
Well-trained Weight (W))
crossbur array
Wir
m
Map W G
m
6
2
GB
:
20-
M
Input Layer
Output Layer
Gy
Gsn
Gyt
Gim
Figure I: Schematic illustration of one layer of neural network performed in a crossbar array.
Weggts dendit
Avarigg mearagg quarsization
Average error
Average quantization
021 551
eneer 0.0110674
Vioight
0.4
-22
0.2
0.4
3.6
0.4
-0.2
0.2
0.4
0.6
0.4
-0.2
0.2
0.4
0.8
(a)
(b)
(c)
Figure 2: The example results of a uniform LAQ (Hou & Kwok 2018 quantizer (a), non-uniform (log-wise) INQ (Zhou et al.] 2017) quantizer (b), and generalized (Lloyd's) quantizer (c). We quantized the last fully connected layer's weight of ResNet- 18 with 8-level quantizers. The gener- alized quantizer can obtain less quantization error.
related to the precision of weights. Therefore, weight quantization method that compresses high- precision weights in DNN is important for an efficient implementation of DNN in CACIM systems.
The most important criterion of weight quantization method is the accuracy loss, which has strong correlation with the quantization error. The quantization error determined by the quantizer used in the quantization method. As far as knowledge is concerned, the generalized quantizer has not been used to quantize DNN weights in CACIM systems. The previous work used either uniform quantizers (Jung et al. 2019 Yang et 2019 or non-uniform quantizers (Tang et al. 2017 Sun et al. 2018). However, an empirical study has shown that the weights in one layer of DNN usually follow bell-shape and long- loigetailed distribution (Han et al. (016)). Meanwhile, we found that weights in the last fully-connected layer of DNN for the classification tasks usually follow an asymmetric distribution. The generalized density-aware quantizer (GDQ), which means the quantization results determined all depends the data without any constraints, can obtain less quantization error than either uniform or non-unifen non-uniform quantizer (Figure 2). Since the weights are stored and operated analog quantities in CACIM systems, using GDQ to quantize the weights won't 't produce extra cost the inference phase.
In CACIM systems, the noise of analog weights is inevitable and can be ignored. The pertur- bations of weights will degrade the performance networks severely. better to quantize the weights and improve the robustness noise in the training process together.
2

---

Under review as a conference paper at ICLR 2021
In this work, we introduced a generalized density-aware quantization method and noise awire train- ing scheme (NATS) for DNN in CACIM systems using 8-level weights series of computer vision tasks. Under the same weight level, our pro- posed method performed hetter than others.
and achieved no degradation of performance by
PRELIMINARY
A quantization method for DNNs consists of parts, one is the quantizer, the other is the quanti- zation algorithm which describes how to use the quantizer in a neural network.
2.1 QUANTIZER
Formulation: A quantizer is a function, / R + q, where q 4= {y RRi- 1,2, v). 3 [ E IRji 1,2, ... d} is the data to be quantized. Each qi has a corresponding domain Q that
1,2,
f(x) qs if x E Qi. where Qu = R. and Q: Qg when i + j. In most cases, {Q: = 1,2, ... 2 are intervals separated by - 1 endpoints e = {er E Ri 1.2, ..,,.... v0 loss of generality, we assume that q
1] on the real axis. Without q2 < < 4, and <e2 ... cx-1, 1. that is.
Q = {x -0 < <
Q2 {i <rSec.
Q = {x (x:6 < x S c
Q {x: 6p-1 < x x))
We use θ {y.ee } to denote a quantizer, and call = |a| the precision of the quantizer. The quantization error of data point =xx;)) defined as
=<x;)= fxx))
%o Ii. if x; E QQ.
quantizer O {g, e) is uniform if g is an arithmetic progression. The quantization method using
uniform quantizer is referred as uniform quantization method, such the BinaryConnect (Cour-
bariaux et al.) 2015). binary weight network (B (BWN) (Rastegari et al.) 2016), which have two levels,
the ternary weight networks (TWN) (Li al.) 2016) and the trained ternary quantization (TTQ)
(/huu et al.
2016), which have three levels, and some other methods that have more levels (He &
Fan 2019
Jung et al. 2019 Yang al. 2019 Shin et al.) 2017 Esser et al., 2019).
The q in a non-uniform quantizer is constrained to be a kind of non-uniform distribution. Such as g is geometric sequence (Li et al., 2020 Miyashita al. 2016 Zhou et al. 2017). These quantization methods work best when the data to be quantized follows the corresponding exponential distribution. The work (Choi et al., 2016) adopts k-means like algorithm quantized the weights.
Beyond the uniform quantization and non- non-uniform quantization, the generalized quantization meth- ods do not constrain the distribution of g. q is determined based on the distribution of the data be quantized, which is robust to all kinds of data distribution.
2.2 QUANTIZATION ALGORITHM
To accelerate the inference process of neural network, the quantizer is usually applied to both the weights and feature maps. In most of CACIM systems, the activations are still implemented by digital circuits, which is significantly different from the analog weights. So in this work, we focused on the weight quantization problem. There are two main strategies for weight quantization. The first one directly quantizes the weights in a trained neural network without fine-tuning or retraining (Zhao et al.) 2019 Banner et al. 2019). This strategy is fast and convenient, but its accuracy loss is usually greater than the other one, which will repeat the training and quantization iteratively until the performance is better enough (Courbariaux et al. 2015 Rastegari al. 2016 Li et al.) 2016 Zhu et al.) 2016). In the iterative quantization scheme, starts from the pre-trained neural network is more likely to get better performance than starts from scratch (Yang et al. 2019).
3

---

Under review as a conference paper at ICLR 2021
3 GENERAL QUANTIZATION ALGORITHMS FOR DNNS IN CACIM SYSTEMS
3.1 LLOYD'S QUANTIZER
quantizer quantize the weights of DNN this work. The - {q. e) is a quantizer as defined in Section 2.1 Quantization distortion E of a quantizer is defined as
We used Lloyd (9822)
(x) dF(x),
(4)
Σ
(9a - 2) QQ.
(5)
where F(x) is the cumulative probability distribution function of x. To minimize E, the quantizer iteratively optimizes the q and e until the relative error of E two consecutive iterations is smaller than given threshold.
3.2 NWARE-AWARE TRAINING SCHEME
We used the noise-aware training scheme (Murray & Edwards 1994) to improve the robustness to weight noise in this work. A Gaussian noise with mean and standard deviation added to each weight when doing the forward calculation. The a is determined by the production of the maximum of quantized weights (W) and a constant ratio 8.
W N(0, 6 IW )
(6)
3.3 TRAIN A WEIGHT QUANTIZED NEURAL NETWORK
Algorithm Training I--layers quantization network:
Input:
mini-batch of inputs and targets(I, Y), the pretrained full precision weights W, distinguish quantized levels, learning rate 1).
Initialize: quantizer @ by Lloyd (982), projection thresh
for i = 1 to L do quantized weight W; is calculated by θ;
noised weight W W; + W end for
compute model output: Y forward([I W) compute loss for i = L to 1 do
ac = Oc
compute weight gradient ow aw:
dC
update the full presicion weight W, according to
OW; and
W. O W ||
if
-1 1| >
then
W; O W; |||
re-initialize θ by Lloyd 1982
end if end for
The training algorithm with quantization and noise awareness is shown in Algorithm X
Σ, Øij { is its p-norm, X O Y denotes the element-wiss wise multiplication. There is one quantizer for each layer in the neural network. As some previous work do (Courbariaux et al. 2015 Rastegari et al. 2016), the quantized weights are used in the forward and backward propagations, but Estimator (STE) method
the update is applied on the full-precision weights. The used during backward propagations. In order to reduce the frequency of optimizing the quantizer, we used the projection of the quantized weight vector on the full full-precision weight vector to determine whether the quantizer need to be updated after each iteration. When the
[Bengio et al.) 2013)
4

---

Under review as a conference paper at ICLR 2021
projection exceeds certain range, the quantizer will be updated. It is found that, if we start from a pre- rre-traine neural network, the distribution of weights won't t have too much change. This means the projection won't 't exceed the proper range during the whole training phase and the quantizer will be optimized only once at the beginning.
EXPERIMENTS
4.1 IMAGE CLASSIHICATION
In this section, we validate our quantization method on the CIFAR-10 Krizhevsky et al. 2009) and the ImageNet-ILSVRC2012 (Russakovsky al. 2015) dataset.
4.1.1 EVALUATION CIFAR-10
ResNet-20: We first evaluated the quantization method with ResNet-20 on the CIFAR-10 10 dataset, which consist of 50k training images and 10k test images in 10 classes. The data augmentation method is same as the previous work (He al. 2016). For fine-tuning, we set the initial learning rate to 0.1, and scaled by 0.1, 0.1, 0.5 at epoch 80, 120, 160. The projection thresh 7 compared the performance of quantized model with several previous works, TTQ (Zhu et al. 2016), He (He & Fan 2019)), Lq-net (Zhang et al. 2018)) and Li (Li
to 1.5. We
2020). As shown in Table | our accuracy which is 0.42% lower than the full-precision model. Our 4-level model achieves 91.40% accuracy which is only 0.08% lower than the full precision model.
ternary model achieves
Table 1: Classification Accuracy (%) of ResNet-20 architecture with 3-level and 4-level weights CIFAR-10. "FP" denotes "Full Precision", "Quan" denotes "Quantization" and "Gap" means the accuracy of FP model minus the accuracy of Quan model.
Accuracy(%)
Method
Levels
FP
Quan
Gap
TTQ Zhu et 2016)*
91.77
91.13
0.64
He & Fan (2019
3
91.7
90.39
1.31
Ours
91.48
91.06
0.42
Ours w/ 1% noise
3
91.48
91.08+00.19
0.40++0.19
Lq-net (Zhagg et al. 2018)*
4
91.6
90.2
1.4
Li al. 2020
91.6
91.0
0.6
Ours
4
91.48
91.40
0.08
Ours w/ noise
91.48
91.45
0.04
'Weights in first and last layer keep full precision.
Weightss in first and last layer use 8-bit precision.
VGG-like Network: We also quantized VGG-like network on CIFAR-10 with ternary weights for evaluation. The model architecture and hyper-parametert are same as Hou's works (Hou & Kwok, 2018). The model has convolutional layers and 2 fully-connected layers. Batch size was set to 50 and the learning rate starts 0.002 and decayed by factor of 0.5 after every 15 epochs. The adam optimizer was used and the maximum number of epoch was set to 200. The projection thresh was set to O.I.
We compared the classification accuracy of our method and several related methods (Table 2p. The average accuracy of five trials using our method is higher than others. For DNNs which don t have pre-trained weights, training and quantizing it with our proposed method from scratch can also obtain good results. However, the quantizer may be updated frequently since the distribution of the weights will change a lot at the early stage of training. The closer to the solution, the more stable the weights distribution, that is, the fewer update times in the method.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Classification Accuracy (%) of VGG-like architecture with 3-1evel weights on CIFAR-10 (FP:89.62).
Method
Accuracy (%)
TWN (Li et al.
2016
89.36
LAT
Hou & Kwok
2018
89.62
TTQ
Zhu et al. 2016
89.41
Ours (from scratch)
89.67 + 0.14
Ours (fine tuning)
89.62 + 0.15
4.1.2 EVALUATION ON IMAGENET
The ImageNet dataset consists of 1.2M training and 50K validation images. We conducted the experiment with the ResNet-18b and ResNet-50b (He et al., 2016). The pre-trained models from the Py Torch model zoo were used The data preprocessing same the origin paper (He et al. 2016). 224 x 224 crop was randomly sampled from an image or its horizontal flip. Stochastic gradient descent (SGD) with the momentum of 0.9 was used optimize weight parameters. The mini-hatch was set to 64 and weight decay to 0.0001. The network was trained up to 90 epochs. The learning rate started from le-4, change 2c-5, 4c-6 epoch 30, 40 correspondingly, and then scaled by 0.5 after every epochs. The projection thresh 7 was set to 0.1.
To make more generalization, first and the last layer in the experiments. For fair comparison with some previous work, we also conducted the experiments that did quantize the first and last layer.
quantized the weights in all the layers of DNN, including the
When the precision is up to 8-level, both the top-1 and top-5 accuracy (70.0%/89.33) outperform the full precision model 099.8%/89.199 The 1 accuracy of ResNet-50b model with 4-vevel precision is only 0.3% lower than the full- precision model. We compared our methods with some previous studies, which were also based on the ResNet architecture. The experimental results are listed in Table 3 In most experiments, our results achieved the least classification accuracy gap.
Similar results were obtained in ResNet-50b network.
Table 3: Comparison the validation Top-1 and Top-5 accuracies gap (%) between ResNet- 18b and ResNet-50b with quantized weights or float weights using various quantization methods on Ima- geNet. The baselines of these two full precision models are 69.8/89.1 and 76. 1/92....
weight levels ResNet-18b
weight levels ResNet-50b
Method
3
3
8
TTQ B (Zhu et al. 2016)
3.0/2.0
ABCnet (Lin
2017
5.6/4.0
Lq-net" (Zhng et a
2018
2.3/1.5
1.0/0.7
1.3/0.9
Jung" (Jung
2019
0.3/0.3
He & Fan 2019
1.7/1.2
He Fan 2019
3.9/2.4
2.1/1.2
Yang et al. (2019)
1.2/0.6
-0.1/-0.1
1.2/0.6
0.2/0
Ours'
1.2/0.7
0.8/0.4
Ours
2.5/1.4
1.3/0.6
-0.2/-0.2
1.3/0.7
0.3/0.2
-0.3/-0.3
Weights in first and last layer keep full precision.
We searched the relative noise during training the ResNet- 18b with ternary weights. No significant difference was involved when å ranges from 1% to 4%, so we used 2% in all following experiments The comparisons the inference performance between our quantized models with weight noise and previous models without weight noise are shown in Figure 3

---

Under review as a conference paper at ICLR 2021
31 ResNot 186 Ours wi 3% noise
ResNes-1Sh wio
ResNet-18b
2%
4L ResNet-18b Lquet wio
8L ResNat 18b Ours w/ 1%
8L ResNet-18b Lypel wio moise
3L ResNer-o0b Ouus wi 19 noise ResNes-50b
4I. ResNer 506 Lenet wio noise
3L SSD Ours wi 15
3L SSD Yang wio moise
0.5
2.5
3.5
4.5
Top-1 Accuracy or mAP gap (76). The less, the better.
Figure 3: Quantization performance comparison between our quantized model with weight noise and others' model without weight noise.
4.2 OBJECT DETECTION
In this section, evaluated proposed approach general object detection task. Our exper- iments were conducted on SSD (single shot multibox detection) architecture (Liu et al. (016)) All methods were fine-tuned on the same pre-trained VGGI6 network. The models trained on the PASCAL VOC 2007 and 2012 training datasets and tested on the PASCAL VOC 2007 test dataset. For fair comparison, except for the final convolutional layers with convolution layer, the parameters of all other layers in the backbone VGG16 are quantized.
kernels and the first
The input images were resized to 300 300. SGD optimizer with the momentum set to 0.9 was used. We used the 1e-3 learning rate for 80k iterations, then continue training for 20k iterations with 1 and 15-5 5 The batch size was set to 16, and the weight decay was 5e-1, The results shown in Table When ternary weights was injected with 1% noise, the mAP gap (1.5% 0.06%) of our method was still not less than ADMM (Leng et al. 2018)'s 's and Yang et al. (2019) without noise. Our 8-level model performed better than full-precision model.
Table 4: mAP(%) of SSD PASCAL VOC object detection. The performance of the full full-precision model 77.8.
Weight levels
Methods
3
4
ADMM Yang Ours
76.2
77.6
76.3
77.7
76.7
77.1
77.9
Ours w/ 1% noise
76.3+10.06
77.1+0003
78.0=00.08
4.3 SEMANTIC SEGMENTATION
We evaluated our method on the PASCAL VOC 2012 segmentation dataset (Everingham al. 2010) and used the PSPNet (pyramid scene parsing network) architecture. Following the same settings in Zhao et al. (2017). we used augmented data with the annotation Hariharan et al. (00I resulting 10,582, 1,449 and 1,456 images for training, validation and testing. The backbone of our model was ResNet50. We evaluated the model with several-scale input and used the average results. The batch size was set to constrained by limited memory. The learning rate initialized by 1e and decayed by power policy with 0.9. SGD optimizer with the momentum set to 0.9 was used. Our 8-lele model can achieve the same performance with full-precision model 5

---

Under review as a conference paper at ICLR 2021
Table 5: Average loU (mloU%) and pixel Accuracy (pAcc%) of PSPNet PASCAL VOC 2012
valid set with our proposed method.
Weight levels
mlou
pAcc
75,5
93.6
4
76.3
93.9
8
77.3
93.9
float
77.3
94.2
4.4 PERFORMANCE ON REAL DEVICES
To further demonstrate the reliability of the results, map the weights to the real memristors and use measurement conductance to verify our method. The conductance (uS) range of our memristor device is [2.1, 17.85]. To represent the negative weights, the differential conductance (G) of two devices is used. G is given by = G - where G (G") is the conductance of a positive (negative) device. To reduce the mapping overhead, used unified reference conductance (Gref) all differential fairs. To represent positive (negative) weight, the negative (positive) device is mapped with Gref- For one layer's quantized weights in DNN model, firstly normalized them by dividing with the maximum value of absolute weights. Then we mapped the normalized weights to G' by multiplying with 15.75μS and used 2. 2.1 as Gref-
Taking an example of a well-trained ResNet- 18b model with 4-level weight, the set of quanta in the first convolutional layer is [-0.25, -0.01, 0.11. 0.38]. After scaling the weight set, we get the G (uS) set which is [-10.36, -0.57, 4.52, 15.75]. The G (uS) set is [2.1, 1. 6.62, 17.85], the G- (μS) set [12.46, 2.67, 2.1, 2.1]. The standard deviation of the measured is 0.32pS,, which approximately equal to 0.02 G maz- Figure 4(a) shows the simulated and measured distribution of the differential conductance. The measured and simulated differential conductance have the same statistical property. As shown in Figure 4(b) compared the inference performance of three types of weights: A) simulated weights training with NATS, B) measured weights training with NATS, C) measured weights training without NATS, using 4L, 8L ResNet- 18b and SSD model. The results of type A and type are very close. The results of type are better than those type indicating that NATS improved the robustness of the DNN model.
Sarakand MMesrred weight brixiog
Masard ovight NATS
2.5
Neasured G Simurated
Bcokin
2-
Sest F-033, 0.57 455 1578)
Densty
1.5
21.21
4L.50
0.5
0
-10
10
15
0.00
3.50
1,00
1,50
2,00
2.50
3.00
3.50 4.00
Differential Conductance (uiS)
mAl The lens, the
(a)
(b)
Figure 4: (a) shows the simulated and measured distribution of differential conductance (uS) which mapped from the 4L weight of the first convolutional layer in ResNet-18hh (b) shows the top- accuracy gap (%) of ResNet- 18b and mAP gap (%) of SSD model using 4I,, 8L. weights with simulated or measured noise.
8

---

Under review as a conference paper at ICLR 2021
5 DISCUSSION
5.1 GENERALIZED QUANTIZATION METHOD IN DIGITAL SYSTEMS
generalized quantizer can obtain less quantization error than a uniform one in theory when the data distribution is non-uniform. However, in digital computers, it needs more memory additional operations to process a set of non-uniform data. As shown in Figure a series of data are quantized {1, 4, 63. In digital system, will use bits to store each number with binary code. Although we can store these numbers also with 2 bits per number, a mapping function, that is "00" 1, "01 2, '10'= 4, '11' 6 must be stored and called whenever using these numbers. This additional cost limits the application of the generalized method in digital systems to certain extent. However, in the CACIM systems, the data is stored analog quantities, and the operation is based on the analog computing scheme. No matter what the exact value is, there is no significant difference in the storing or computing operation. This is why the generalized quantization method is more suitable for the CACIM system.
Additional memory overhead: use 3-bit representation
001
010
100
110
1
2
4
6
Quanta set of non- uniform quantizer
00
01
10
11
2
4
6
Additional operation of projection
Processing of series of non-uniform data in digital computers. overhead or additional operation of projection needed.
Figure 5:
Additional memory
5.2 PRINCIPLES OF CACIM-ORIENTED ALGORITHM DESIGN
Different from the digital system, CACIM system mainly based on analog computing, which may introduce great difference when designing and using neural network algorithms. For the quantization methods, the analog computing scheme means that the weights are represented and calculated on real quantities, not binary numbers implemented by high and low levels in the digital system. No matter what the exact weight is, the read or the computing is the same process, that is, apply voltage and collect the current. Therefore, more powerful quantization algorithm, such as the GDQ, which may obtain non-uniform results. Besides the quantization, the characteristics of analog computing may also play important roles in other scenarios. Since the current is directly accumulated and collected along the column of the crossbar, the behavior of each device the column may influence the results. If ignore these behaviors when designing the algorithm, they will degrade the performance of the algorithms. That why we usually call them non-ideal characteristics. However, we have a clear understanding of these characteristics, we can overcome them. Such as we used the noise-aware training scheme to improve the accuracies in this work. Furthermore, we can even utilize them to achieve better results, such using the variation of the device provide the stochasticity needed in the algorithm (Lin) et al. 2018 2019), using the I-V nonlinearity of the device to introduce the nonlinearity and displace the activation functions, weight decay operation the training process. As demonstrated in this work, the understanding of the hardware system is helpful for to design better algorithms.
utilize this characteristic select
using the relaxation behavior of the device to efficiently implement the

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional net-
works for rapid-deployment. In Advances in Neural Information Processing Systems, pp. 7948- 7956, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:130..3432, 2013.
Yoojin Choi, Mostafa El- El-Khamy, and Jungwon Lee. Towards the limit of network quantization. arXiv preprint ar arXiv:1612.011533 2016.
Matthieu Courbariaux, Yoshua Bengio, and Jean- David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint 1902.0899000055555 2019.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, pp. 1764-1772, 2014.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649 IEEE, 2013.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations, 2016
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 201 International Conference on Computer Vision, pp. 991-998. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 770-778, 2016.
Zhezhi He and Deliang Fan. Simultaneously optimizing weight and quantizer ternary neural network using truncated gaussian approximation. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11438-11446, 2019.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
Lu Hou and James T. Kwok. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations, 2018.
Miao Hu, Hai Li, Qing Wu, and Garrett S Rose. Hardware realization of bsb recall function using memristor crossbar arrays. In Proceedings of the 49th Annual Design Automation Conference, pp. 498-503. ACM, 2012.
Daniele lelmini and H-S Philip Wong. In-memory computing with resistive switching devices. Nature Electronics, 1(6):333-343, 2018.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Proceedings of the IEEE Conference on Computer Vision and Pattenn Recognition, pp. 4350 4359, 2019.
10

---

Under review as a conference paper at ICLR 2021
Yoon Kim. Convolutional neural networks for sentence classification. EMNLP 2014 2014 Con- ference on Empirical Methods in Natural Language Processing. Proceedings of the Conference, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In Twenty-ninth AAAI conference artificial intelligence, 2015.
Cong Leng, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural network: Squeeze the last bit out with admm. AAAI, 2018.
Fengfu Li, Bo Zhang, and Bin Liu. Temary weight networks. arXiv preprint arXiv:1605.04711, 2016.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: A non-uniform dis-
cretization for neural networks. International Conference on Learning Representations, 2020.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Yudeng Lin, Huaqiang Wu, Bin Gao, Peng Yao, Wei Wu, Qingtian Zhang, Xiang Zhang, Xinyi Li, Fuhai Li, Jiwu Lu, et al. Demonstration of generative adversarial network by intrinsic random noises of analog rram devices. In 2018 IEEE International Electron Devices Meeting (IEDM), pp. 3-4. IEEE, 2018.
Yudeng Lin, Qingtian Zhang, Jianshi Tang, Bin Gao, Chongxuan Li, Peng Yao, Zhengwu Liu, Jun Zhu, Jiwu Lu, Xiaobo Sharon Hu, et al. Bayesian neural network realization by exploiting inherent stochastic characteristics of analog rram. In 2019 IEEE International Electron Devices Meeting (IEDM), pp. 14-6. IEEE, 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yart Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
Stuart Lloyd. Least squares quantization pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Daisuke Miyashita, Edward H Lee. and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint /603..6603.000 (0055.00.5. 2016
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 518(7540):52-5533 2015.
Alan F Murray and Peter J Edwards. Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training. IEEE Transactions on neural networks, 5(5):722-802, 1994.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11(())211-252, 2015.
Sungho Shin, Yoonho Boo, and Wonyong Sung. Fixed-point optimization of deep neural networks with adaptive step size retraining. In 2017 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp. 1203-1207. IERE, 2017.
11

---

Under review as a conference paper at ICLR 2021
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Xiaoyu Sun, Xiaochen Peng. Pai- Yu Chen, Rui Liu, Jae-sun Seo, and Shimeng Yu. Fully parallel rram synaptic aray for implementing binary neural network with (+ 1,- 1) weights and (+ 1, 0) neurons. Proceedings of the 23rd Asia and South Pucific Design Automation Conference, pp- 574-579. IEEE Press, 2018.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2255-2229, 2017.
Tianqi Tang, Lixue Xia, Boxun Li, Yu Wang, and Huazhong Yang. Binary convolutional neural network on rram. In 2017 22nd Asia and South Pacific Design Automation Conference (ASP- DAC), pp. 782-787. IEEE, 2017.
J Joshua Yang. Dmitri B Strukov, and Duncan R Stewart. Memristive devices for computing. Nature nanotechnology, 8(1):13, 2013.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian- sheng Hua. Quantization networks. In The IEEF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. Proceedings of the 2016 conference of the North American chapter of the association for computational linguisties: human language technologies, pp. 1480-1489, 2016.
Peng Yao, Huaqiang Wu, Bin Gao, Jianshi Tang, Qingtian Zhang, Wenqiang Zhang, Joshua and He Qian. Fully hardware-implemented memristor convolutional neural network. Nature, 577 (7792):641-645 2020.
Dongging Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. ne-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2881-2890, 2017.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning. pp. 7543-7552, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. International Conference on Learning Representations, 2017.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization, Interna- tional Conference on Learning Representations, 2016.
12


Reviews:
[{"review_id": "e9Q3FNkHk1t", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In order to improve robustness of analog written weights to circuit variation, quantization is used. My first question is why is that the case? The claim that quantization reduces analog noise does not seem to be correct. As far as I can tell, this setup leads to quantization noise on top of analog noise. This is a very serious point since the authors just assume this claim to be true (first para of page 3) and that is the whole motivation. The claim should be justified, or at least a reference should be provided. P.S., I do not think the claim is correct; two noise sources (quantization + analog) is worse than one noise source (analog only).\n\nSecond issue: what is the contribution of this paper? As far as I can tell, two old methods (LLoyd max and dithering) are used and that's it. Sure, the experimental section is massive but this paper does not claim to be a pure empirical study. In fact, the claim is that for the first time, a general quantizer is used. Those methods have been around for decades!\n\nThe experiments only use quantization. One would expect that CACIM would be emulated somehow and we would see that indeed the proposed method improves CACIM's accuracy (which I don't think it would as discussed above). Even the conclusion talks about digital systems. So CACIM was (erroneously) used to motivate the work but then completely forgotten. \n\nPost Rebuttal Comments:\n\nI thank the authors for their feedback. I have no modification to make to my original review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bbLRFeeozEq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes a method to train weight-quantized neural networks.  The authors propose to directly calculate the endpoints that minimize the quantization error according to the weight distribution of each layer. Empirical results on image classification tasks and object detection tasks show that the proposed method outperforms other compared weight quantization methods under the same number of bits.\n\nStrengths: \n- The paper is clearly written and the proposed method is simple.\n- Experiments are performed on image classification tasks CIFAR-10 and ImageNet, with extensive comparisons with other methods. \n\nWeaknesses: \n- Since the Floyd algorithm requires alternating minimization between q and e, one concern is that this may be costly. How often is the Floyd algorithm called in Algorithm 1? What is the training time compared with other methods?\n\n- Comparison with one important reference [1] is missing. [1] also learns the quantized values, but (i) through the step size instead of the endpoints,  and (ii) uses uniform distributed quantized values instead of non-uniformly distributed ones. However, the reported results in [1] show that their 3-bit, 4-bit and 8-bit quantized models on ImageNet have  smaller accuracy gaps with the full-precision baseline compared to the proposed method. \n\n- One other concern is that while the authors claim that this method is proposed for compute-in-memory (CIM) system, and discussed in section 5 that the property of the CIM does not restrict the quantized values to be uniformly strictly, compared to digital systems, there is no empirical comparison of the efficiency (e.g. storage, latency) of the proposed method and other quantization methods in CIM. Do they just perform similarly in terms of computation and memory efficiency? If so, when the uniform quantization in [1] already has good accuracy, using generalized quantization as in the proposed method does not seem quite well-motivated?\n\n[1] Esser, Steven K., et al. \"LEARNED STEP SIZE QUANTIZATION.\" International Conference on Learning Representations. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SddXRcJ0Izk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contribution:\n1. Using quantization to make the deployment of CACIM feasible is straightforward and interesting. \n\n2. The proposed method was validated on CIFAR-10 and ImageNet with different models. \n\n\nCons:\n1. Combining quantization and CACIM is interesting. However, I am concerning the novelty of methods used in this paper. The 'general quantization' used in this paper is quite similar to Product quantization. [Stock, Pierre, et al. \"And the bit goes down: Revisiting the quantization of neural networks.\" arXiv preprint arXiv:1907.05686 (2019).] Quantization by grouping and k-means is well-known in this community. \n\n2. In the abstract, the authors claim that 'The analog weight has its unique advantages when doing quantization. Because there is no encoding and decoding process, the range of quantization function will not affect the computing process.' When we do quantization for the digital device, we quantize (almost) continual (high precision) numbers into discrete numbers. The range of quantizer is telling us which range we want to pay more attention to during quantization. Although I agree with the claim that 'there is no encoding and decoding.', I didn't follow the sentence 'the range of quantization function will not affect the computing process'. More explain should be provided about 'what is the definition of the range of quantizer and why it will impact quantization in the digital system and it will not impact quantization in the analogy system.'", "extra_scores": {}, "extra_text": {}}, {"review_id": "S3F2uxglNtX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Starting from the property of CACIM, this paper found that weight quantization fits CACIM's analog calculation process and proposed a general quantization method. Specially, it use Lloyd;s quantizer to quantize weights and followed the commonly used quantization training method (STE) to update the full-precision weights.\n\nPros:\n1. Find a good usage of model quantization in a new hardware system.\n\nCons:\n1. The biggest problem from my perspective is that: the proposed quantization method is disentangled with CACIM: It only considers that quantized weight has its unique advantages in analog computation. But CACIM is not involoved in how to acchieve the quantized weight. Other quantization method can also be applied to CACIM. Although experiments show outperformance over certain baselines, it only shows that the proposed method can achieve satisfying quantization. Overall, property of CACIM does not contribute to the design of quantization method.\n2. Besides, the proposed quantization method is not innovative. Lloyd's formula has been used in quantization before [1][2]. Not to mention the noise-aware training and STE.\n3. This paper's focus is on CACIM, but experiments are not conducted on this hardware. It will be much convincing if the proposed method is applied on CACIM.\n4. Is $|\\frac{\\bar{W}_i W_i}{\\bar{W}_i \\bar{W}_i} - 1| > T$ in Algorithm 1 typo?\n\n[1] https://github.com/spencerkent/generalized-lloyd-quantization\n[2] https://openreview.net/forum?id=rJ8uNptgl", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work develops a weight-quantization method for deep neural networks that is suitable for a type of analog hardware system known as crossbar-enabled analog computing-in-memory (CACIM).  The goal of this work is to train models on GPUs in such a way that they retain their predictive accuracy during inference when deployed on the analogue hardware system.

Pros:
* Good adaptation of quantization methods to the CACIM system
* Simple method
* Validation of the proposed method on multiple datasets and models

Cons:
* Lack of novelty: the proposed method is a simple combination of two popular methods, LLoyd's quantization and noise-aware training

All reviewers appreciate the simplicity of the method and the good fit to the hardware.  The authors responded to all reviews and two reviewers acknowledged the authors' response.  The authors acknowledge some reviewer observations (motivation of quantization as reducing analogue noise, lack of experiments on the actual CACIM system), and the authors added an experimental evaluation on the actual physical CACIM system showing that their method performs well.

Overall the work is well-executed and the proposed method is a good fit to the CACIM system.  However, the proposed quantization method is a straightforward adaptation of popular quantization methods.
Decision: Reject