A Weight Variation-Aware Training Method for Hardware Neuromorphic Chips
Authors: Min-Hye Oh
Venue: Submitted to ICLR 2023 (2023)
Keywords: edge computing systems
neuro-inspired computing
hardware implementation
synaptic device
hardware-oriented neural network

Abstract: Hardware neuromorphic chips that mimic the biological nervous systems have recently attracted significant attention due to their ultra-low power and parallel computation. However, the inherent variability of nano-scale synaptic devices causes a weight perturbation and performance drop of neural networks. This paper proposes a training method to find weight with robustness to intrinsic device variability. A stochastic weight characteristic incurred by device inherent variability is considered during training. We investigate the impact of weight variation on both Spiking Neural Network (SNN) and standard Artificial Neural Network (ANN) with different architectures including fully connected, convolutional neural network (CNN), VGG, and ResNet on MNIST, CIFAR-10, and CIFAR-100. Experimental results show that a weight variation-aware training method (WVAT) can dramatically minimize the performance drop on weight variability by exploring a flat loss landscape. When there are weight perturbations, WVAT yields 85.21% accuracy of VGG-5 on CIFAR-10, reducing accuracy degradation by more than 1/10 compared with SGD. Finally, WVAT is easy to implement on various architectures with little computational overhead.

Under review as a conference paper at ICLR 2023
A WEIGHT PARIATION-AWART TRAINING METHOD FOR HARDWARE NEUROMORPHIC CHIPS
Anonymous authors Paper under double-blind review
ABSTRACT
Hardware neuromorphic chips that mimic the biological nervous systems have re- cently attracted significant attention due to their ultra-low power and parallel com- putation. However, the inherent variability of nano-scale synaptic devices causes a weight perturbation and performance drop of neural networks. This paper pro- poses training method to find weight with robustness to intrinsic device variabil- ity. A stochastic weight characteristic incurred by device inherent variability is considered during training. We investigate the impact of weight variation both Spiking Neural Network (SNN) and standard Artificial Neural Network (ANN) with different architectures including fully connected, convolutional neural net- work (CNN), VGG, and ResNet on MNIST, CIFAR-10, and CIFAR- 100. Exper- imental results show that weight variation-aware training method (WVAT) dramatically minimize the performance drop on weight variability by exploring a flat loss landscape. When there are weight perturbations, WVAT yields 85.21% % accuracy of VGG-5 on CIFAR-10, 10, reducing accuracy degradation by more than 1/10 compared with SGD. Finally, WVAT easy to implement on various archi- tectures with little computational overhead.
INTRODUCTION
Deep Neural Networks (DNN) have achieved remarkable breakthroughs in computer vision, auto- matic driving, and image/voice recognition (LeCun al.) (0155). With this success, neuromorphic technology, which mimics the human nervous system, has recently received significant attention in the semiconductor industry. Compared with the conventional von Neumann architecture which has limitations in power consumption and real-time pattern recognition (Schuman et al.) 2017;; Indiveri ict al. 2015), neuromorphic chips, biologically inspired from the human brain, are new compact semiconductor chips that collocate processing and memory (Chicca et al., 2014; Catherine D. Schu- man & Kay| 2022). Therefore, neuromorphic chips can process highly parallel operations and be suitable for real-time recognizing images, videos, and audios with ultra-low power consumption [Iniiveri & Liu 2015).
Neuromorphic chips also suitable for "Edge Al computing." " which process data in edge devices rather than in the cloud at a data center (Nwakanma et al.) (021). In other words, tasks that require a large amount of computation, such training, are performed in the cloud and inference in edge devices. Traditional cloud AI processing requires sufficient computing power and network connec- tivity. This means that an enormous amount of data transmission is required, likely increasing data latency and transferring disconnections (Li et al.) 2020). It causes severe problems in autonomous driving, robotics, and mobile VR/AR that require real-time processing. Therefore, there a grow- ing need for data processing on edge devices. Neuromorphic devices are compact, mobile, and energy-efficient, promising candidates for edge computing systems.
However, despite enormous advances in semiconductor integrated circuit (IC) technology, hard- ware neuromorphic implementation and embedded systems with numerous synaptic devices remain challenging (Prezioso et al.) 2015 Esser et al.) 2015 Catherine D. Schuman & Kay 2022). De- sign considerations such as multi-level state, device variability, programming energy, speed, and arvay-level connectivity, variability an inevitable issue originating from manufacturing fabrication (Prezioso et al. 2010). Although there many kinds of nano-electronic devices for neuromorphic systems and in-memory
required. (Eryilmaz et al.) 2015). In particular, nano-electronic device

---

Under review as a conference paper at ICLR 2023
computing -including memristor, flash memory, chase-change memory, and optoelectronic devices- we call them "devices" for readability in this paper.
Device variability causes mapped synaptic weight values in hardware to differ undesirably from software weight. This gap between hardware synapse and software weight makes it challenging to implement neural networks in real-world applications. Many recent studies have reported that device variability can significantly reduce the accuracy of neuromorphic hardware and DNN accelerators (Catherine D. Schuman & Kay 2022; Peng et al. [020: Joshi et al. 2020b Sun & Yu 2019; Kim et al,, 2019; (018). Although there are various studies to solve this problem, they focus on the unique behaviors of devices (Hennen et al., 2022 vis Fu al. 2022). The diversity of devices used to implement neuromorphic hardware results in the customized solutions required for a given device variation. Therefore, the versatility of customized solutions at the device level is limited.
There is a growing need for hardware-oriented training method to learn parameters robust to device variability. It is widely known that wide and flat loss landscapes lead improved generalization (Keskar et al., 2017 Li et al., 2018)). It is natural to think that wide and flat loss landscapes with respect to weight will mitigate the accuracy drop caused by device variability. However, we experi- mentally confirm that related studies (Izmailov et al.) 2018 Wu et al. [2020] Foret et al. 202T) can not significantly reduce the accuracy drop by device variation (Experiments are provided later in section 2). This observation reminds us of the need for hardware-orientee neural network training method.
Motivated by this, we propose a weight variation-awar training method (WVA7) that alleviates performance drops induced by device variability at the algorithmic level rather than the device level. This method explores wide and flat weight loss landscape through the ensemble technique and the hardware-simulate variation-aware update method, which is more tolerant to the software weight perturbation caused by hardware synaptic variability. WVAT can effectively minimize performance drops with respect to weight variations with little additional computational overhead in the training phase. Our contributions include the following:
For the first time to the best of our knowledge, we investigate and analyze the impact of variations in model parameters on performance in several architectures, including standard Artificial Neural Networks (ANN) and Spiking Neural Networks (SNN), which are suitable for hardware neuromorphic implementation due to event-driven spike properties.
By exploring the flatter weight loss landscape, we propose WVAT that is tolerant to in- trinsic device variability. We introduce an ensemble technique for better generalization and present intuitive weight update method with hardware-simulate variation. This method is also efficient for quantization and input noise, which is one of the hardware implemen- tation issues besides weight perturbations.
We experimentally demonstrate that WVAT achieves nearly similar performance to the typical training method stochastic gradient descent (SGD) while having robustness to vari- ations in model parameters. WVAT is easy to implement with little computational cost.
By presenting an algorithm-level hardware-oriented training method, we expect that WVAT will help the development of society related to hardware implementation, including neuro- morphic chips"]
2
BACKGROUND
Many studies have been conducted to develop training methods robust device variability (Liu
et al.
2015 Long et al. 2019) Zhu et al. 2020 Joshi et al. 22206;; Joksas et al.) 2022, Huang et al.,
2022 Liu et al. (2015) proposed adding penalty for variations in model parameters to training loss. and Long et al. (2019); Zhu et al. (2020) generated noise model to reflect device variability during training phase. Although Long et al. (2019) achieved good performance, this method has
a limitation, a binary device (1 bit per cell). However, as mentioned in section [ the customized solutions for the given device has limitation in applying to the general case.
Recently, there have been many studies investigating the effect of loss landscape generalization (Garipov et al. 2018 Izmailov et al. 2018, [Wu et al. 2020 Foret et al. 2021 Liu et al. 2022). is
A source code will be available soon.
2

---

Under review as a conference paper at ICLR 2023
widely known that flat loss landscape reduces the generalization gap. Stochastic weight averaging (SWA) (lzmailov et al. 2018) is an ensemble technique that averages the weights as a time axis instead of storing multiple models with different weights. It has been experimentally demonstrated that the ensemble method brings a flatter loss landscape, resulting in a lower test error than SGD. Adversarial weight perturbation (AWP) (Wu al. 2020) and sharpness-awan minimization (SAM) (Foret et al. 2021) were proposed to improve model generalization by seeking the flat loss landscape. These methods explore the direction of weight perturbation in the worst case and update the weight based on that direction.
AWP AWP proposed adversarial weight perturbation v based on the generated adversarial exam- ples z for adversarial training:
V. 1 Ex iil ([w++o<z)).b) |w|||
V m Ei=1 ((fu+o(a)) ys)ll
where ye is label, and is batch size. (-) is neural network with weight w, and is standard classification loss.
SAM SAM seeks model parameters whose entire neighborhoods have uniformly low training loss value. Neighborhood maximizzii loss value- is given by the solution of a classical dual norm problem:
< ( =
where 1/p + 1/9 = 1, p is neighborhood size, and Ls(-) is training set loss. In the case of p = 2, c(w) is norm of the gradient scaled by p.
Both methods have similarities in finding weight perturbations the gradient-ascent direction, ex- cept the perturbation is scaled by norm of weight (AWP) neighborhood size (SAM). Although both methods yielded state-of-the-ar performance, they have drawbacks in terms of computational overhead A update rule in AWP and SAM requires two sequential gradient computations, one for obtaining weight perturbation and another for computing the gradient descent update (Liu et al. 2022). This has twice the computational overhead compared with SGD.
The effect of weight perturbation induced by device vari- ability on model performance can also be considered kind of generalization problem. We start with a per- spective that the flat loss landscape will have robustness against variations in model parameters. A naive approach to solve device variability issues applying techniques related to generalization. Unfortunately, as shown in Figure M these techniques did not lead to significant improvement in weight variation. As mentioned above, since AWP and SAM are similar algorithms in terms of computing weight perturbations, the experiment conducted with AWP. When there are variations in the
Ascuracy (N)
saradarr wi'veriation
Figure 1: Accuracy of VGG5 on CIFAR- 10 using SGD, SWA, and AWP.
weight, an accuracy drop of 36.58% for SGD, 34.17% for SWA, and 14.89% for AWP occurs compared with a case without variation. The accuracy drop caused by device variability a challenge. This motivates us to explore a new hardware-orient Therefore, we proposed efficient and intuitive WVAT by considering stochastic weight characteris- ties incurred by device inherent variability in the training phase and in the weight update to flatten the loss landscape.
one of the major issues that make hardware implementation
neural network training method.
WEIGHT VARIATION-AWAL TRAINING (WVAT)
Our goal is to find parameters that are tolerant perturbations of model parameters (c.g., weights) caused by the variability of synaptic hardware devices. We aim to minimize the loss, along with the difference the loss with respect to perturbed weights. Thus, objective follows:
min [[(w) + (L(w + w.) L(w))) min L(w w.)
3

---

Under review as a conference paper at ICLR 2023
where where w and we denotes weight and weight variation, respectively. L(-) represents loss function. In order to explore the flat loss landscapes with respect to weight variation, the difference term L(w + w.)
L(w) should be minimized. When the weight trained in software (cloud) is transferred to a hardware device (edge device), the mapped weight is likely to be different from the software weight due device variability. This results in performance degradation. For these reasons, here we devise two types of variations that reproduce device variations during training phase. One is hardware-simulated variation (HSV) reflecting device variability, and the other is gradient-ascent variation (GAV).
Hardware-simulated variation (HSV)
Assume "analog" synaptic devices, as shown in Figure 2 the layer-wise weight range is set [μ 30, / + 30] across the weight distribution, and we take a weight vari- ation on A.
<000
2006
2000
Av -w=-x: >N(, Ao.²)
where Aw is hardware-simularte variation (HSV) to imitate intrinsic hardware device variability, which is a random sample of the same size as w from Gaussian distribution with mean and variation Ao,, Y is a range coefficient, scaling factor, to determine the variation size. In general, when mapping software weights to a synaptic hardware device, clipping method is widely used rather than the full range of software weights ([wmin, Wmax D) due the memory window limit of the synaptic device (Kwon et al. 2019 Joshi et al., 20205).. Therefore, A is the range of weights that can be expressed in synaptic hardware devices. HSV refers to how much variation occurs within the range that can be expressed in the device.
Wmn
Wins
Weight
Figure 2: Weight distribution of a layer. A graph indicates the histogram of the software weight. Due to the device variability, the original software weight is perturbed. HSV is drawn from the weight range A to mimic the variation.
Many studies have reported that fabricated synaptic devices have Gaussian distribution (Gong al. 2018 Boybat et al.) 2018 Yu et al. 2013); therefore, we use a Gaussian distribution when generating HSV. addition, considering that the standard deviation of the fabricated devices usually ~5%
(Jshi et al., 2020a Wan et al.) 2019), we set that 10% standard deviation was simulated during training for more stable results. For example, oa of 10% means that the weight has changed by
10% of A. In order to minimize objective L(w + Aw) using SGD as a optimizer, differentiating it
as follows:
V VwL(w- + Aw) d(w + Aw) dL(w)
dw
dw
w=w+AW
= -V w== VwL(w + Aw) can be calculated as the differentiation at the value in which the weight variation occurs by the differentiation of the composite function.
Gradient-ascent variation (GAV) In addition to reproducing HSV during the training phase, the weight variation corresponding to the worst-case manit-nne-mmninn the greatest the difference term -shuld also be reflected to find flat loss landscapes. Recalling our objective, this is modified as maximiza- tion problem.
mnn]L(ww)) + mmax(L(w + w.,) L(w))] + min max L(w + w.)
=(w))
L(w + w.) can be approximated by first-order Taylor expansion find the weight variation w. that maximizes the loss.
L(w w.) L(w av(w)) ~ L(w) + V L(w)
where a is a step-length parameter. v(w)² i V... [(w)) is the rate of change in I, along the direction o (ur) at u. Therefore, the most rapidly increasing direction the solution to the problem.
max v(w)" V L(w) ((r)
subject to [|v(w) ||
|
V Tam (w)
v(w)
Aww|
V L (w)
4

---

Under review as a conference paper at ICLR 2023
SWA
WVAT
WVAT SWA
W2
dalt gradient on moved position
kuloott dralent noved position
DD upelite gradiont
sampling @D weath vanadion
varsyfion
haa- uodatt weight
2 updert avarage
Myeu
Figure 3: Schematic of the weight update of the each method.
where the magnitude of the weight variation is set to be the same as that of HSV. VLw (1r) is the steepest ascent direction for line search method (Nocedal & Wright 1999). For this reason, we name v(w) a gradient-ascent variation (GAV). This is a similar approach to AWP and SAM.
Weight update Weight variation simulated during training and updates the weight via SGD, The training process is as follows:
Weight variation: u 4- w + wy Weight update:
V L w w w.)
Weight reverse: w <- w we
where v) a learning rate. In this case, Wy be of the two proposed HSV and GAV. Depending on the probability p. it determine whether or not to adding wa for each batch. If no variance is added, the weight update is equivalent SGD and no weight reverse is performed.
2 (0, 1)
HSV Aw is generated æ < With GAV vww) is generated if x 2 with
wit is a threshold of what kind of weight variation will reproduce during training. xr is randomly sampled value from uniform distribution for each batch. For each batch, determines what kind Seuuresschemttialll illustrates the weight update according to each
of variation will be generated. method. SWA averages multiple points along the trajectory of SGD, leading to better generalization than SGD. By applying SWA to WVAT, wider loss landscapes can be explored.
EXPERIMENTS
In this section, we conduct experiments to evaluate the proposed WVAT on both artificial neu- ral network (ANN) and spiking neural network (SNN) with different architectures, including fully connected (FC), convolutional neural network (CNN), VGG, and ResNet on benchmark datasets (MNIST, CIFAR-10, CIFAR-100). These benchmarks and models have been widely used in hard- ware implementations (Kim et al.] [2018] Long et al. 2019) Zhu et al. 2020) Joshí et al.) 20206] Joksas et al.] 2022, Huang al.) 2022] Jung et al. (022).. Edge devices are mainly implemented using small models; moreover, small models are vulnerable to performance degradation due to de- vice variability. Hence, we focus experiments on small models, including ablation studies and comparisons with SGD, temporal spike sequence learning via backpropagation (TSSL-BP) (Zhang Li 2020), and SWA. Although SWA yields better performance via longer training, all method trained for the same epochs for fair comparison. We report the mean and standard deviation of test accuracy over 5 runs.
Figure g (a) and (b) shows the effect of two hyperparmeters on model performance for VGG5 on CIFAR- 10 and VGG16 on CIFAR-I00. respectively. We test WVAT for hyperparameter wis E (0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1). If wrs 0. only HSV is reflected, and if wts 1. only GAV is generated during training. For baseline, we test SGD and SGD Test Time Variation (SGD-TTV), which means when the weight of the trained model perturbs during a test phase. As shown in Figure D (a), SGD achieved 90.09% accuracy, while SGD-TTV 53.51 % when o, of 10% (36.58% performance drop by the variation). In the of Wish 1, only GAV is reproduced
5

---

Under review as a conference paper at ICLR 2023
(9)
VOG en CP
(6)
VGG-16 on CRPNN
(6)
VG CIPAR-10
(4)
PEESNET-2 en CIMA/6-10
Figure 4: Accuracy of SGD and WVAT. (a), (b) Effect of HSV and GAV, and (c), (d) range coeffi- cient y model performance.
during the training phase. Although it achieved the highest accuracy, it did not effectively prevent the degradation in accuracy when there was a variation in the weight. On the other hand, in the case of Web = 0, WVAT minimized the degradation in accuracy during TTV, but the model performance was poor than SGD. Therefore, we set Weh = 0.5 as the default value, which achieves nearly the similar performance
SGD and is robust weight perturbation induced by device variability. Since HSV requires one gradient computation, like SGD, and GAV requires two sequential gradient computations, our proposed WVTT, which is a mixture of HSV and GAV, has less computational overhead compared to AWP and SAM. Through this experiment, we experimentally confirmed that reflecting both HSV and GAV is a key factor in model performance and robustness against stochastic weight characteristics.
Range coefficient ? is hyperparameter that determines the variation size simulated in training. 1 means that the variation size during TTV is equally reflected during training. As shown in Figure 4 (c) and (d), HSV with the same variation size as the test minimizes the accuracy drop. We set 7 1 as the default value. Engineers can choose Web and y by considering the trade-offs for accuracy when variations occur and when they do not, according to the characteristics of each device.
4.1 MNIST
For MNIST, we compare the accuracy of the ANN with the FC network and spiking CNN trained by each method in Table [
FC network are trained using SGD optimizer with learning rate 0.0005. When applying SWA, we use SWA training with fixed learning rate schedule from scratch. We train spiking-CNNs by TS for training SNN, and all models for 100 epochs. For TSSL-BP, we use Adam optimizer with a learning rate of 0,0005 and other settings are the same as in Zhang & Li (2020). Hyperparameters for WVAT are set as default value, 7 = 1, Wha 0.5, and a = 0.1. For each network architecture, the proposed WVAT achieves nearly similar performance as SGD and SWA and minimizes accuracy degradation when weight variation occurs.
Hardware devices inevitably have inherent variability. Therefore, in most cases, the model perfor- mance in software cannot be directly reproduced in hardware implementation. From the hardware implementation point of view, it is preferable refer to the accuracy when there is weight variation rather than the accuracy when there is no variation. In the case of spiking-C 1CN11, when dg = 10% WVAT yields 96.02% accuracy achieving 1/5 reduction of accuracy drop compared with TSSL-BP (16.97% % performance drop for TSSL-BP, 3.29% for WVAT).
4.2 CIFAR-10
We test the proposed WVAT on CIFAR-10 in Table For comparison experiments, we use VGG-5, VGG-16, ResNet-20, and ResNet- 110 models. VGG-5, VGG-16 and ResNet-20 are trained for 200 epochs using SGD with momentum 0.9, weight decay 0.0005, and initial learning rate of 0.01. We use Preactivation ResNet-110 10 in Garipov et al. (018). ResNet- 10 are trained for 150 epochs using SGD with momentum 0.9, weight decay 0.0003, and an initial learning rate of 0.I. When applying SWA, we first run SGD optimizer with decaying learning rate schedule for 75% of the training budget, and then apply SWA with a fixed learning rate of 0.005 for all models except for 0.01 for ResNet- 110. We adopt hybrid conversion method (Rathi al.) 2020) for SNN implementation. Hyperparameters for WVAT are set as = 1, Wis = 0.5, and a 0.1 for VGG-5 and 0.01 for VGG-16, ResNet-20, and ResNet-110.

---

Under review as a conference paper at ICLR 2023
Table 1: Accuracy on MNIST for different model and method. 15C5 means convolution layer with 15 of the 5 x 5 filters, and P2 pooling layer with 2 x 2 filters.
Ace (%)
Acc (%) 0g = 10%
Model
Method
Acc (%)
Ge = 5%
SGD
98.13 + 0.09
97.86 + 0.14
96.54 + 0.23
FC 2-layer'
SWA
98.56 + 0.12
98.31 + 0.71
97.20 + 1.54
(ANN)
WVAT
98.47 0.05
98.39 + 0.04
97.98 + 0.19
TSSL-BP
99.37 + 0.02
98.08 + 0.15
82.40 = 8.03
Spiking-CNin CNN12 (SNN)
SWA
99.29 0.11
98.62 0.60
90.24 = 2.34
WVAT
99.31 0.07
99.07 + 0.07
96.02 + 0.82
TSSL-BP
99.38 0.05
98.81 0.06
86.17 + 8.55
Spiking-CNN2*
SWA
99.44 0.03
98.23 + 0.08
89.73 + 2.69
(SNN)
WVAT
99.39 + 0.03
99.24 + 0.08
95.57 + 1.87
784-400-200-10.
16C5-P2-Concnr322C3,
8C1)-8C1-288.
1SC5-P2--0CC--P2-3000
Table 2: Accuracy CIFAR-10 for different model and method.
Acc (%)
Acc (%)
Model
Method
Acc (%)
du 5%
0, 10%
SGD
89.96 + 0.16
79.08 0.95
53.39
3.92
VGG-5
SWA
89.97 + 0.22
80.62 + 1.51
57.14 + 2.42
(ANN)
WVAT
87.76 + 0.24
87.05 + 0.25
85.21 + 0.19
SGD
92.68 + 0.11
90.49 + 0.08
79.71 = 1.27
VGG-16
SWA
92.77 = 0.08
91.05 0.16
84.10 = 0.75
(SNN conversion)
WVAT
91.79 0.144
91.20 0.12
89.25 0.10
SGD
92.88 + 0.23
90.57 + 0.28
80.36 + 1.17
ResNet-20 (ANN)
SWA
92.98 + 0.19
91.51 0.18
84.54 0.57
WVAT
91.88 + 0.11
91.43 + 0.12
90.06 + 0.12
SGD
95.06 + 0.17
93.58 + 0.12
84.68 0.71
ResNet-1 10
SWA
95.47 + 0.06
94.26 +
87.26 + 0.95
(ANN)
WVAT
94.45 + 0.16
93.64 0.13
90.47 0.12
We experimentally confirm that WVAT minimizes the decrease in accuracy when there is variation induced by device variability most models. Compared to MNIST, the performance degradation due to weight perturbation is more pronounced. In the case of VGG-5, when 0, 10%, W yields 85.21% 1% accuracy, while 53.39% for SGD and 57. 14% for SWA. WVAT reduces the perfor- mance drop by more than I/10 compared with SGD and SWA. Therefore, we demonstrated that the advantage of WVAT in small models more significant.
4.3 CIFAR-100
We compare the accuracy of each method on CIFAR-100, and the results are summarized in Table For comparison experiments, we use VGG-16. ResNet-34, and ResNet-110 10 models. It is basically the same as the experimental setting of CIFAR-10, 10, and the different parts is as follows. Experimental settings for ResNet-34
CIFAT- 100 are the same ResNet-20
CIFAR-10. 10. ResNet-110 on
CIFAR-100 100 is trained with an initial learning rate of 0.05. As mentioned in section 4.2 WVAT outperforms other methods only in the small model but also in Resnet- 10. It important

---

Under review as a conference paper at ICLR 2023
to note that the standard deviation of aceuracy using WVAT is also the smallest for each network architecture, which means that the proposed WVAT is tolerant to weight variation.
Table 3: Accuracy on CIFAR-100 for different model and method.
Acc (%)
Acc (%)
Model
Method
Ace (%)
0, = 5%
0, = 10%
SGD
70.17 + 0.19
63.88 0.41
43.02 0.76
VGG-16
SWA
70.46 + 0.32
66.33 = 0.25
51.87 + 1.25
(ANN)
WVAT
68.13 0.12
66.51 0.14
62.16 + 0.11
SGD
61.40 + 0.68
58.73 + 0.63
49.80 + 1.04
ResNet-34 (ANN)
SWA
61.28 + 1.46
59.57
1.38
53.33 + 1.33
WVAT
63.65 + 0.43
62.49 0.40
59.25 0.65
SGD
76.72 0.55
69.22 0.42
32.68 0.75
ResNet-110
SWA
78.52 + 0.19
74.33 0.14
52.53 0.32
(ANN)
WVAT
78.13 + 0.94
74.59 + 0.87
54.25 + 0.82
DISCUSSION
5.1 WRIGHT LOOSS LANDSCAPES
In this section, an experiment is first conducted to investigate the effect of the proposed WVAT on the geometry of weight loss landscapes. We visualize the loss landscapes by plotting the change in loss the weight moves along the direction of the weight variation Aw.
L(w + Aw) = L(w + (0, Ao,
where ou varies in the range of [0, 15)%. and moving direction is adjusted with y -1 and y 1. We visualize it as drawn from 10 different samplings.
Spiking-CNN2
VGG-5
5CD
5:D
SWA WANTT
SWA
WVAT
16
1551
(7,5%
Figure 5: Loss landscapes with respect to model weights. (left) Spiking- CNN2 on MNIST, and (right) VGG-5 on CIFAR-10.
Figure 5 shows the test loss as a function of the magnitude of the weight variation. Weight loss landscapes become broader and flatter in the order of SGD, SWA, and WVAT, and are uniform for multiple trials. We are taken together with the experimental results in section 4 these experiments show the connection between flat weight loss landscapes and robustness to weight perturbations.
Secondly, we carefully investigate the effect of the proposed WVAT on the weight distribution. As shown in Figure 6 in Appendix A we verify that WVAT produces larger weights than SGD and SWA. Li 2018) argued that small weights are more sensitive to weight perturbations and make sharper loss landscape. This claim is consistent with our experimental results. Thus, based these observations, we demonstrate that WVAT produces larger weights, which makes it less sensitive to weight perturbations and leads to flatness of the weight loss landscapes.
8

---

Under review as a conference paper at ICLR 2023
VGG-5
VGG-16
ResNet-29
WNAT
WVAT
150
WUAT
353
SWA
28
1.5
-1.00
-0.5
-0.00
0.5
-0.15
-4.50
-005
6:0
4.5
Figure 6: Weight distribution of the first convolutional layer in various network architectures on CIFAR-10.
5.2 ROBUSTNESS TO QUANTIZATION
Although WVAT is not for quantization technique, the fact that WVAT robust to weight perturba- tion implies that it also effective against quantization and noise, another hardware implementation issue. Therefore, we evaluate the robustness of WVAT to quantization and input noise. There are
various quantization techniques (Zafrir et al. 2019 van Baalen al. [022] Jacob et al. 2018 Kirtas iet al. 2022) for small models, but post-training quantization is used to simply investigate the effect of quantization.
Qunatization
input noise
Accuracy
SCD
SGD
1941
1396
708
Figure 7: Accuracy of (Left) VGG-16 and (Right) VGG-9 on CIFAR-10.
For quantization and input noise, Figure shows that WVAT provides robustness compared with SGD. We confirm that the proposed method comprehensively effective for issues for implementing neuro-inspired computing in addition to weight perturbations.
6 CONCLUSION
This paper proposes a weight variation-aware training method that is robust to weight perturbations incurred by device variability. For the first time to knowledge, investigate and analyze the impacts of weight variations on various benchmark datasets and network architectures. The pro- posed WVAT effectively minimizes performance degradation by more than 1/10 compared to SGD when there is a weight variation. We propose HSV and GAV to mimic weight variations during the training and present weight update method that considers the stochastic weight characteristics. We experimentally confirm that WVAT is tolerant to weight perturbations by finding the flat loss land- scapes with respect to weight. This method is a hardware-oriented training method at the algorithm level rather than a custom solution at the device level to reduce the performance degradation for the stochastic weight characteristic caused by the inherent variability of the device. Therefore, when the weights trained by WVAT are transferred to hardware, accuracy drop due device variability can be prevented. especially effective in small models.
For limitations and future works, the proposed WVAT computes the gradient-ascent variation, quiring two sequential gradient computations, like AWP and SAM, so there is room for computa- tional improvement. Further research is needed how to achieve similar performance to WVAT with less computational overhead. We think that studies like Liu et al. (2022) can inspire us to solve this problem.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Irem Boybat, Manuel Le Gallo, SR Nandakumar, Timoleon Moraitis, Thomas Parnell, Tomas Tuma, Bipin Rajendran, Yusuf Leblebici, Abu Sebastian, and Evangelos Eleftheriou. Neuromorphic computing with multi-memristive synapses. Nature communications, 9():1-112 2018.
Maryam Parsa J. Parker Mitchell Prasanna Date Catherine D. Schuman, Shruti R. Kulka- mi and Bill Kay. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2:10-19, 2022. ISSN 2662-8457. doi: hups://doi.org/ 10. 10.10088/433888001-00004444
URL https:: /www sci encedi rect. com wwt.ecieecce......c
atticle/psI/224555552200000
Elisabetta Chicca, Fabio Stefanini, Chiara Bartolozzi, and Giacomo Indiveri. Neuromorphic elec- tronic circuits for building autonomous cognitive systems. Proceedings of the IEEE, 102(9): 1367-1388, 2014. doi: 10.1100111110 1wL.1009JR...221..23333333
S. Bure Eryilmaz, Duygu Kuzum, Shimeng Yu, and H.-S. Philip Wong. Device and system level design considcrations for hoalloo-oon-olaile---no In 2015 IEEE International Electron Devices Meeting (IEDM), pp. 4.1.1-4.1.4, 2015. doi: 10.1 1009/1/DM..01..4400000
based neuromorphic architectures.
Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V. Arthur, and Dharmendra S Modha. Backpropagation for energy-efficient neuromorphic computing. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gareet (eds.). Advances in Neural Information Processing Sys- tems, volume 28, Curran Associates, Inc., 2015. URL /ttpt:pedeeppoooeeeinngggggee/ www.innnnnnnriiggg neurips. Paper.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur, Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2021. httpstttt////oerreettt
net/ forum fncrrmm?id ?id= 6m1mpos 1rM
Jingyan Fu, Zhiheng Liao, and Jinhui Wang. Level scaling and pulse regulating to mitigate the impact of the cycle-to-cycle variation in memristor-basee edge ai system. IEEE Transactions 468801.
Electron Devices, 69(4))1752-1762 2022. doi: 10.1
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances neural information pro- cessing systems, 31, 2018.
Nanbo Gong, T Idé, S Kim, Irem Boybat, Abu Sehastian, V Narayanan, and Takashi Ando. Sig- nal and noise extraction from analog memory elements for neuromorphic computing. Nature communications, 9(1):1-8, 2018
Tyler Hennen, Alexander Elias, Jean-François Nodin, Gabriel Molas, Rainer Waser, Dirk Wouters, and Daniel Bedau. A high throughput generative vector autoregression model for stochastic synapses. Frontiers in Neuroscience, pp. 1423, 2022.
Chenglong Huang, Nuo Xu, Junwei zeng, Wenqing Wang, Yihong Hu, Liang Fang, Desheng Ma, and Yanting Chen. Rescuing reram-based neural computing systems from device variation. ACM Transactions on Design Automation of Electronic Systems (TODAES), 2022.
Giacomo Indiveri and Shih-Chin Liu. Memory and information processing in neuromorphic systems. Proceedings of the IEEE, 103(8):1379-1397, 2015. doi: 10.11099/PRO..205552444444
Giacomo Indiveri, Federico Corradi, and Ning Qiao. Neuromorphic architectures for spiking deep neural networks. 2015 IEEE International Electron Devices Meeting (IEDM), pp. 4.2.144.2.4, 2015. doi: 10.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05440 2018.
10

---

Under review as a conference paper at ICLR 2023
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-rithmetii--ee inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Dovydas Joksas, Erwei Wang, Nikolaos Barmpatsalos, Wing H Ng, Anthony J Kenyon, George A Constantinides, and Adnan Mehonic, Nonidcality-awan training for accurate and robust low- power memristive neural networks. Advanced Science, pp. 2105784, 2022.
Vinay Joshi, Manuel Le Gallo, Simon Haefeli, Irem Boybat, Sasidharan Rajalekshmi Nandakumar, Christophe Piveteau, Martino Dazzi, Bipin Rajendran, Abu Sebastian, and Evangelos Eleftheriou. Accurate deep neural network inference using computational phase-change memory. Nature com- munications, 1 11(1): 1-11):111,, 2020a.
Vinay Joshi, Manuel Le Gallo, Simon Haefeli, Irem Boybat, Sasidharan Rajalckshmi Nandakumar, Christophe Piveteau, Martino Dazzi, Bipin Rajendran, Abu Sebastian, and Evangelos Eleftheriou. Accurate deep neural network inference using computational phase-change memory. Nature com- munications, 1 11(1):1-13 2020b.
Seungchul Jung, Hyungwoo Lee, Sungmeen Myung. Hyunsoo Kim, Seung Keun Yoon, Soon-Wan Kwon, Yongmin Ju, Minje Kim, Wooseok Yi, Shinhee Han, et al. crossbar array of magnetore- sistive memory devices for in-memory computing. Nature, 601(7892)211-216,, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017. htts::/ /oppereveee......eo
Sungho Kim, Meehyun Lim, Yeamin Kim, Hee-Dong Kim, and Sung-Jin Choi. Impact of synaptic device variations on pattern recognition accuracy in a hardware neural network. Scientific reports, 8(1):1-7,201
Sungho Kim, Hee- Dong Kim, and Sung-Jin Choi. Impact of synaptic device variations on classifi- cation accuracy in binarized neural network. Scientific reports, 9()):177, 2019.
M. Kirtas, A. Oikonomou, N. Passalis, G. Mourgias-Alexandie M. Moralis-Pegios, N. Pleros, and A. Tefas. Quantization-aween training for low precision photonic neural networks. Neural Net- works, 2022. ISSN 0893-6080. doi: 00t0:0/0000......///////oooo/000
015. URL AtRhhttss
77 7/www. wwt.:ccciencccc rect cwm wtt.c/ooo.cccceeeee http:///cc.eeeeeaaa..cee////////
Dongseok Kwon, Suhwan Lim, Jong- Ho Bae, Sung-Tae Lee, Hyeongsu Kim, Chul-Heung Kim, Byung-Good Gook Park, and Jong-Ho Ho Lee. Adaptive weight quantization method for nonlinear synaptic devices. IEEE Transactions on Electron Devices, 66(1):395-40) 2019. doi: 10.1 1.1.T............ 2879821.
sonlinearsynaptiee
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 444, 2015. doi: https://doi.org/IW 1t.t./o/////...../aataiiieo 1038/nature 15539.
En Li, Liekang Zeng, Zhi Zhou, and Xu Chen. Edge ai: On-demand accelerating deep neural network inference via edge computing. IEEE Transactions on Wireless Communications, 19(1):
447-457, 2020. doi: 10. 10.1109/TC.2201.29444
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Beiye Liu, Hai Li, Yiran Chen, Xin Li, Qing Wu, and Tingwen Huang. Vortex: Variation-awan aware training for memristor x-bar. In Proceedings of the 52nd Annual Design Automation Conference, pp. 1-6, 2015.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-awe aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12360-12370 June 2022.
11

---

Under review as a conference paper at ICLR 2023
Yun Long, Xueyuan She, and Saibal Mukhopadhyay. Design of reliable dnn accelerator with un- reliable reram. In 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1769-1774. IEEE, 2019.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Cosmas Ifeanyi Nwakanma, hee-Woo Kim, Jae-Min Lee, and Dong-Seong Kim. Edge ai prospect using the neuroedge computing system: Introducing a novel neuromorphic technol- ogy. ICT Express, 7(2):152-157, 2021. ISSN 2405-9595. doi: https://doi.org/100.000
2021.05.003 URL https::/ http://wwww.iiieeeereree.....o http://rww..ooeeeeeeeeeeeiiaeaaa 01405959522000500
Xiaochen Peng, Shanshi Huang, Hongwu Jiang, Anni Lu, and Shimeng Yu. Dnn+neurosim v2.0: An
end-to-end benchmarking framework for compute-in-memory accelerators for on-chip training, 2020. URL https://aaxii: http://wwxvvoorg/aaoaaaaa org/abs/////o..ooooo...........r 2003. 0600b
M. Prezioso, F. Seerikk-BByarn B. D. Hoskins, G. C. Adam, and K. K. Likharev D. B. Strukov. Nan- otube devices based crossbar architecture: toward neuromorphic computing. Nanotechnology, (17):175202, 2010.
doi: https://doi.org// 00. http:////..000007-444450000 108000995--444221177750000
M. Prezioso, F, Merrikh-Bayal Bayat, B. D. Hoskins, G. C. Adam, and K. K. Likharev D. B. Strukov. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature, 521:61-64 64, 2015. doi: https://doi.org/ org/ 0. 1038/nature 14441.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. International Conference on Learning Representations, 2020.
Catherine D. Schuman, Thomas E. Potok, Robert M. Patton, J. Douglas Birdwell, Mark E. Dean, Garrett S. Rose, and James S. Plank. A survey of neuromorphic computing and neural networks in hardware. CoRR, abs/1705, 705.06963. 2017.
URL BRLhhtp:: http://aaxivvv.o http://aaaww..ooaa/////abb wttp://g///ab//.....0.0000
Xiaoyu Sun and Shimeng Yu. Impact of non-ideal characteristics of resistive synaptic devices on implementing convolutional neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, (3):570--579 2019.
Mart van Baalen, Brian Kahne, Eric Mahurin, Andrey Kuzmin, Andrii Skliar, Markus Nagel, and Tijmen Blankevoort. Simulated quantization, real power savings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2757-2761,
June 2022.
Qingzhou Wan, Mohammad T. Sharbati, John R. Erickson, Yanhao Du, and Feng Xiong. Emerg- ing artificial synaptic devices for neuromorphic computing. Advanced Materials Technolo- gies, 4(4):1900037, 2019. doi: http://two..rg///..00/0///mm0000
URL https://
online malinellbbary wi wwley.y....m cww. doi /00p admt 201900037]
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. In NeurIPS, 2020.
Shimeng Yu, Bin Gao, Zheng Fang, Hongyu Yu, Jinfeng Kang, and H-S Philip Wong. Stochastic learning in oxide binary synaptic device for neuromorphic computing. Frontiers neuroscience, 7:186, 2013.
Ofrr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Leurning and Cognitive Computing NeurlPS Edition (EMC2-NIPS), pp. 36-39, 2019, doi: 10. 11110MCCIIISSSS30000000000
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances Neural Information Processing Systems, 33: 13:12222-1200 2020.
Ying Zhu, Grace Li Zhang, Tianchen Wang, Bing Li, Yiyu Shi, Tsung- Yi Ho, and Ulf Schlichtmann. Statistical training for neuromorphic computing using coamistoorbbssed crossbars considering pro- cess variations and noise. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1590-1593 IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
convolutional leyer
2 convolutional leyer
3 convolutional leyer
WVAT SWA
WVAT
WHAT
SMA
SYA
5GD
SG2
SGD
30
36.04
300
-24
15 Tull conneted layer
2 fully conneted layer
fully conneted layer
WVAT SWA SGB
WHAT SWAA
13:00
WAAT
1.7
SMA
SCD
SGD
-1.5
-1.00
0.55
035
1V
Figure 8: Weight distribution of the entire network layer of VGG-5 on CIFAR-10 10.
VGG-16
ResNet-20
WHAT SINA
WVRT
10000
SGD
1004
SNA
090
1050
66.0
3.00
-0.3
Figure 9: Weight distribution of the last FC layer of VGG-16 and ResNet-20 CIFAR-10.
13


Reviews:
[{"review_id": "xOA-GkEQQI", "reviewer": "Reviewer_RFsq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs an algorithm for neuromorphic chips, where weights in the final product can look different due to manufacturing imperfections. Assuming the weights are perturbed with a known distribution, this devices an alternative to SGD that aims to minimize the generalization loss with expectation not just over the data distribution, but also the weights distribution. It does this showing improvements over SGD, as well as other methods that would seem to be beneficial for this type of generalization.", "summary_of_the_review": "This presents a novel algorithm specifically targeting neural network settings where final weights are hard to control exactly due to manufacturing imperfections. This does not affect the vast majority of neural networks running on smart devices today, which does limit the impact, but it could become increasingly important in the future and work in this direction seems motivated. It is sound work with good experimental results. Due to some question marks around potential practical impact and the methods being fairly straightforward, I rate this as a weak accept.", "strength_and_weaknesses": "Strengths\n- The training strategy is a combination of weight noise sampling during training (HSV, the more obvious training method) and a minimax loss that assumes perturbation in the worst direction. A combination of both gives the best results and tests on both artificial NNs as well as spiking NNs.\n- It is interesting that methods like AWP and SAM do not exhibit better robustness in this regard. This motivates the paper.\n- Ultra-low power devices that this is discussing can have significant societal benefits, not the least in the medical domain.\n\nWeaknesses:\n- It is never verified on real hardware. It is unknown how this Gaussian perturbation distribution actually corresponds to real hardware. One of the criticisms of prior art was that it was too specific to a particular hardware. That must mean there is a great variation, so does this simple synthetic Gaussian really represent the real world? If the benefit of this is that one training algorithm can work for many types of hardware, shouldn't that also be simulated in the evaluation protocol?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ntiCWPqZ41", "reviewer": "Reviewer_oSwx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a more robust training method for neurotrophic devices ", "summary_of_the_review": "This paper does propose a new training method to improve robustness towards wright variation, the applicability does not occur very broad to me other than for memristive devices. A few claims and comparison in quantisation and input noise should be fairer. ", "strength_and_weaknesses": "Strength:\n* The paper is well struct and easy to follow. Results are clearly listed in tables and figures.\n\nWeaknesses:\n* I think the claim of \"hardware-oriented\" is a bit of stretch as the core of the target problem in improving robustness on process variation mostly occurs in neuromorphic computing with analog in particular.\n* For having a process variation of 10%, does this mean testing with all weights with 10% shift, or up to 10% shift?\n* For figure 7, while I do understand that quantisation is an issue, comparing the performance against SGD rather than quantisation techniques seems a bit misleading. The same applies to input noise, these are covered by network robustness and attack. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QrMZtbBsMJ", "reviewer": "Reviewer_giaL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Spiking Neural Networks mimic the biological nervous systems for ultra-low-power NN execution.\nHowever, it can be sensitive to the weight perturbations that may lead to significant performance drop.\nThe paper aims to understand the causal relationship between the perturbations and the performance drop to devise a Weight Variation Aware Training Method.\nThe paper provides experimentation on small networks to show that it can reduce accuracy degradation.", "summary_of_the_review": "The paper aims to close the gap between the ideal and the real implementation of the SNNs by devising a variation-aware training. Considering the potential upsides of the SNNs such as the low power NN execution, the paper works on an important topic. While some improvements can be made, the paper also does a good job in explaining the problem and the key ideas. \n\nThe paper provides two ideas Hardware-simulated variation (HSV) which acts as a imitation model of the hardware variability. Similar idea is presented in the following paper which should be noted. However, considering the difference (DNN vs SNN) this seems like a neat contribution.\n* Ghodrati, Soroush, et al. \"Mixed-Signal Charge-Domain Acceleration of Deep Neural Networks through Interleaved Bit-Partitioned Arithmetic.\" Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques. 2020.\nIn addition to HSV, the paper develops Gradient-ascent variation which considers the worst case.\n\nThe paper runs experiments to show the effectiveness. It seems to make a thorough comparison against baselines (SGD and SWA). The paper also provides some discussions. While the experimentation on SNNs seem reasonable, experiments with ANNs seem rather weak. Adding some datapoints (larger networks, datasets) would make the claims stronger in the paper.\n\n* Typo in Figure 7: Qunatization -> Quantization", "strength_and_weaknesses": "+ SNNs are interesting research topic with huge potential for ultra-low-power NN execution.\n+ Closing the gap between the ideal vs real execution (with perturbation) is important.\n\n- More details on Section 3 would help.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2L0wKhdLs9", "reviewer": "Reviewer_XxLX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces two sources of noise applied during training to flatten the loss landscape and make weight more resilient to device variability: Hardware-simulated variation (HSA, gaussian noise which scales with the width of the weight distribution of each layer) and Gradient-ascent variation (GAV, added noise in the direction of steepest gradient). The proposed algorithm selects randomly between HSA and GAV during training, using (a hyperparameter $w_{th}$ as threshold and redrawing the selection at every batch. Results are presented on several neural network and spiking neural network models, on basic image classification datasets. ", "summary_of_the_review": "The authors demonstrated that there is value in the idea of alternating sources of noise during training to achieve flatter losses which make the model more robust to some type of noise at inference time. A correlation between actual noise encountered in hardware and this software simulated noise is lacking entirely, so it is hard to draw a conclusion on the actual effectiveness of the proposed technique in real world scenarios. The paper also necessitates extensive proofreading and polishing.", "strength_and_weaknesses": "Strengths:\n- although the two formulation of the proposed noise is definitely not novel, their threshold-driven combination might be\n- experiments compare a range of models across fundamentally different architectures\n- experiments do show (Fig. 5) that the proposed technique achieves remarkably flatter losses than the alternatives\n\nWeaknesses:\n- paper is often confusing, often lacking in details and clear explanations, it requires further proofreading\n- the method introduces two hyperparameters, $w_{th}$ and $\\gamma$, for the noise source selection and noise scaling, respectively. It's unclear and never motivated why the selected default values (chosen on the basis of VGG5, CIFAR-10 experiments) would be the appropriate selection across different models\n- some level of noise are simulated ($\\sigma_v = 5 or 10%$ in Table 1-3) but there is no correlation of this additional parameter with the incidence of hardware noise, which this technique is supposed to address. There is not attempt to establish this relationship at all (not even with a comparison with the literature). The shape of the real hardware noise may also dramatically differ from what is being used here at test time\n\nOther comments and corrections:\n- I don't understand the meaning of this sentence in section 1: \"we expect that WVAT will help the development of society related to hardware implementation\"\n- fig 1 shows accuracy drop in the presence of \"weight variation\". I assume these variations are weight noise added entirely at the software level (simulated device variability). Is this correct? What variations are being simulated in this figure?\n- section 3: when $\\gamma$ is introduced, it should be specified that it is an additional hyperparameter (this is only mentioned later).\n- section 4: SGD-TTV explanation is unclear: \"which means when the weight of the trained model perturbs during a test phase\". Does this mean the perturbation is only applied at test time? What level of perturbation is being applied?\n- figure 4a does not specify what $\\gamma$ is being used. figure 4b results do not specify what $w_{th}$ is used and do not appear to match numerically with those in figure 4a (for example, $\\gamma=1$ gives $WVAT-TTV \\sim 88%$, which in no case matches fig. 4a WVAT-TTV curve). \n- section 4.1: I suppose $\\alpha$ is the learning rate but it is never defined\n- fig. 6 shows a single layer per network. Some visualization of the distribution broadening across the full networks would be more appropriate\n- section 5.2: what does it mean that post training quantization \"is mainly used as a baseline\"?\n\nOther typos:\n- fig 4b: \"range coefficeint\"\n- fig 7: \"qunatization\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject