NETWORK INSENSITIVITY TO PARAMETER NOISE VIA PARAMETER ATTACK DURING TRAINING
Authors: Julian Büchel
Fynn Firouz Faber
Dylan Richard Muir
Venue: ICLR 2022 Poster (2022)
Keywords: parameter attack
adversarial attack
neural network
deep learning
optimisation
neuromorphic processor

Abstract: Neuromorphic neural network processors, in the form of compute-in-memory crossbar arrays of memristors, or in the form of subthreshold analog and mixed-signal ASICs, promise enormous advantages in compute density and energy efficiency for NN-based ML tasks. However, these technologies are prone to computational non-idealities, due to process variation and intrinsic device physics. This degrades the task performance of networks deployed to the processor, by introducing parameter noise into the deployed model. While it is possible to calibrate each device, or train networks individually for each processor, these approaches are expensive and impractical for commercial deployment. Alternative methods are therefore needed to train networks that are inherently robust against parameter variation, as a consequence of network architecture and parameters. We present a new network training algorithm that attacks network parameters during training, and promotes robust performance during inference in the face of random parameter variation. Our approach introduces a loss regularization term that penalizes the susceptibility of a network to weight perturbation. We compare against previous approaches for producing parameter insensitivity such as dropout, weight smoothing and introducing parameter noise during training. We show that our approach produces models that are more robust to random mismatch-induced parameter variation as well as to targeted parameter variation. Our approach finds minima in flatter locations in the weight-loss landscape compared with other approaches, highlighting that the networks found by our technique are less sensitive to parameter perturbation. Our work provides an approach to deploy neural network architectures to inference devices that suffer from computational non-idealities, with minimal loss of performance. This method will enable deployment at scale to novel energy-efficient computational substrates, promoting cheaper and more prevalent edge inference.

Published as a conference paper at ICLR 2022
NETWORK INSENSITIVITY TO PARAMETER NOISE VIA ADVERSARIAL REGULARIZATION
Julian Büchel
Fynn Faber ETH Zürich, Switzerland faber r f@ethz.ch ch
Dylan R. Muir SynSense, Zürich, Switzerland dy an. mu resynsense at
IBM Research Zurich SynSense, Zürich, Switzerland ETH Zürich, Switzerland jbu @zurich. i.m. com
ABSTRACT
Neuromorphic neural network processors, in the form of compute- in-memory cross- bar arrays of memristors, or in the form of subthreshold analog and mixed-signal ASICs, promise enormous advantages in compute density and energy efficiency for NN-based ML tasks. However, these technologies are prone to computational non-idealities, due to process variation and intrinsic device physics. This degrades the task performance of networks deployed to the processor, by introducing param- eter noise into the deployed model. While it is possible to calibrate each device, or train networks individually for each processor, these approaches are expensive and impractical for commercial deployment. Alternative methods are therefore needed to train networks that are inherently robust against parameter variation, as a consequence of network architecture and parameters. We present new network training algorithm that attacks network parameters during training, and promotes robust performance during inference the face of random parameter variation. Our approach introduces a loss regularization term that penalizes the susceptibility of network to weight perturbation. We compare against previous approaches for producing parameter insensitivity such as dropout, weight smoothing and introduc- ing parameter noise during training. We show that our approach produces models that more robust to random mismatch-induced parameter variation well to targeted parameter variation. Our approach finds minima in flatter locations the weight-loss landscape compared with other approaches, highlighting that the networks found by our technique are less sensitive to parameter perturbation. Our work provides an approach to deploy neural network architectures inference devices that suffer from computational non-idcalities, with minimal loss of per- formance. This method will enable deployment at scale novel energy-efficing computational substrates, promoting cheaper and more prevalent edge inference.
INTRODUCTION
There is increasing interest in NN and ML inference on IoT and embedded devices, which imposes energy constraints due to small battery capacity and untethered operation. Existing edge inference solutions based on CPUs or vector processing engines such as GPUs or TPUs are improving in energy efficiency, but still entail considerable energy cost (Huang al., 2009). Alternative compute architectures such as memristor crossbar arrays and mixed-signal event-driven neural network acceler- ators promise significantly reduced energy consumption for edge inference tasks. Novel non-volatile memory technologies such as resistive RAM and phase-change materials (Chen, 2016; Yu & Chen, 2016) promise increased memory density with multiple bits per memory cell, as well as compact compute-in caaputt-ii-mmmoyy for NN inference tasks (Sebastian et al., 2020). Analog implementations of neurons and synapses, coupled with asynchronous digital routing fabrics, permit high sparsity in both network architecture and activity, thereby reducing energy costs associated with computation.
However, both of these novel compute fabrics introduce complexity in the form of computational non-idealities, which do not exist for pure synchronous digital solutions. Some novel memory technologies support several bits per memory cell, but with uncertainty about the precise value stored on each cycle (Le Gallo al., 2018b; Wu al., 2019). Others exhibit significant drift in stored

---

Published as a conference paper at ICLR 2022
states (Joshi al., 2020). Inference processors based on analog and mixed signal devices (Neckar et al., 2019; Moradi et al., 2018; Cassidy et al., 2016; Schemmel et al., 2010; Chaddam-Aljamed et al., 2022) exhibit parameter variation across the surface of a chip, and between chips, due to manufacturing process non-idealities. Collectively these processes known as "device mismatch" manifest frozen parameter noise in weights and neuron parameters.
In all cases the mismatch between configured and implemented network parameters degrades the task performance by modifying the resulting mapping between input and output. Existing solutions for deploying networks to inference devices that exhibit mismatch mostly focus on per-device calibration or re- re-training (Ambrogio al., 2018; Bauer et al., 2019; Nandakumar et al., 2020a). However, this, and other approaches such as few-shot learning or meta learning entail significant per-device handling costs, making them unfit for commercial deployment.
We consider a network to be "robust" if the output of network to given input does not change in the face of parameter perturbation. With this goal, network architectures that are intrinsically robust against device mismatch can be investigated (Thakur et al., 2018; Büchel et al., 2021). Another approach to introduce parameter perturbations during training that promote robustness during inference, for example via random pruning (dropout) (Srivastava et al., 2014) or by injecting noise (Murray& & Edwards, 1994).
In this paper we introduce novel solution, by applying adversarial training approaches to parameter mismatch. Most existing adversarial training methods attack the input space. Here we describe an adversarial attack during training that seeks the parameter perturbation that causes the maximum degradation in network response. summary, we make the following contributions:
We propose a novel algorithm for gradient- trasee supervised training of networks that are robust against parameter mismatch, by performing adversarial training in the weight space.
We demonstrate that our algorithm flattens the weight-loss landscape and therefore leads to models that are inherently more robust to parameter noise.
We show that approach outperforms existing methods in terms of robustness.
We validate our algorithm on a highly accurate Phase Change Memory (PCM)- based Compute- in- in-Memory (CiM) simulator and achieve new state-of-the-ard results in terms of performance and performance retention over time.
2 RELATED WORK
Research date has focused mainly on adversarial attacks in the input space. With increasing number of adversarial attacks, an increasing number of schemes defending against those attacks have been proposed (Wang et al., 2020; Zhang al., 2019; Madry al., 2019; Moosavi-Dezfooli et al., 2018). In contrast, adversarial attacks in parameter space have received little attention. Where parameter-space adversaries have been examined, has been to enhance performance in semi- supervised learning (Cicek & Soatto, 2019), to improve robustmess to input-space adversarial attacks (Wu et al., 2020), or to improve generalisation capability (Zheng et al., 2020).
We define "robustness" to mean that the network output should change only minimally in the face of a parameter perturbation - in other words, the weight-loss landscape should be as flat as possible at loss minimum. Other algorithms that promote flat loss landscapes may therefore also be useful to promote robustness to parameter perturbations.
Dropout (Srivastava et al., 2014) is a widely used method to reduce overfitting. During training, random subset of units are chosen with some probability, and these units are pruned from the network for single trial or batch. This results in the network learning to distribute its computation across many units, and acts as a regularization against overfitting.
Entropy-SCr -SD (Chaudhari et al., 2019) is network optimisation method that minimises the local entropy around a solution in parameter space. This results in a smoothed parameter-loss landscape that should penalize sharp minima.
Adversarial Block Coordinate Descent (ABCD) (Cicek & Soatto, 2019) was proposed in order to complement input-space smoothing with weight-space smoothing in semi-supervised learning.
2

---

Published as a conference paper at ICLR 2022
ABCD repeatedly picks half of the network weights and performs one step of gradient ascent them, followed by applying gradient descent on the other half.
Adversarial Weight Perturbation (AWP) (Wu et al., 2020) was designed to improve the robustness of a network to adversarial attacks in the input space. The authors use Projected Gradient Ascent (PGA) the network parameters to approximate worst case perturbation of the weights @'. PGA repeatedly computes the gradient of a loss function and updates the parameters in the direction of the (positive) gradient. After each update, the parameters are projected back onto a ball (e.g. in (2) around the original parameters to ensure that maximum distance is kept. Having identified an adversarial perturbation in the weight-space, an adversarial perturbation in the input-space is also found using PGA. Finally, the original weights è are updated using the gradient of the loss evaluated the adversarial perturbation O'.
Adversarial Model Perturbation (AMP) (Zheng et al., 2020) improves the generalisation of conven- tional neural networks by optimizing a standard loss evaluated using parameters that were perturbed adversarially using PGA. Unlike our method, (Zheng et al., 2020) did not formulate the loss function as a trade-off between performance and robustness. Furthermore, the presented algorithm, unlike our method, treats the perturbation Ag to the parameters & as constant during backpropagation.
TRadeoff- -inspirrddeedddddddeeee Adversarial DEfense via Surrogate-loss minimization (TRADES) (Zhang et al., 2019) is a method for training networks that are robust against adversarial examples in the input space. The method consists of adding a boundary loss term to the loss function that measures how the network performance changes when the input is attacked. The boundary loss does not take the labels into account, so scaling it by a factor Soub allows for a principled trade-off between the robustness and the accuracy of the network.
Noise injection during the forward pass (Murray & Edwards, 1994) is a simple method for in- creasing network robustness to parameter noise. This method adds Gaussian noise to the network parameters during the forward pass and computes weight gradients with respect to the original parameters. This method regularizes the gradient magnitudes of output units with respect to the weights, thus enforcing distributed information processing and insensitivity to parameter noise. We refer this method as "Forward Noise".
A recent paper proposed method for improving the resilience to random and targeted bit errors in SRAM cells on digital Deep Neural Network (DNN) accelerators (Stutz al., 2021). By employing adversarial or random bit flips during training, the authors significantly improved the robustness to bit perturbations, enabling the accelerators to be operated below the conventional supply voltage.
METHODS
We use θ to denote the set of parameters of a neural network f(x Q) that are trainable and susceptible to mismatch. The adversarial weights are denoted O", where θ; are the adversarial weights at the t-th iteration of PGA. We denote the PGA-adversary as function that maps parameters to attacking parameters O* We denote mini-batch of training examples as X with y being the corresponding ground-truth labels. I (m) denotes the projection operator on the ç-ellipsoid in space. The operator C denotes elementwise multiplication.
The effect of component mismatch on a network parameter can be modelled using a Gaussian distribution where the standard deviation depends the parameter magnitude (Joshi al., 2020; Büchel et al., 2021). In this paper we restrict ourselves to mismatch-driven perturbations in the network weights. For complex Spiking Neural Networks (SNNs), "network parameters" can refer additional quantities such as neuronal and synaptic time constants or spiking thresholds. Our training approach described here can be equally applied these additional parameters.
We define the value of an individual parameter when deployed on a neuromorphic chip as
miismatch ~ Ae( diag(C|o1)
(I)
where governs the perturbation magnitude, referred to as the "mismatch level". The physics underly- ing the neuronal- and synaptic circuits lead to model where the amount of noise introduced into the system depends linearly on the magnitude of the parameters. If mismatch-induce perturbations had constant standard deviation independent of weight values, could use the weight-scale invariance
3

---

Published as a conference paper at ICLR 2022
of neural networks means achieve robustness, by simply scaling up all network weights (see Figure S4). The linear dependence weight magnitude and mismatch noise preeludes this approach.
In contrast to adversarial attacks in the input space (Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2015; Madry et al., 2019; Goodfellow et al., 2015), our method relies on adversarial attacks in parameter space. During training, approximate the worst case perturbation of the network parameters using PGA and update the network parameters in order to mitigate these attacks. To trade-off robustness and performance, use surrogate loss (Zhang al., 2019) capture the difference in output between the normal and attacked network. Algorithm illustrates the training procedure in more detail.
begin - - + 19| 3 R; R ~ NN(,,1
for l to N Nueps do
g Vor, Crob E (,,
v t arg max TTo g
v:llolln $
0 Il (0 + a 3
end
θ - θ V/ (0,, v) + Fobb Nam X)
end
Nap lee (m) de- notes the projection operator on the -alus-ellipoid in IP space. In 1 this corresponds to min min(max((m, θ c), θ c) with Çatack ( Sanaac and Boob are hyperparameters of our model.
Algorithm 1: In 1xo. a corresponds to sign(g) and the step size is Ol
Unlike adversarial training in the input space, where adversarial inputs can be seen as a form of data augmentation, adversarial training in the parameter space poses the following challenge: Because the parameters that are attacked are the same parameters being optimized, performing gradient descent using the same loss that was used for PGA would simply revert the previous updates and no learning would occur. ABCD circumvents this problem by masking one half of the parameters in the adversarial loop and masking the other half during the gradient descent step. However, this limits the adversary in its power, and requires multiple iterations to be performed in order to update all parameters least once. AWP approached this problem by assuming that the gradient of the loss with respect to the attacking parameters can be used in order to update the original parameters to favor minima in flatter locations in weight-space. However, it not clear whether this assumption always holds since the gradient of the loss with respect to the attacking parameters is not necessarily the same direction that would lead to flatter region the weight loss-landscape.
We approach this problem slightly differently: Similar to the TRADES algorithm (Zhang et al., 2019). our algorithm optimizes a natural (task) loss and separate robustness loss.
gen O X, Lm (e, X,y) v) + Pro A(O),
Using a different loss for capturing the susceptibility of the network to adversarial attacks enables us simultaneously optimise for performance and robustness, without PGA interfering with the gradient descent step. our experiments, Lish defined as
Lro -*, x) KL. (), ((0",
This formulation comes with large computational overhead since it requires computing the Jacobian Je- (+) of a complex recurrent relation between θ and O* To make our algorithm more efficient we assume that the Jacobian diagonal, meaning that O = + AO for some AO given by the adversary. In 1... the Jacobian can then be calculated efficiently using (see suppl. material for details):
Jo (+) = I + diag sign O)@ (Gamactee-B
Naup sign (Vo; (O, 0 x) D
Nasps
By making this assumption, our algorithm effectively multiplies the original training time by the number of PGA steps, similar to (Wu al., 2020; Cicek & Soatto, 2019; Zheng et al., 2020).
4

---

Published as a conference paper at ICLR 2022
Because component mismatch is independently proportional to the magnitude of each parameter. one has to model the space in which the adversary can search for a perturbation using an axis-aligned ellipsoid in l2 and an axis-aligned box in lo Using an e-ball where the radius depends linearly the individual parameter sets (Li et al., 2018; Cicek & Soatto, 2019; Wu et al., 2020) would either give the adversary too little too much attack space. Projecting onto axis-aligned ellipsoid in l2 corresponds to solving the following optimization problem (Gabay & Mercier, 1976; Dai, 2006), which does not have a closed-form solution:
x" arg min =llm x 2
s.t. (x - w 2 x c) 1
where W = diag([@) +I Seonst: c θ and = 0 + a o Because of the computational overhead this would incur, we only consider the 1xx case in our experiments.
4 RESULTS
The ultra-low power consumption of mixed-signal neuromorphic chips make them suitable for edge- applications, such always-on voice detection (Cho al., 2019), vibration monitoring (Gies et al., 2021) or always-on face recognition (Liu et al., 2019). For this reason, we consider two compact network architectures in our experiments: A Long Short-term spiking recurrent Neural Network (LSNN) with roughly 65k trainable parameters; a conventional CNN with roughly 500k trainable parameters; and Resnet32 architecture (He et al., 2015) (see Supplementary Material S1 for more information). We trained models to perform four different tasks:
Speech command detection of classes (Warden, 2018);
ECG-an GCGananmyly detection on 4 classes (Bauer et al., 2019);
MaSSIon-MNIST (F-MNIST): clothing-image classification 10 classes (Xiao et al., 2017);
and
The Cifar10 colour image classification task (Krizhevsky, 2009).
We compared several training and attack methods, beginning with a standard Stochastic Gradient Descent (SGD) approach using the Adam optimizer (Kingma & Ba, 2015) ("Standard").. Learning rate varied by architecture, but was kept constant when comparing training methods on an architecture. We examined networks trained with dropout (Srivastava et al., 2014), AWP (Wu et al., 2020), AMP (Zheng et al., 2020), ABCD (Cicek Soatto, 2019), and Entropy-SGD (Chaudhari et al., 2019). The adversarial perturbations used in AWP and ABCD were adapted to our mismatch model (i.e. magnitude-deppnndent in lec) unless stated otherwise. AMP was not adapted.
A dropout probability of 0.3 was used in the dropout models and in AWP was set to 0. When Gaussian noise was applied to the weights during the forward pass (Murray & Edwards, 1994) relative standard deviation of 0.3 times the weight magnitude
used (Muain 0.3). For Entropy- SGD, we set the number of inner iterations to 10 with Langevin learning rate of 0.1. Because Entropy-SGD and ABCD have inner loops, the number of total epochs were reduced accordingly. All other models were trained for the same number of epochs (no early stopping) and the model with the highest validation accuracy was selected.
Effectiveness of adversarial weight attack We examined the strength of our adversarial weight attack during inference and training. Standard networks trained using gradient descent alone with no additional regularization (Fig. S5a, "Standard")) were disrupted badly by our adversarial attack during inference (( 0.1; final mean test accuracy 91 91.40 >> 17.50%), and this was not ameliorated by further training. When our adversarial attack was implemented during training (Fig. S5a, Blob 0.1), the trained network was protected from disruption both during training and during inference (final test accuracy 91.97% 78.41%).
Our adversarial attack degrades network performance significantly more than a random perturbation. Because adversary uses PGA during the attack, it approximates a worst-case perturbation of the network within an ellipsoid around the nominal weights e. We compared the effect of our attack against random weight perturbation (random point on <-ellipsoid) of equal magnitude.
5

---

Published as a conference paper at ICLR 2022
F-MNIST CNN
EGG LSNN
Speech LSNN
Manddard
0.77
3.3
1122
0.0
0:15
3.5 50 -1.
-0.5
110
0.5
-2.
011
1.5
15
Figure 1: Our training method flattens the test weight-loss landscape. When moving away from the trained weight minimum (a randomly-chosen directions, we find that our adversarial training method (Beta; Feta+Forward) finds deeper minima (for F-MNIST and Speech tasks) flatter locations in the cross-entropy test loss landscape. See text for further details, and Fig. S2 for visualisation over several random seeds.
For increasing perturbation size during inference (Fig. S5c; (), our adversarial attack disrupted the performance of the standard network significantly more than random perturbation (test acc. 91.40% + 17.50% (attack) vs. 91.40% - 90.63% (random) for Ç 0.1). When our adversarial attack was applied during training (Fig. S5d; 8mh 0.1), the network was protected against both random and adversarial attacks for magnitudes up to 0.7 and Ç 0.1, respectively.
Flatness of the weight-loss landscape Under our definition of robustness, the network output should change only minimally when the network parameters are perturbed. This corresponds to loss surface that is close to flat weight space. We measured the test weight-loss landscape for trained networks, compared over alternative training methods and for several architectures (Fig. 1). We examined only cross-entropy loss over the test set, and not the adversarial attack loss component chose random vector (O (/ θ + a v), Vhea) for many evenly-spaced E [-2, 2]. This process
(KL divergence loss; see Eq. 2). For each trained network, and calculated (f was repeated 5 times for 0.2, and the means plotted in Fig. 1. Weight-loss landscapes for the individual trials are shown in Fig. S2.
Our adversarial training approach found minima of trained parameters Ô in flatter areas of the weight- loss landscape, compared with all other approaches examined (flatter curves in Fig. 1), In most cases our training approach also found deeper minima lower categorical cross-entropy loss (fece), reflecting better task performance. These results are reflected in the better generalization performance of our approach (see Table 1). Not surprisingly, dropout and AWP also lead to flatter minima than the Standard network with no regularization. ABCD and Entropy-SGD were not included in Fig. 1 because they did not outperform the Standard model.
Network robustness against parameter mismatch We evaluated the ability of our training method to protect against simulated device mismatch. We introduced frozen parameter noise into models trained with adversarial attack, with noise modelled on that observed in neuromorphic pro- cessors (Joshi al., 2020; Büchel et al., 2021). In these devices, uncertainty associated with each weight parameter is approximately normally distributed around the nominal value, with a standard deviation that scales with the weight magnitude (Eq 1). We measured random mismatch for 100 samples across two model instances. A comparison of our method against standard training shown in Fig. 2. For mismatch levels up to 70% (Ç 0.7), our approach protected significantly against simulated mismatch for all three tasks examined (p < 2 x 10 in all cases; U test). A more detailed comparison between the different models given Table
accuracy under simulated
Network robustness against direct adversarial attack on task performance Our training ap- proach improves network robustness against mismatch parameter noise. We further evaluated the robustness our trained networks against a parameter adversary that directly attacks task perfor- mance, by performing PGA the cross-entropy loss Lece. Note that this is separate from the adversary used in our training method, which attacks the boundary loss (Eq.2). The AWP method uses the cross-entropy loss to find adversarial parameters during training. Nevertheless, we found that

---

Published as a conference paper at ICLR 2022
Manstch ked (Q)
Mismarih lovel (c)
Mianatex (C)
Figure Adversarial attack during training protects networks against random mismatch- induced parameter noise. Networks were evaluated for the Speech (a). ECG (b) and F-MNIST tasks (c), under increasing levels of simulated mismatch ((). Networks trained using standard SGD were disrupted by mismatch levels < > 0.1. At all mismatch levels adversarial training approach performed significantly better in the presence of mismatch (higher test accuracy). Red boxes highlight misclassified examples.
300
20
80
23
03
0.02 0.23
0.25
011
03
8:05 2.01
0.55
03
0.2
0.3
0.00 0.11
5.00
011
02
03
05
Attack
Abaak
Figure 3: Our training method protects against task-adversarial attacks in parameter space. Networks trained under several methods were attacked using a PGA adversary that directly attacked the task performance Looe. Our adversarial training approach (bold; dashed) outperformed all other methods against this attack.
our method consistently outperforms all other compared methods for increasing attack magnitude Ç (Fig. 3). In networks trained with method, the adversary needed to perform considerably larger attack to significantly reduce performance (test accuracy < 70%). ABCD and Entropy-SOO -SD were not included the comparison because they did outperform the standard network.
Robustness against parameter drift for PCM based CIM CiM devices based on memristor technologies such as PCM promise deliver energy- and space-efficient accelerators. With the increasing interest in CiM devices for accelerated inference and energy seeegy-efficcent edge computing (Sebastian al., 2020), the problem of deploying a model that robust noise originating from the device physics of PCM cells has gained in significance. We investigated the effect of our training method on the robustness of networks that are simulated to run on PCM-based CiM hardware (for details the simulator, see SM 5).
Networks trained with our method outperform state-of-the-art networks deployed on PCM-based CiM. Currently, the method that has proven to yield the best performance CiM hardware is training with noise on the network parameters during the forward pass. We adapted this method by adding algorithm and show that consistently outperform the conventional method (see Figure S S14) for a wide range of hyperparameters, and even surpass the FP baseline (a model trained without noise injection and evaluated standard PC) for some configurations (see Figure 4). Following
7

---

Published as a conference paper at ICLR 2022
Grack m 0.01
Grick m 0.03
Grayy m 0.05
Crros m 0.10
0010-
Test
Test
Test
F7 Sauetne
O.O. 0:110
(4)
Tac
Figure 4: Networks trained with our method show overall better performance when deployed on PCM-based CiM hardware. This figure shows the performance degradation as consequence of the PCM devices drifting over time (x-axis, up to one year) of networks deployed on CiM hardware. Each subplot shows networks trained with different attacking magnitude (Sattck) that are trained with different values of 3rbb. Each network is compared to the FP-baseline and a network trained with Gaussian noise injection on the weights.
Cenues 0.01
(uroac 0.10
Tor: SCC (8)
Saudlin
As 0.0
440
240 car
<en
3an 200
Tea C
Osee
Camees 0.01
Curues 0,03
Zuneci - 0.05
Zettace 0,10
Jcc 241
Best seC.
-101
440 Phau
cao
Add
AG
Figure 5: Our method consistently yields networks that outperform training with Gaussian noise injection. This figure compares the robustness to Gaussian noise at various levels (() for networks trained with method (blue) and networks trained with Gaussian noise injection (red), where the level of noise used during training (Mrain) matches the noise used during inference. Each row represents a different type of noise: The first models Gaussian noise with a standard deviation that is proportional to the largest absolute weight in the individual weight kernels (Joshi et al., 2020) and the second row follows the model presented in this paper (see Eq. 1).
experiments conducted in (Joshi et al., 2020), we used Resnet32 (He et al., 2015) trained on Cifar10 (Krizhevsky, 2009).
Injecting Gaussian noise the weights during the forward pass yields strong improvements compared to the standard network. We show that by adding our method, we consistently improve this robustness by significant amount (see Figure 5).
We furthermore improve the scalability of our method by using a pretrained model and fewer steps for the adversary. Our algorithm incurs an additional training time that scales linearly with the number of attack steps used in the adversary (note that we cache the necessary gradients for the Jacobian calculation). To alleviate this additional time, we show that our method produces good results even for just one single adversarial step. Figure S13 shows the resulting performance when varying the number of attack steps used by the adversary. should be noted that all results reported on PCM robustness were obtained using three ad adversarial steps and a pretrained model.
Verifiable robustness for LSNNs We investigated the provable robustness of LSNNs trained using our method using abstract interpretation (Cousot & Cousot, 1977; Gehr et al., 2018; Mirman et al., 2018). In this analysis function f(x, θ) (in our case, neural network with input and parameters θ) is overapproximated using an abstract domain. We specify the weights θ in our network as an interval parameterised by the attack size Ç, spanning ( < θ + [[+]]. We examined the proportion of provably correctly classified test samples for the Speech and ECG tasks, under a range of attack sizes (, and comparing our approach against standard gradient descent and against training
8

---

Published as a conference paper at ICLR 2022
with forward-pask pass noise only (Fig. S9). We found that our approach is provably more correct over increasing attack size { (higher verified test accuracy).
DISCUSSION
We proposed new training approach that includes adversarial attacks the parameter space during training. Our proposed adversarial attack was significantly stronger than random weight perturbations at disrupting the performance of trained network. Including the adversarial attack during training significantly protected the trained network from weight perturbations during inference. Our approach found minima in the weight-loss landscape that usually corresponded to lower loss values, and were always in flatter regions of the loss landscape. This indicates that our approach found network solutions that are less sensitive to parameter variation, and therefore more robust. Our approach was more robust than several other methods for inducing robustness and good generalisation. To the best of our knowledge, our work represents the first example of interval bound propagation applied to SNNs, and the first application of parameter-space adversarial attacks to promote network robustness against device mismatch for mixed-signal compute. Our experiments only considered the impact of weight perturbations, and did not examine the influence of uncertainty in other parameters of mixed-signal neuromorphic processors such as time constants or spiking thresholds. Our approach can be adapted to include adversarial attacks in the full network parameter space, increasing the robustness of spiking networks. The technique of interval bound propagation can also be applied to these additional network parameters. We did not quantize network parameters either during after training, in this work. On some platforms (Moradi et al., 2018) it is necessary to deploy quantized weights and it is unclear how adversarial attacks would interact with quantization during the training process. However, most PCM-based CiM hardware does not require quantization during training in order get good performance (Joshi et al., 2020). Per-device training for a device with known calibrated parameter noise is likely to achieve the highest possible deployed performance on that single device. However, this approach has significant drawbacks. Firstly, each device must be either measured/ calibrated accurately - not a trivial requirement or trained with the device in the forward inference pass of the training loop. Secondly, training must be performed individually for each device, entailing significant logistical problems if the training is conducted in the factory or inside a consumer product. Thirdly, this approach will retain full sensitivity to parameter variation on the device. Our method improves the performance of neural networks deployed to inference hardware that include computational non-idealities. For example, NN processors with crossbar architectures based on novel memory devices such RRAM and PCM (Sebastian et al., 2020) display uncertainty in stored memory values as well as conductance drift over time (Le Gallo et al., 2018b; Wu et al., 2019; Joshi et al., 2020). Our method could also address in-memory computing-basee NN processors based on SRAM and switched capacitors (Verma et al., 2019). Analog neurons and synapses in mixed-signal NN processors, for example SNN inference processors (Moradi al., 2018), exhibit variation in weights and neuron parameters across a processor. We showed that our training approach finds network solutions that are insensitive to mismatch- induced parameter variation. Our networks can therefore be deployed to inference devices with computational reduction task performance, and without requiring per-device calibration or model training. This reduction in per-device handling implies a considerable reduction in expense when deploying at commercial scale. Our method therefore brings low-power neuromorphic inference processors closer to commercial viability.
idealities with only minimal
Ethics statement The authors declare conflicts of interest.
Reproducibility statement
Code for reproduce all experiments described in this work are provided at https:: /ttit//itttttttt com/ jwwuecheuucceee/eee /BPTT-/A/ psch tzness and https: / / thub. RRw./eeeneeettteeeeeeeeeeee 32-
This work was partially supported by EU grants 826655 "TEMPO"; 871371 "MEMSCALES" and 876925 "ANDANTE" to DRM. JB would also like to thank Manuel Le Gallo- Bourdeau, Irem Boybat and Abu Sebastian from IBM Research Zurich, for insightful discussions and technical support.
Acknowledgments

---

Published as a conference paper at ICLR 2022
Table Results of training multiple networks using several methods over three tasks. Networks were evaluated under different levels of mismatch ()).
Derand Note.
4.1
Sumberd
Drapest
18%
Nox
54
Xic.
61
01.31
0:11
21
0.5
9356
96-45
88:10
$154
6.97
$1.60
90.30
8.16
91.5
00.13
015
$5.33
15:42
0.38
08.21
19.00
0.34
13W
1.0
82148
42.44
$732
8.
B0.55
06.22
71.78
534
13.13
7.56
55.31
Froved Note: Ale 1.1
Sumbed
ANP
1111
Dispud
AND 5 0400
Mixe a
Siaa Aat.
$-4
16
Mow
36
$4.6
50
Monn Aec.
S44
$4.6
5-4
56..
Moat
34
14.8
$4
M4c
66.35
06.45
65.13
M4.04
18.10
21.22
52.33
07
13.05
98.90
01
98.31
6.5
94.66
2.58
03:39
15.23
0.66
6A155
73.55
25.75
2.50
$2.4
00:32
1.38
11.04
$3.56
026
64.13
73.35
11.37
28.30
Au
(153
Dippal
M a-
Minc
8.4
16.
941
34m
14-m
14x
Mmn 54
16x
Mon
94
644
Mm.
8-2
08.33
1011
00.34
0.1
3..0
12.53
00.03
8.3
16.14
0.0
33.0
13.0
060
1555
21.50
0.01
79.01
0110
2718
14.00
09:00
5535
25.00
REFERENCES
Stefano Ambrogio, Pritish Narayanan, Hsinyu Tsai, Robert M. Shelby, Irem Boybat, Carmelo di Nolfo, Severin Sidler, Massimo Giordano, Martina Bodini, Nathan C. P. Farinha, Benjamin Killeen, Christina Cheng, Yassine Jaoudi, and Geoffrey W. Burr. Equivalent-accuracy accel- erated network training using analogue memory. Nature, 558(7708):60-67,, June 2018. ISSN 1476-4687.
doi: 10. 10038/4418800018-0111111 0-0-000-.. URL https: /toii/ooi./////// wow.:/o.orrr//.. 038/ s41586- 018001888000 010---018881000000
C. Bauer, D. R. Muir, and G. Indiveri. Real-time ultra-low power ecg anomaly detection using an event-driven neuromorphic processor. IEEE Transactions on Biomedical Circuits and Systems, 13 (6):1575-1582, 2019. doi: 10. 1,11100/BCASS00199 2953300.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert A. Legenstein, and Wolfgang Maass. Long short-term memory and leaming-ttolear in networks of spiking neurons. CoRR, abs/ abs/1803.09574, 2018. URL http: //rxiv. .rg. abs///b///// /1003. 01514188333........
Irem Boybat, Manuel Le Gallo, SR Nandakumar, Timoleon Moraitis, Thomas Parnell, Tomas Tuma, Bipin Rajendran, Yusuf Leblebici, Abu Sebastian, and Evangelos Eleftheriou. Neuromorphic computing with multi-memrritive synapses. Nature communications, 91):25514, 2018.
Julian Büchel, Dmitrii Zendrikov, Sergio Solinas, Giacomo Indiveri, and Dylan R. Muir. Super- vised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors, 2021.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. CoRR, abs/1608.0466 2016. URL http: //tp//aaaivvv//////////aa org/ abs/ 2tls/1600 1160.. 04644.
Andrew S. Cassidy, Jun Sawada, Paul Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Filipp Akopyan, Bryan L. Jackson, and Dharmendra S. Modha. Truenorth: A gigh-performance, low- power neurosynaptic processor for multi-sensory perception, action, and cognition. 2016.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018. dec 2019. doi: 10. 10. 11088//742-5448//339000 URL https: org./.orrg/a 1088 2P1742-5444 68 FF ab39d9.
An Chen. A review of emerging non-volatile memory (nvm) technologies and applications. Solid-State Electronics, 125:25-38, 2016. ISSN 0038-1101. doi: https://doi.org/10..000 10e 2016.07.006. URL https: www. sciencedirect. com/ wwteeeeneeeeeeeeeeeeeeeeeeeeeeee S0C38110116300865 Extended papers selected from ESSDERC 2015.
Minchang Cho, Sechang Oh, Zhan Shi, Jongyup Lim, Yejoong Kim, Seokhyeon Jeong, Yu Chen, David Blaauw, Hun-Seok Kim, and Dennis Sylvester. 17.2 142nw voice and acoustic activity
10

---

Published as a conference paper at ICLR 2022
detection chip for mm-scale sensor nodes using time-interleaved baieer-based frequency scanning. In 2019 IEEE International Solid- State Circuits Conference (ISSCC), pp. 278-280, 2019. doi: 10.1 (LL0///SSC..21..86666393
Safa Cicek and Stefano Soatto. Input and weight space smoothing for semi-supervised learning, In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 1344-1353, 2019. doi: 10. 109/1CCW..201900011111
Patrick Cousot and Radhia Cousot. Abstract interpretation: A unified lattice model for static analysis of programs by construction approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, POPL '77. pp. 238-252, New York, NY, USA, 1977. Association for Computing Machinery. ISBN 978145333500. doi: 00.1145/51250.5129777 URL https: /tt. org/ 10. 1145/ 512950.22550 0ttp///...1115512550101 512973.
Yu-Hong Dai. Fast algorithms for projection on an ellipsoid. SIAM Journal on Optimization, 16(4):98--1006 2006.
doi: 10.1137/040613300. URL https: http:::/doo.oggggg.
040613305.
Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications, 2(1): 17-40, 1976. ISSN 0898-1222 1221. doi: http://ddi.orr//00111609999-- (22167))000 URL https: //www. sci encedi rect. com/ sssssiieese /ttps//enccaaaaacleppppp......pp http:///..aaaee/e/////////211100
Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen. Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP), pp. 3-18, 2018. doi: 10.1 1..100//P.2018 8.00058.
Valentin Gies, Sebastián Marzetti, Valentin Barchasz, Hervé Barthélemy, and Hervé Glotin. Ultra-low power embedded unsupervised learning smart sensor for industrial fault classificatio. 2020 IEEE International Conference on Internet of Things and Intelligence System (IoTaIS), pp. 181-187,
2021. doi: 10. T11////TTS130099222222222 200109949..22113399717
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Tch and Mike Titterington (eds.). Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. JMLR Workshop and Conference Proceedings. URL http::tpp/ooceeednng//// mlr press/ v glooottoootooooooooooooooooooooo 0a .ttml.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/15 1at.01352.0385., 2015. URL http: /ttp/varxvvv//////////.././.a orq/ aaba//////// ha/aabba11122/ /11./111..00 03385.
Huang, S. Xiao, and Feng. On the energy efficiency of graphies processing units for scientific computing. In 2009 IEEE International Symposiun on Parallel Distributed Processing, pp. 1-8, 2009. doi: 10. 109/IPDPS.2009, 1609900.
Vinay Joshi, Manuel Le Gallo, Simon Haefeli, Irem Boybat, S. R. Nandakumar, Christophe Piveteau, Martino Dazzi, Bipin Rajendran, Abu Sebastian, and Evangelos Eleftheriou. Accurate deep neural network inference using computational phase-change memory. Nature Communications, doi:
11(1):2473, May 2020. ISSN 2041-1723.
11107-000-1608-99- 16108-9. URL https:
//doi. 100 1038 4 44144444000000 - -200--200---500 6100899
Riduan Khaddam-Aljameh, Milos Stanisavljevic, Jordi Fornt Mas, Geethan Karunaratne, Matthias Brändli, Feng Liu, Abhairaj Singh, Silvia M Müller, Urs Egger, Anastasios Petropoulos, et al. HERMES-core-a T.Pm/m PCM 14-nm CMOS in-memory compute using 300- ps/LSB linearized CCO-based ADCs. IEEE Journal of Solid-Statt State Circuits, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv. www.iiivrggaas///o 412. 6980
11

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Manuel Le Gallo, Daniel Krebs, Federico Zipoli, Martin Salinga, and Abu Sebastian. Col- lective structural relaxation phase-change memory devices. Advanced Electronic Mate- rials, 1700677 2018a. doi:
URL https: //onlinelibrary wiley com/ htt//doi/aab//// 1002 aelm. 201700627.
Manuel Le Gallo, Abu Sebastian, Roland Mathis, Matteo Manica, Heiner Giefers, Tomas Tuma, Costas Bekas, Alessandro Curioni, and Evangelos Eleftheriou. Mixed-precision in- in-memory computing. Nature Electronics, 1(4):246-253, April 2018b. ISSN 2520-1131. doi: 10.1038/ 141928-018-0054- URL https: //tp::://///iiee org 10. . 1088 s 928 018-0054- 8.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL htt.t:tss///ae httppsts//pprocceeiiiiiie wew.reediinnnnuuunneeeeeeeee neurips cc/paper//ae pdf.
a41b3bb3e
Qian Liu, Ole Richter, Carsten Nielsen, Sadique Sheik, Giacomo Indiveri, and Ning Qiao. Live demonstration: Face recognition on an ultra-low power event-driven convolutional neural network asic. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1680-1681, 2019, doi: 10.1109/VVP..2219.000
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
Matthew Mirman, Timon Gehr, and Martin T. Vechev. Differentiable abstract interpretation for provably robust neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3575- 3583. PMLR, 2018. URL http: pttppeeecceegsssee mlr, press /veess//v/// / mi rman 8b html.
Seyed-Mohsen Moosavi-Dezfoolii Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. CoRR, abs/1511.04500 2015. URL http: //aaxaavvv.//////////////////aaa o aabs/1511 04599.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. CoRR, abs/1811.0977 1,9916,, 2018. URL http:// /ttp:///axivv../ wttg/bs/////reeeeeeeee 09716..
S. Moradi, N. Qiao, F. Stefanini, and G. Indiveri. A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dy- naps). IEEE Transactions on Biomedical Circuits and Systems, 12(1):106-122, 2018. doi: 10.1009TBBCAS222727277777
A.F. Murray and P.J. Edwards. Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training. IEEE Transactions on Neural Networks, 5(5):792 802, 1994. doi: 10.1 10.1109/72.317330.
S. R. Nandakumar, I. Boybat, V. Joshi, C. Piveteau, M. Le Gallo, B. Rajendran, A. Sebastian, and E. Eleftheriou. Phase-change memory models for deep learning training and inference. In 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pp. 727-730, 2019. doi: 10.1109/CCCCS44459..0099 89644552.
R. Nandakumar, Manuel Le Gallo, Christophe Piveteau, Vinay Joshi, Giovanni Mariani, Irem Boybat, Geethan Karunaratne, Riduan Baadddam-Ajjamed Urs Egger, Anastasios Petropoulos, Theodore Antonakopoulos, Bipin Rajendran, Abu Sebastian, and Evangelos Eleftheriou. Mixed- precision deep learning based on computational memory. Frontiers in Neuroscience, 14:406, 2020a. ISSN 1662-453X. doi: 20.3389 00400000000...0000000000 URL https: :ttps:///wwwww //www. www.:/wfooo.eeeeeeeeeeoeeoe frontiersin.ozg. article/100 3389 fni fn..n.n.........mmmm............ 2020 00406.
12

---

Published as a conference paper at ICLR 2022
SR Nandakumar, Irem Boybat, Jin-Ping Han, Stefano Ambrogio, Praneet Adusumilli, Robert L Bruce, Matthew BrightSky, Malte Rasch, Manuel Le Gallo, and Abu Sebastian. Precision of synaptic weights programmed in phase-change memory devices for deep learning inference. In 2020 IEEE International Electron Devices Meeting (IEDM), pp. 29-4. IEEE, 2020b.
Alexander Neckar, Sam Fok, Ben Benjamin, Terrence Stewart, Aaron Voelker, Chris Eliasmith, Rajit Manohar, and Kwabena Boahen. Braindrop: mixed-signal neuromorphic architecture with dynamical systems-based programming model. Proceedings of the IEEE, 107:144-164, 01 2019. doi: 10. 10.1109/PPOC.200 2881432.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate O(1/k2). Proceedings of the USSR Academy of Sciences, 269:543-547, 1983.
Schemmel, Brüderle, A. Grübl, M. Hock, K. Meier, and S. Millner. wafer-scale neuromorphic hardware system for large-scale neural modeling. In 2010 IEEE International Symposium Circuits and Systems (ISCAS), pp. 1947-1950, 2010. doi: 10. 109/ISCAS.000.55566777
Abu Sebastian, Manuel Le Gallo, Riduan Khaddam-Aljameh, and Evangelos Eleftheriou. Mem- ory devices and applications for in-memory computing. Nature Nanotechnology, 15(7): 529-544, 2020. doi: 10. 00.10888/41555-000-0665533 s41565- 1011665-22-0655-22
URL https: /top:://do...o http:////o..oo//.111//1333
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929- 1958, 2014. URL http:// www.iiill....a.e.eee.eeooeeeeeee wttgpaapps///l......
srivastaval4a. html.
David Stutz, Nandhini Chandramoorthy, Matthias Hein, and Bernt Schiele. Random and adversarial bit error robustness: -nfrry--ffficient and secure dnn accelerators. CoRR, abs/2104.08323, 2021.
C. S. Thakur, R. Wang, T. J. Hamilton, R. Etienne-Cummings, J. Tapson, and A. van Schaik. An analogue neuromorphic co-processor that utilizes device mismatch for learning applications. IEEE Transactions Circuits and Systems I: Regular Papers, 65(4): 1110:1174--184,, 2018.
Naveen Verma, Hongyang Jia, Hossein Valavi, Yinqi Tang, Murat Ozatay, Lung- Lung-Yen Chen, Bonan Zhang, and Peter Deaville. In-memory computing: Advances and prospects. IEEE Solid-State Circuits Magazine, 11(3):43-55 2019. doi: 10. 19/MSSS02222119999988888
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adver- sarial robustness requires revisiting misclassified examples. In International Conference Learn- ing Representations, 2020. URL https: http///pppee..eeeeeeeeeeee net /tt/ htt.:/ooommmmmi1dogee 2id=rkdoog6E
P. Warden. Speech Commands: A Dataset for Limited- Speech Recognition. ArXiv e-prints, April 2018. URL httpsts:: http://wxvvv..../...... abs absp///as//110...//. 110///8144440 03209.
Dongxian Wu, Yisen Wang, and Shutao Xia. Revisiting loss landscape for adversarial robustness.
CoRR, abs/2004.05884, 2020. URL https: https///aaxxvvv......e
05884.
Lei Wu, Hongxia Liu, Jiabin Li, Shulong Wang, and Xing Wang. A Multi-level Memristor Based on Al-Doped HfO2 Thin Film. Nanoscale Research Letters, 14(1):177, May 2019, ISSN 1556-276X. doi: 1..11866116711-19-30015 URL s
Han Xiao, Kashif Rasul, and Roland Vollgraf. fashion-mmise novel image dataset for benchmarking machine learning algorithms, 2017.
Shimeng Yu and Pai- Yu Chen. Emerging memory technologies: Recent trends and prospects. IEEE Solid-Stare Circuits Magazine, 8(2):43-56, Spring 2016. ISSN 1943-0590. doi: 10.1109/MSSC. 2016.2546199.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing. Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. CoRR, abs//190..00000 990100857,, 2019. URL http::///// //axiiv org/ abs // 901 08573.
13

---

Published as a conference paper at ICLR 2022
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. CoRR, abs/2010.04925,, 2020. URL htt.t::://axx..cg/////o/ 2010. 04925.
14

---

Published as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
SPIKING RNN ARCHITECTURE
The dynamies of the spiking model (Bellec al., 2018) can be summarised by set of differential equations (Eq. 3).
B' = bd + 4,3b'
o' 1(V > B*) unless refractory leefr
b+ ppa + (1 pp)
dt
(3)
o'
I
B'dt dt
Resel
V++1= = = V + (1 pv) Wn + o' Wree) I
dt
d where Pa di/T and pV e- The variables B describe the spiking thresholds with adaptation. The vector o' denotes the population spike train at time t. The membrane potentials V have a time constant and the adaptive threshold time constant denoted by Tada- The speech signals and ECG traces fed into the network are represented as currents Ii- Since the derivative of the spiking function with respect its input mostly we use surrogate gradient that is explicitly defined as
OE
OE Oz
OE Oz d max(1
V
1.0)
(4)
OV
Oz oV:
B
where E is the error and d is the dampening factor. To get the final prediction of the network, average the population spike trains along the time-axis Zavg and compute
1 = softmax (Favg
Favg W out + bour)
/ arg max1,
CNN ARCHITECTURE
Our architecture comprises two convolutional blocks (2 x [4 x 4, 64 channels, MaxPool, ReLUJ) followed by three dense layers (N 1600, 256, 64, ReLU) and softmax layer. All weights and kernels are initialized using the Glorot normal initialisation (Glorot & Bengio, 2010). Using this architecture, we achieved all the kernel weights, as well as all the dense layer parameters.
accuracy of 93%. Attacked parameters for this network included
15

---

Published as a conference paper at ICLR 2022
DERIVATION OF JACOBIAN
Under the assumption that a
Smes 6
and p = co, we can rewrite the inner loop of Algorithm to
Na
begin
O+ + RR; R 1),
for t = 1 to Nagps do
6 < - a sign (er Loo (7, x))
end end
By rewriting O' in the form of O* = - + AO we get
Nag
θ' - e R
sign
From this the Jacobian can be easily calculated. Plugging in the defintion for get
Je- (G) + diag
sign (Gma te R2)
Nates
sign
Deps
16

---

Published as a conference paper at ICLR 2022
MISMATCH MODEL
To model the parameter noise introduced by component mismatch we used a Gaussian distribution where the mean is the nominal noise-free weight value and the standard deviation depends linearly on the weight value. This model realistically captures the behaviour of parameter mismatch on mixed- signal neuromorphic SNN inference processor (Moradi et al., 2018). Fig. SI shows the quantified parameter mismatch recorded directly from neuromorphic HW. over a range of nominal parameter values and for several neuronal and synaptic parameters. The measured mismatch parameter variation follows an approximately Gaussian distribution where the standard deviation depends linearly on the mean.
14
59
Valnoe peak (mk)
(11a)
d
s0l. dev. mV)
6-
4-
Malow
Pout
12
11 Goot: peak (mV)
volue (me ml/)
Figure S1: Quantification of mismatch on analog neuromorphic hardware. Parameter values for several weight parameters and membrane time constants were measured for range of nominal parameter values, using an oscilloscope directly connected to a mixed-signal neuromorphic SNN processor. (a-c) Various parameters of the chip follow Gaussian distribution with increasing width. (d) The mismatch standard deviation depends linearly on the nominal value of each parameter.
17

---

Published as a conference paper at ICLR 2022
WEIGHT LOSS-LANDSCAFE VISUALIZATION
We characterized the shape of the weight loss-landscape by plotting the categorical cross entropy loss for varying levels of noise added the weights of the trained network. In each trial, we picked a random vector ~ (0,, <|@l) and evaluated the categorical cross entropy of the whole test set given the weights θ + a v, where of the individual ID weight loss-landscapes is small.
[-2, 2] and Ç 0.2. As we show in Figure S2, the variance
Table SI quantifies the flatness of the illustrated weight loss-landscapes. As can be seen, our method combined with adding noise during the forward pass yields the flattest landscapes.
Table S1: Average slope of estimated ID weight loss-landscapes. Slopes were calculated as the mean absolute differences between sample points divided by the sampling distance.
F-MNIST CNN
ECG LSNN
Speech LSNN
Standard
0.3375
0.1605
0.2702
Beta
0.0480
0.0879
0.0633
Forward Noise
0.0332
0.0180
0.1009
Forward Noise Beta
0.0190
0.0187
0.0540
Dropout
0.3022
0.0952
0.0707
AWP
0.0841
0.1013
0.1389
F-NNIST CNN
ECG LSMN
Speech 15NN
1.4
Famaet
2.5
0.2
0.5
0.4
Croek
-201
-1.0
-1.5
11.0
112
1.0
1.5
2.
-20
-1.
-1.
-1.5
100
1.5
1/2
1.5
20
220
1.5
-10
-1.
0.11
0.5
2.01
15
Figure S2: Illustration of the test weight loss-landscape highlighting individual trials to mea- sure the loss landscapes. See the results text for more details.
18

---

Published as a conference paper at ICLR 2022
ATTACKING KL-DIVERGENCE LOSS DURING INFERENCE
While the adversary in AWP attacks the task loss directly, i.e. max (ece "1,"",)),) x), in our training algorithm attacks the KL divergence max KL()(O, (() X))). This implies that our parameter attack seeks simply to change the response of the network in any way, and is agnostic the task itself. In the main results we attack the cross-entropy task loss during inference, order to not give an undue ad advantage to our training approach. Here we show that our training approach also provides robustmess against parameter attacks on the Kl--divergence loss during inference.
the adversary
Fig. S3 shows the adversarial robustness of the methods for an adversary that attacks the KL- divergence rather than the cross-entropy loss. When the network is attacked by maximizing the KL-divergence between the normal and attacked network, our adversarially-traine more robust than AWP, SGD and forward-noise. Because the parameter attack used here during inference well as in the inner optimization loop of the training procedure is the same, this result is expected, and serves as a sanity check that our networks indeed learn to defend against the attack they were trained against.
networks are
F.MNIST CNN
ECG LSNN
Spanch LSNN
100
80
80
70
60
00
Avt.
Test Aoc
50
30-
Standed
Fonmeed Rotes
30-
0.005
0.01
0.05
0.1
0.005
0.5
0.035
D.01
0.05
0.1
0.2
Attack
size
sizw
Figure S3: Robustness weight attack targeting KL divergence during inference. When the network is attacked by maximizing the KI-diiergence between the normal and attacked network, our adversarially trained networks are more robust than standard SGD or AWP.
19

---

Published as a conference paper at ICLR 2022
EFFECT OF CONSTANT VERSUS RELATIVE PARAMETER NOISE
As described in the main text, parameter noise that has constant magnitude (for example, Gaussian noise with fixed standard deviation) is trivial to protect against by increasing weight magnitudes. We examined this effect by training MLPs with an adversary that employs Gaussian noise with fixed std. dev. 0.2. Figure S4 (right) illustrates the accuracy of two MLPs trained on F-MNIST over the course of training. Using an inverse weight decay term, the miighttmagiiudee of one network is forced increase to 2.0 over the course of training (black crosses, )*). The weight magnitude of the other network (red crosses) is limited during training to 0.2 (red crosses; θ). One can observe that as the weight magnitude the increasing magnitude network O* increases, also the robustness to Gaussian noise (e = 0.2) increases (blue), while the performance of the small magnitude network Q remains poor (cyan).
V roc
MLP MNIST
Contuct
Tel
2.00
Rebavee
Tel
1.75
Sott st..d
1.50
0.8
0.1
0.75
0.00
0.50
0.2
000
0.180
2.0
255
230
3.50
3.21
1105
310
12.5 37.00
Figure S4: Constant magnitude versus magnitude-relative parameter noise. (left, middle) When parameter noise of constant magnitude (blue) is introduced by the adversarial attack during training, the networks learn to increase the magnitude of the weights to trivially improve robustness to the constant-magniuun
attack. When the parameter attack is relative to each parameter magnitude, as in the main text (red), the weight magnitudes do increase. These networks were trained for range of Bmb, i.e. varying emphasis on robustness.
(right) One can also trivially increase the robustness to mned-magnitude noise by introducing an inverse weight decay term that causes the weights to increase in magnitude (black, cross marker) while retaining performance. This causes the MI.P trained on MNIST become increasingly robust to fixed- -ngditag parameter noise (blue dots; 6-test acc., θ*). The network where the weight magnitude was not increased over time (red, cross marker) did not improve in terms of robustness (cyan; e-test acc., θ), although both models perform similarly when there is parameter noise applied (red and black dots).
20

---

Published as a conference paper at ICLR 2022
EFFECT OF ADVERSARIAL REGULARIZATION DURING TRAINING
Training 2ob =0.0
Training Brob 0.1
0.8
0.8
0.6-
8.0
Valntation
0.0
0.44
0.4
0.2
20
30
40
50
10
20
50
Training =0.0
Training Srob 9011
Randon
0.8
0.6
Tex
Text
0.4
0.2
C
sizn
Figure S5: Our parameter attack effective at decreasing the performance a network dur- ing inference, and using our attack during training protects a network from later disruption. (a) network trained using standard SGD (i.e. frob = 0.0) on the F-MNIST task disnupted badly by the parameter noise adversary during inference (ç 0.1; black curve). (b) When the same network is trained with parameter attacks during training (Brab = 0.1), the network is protected from parameter attacks during inference (high accuracy of attacked network; black curve). (c) When trained with standard SGD (Bzob = 0.0), both random noise (red) and parameter attacks (black) disrupt network performance for increasing attack size Ç. (d) Under our training approach (Breb 0.1), networks significantly protected against random and adversarial weight perturbations during inference.
21

---

Published as a conference paper at ICLR 2022
EFFECT OF VARYING Bzob
We additionally quantify the trade-off between test loss and robustness of our algorithm by repeating the experiment in Figure at different values of 3mb. As shown Figure S6, increasing Sob flattens the weight loss-landscape, effectively increasing the robustness of the model. To further substantiate this claim, we repeated the experiment of Figure S5 with the same values of Brob. As Figure S8 shows, increasing Prob and therefore increasing the flatness of the landscape yields increased robustness to random, as well as, adversarial perturbations.
We note that, relative to the baseline, the test loss does not consistently increase with increasing values of Brob- We hypothesize that this is due to the increased generalization capability of our networks. To check whether this is indeed the case, we repeated the experiment from Fig. S6 using the loss computed on the training set. As Fig. shows, increasing values Sra lead flatter minima and higher loss the training set, which is the exact trade-off to be expected from the formulation of our loss function. However, the increased generalization capability that follows from a flatter loss-landscape seems to disrupt this trade-off. As result, when choosing Plub one should aim at choosing the highest value that still yields good performance on the validation set.
F-MNIST CNN
FCG
15NN
0.40
0
022
0.75
0.55-
0.15
0.70
E.00
0.20
0.23
Doox
-.5
-10
-0.5
0.0
305
1.5
-15
025
0.5
1.5
-35
-10
-35
60
0.5
1.0
1.5
Figure S6: Effect of Boob on the test loss landscape. Increasing Brob promotes robustness by flattening the test loss landscape. However, increasing Brob does not lead to systematic rise in loss on the test set.
3.5
Eerecty Lose
0.33
0.0
0.30
0.0
Figure S7: Effect of Sub on the training loss landscape. Weight loss landscape computed on the training (c.f. Fig. S6). Increasing Bobb leads flatter loss-landscapes (increased robustness) as before, but also a systematic increase in training loss (higher values for cross-entropy loss; note ordering of curves).
22

---

Published as a conference paper at ICLR 2022
We additionally trained the CNN the F-MNIST data for range of Boob. and measured the network robustness to attacks of varying magnitude Ç (Fig. S8). We compared the effectiveness of random attack versus adversarial attack during inference, evaluated the test set. We found that increasing Prob improved the robustness of the network to both random and adversarial attack during inference.
Training 9 20.00. -000
Training Robb
Advanaai
0.8
Ranker
0.3
0.6
0.2
0.2
0.0
0.7
0.01
0.05
0.1
0.2
0.3
Training Srob= =0.3
4
Training 9rob 0055
0.8
0.8
0.6
Text
0.4
0.2
0.0
Attack sine C
Attack sizn C
Figure S8: Increasing Boob during training improves robustness during inference. The CNN was trained using different values of Brob and evaluated on the test set after the weights were perturbed either randomly (red) or adversarially (black).
23

---

Published as a conference paper at ICLR 2022
VERIFIABLE ROBUSTNESS FOR LSNNS
Computations through the network are performed on intervals rather than on discrete values. By propagating the intervals through network over set, obtain output logits that are also expressed as intervals and can therefore determine whether a sample will always be classified correctly.
The final classification of the network is made using an arg max operator. For provability of network performance, we consider that a test sample æ is correctly classified when the lower bound of the logit interval for the correct class is the maximum lower bound across all logit intervals, and when the logit interval for the correct class is disjoint from the other logit intervals. When the logit interval for the correct class overlaps with another logit interval, that test sample is not considered to be provably correctly classified. Note that interval domaín analysis provides a relatively loose bound (Gehr et al., 2018 8), with the implication that the results here probably underestimate the true performance of our method.
Sperch LSNN
ECG LSNN
Noise Standard
Verifee best oec.
0.6
Verffee bast
0.4
0.2
0.0
1s:5
56:05
0.0001
0.0005
0.001
0.0
1c:05
Se:05 0.0001
0.0005
0.001
Figure $9: Networks trained with our method are provably more robust than those trained with standard gradient descent or forward noise alone. We used interval bound propagation to determine the proportion of test samples that are verifiably correctly classified, under increasing weight perturbations For both the Speech and ECG tasks, computed on the trained LSNNs, our method was provably more robust for Ç < x 10-4
24

---

Published as a conference paper at ICLR 2022
WIDE-MARGIN NETWORK ACTIVATIONS
Murray et al. show that adding random forward noise to the weights of a network during the forward
Bou pass implicitly adds regularizer of the form ( (ee to the network weights (Murray & Edwards, 1994). When sigmoid activation functions are used, this regularizer favors high or low activations. When implementing interval bound propagation for LSNNs, intervals over spiking activity must be computed by passing intervals through the spiking threshold function. As a result, intervals for spiking activity become either [0. for neurons that never emit a spike regardless of weight attack; [1, 1] for neurons that always emit spike; and [0, 1] for neurons for which activity becomes uncertain the presence of weight attack. By definition, a robust network should promote bounds [0, 0] and []. where the activity of the network is unchanged by weight attack. Robust network configurations should therefore avoid states where the membrane potentials of neurons are close to the firing threshold.
To see whether this was also the case for our LSNNs, we investigated the distribution the membrane potentials on batch of test examples for network that was trained with and without noise during the forward pass. We found that robust networks cxhibited a broader distribution of membrane potentials, with comparatively less distribution mass close to the firing threshold (Fig. S10). This indicates that neurons in the robust network spend more time in a "safe" regime where small change in the weights cannot trigger unwanted spike, remove desired spike.
Forward Noise I Beta Standard
Forward Noise
Beta
Standard
2.5
Normalized bin count
0.5
2.0
Normalized bin
0.4
1.5
0.3
0.2-
0.0
10
Membrane potential
Membrane potential
Figure S10: Membrane potentials are distributed away from the firing threshold for networks trained with forward noise. We measured the distribution of membrane potentials in L.NNNs trained using standard gradient descent ("Standard"),, and the presence of weight noise injected in the forward pass ("Forward Noise Beta"). As predicted, networks trained with forward noise have membrane potentials distributed away from the firing threshold. This implies that weight perturbations less likely to inject delete a spike erroneously, improving the robustness of the network.
25

---

Published as a conference paper at ICLR 2022
EFFECT OF VARYING EPGΛ IN AWP
In addition to attacking the network parameters, AWP (Wu et al., 2020) also attacks the input using PGA. Since the relation between robustness weight- and input-space perturbations is still unclear, we performed additional sweeps over the attack size the input space. Figure S11 demonstrates that attacking the input during training generally does not improve the robustness, with the exception for the network trained on the speech dataset. We also note that attacking the imputs improved robustness for larger mismatch values, but generally degraded performance for the small values.
Speech
FNNIST
1.0
0.30
0.50
100
85-
10.00
0.90
-LU00
-0.00
10.00
-0010
11.00
11.00
10.100
Figure S11: Attacking the input using conventional PGA generally has a limited effect on robustness to weight-space perturbations. We swept various values of Epas the parameter that determines the maximum perturbation in 1 for the AWP algorithm.
26

---

Published as a conference paper at ICLR 2022
TRAINING OF CNN USED FOR PCM- BASED CIM SIMULATION
The CNN that was used for this series of experiments is Resnet32 (He et al., 2015) trained on the 10 (Krizhevsky. 2009) dataset. The CNN generally trained for 300 epochs using batch size of 256. We used SGD with an initial learning rate of 0.001 that was decreased by multiplicative factor of 0.2 after epochs 60, 120 and 160. Additionally, Nesterov momentum (Nesterov, 1983) was used with value of 0.9 and weight decay with value of 5e-4.
Oed,
Reaas
383
1.0
291
259
3.57
0,1
633
Epochs
Ove
Fal Pavan |M(_]
82-
22.
Figure S12: The left panel illustrates the validation accuracy over the course of finetuning pretrained model for additional 240 epochs. One should note that convergence usually achieved much quicker (roughly after 150 additional epochs). The right panel illustrates the performance the test set of each model trained with noise injection of magnitude Vrriin (x-axis). Each row depicts different noise model. It should be noted that the weights in each filter were clipped two standard deviations during training to avoid outliers causing excessive amounts of noise following the model that relies on the maximum weight value.
VARYING THE NUMBER OF ATTACK STEPS FOR THE PCM- CIM SIMULATION
Tattack 0.10
93.0
Test acc. (%)
N steps
91.5
90.5
Baseline
Brob 0.0
90.0
9.026
0.037
0.096
0,075
0.110
Atrain = (
Figure S13: One can greatly reduce the number of attack steps used during training. Our method still produces strong results for very few number of attack steps (blue) when compared to the baseline model (red, trained with Gaussian noise (Mmmin))
27

---

Published as a conference paper at ICLR 2022
PERFORMANCE OF VARYING HYPERPARAMETEN FOR THE BEM-BASED CIM SIMULATION
In this experiment we show that the choice of hyperparameters is generally not very important to outperform the baseline that was trained with noise injection. However, in order surpass the FP baseline, i.e. the model that was trained without noise injection and evaluated on a standard PC, one has to tune the hyperparameters in order to obtain a combination that yields the highest performance. Figure S14 illustrates this sweep. Each row represents a different value of Taain that was used for training the baseline model (red). Each column represents a different attack size and the different hues of blue correspond to varying values of Pobb
Cross 0.03
Zanoes - 0.03
Cono 0. 2.05
Crooe - 0:10
Saselne
O.O.
Yor 64)
Tar (5)
Tae (6)
10
Carae 0.01
0.03
Cra 0.05
Cattee 0.10
Not aec 2961
Tolk 2661
12
Cauaaa - 0.02
Cas - 0.03
Cama 0.05
Gitaas - 0.20
1.01
F Saselin
Zap -0 O.O. 0.056
65
Tax [5)
Tor10 Crove 0.10
Geoe 0.03
Zyow 0.05
0.4
0.3
(4)
Te
Gataee = 0.01
Cattece = 0.03
Cod 0.05
Cattr = =0.10
aCC 1941
Teak
1.00
F Baselne
0
0.130
Figure S14: The choice of hyperparameters is not critical in order to beat the baseline model. Our method is resilient variations in hyperparameters. However, to obtain configurations where even the FP baseline is surpassed, one has to fine-tune the method.
28

---

Published as a conference paper at ICLR 2022
PCM NOISE MODEL
Analog CiM comes in various flavors, depending on the memory technology used. In this paper, assume the of PCM devices, which have been heavily studied in the context of analog CiM accelerators (Joshi et al., 2020; Nandakumar et al., 2019; Boybat et al., 2018). PCM-based, or, more generally, Non-Volatile Memory (NVM)-based architectures, essentially perform Matrix- Vector- Multiplications (MVMs) using Kirchhoff's current law. The weights of the matrix are organized as differential pairs in order account for positive and negative weights. When storing neural network, each weight matrix is programmed into the NVM devices by applying short electrical pulses (Nandakumar et al., 2020b). Because of various noise sources, this process is often imprecise and exhibits noise on the weights, termed "programming noise". Additionally, PCM devices suffer from 1/f and telegraph noise, adding even more noise during inference ("read noise"). At last, PCM devices also drift due the underlying physical properties (Le Gallo et al., 2018a). Although the effect drift can mostly be alleviated by scaling the output of the MVM (a method called Global Drift Compensation (GDC)), the non-uniform drift of the devices still leads to performance degradation over time. In the simulator that we used, we model these three main sources of noise, sealog-to-digiin and digital-to-analog converters, GDC and splitting of the MVM account for smaller tile sizes (lypically each crossbar is 256 x 256).
Initially, the clipped weights are mapped to target conductances in a differential manner, i.e. the weight matrix is split into two conductance matrices that both represent the positive and negative weights as conductances. The target conductances typically range from zero to Gmax, where Gmax is assumed to be 25pS. After mapping the weights to the target conductances GT. the programming noise is simulated (these statistical models assume the conductances to normalized): Gp = +NN a)) where op max (maxx_1177311;; + 1.9650GT + 0.2635, 0.0).
After the conductances have been programmed they drift over time, with the conductance of a device typically following GD (r(ttte) where is the drift coefficient, is the time at inference, and te is the time the conductances were programmed. Additionally, the drift coefficient is modelled follow Gaussian distribution. This makes typically hard to correct for drift and is the main reason why drift is a problem in PCM based CiM devices.
(nG(t))) where GxG()) (og()Q//og((t + t,,tt) with Q min G 0.2) and t, 250ns and t is the time at inference. For the experiments in this paper, we simulated the performance of the networks deployed on CiM hardware for up to one year.
Finally, the read noise is modelled using Gaussian: GR
29


Reviews:
[{"review_id": "ZAYUTqJnmkk", "reviewer": "Reviewer_vgst", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, this paper is well-written and easy to follow. \nThe authors provided clear explanations on the prior works, and the proposed method is clearly described.\nThe proposed method that uses the robustness loss seems like a simple idea, but the experimental results showed clear improvement compared to existing methods with several different benchmarks.\n\nHowever, the datasets used for experiments seem too small in terms of the network size and the number of parameters, so it is doubtful that the proposed method can be a scalable solution.\nSince there seems no reason for the proposed method to be limited to the spiking neural networks, it would be better if the authors can showed the improvement from the proposed method with larger-scale datasets commonly used for non-spiking neural networks.\n", "summary_of_the_paper": "In this paper, the authors proposed an adversarial training method that minimizes the robustness loss so that the network parameter can be trained to be robust to the parameter perturbation. \nThe motivation of the work comes from the parameter mismatch that exists when using analog devices for computation. \nThe robustness loss calculates the difference between the network output before the weight perturbation and after the weight perturbation, and the direction of perturbation is decided in adversarial manner with Projected Gradient Ascent (PGA). \nExperimental results with several benchmarks showed that minimizing such loss resulted in flattened weight-loss landscape, and the proposed method outperformed existing methods in terms of the robustness to parameter mismatch.", "summary_of_the_review": "This paper proposed a simple yet effective solution for parameter mismatch with gradient-based adversarial training method.\nHowever, since I am not familiar with the prior works on the parameter mismatch, I am not very confident about my assessment.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XWGGEqQmU4-", "reviewer": "Reviewer_9GjF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a novel application of a loss formulation (task loss and robustness loss) and projected gradient descent training to build resilience to parameter noise into neural networks.  The paper is presented well and provides sufficient evidence on the performance of the method.  While the grammar and layout is generally good, Figure caption text size does need to be increased in some cases (e.g. Fig 1 and 2 legends).  It appears details are present to reproduce results from a casual glance at supplementary code in conjunction with details provided in the paper.   Repeated and averaged results in several figures strengthens confidence of reported figures.cce\n\nI am unsure of the implications of assuming the Jacobian of the parameters w.r.t. the attack parameters is diagonal but I believe this would limit the device applications to those where noise is uncorrelated between parameters.  Correlated noise may also have a small effect on final network performance even when trained under a diagonal Jacobian assumption. \n\nL_{cce} is referred to twice in the paper but is undefined -- is this the same as L_{nat} task loss?\n\n", "summary_of_the_paper": "This paper introduces a special loss formulation and training algorithm leveraging projected gradient ascent to build robustness to parameter noise into training for neural networks (both spiking and convolutional neural networks are tested).  The authors show that the proposed method flattens the loss landscape for small parameter perturbations with a tradeoff of increased overall cross entropy (task) loss.  It is shown that this alternative loss, when utilized during training, provides stronger resiliency to adversarially injected parameter noise than competing methods such as dropout or adversarial weight perturbation.  Tests relating to device mismatch due to manufacturing process deficiencies are also performed and the authors show their training method would be useful for building networks that don't require per-device tuning in such cases (neuromorphic/analog computing).", "summary_of_the_review": "Techniques in this paper appear to be novel modifications to neural network training that combine projected gradient ascent with a loss specifically designed to promote robustness to small perturbations in network weights (or any similarly learned parameter).  The authors provide evidence to the strengths of this technique when compared with other mitigation measures for noise in the parameter space (either adversarial or as a byproduct of physical properties).  This research provides an interesting path toward making imperfect analog computing devices practical for large scale use and deployment.  Though I am not an expert in the specifics of this adversarial and noise related network training methods, the paper layout and methodology appear sound.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7JDwF9f0Arx", "reviewer": "Reviewer_BXMW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear to understand in most parts. Experiments are well presented, including successful demonstrations of better flattening of the weight-loss landscape with respect to the competitive methods. My main concern relates to the ambiguity on explicitly putting the novelty of this paper as the proposed problem is recently addressed by similar studies. In fact, adversarial parameter perturbations for regularization is an approach that was demonstrated in the context of model generalization with similar results (see below). The novelty of this paper however, in my opinion, comes from the fact that the authors applied this idea for the first time to recurrent spiking neural networks (SNNs).\n\nGiven the current narrative, title, and manuscript organization, the methodological contribution of this work is not directly put to the domain of compact SNN architectures and utility in the context of neuromorphic hardware, however sort of generally implied. As CNN experiments are also demonstrated to show that the proposed idea extends to conventional networks, one obvious concern would be on the scalability of this approach on the parameter space to larger networks as training would get costly to estimate Eq (2). So far the conventional CNN that the authors used only has 500k trainable parameters, and is not even demonstrated to operate on RGB images such as CIFAR-10. Did the authors investigate this with respect to the competitive methods?\n\nFrom various aspects the work has similarities to the adversarial model perturbation (AMP) framework that was recently proposed [Zheng et al., Regularizing Neural Networks via Adversarial Model Perturbation, CVPR 2021]. A very similar goal was reached to obtain robust parameters to variations, including convergence to flatter local minima regions for empirical risk minimization. While I agree on the choice of all the other competitive methods that the authors compared their approach with in the paper, one needs to compare/discuss AMP with the proposed method (which uses a novel TRADES-like loss objective).\n\nIt is ambiguous what does the following statement mean on page 5: “We further adapted AWP [Wu et al, 2020] so that the input data is never attacked.” Does this mean that there were no adversarial input robustness evaluations, or that AWP [Wu et al, 2020] was implemented by the authors in a way that did not harness adversarially crafted input examples during training with adversarial weight perturbation? If it is the latter, than the AWP approach may possibly be under-performing as well. Did the authors investigate that?\n\nOne recent concept that the authors should also discuss the similarities in their paper: In principle the idea is similar to training quantized neural networks with adversarial bit errors [Stutz et al., “Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators”, arXiv preprint arXiv:2104.08323 (2021)], hence perturbing the weights during training for generalization to random weight perturbations at test time. Present study approaches this problem from the perspective of a network which operates with floating-point weights.\n\nAuthors use the abbreviation LSNN in the figures and tables, however this was never described in the paper. Instead the text used the abbreviation SRNNs. It is later discovered by the reader that LSNN abbreviation corresponds to the one by [Bellec et al. 2018], when one dives into the supplementary materials.\n", "summary_of_the_paper": "Authors investigate the problem of network robustness in the presence of parameter variations. To that end, authors propose an adversarial parameter perturbation based robust training method. Proposed training approach iteratively performs adversarial attacks on the parameter space during training to regularize the model by penalizing parameter vulnerability. Experiments are performed on F-MNIST, ECG data and a speech command detection dataset, with both conventional CNNs and recurrent SNNs, where weight-space perturbations during training showed generalization to weight perturbations at inference time.", "summary_of_the_review": "The paper has solid contributions by presenting an adversarial parameter perturbation based model regularization scheme that is successfully applied to recurrent SNNs. However there are important methodological comparisons/discussions missing with respect to similar methods that were recently proposed. My major concerns are listed in the above reviews, which concludes that I find this work currently to be below borderline.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an adversarial training method to increase network robustness to parameter variations. The proposed approach performs adversarial attacks on network parameters during training. They demonstrate that their method flattens the loss landscape of the network. Experiments were performed on F-MNIST, ECG data, and speech command detection datasets using a conventional CNN and a recurrent spiking neural networks (SNNs).

The manuscript is well-written and the method is interesting.

One reviewer was somewhat concerned about the novelty of the work, but acknowledged that the application to recurrent SNNs was new.
The main initial criticism was the question of scalability of the method, as it was tested only on networks with a relatively small number of parameters.

In the revision, the authors addressed these issues. Their method was compared to related approaches, and experiments on CIFAR-10 with a ResNet32 were performed.
The reviewers acknowledged these larger-size experiments, but were not fully convinced as much larger models are typically used today.

Nevertheless, the reviewers acknowledged the improvements and ratings were increased, so all are voting for acceptance.
Decision: Accept (Poster)