SAFE-DNN: A Deep Neural Network with Spike Assisted Feature Extraction for Noise Robust Inference

1 INTRODUCTION . There is a growing interest in deploying DNNs in autonomous systems interacting with physical world such as autonomous vehicles and robotics . It is important that an autonomous systems make reliable classifications even with noisy data . However , in a deep convolutional neural networks ( CNN ) trained using stochastic gradient descent ( SGD ) , pixel level perturbation can cause kernels to generate incorrect feature maps . Such errors can propagate through network and degrade the classification accuracy ( Nazaré et al . ( 2017 ) ; Luo & Yang ( 2014 ) ) . Approaches for improving robustness of a DNN to pixel perturbation can be broadly divided into two complementary categories . First , many research efforts have developed image de-noising ( or filtering ) networks that can pre-process an image before classification , but at the expense of additional latency in the processing pipeline ( Ronneberger et al . ( 2015 ) ; Na et al . ( 2019 ) ; Xie et al . ( 2012 ) ; Zhussip & Chun ( 2018 ) ; Soltanayev & Chun ( 2018 ) ; Zhang et al . ( 2017 ) ) . De-noising is an effective approach to improve accuracy under noise but can degrade accuracy for clean images ( Na et al . ( 2019 ) ) . Moreover , de-noising networks trained on a certain noise type do not perform well if the a different noise structure is experienced during inference ( Zhussip & Chun ( 2018 ) ) . Advanced de-noising networks are capable of generalizing to multiple levels of a type of noise and effective for different noise types ( Zhussip & Chun ( 2018 ) ; Soltanayev & Chun ( 2018 ) ; Zhang et al . ( 2017 ) ) . But high complexity of these network makes them less suitable for real-time applications and lightweight platforms with limited computational and memory resources . An orthogonal approach is to develop a classification network that is inherently robust to input perturbations . Example approaches include training with noisy data , introducing noise to network parameters during training , and using pixel level regularization ( Milyaev & Laptev ( 2017 ) ; Nazaré et al . ( 2017 ) ; Luo & Yang ( 2014 ) ; Na et al . ( 2018 ) ; Long et al . ( 2019 ) ) . These approaches do not change the processing pipeline or increase computational and memory demand during inference . However , training-based approaches to design robust DNNs also degrade classification accuracy for clean images , and more importantly , are effective only when noise structure ( and magnitude ) during training and inference closely match . Therefore , a new class of DNN architecture is necessary for autonomous system that is inherently resilient to input perturbations of different type and magnitude without requiring training on noisy data , as well as computationally efficient . Towards this end , this paper proposes a new class of DNN architecture that integrates features extracted via unsupervised neuro-inspired learning and supervised training . The neuro-inspired learning , in particular , spiking neural network ( SNN ) with spike-timing-dependent plasticity ( STDP ) is an alternative and unsupervised approach to learning features in input data ( Hebb et al . ( 1950 ) ; Bi & Poo ( 2001 ) ; Diehl & Cook ( 2015 ) ; She et al . ( 2019a ) ; Querlioz et al . ( 2013 ) ; Srinivasan et al . ( 2016 ) ) . STDP based SNN optimizes network parameters according to causality information with no labels ( Moreno-Bote & Drugowitsch ( 2015 ) ; Lansdell & Kording ( 2019 ) ) . However , the classification accuracy of a STDP-learned SNN for complex datasets is much lower than a that of a DNN . The fundamental premise of this paper is that , augmenting the feature space of a supervised ( trained ) DNN with features extracted by an SNN via STDP-based learning increases robustness of the DNN to input perturbations . We argue that stochastic gradient descent ( SGD ) based back-propagation in a DNN enables global learning between low-level pixel-to-pixel interactions and high-level detection and classification . On the other hand , STDP performs unsupervised local learning and extracts low-level features under spatial correlation . By integrating features from global ( supervised training ) and local ( STDP ) learning , the hybrid network “ learns to ignore ” locally uncorrelated perturbations ( noise ) in pixels while extracting the correct feature representation from the overall image . Consequently , hybridization of SGD and STDP enables robust image classification under noisy input while preserving the accuracy of the baseline DNN for clean images . We present a hybrid network architecture , referred to as Spike Assisted Feature Extraction based Deep Neural Network ( SAFE-DNN ) , to establish the preceding premise . We develop an integrated learning/training methodology to couple the features extracted via neuro-inspired learning and supervised training . In particular , this paper makes the following contributions : • We present a SAFE-DNN architecture ( Figure 1 ) that couples STDP-based robust learning of local features with SGD based supervised training . This is achieved by integrating a spiking convolutional module within a DNN pipeline . • We present a novel frequency-dependent stochastic STDP learning rule for the spiking convolutional demonstrating local competitive learning of low level features . The proposed learning method makes the feature extracted by the spiking convolutional module robust to local perturbations in the input image . • We develop a methodology to transform the STDP-based spiking convolution to an equiv- alent CNN . This is achieved by using a novel special neuron activation unit ( SAU ) , a non-spiking activation function , that facilitates integration of the SNN extracted features within the DNN thereby creating a single fully-trainable deep network . The supervised ( SGD-based ) training is performed in that deep network after freezing the STDP-learnt weights in the spiking CNN module . We present implementations of SAFE-DNN based on different deep networks including MobileNet , ResNet and DenseNet ( Sandler et al . ( 2018 ) , He et al . ( 2015 ) , Huang et al . ( 2016 ) ) to show the versatility of our network architecture . Experiment is conducted for CIFRA10 and ImageNet subset considering different types of noise , including Gaussian , Wald , Poisson , Salt & Paper , and adversarial noise demonstrating robust classification under input noise . Unlike training-based approaches , SAFE-DNN shows improved accuracy for a wide range of noise structure and magnitude without requiring any prior knowledge of the perturbation during training and inference and does not degrade the accuracy for clean images ( even shows marginal improvement in many cases ) . SAFE-DNN complements , and can be integrated with , de-noising networks for input pre-processing . However , unlike de-noising networks , the SAFE-DNN has negligible computation and memory overhead , and does not introduce new stages in the processing pipeline . Hence , SAFE-DNN is an attractive architecture for resource-constrained autonomous platforms with real-time processing . We note that , SAFE-DNN differs from deep SNNs that convert a pre-trained DNN to SNN ( Sengupta et al . ( 2019 ) , Hu et al . ( 2018 ) ) . Such networks function as a spiking network during inference to reduce energy ; however , the learning is still based on supervision and back-propagation . In contrast , SAFE-DNN hybridizes STDP and SGD during learning but creates a single hybrid network operating as a DNN during inference . 2 BACKGROUND ON SNN . Spiking neural network uses biologically plausible neuron and synapse models that can exploit temporal relationship between spiking events ( Moreno-Bote & Drugowitsch ( 2015 ) ; Lansdell & Kording ( 2019 ) ) . There are different models that are developed to capture the firing pattern of real biological neurons . We choose to use Leaky Integrate Fire ( LIF ) model in this work described by : dv/dt = a+ bv + cI ; and v = vreset , if v > vthreshold ( 1 ) where , a , b and c are parameters that control neuron dynamics , and I is the sum of current signal from all synapses that connects to the neuron . In SNN , two neurons connected by one synapse are referred to as pre-synaptic neuron and postsynaptic neuron . Conductance of the synapse determines how strongly two neurons are connected and learning is achieved through modulating the conductance following an algorithm named spike-timingdependent-plasticity ( STDP ) ( Hebb et al . ( 1950 ) ; Bliss & Gardner-Medwin ( 1973 ) ; Gerstner et al . ( 1993 ) ) . With two operations of STDP : long-term potentiation ( LTP ) and long-term depression ( LTD ) , SNN is able to extract the causality between spikes of two connected neurons from their temporal relationship . More specifically , LTP is triggered when post-synaptic neuron spikes closely after a pre-synaptic neuron spike , indicating a causal relationship between the two events . On the other hand , when a post-synaptic neuron spikes before pre-synaptic spike arrives or without receiving a pre-synaptic spike at all , the synapse goes through LTD. For this model the magnitude of modulation is determined by ( Querlioz et al . ( 2013 ) ) : ∆Gp = αpe −βp ( G−Gmin ) / ( Gmax−Gmin ) and ∆Gd = αde−βd ( Gmax−G ) / ( Gmax−Gmin ) ( 2 ) In the functions above , ∆Gp is the magnitude of LTP actions , and ∆Gd is the magnitude of LTD actions . αp , αd , βp , βd , Gmax and Gmin are parameters that are tuned based on specific network configurations . 3 MOTIVATION BEHIND SAFE-DNN . The gradient descent based weight update process in a DNN computes the new weight as W ′ = W − η∇L , where the gradient of loss function L is taken with respect to weight : ∇wL = 〈 ∂L∂Wi , ... , ∂L ∂Wk 〉 . Consider cross entropy loss as an example for L , weight optimization of element i is described by : W ′i = Wi − η − 1N ∂ { N∑ n=1 [ ynlog ( ŷn ) ] } ∂Wi ( 3 ) Here η is the rate for gradient descent ; N is the number of classes ; yn is a binary indicator for the correct label of current observation and ŷn is the predicated probability of class n by the network . For equation ( 3 ) , gradient is derived based on the output prediction probabilities ŷ and ground truth . Such information is available only at the output layer . To generate the gradient , the output prediction ( or error ) has to be back-propagated from the output layer to the target layer using chain rule . As ŷ = g ( W , X ) with g being the logistic function and X the input image , the prediction probabilities are the outcome of the entire network structure . Consider the low level feature extraction layers in a deep network . Equation ( 3 ) suggests that gradient of the loss with respect to a parameter is affected by all pixels in the entire input image . In other words , the back-propagation makes weight update sensitive to non-neighboring pixels . This facilitates global learning and improve accuracy of higher level feature detection and classification . However , the global learning also makes it difficult to strongly impose local constraints during training . Hence , the network does not learn to ignore local perturbations during low-level feature extraction as it is trained to consider global impact of each pixel for accurate classifications . This means that during inference , although a noisy pixel is an outlier from the other pixels in the neighbourhood , a DNN must consider that noise as signal while extracting low-level features . The resulting perturbation from pixel level noise propagates through the network , and degrades the classification accuracy . The preceding discussion suggests that , to improve robustness to stochastic input perturbation ( noise ) , the low level feature extractors must learn to consider local spatial correlation . The local learning will allow network to more effectively “ ignore ” noisy pixels while computing the low-level feature maps and inhibit propagation of input noise into the DNN pipeline . The motivation behind SAFE-DNN comes from the observation that STDP in SNN enables local learning of features . Compared to conventional DNN , SNN conductance is not updated through gradient descent that depends on back propagation of global loss . Consider a network with one spiking neuron and n connected input synapses , a spiking event of the neuron at time tspike and timing of closest spikes from all input spike trains Tinput , the modulated conductance is given by : G′i = Gi + sign ( ∆ti ) · r ( Gi ) · p ( ∆ti , fi ) ( 4 ) Here ∆ti = tspike − T iinput is spike timing difference , r is the magnitude function ( Equation 2 ) and p is the modulation probability function ( Equation 5 ) . The value of tspike is a result of the neuron ’ s response to the collective sum of input spike trains in one kernel . Hence , the modulation of weight of each synapse in a SNN depends only on other input signals within the same ( local ) receptive field . Moreover , as the correlation between the spike patterns of neighboring pre-synaptic neurons controls and causes the post-synaptic spike , STDP helps the network learn the expected spatial correlation between pixels in a local region . During inference , if the input image contains noise , intensity of individual pixel can be contaminated but within a close spatial proximity the correlation is better preserved . As the SNN has learned to respond to local correlation , rather than individual pixels , the neuron ’ s activity experiences less interference from local input perturbation . In other words , the SNN “ learns to ignore ” local perturbations and hence , the extracted features are robust to noise . 4 SAFE-DNN ARCHITECTURE AND LEARNING PROCESS 4.1 NETWORK ARCHITECTURE Table 1 : Network Complexity Model Params ( M ) MACs ( G ) Baseline MobileNetV2 3.50 0.33 Baseline ResNet101 44.55 7.87 Baseline DenseNet121 7.98 2.90 SAFE-MobileNetV2 3.57 0.36 SAFE-ResNet101 44.62 7.90 SAFE-DenseNet121 8.04 2.94 Fig . 1 ( a ) shows an illustrative implementation of SAFE-DNN . The network contains spiking layers placed contiguously to form the spiking convolution module , along with conventional CNN layers . The spiking convolution module is placed at the front to enable robust extraction of local and low-level features . Further , to ensure that the low-level feature extraction also considers global learning , which is the hallmark of gradient backpropagation as discussed in section 3 , we place several conventional CNN layers of smaller size in parallel with the spiking convolution module . This is called the auxiliary CNN module . The output feature map of the two parallel modules is maintained to have the same height and width , and concatenated along the depth to be used as input tensor to the remaining CNN layers , referred to as the main CNN module . Main CNN module is responsible for higher level feature detection as well as the final classification . The main CNN module can be designed based on existing deep learning models . The concatenation of features from auxilary CNN and spikining convolutional module helps integrate global and local learning . Fig . 2 shows the process of implementing SAFE-MobileNetV2 based on the original MobileNetV2 . The first convolution layer and the following one block from the original network architecture are dropped and the remaining layers are used as the mian CNN module of SAFE-MobileNetV2 . We show that SAFE-DNN is a versatile network by testing three configurations in this work , which have the main CNN module based on MobileNetV2 , ResNet101 and DenseNet121 , respectively . The storage and computational complexity of the networks are shown in Table 1 . It can be observed that SAFE-DNN implementations do not introduce a significant overhead to the baseline networks . In the dynamical system of SNN , neurons transmit information in the form of spikes , which are temporally discrete events that spread across multiple simulation time steps . This requires input signal intensity to be converted to spike trains , and a number of time steps for neurons to respond to input stimulus . Such mechanism is different from that of the conventional DNN , which takes only one time step for data to propagate through the network . Due to this reason the native SNN model can not be used in spiking convolution module of SAFE-DNN . Two potential solutions to this problem are , running multiple time steps for every input , or , adapting the spiking convolution module to single-time-step response system . Since the first slows down both training and inference by at least one order of magnitude , we choose the latter . Training Process . We separate STDP-based learning and DNN training into two stages . In the first stage , the spiking convolution module operates in isolation and learns all images in the training set without supervision . The learning algorithm follows our novel frequency dependent STDP method described next in section 4.2 . In the second stage , network parameters are first migrated to the spiking convolution module of SAFE-DNN . The network building blocks of the spiking convolutional module go through a conversion process shown in Fig . 1 ( b ) . The input signal to spike train conversion process is dropped , and conductance matrix is re-scaled to be used in the new building block . Batch normalization is inserted after the convolution layer . In order to preserve the non-linear property of spiking neurons , a special activation unit ( SAU ) is designed to replace the basic spiking neuron model . Details about SAU is discussed later in section 4.3 . Once the migration is completed , the entire SAFE-DNN is then trained fully using statistical method , while weights in the spiking convolution module are kept fixed to preserve features learned by SNN . Network inference is performed using the network architecture created during the second stage of training i.e . instead of the baseline LIF , the SAU is used for modeling neurons .

Summary: The paper develops a method to augment deep neural networks with Spike-Time-Dependent-Plasticity (STDP) aiming at improving noise robustness of the network learned features. In the hybrid network, learned feature is the concatenation of traditionally supervised-ly learned features and those from an auxiliary module trained locally and unsupervised-ly by STDP. The new network demonstrates improved noise robustness via improved classification accuracy on Cifar10 and ImageNet subset when input data have noise, on different network architectures.