Localized random projections challenge benchmarks for bio-plausible deep learning

1 INTRODUCTION . While learning a new task , synapses deep in the brain undergo task-relevant changes ( HayashiTakagi et al. , 2015 ) . These synapses are often many neurons downstream of sensors and many neurons upstream of actuators . Since the rules that govern such changes deep in the brain are poorly understood , it is appealing to draw inspiration from deep artificial neural networks ( DNNs ) ( LeCun et al. , 2015 ) . DNNs and the cerebral cortex process information in multiple layers of many neurons ( Yamins & DiCarlo , 2016 ; Kriegeskorte , 2015 ) and in both , the artificial and the biological neural networks , learning depends on changes of synaptic strengths ( Hebbian theory , Hebb ( 1949 ) ) . However , learning rules in the brain are most likely different from the backpropagation algorithm ( Crick , 1989 ; Marblestone et al. , 2016 ; Rumelhart et al. , 1986 ) . Furthermore , biological neurons communicate by sending discrete spikes as opposed to real-valued numbers used in DNNs . Differences like these suggest that there exist other , possibly equally powerful , algorithms that are capable to solve the same tasks by using different , more biologically plausible mechanisms . Thus , an important question in computational neuroscience is how to explain the fascinating learning capabilities of the brain with bio-plausible network architectures and learning rules . On the other hand , from a pure machine learning perspective there is increasing interest in neuron-like architectures with local learning rules , mainly motivated by the current advances in neuromorphic hardware ( Nawrocki et al. , 2016 ) . Image recognition is a popular task to test the proposed models . Because of its relative simplicity and popularity , the MNIST dataset ( 28×28-pixel grey level images of handwritten digits , LeCun ( 1998 ) ) is often used for benchmarking . Typical performances of existing models are around 97- 99 % classification accuracy on the MNIST test set ( see section 2 and Table 8 ) . This value lies in the region of the benchmarks for a large class of classical DNNs trained with backpropagation but without data-augmentation or convolutional layers ( see table in LeCun ( 1998 ) ) . Thus , accuracies around this value are assumed to be an empirical signature of backpropagation-like deep learning ( Lillicrap et al. , 2016 ; Sacramento et al. , 2017 ) . It is noteworthy , however , that several of the most promising approaches that perform well on MNIST have been found to fail on harder tasks ( Bartunov et al. , 2018 ) . An alternative to supervised training of all layers with backpropagation are fixed random weights , as proposed by general approximation theory ( Barron , 1993 ) and the extreme learning field ( Huang et al. , 2006 ) , or unsupervised training in the first layers , combined with supervised training of a readout layer . Unsupervised methods are appealing since they can be implemented with local learning rules , see e.g . “ Oja ’ s rule ” ( Oja , 1982 ; Sanger , 1989 ) for principal component analysis or algorithms in Olshausen & Field ( 1997 ) ; Rozell et al . ( 2008 ) ; Liu & Jia ( 2012 ) ; Brito & Gerstner ( 2016 ) for sparse coding . A single readout layer can also be implemented with a local delta-rule ( also called “ perceptron rule ” ) , which may be implemented by pyramidal spiking neurons with dendritic prediction of somatic spiking ( Urbanczik & Senn , 2014 ) . Since it is pointless to simply stack multiple fully connected layers trained with principal component analysis or sparse coding ( Olshausen & Field , 1997 ) we investigate here networks with a single hidden layer . The main objective of this study was to see how far we can go with a single hidden layer and local learning rules in networks of spiking neurons . To support the design choices of the spiking model , we compared the classification performance of different rate networks : networks trained with backpropagation , networks where the hidden layer is trained with unsupervised methods , and networks with fixed random projections in the hidden layer . Since sparse connectivity is sometimes superior to dense connectivity ( Litwin-Kumar et al. , 2017 ; Bartunov et al. , 2018 ) and successful convolutional networks leverage local receptive fields , we investigated also sparse connectivity between input and hidden layer , where each hidden neuron receives input only from a few neighboring pixels of the input image . 2 RELATED WORK . In recent years , many bio-plausible approaches to deep learning have been proposed ( see e.g . Marblestone et al . ( 2016 ) for a review ) . For achieving performances similar to deep learning methods , existing approaches usually use either involved architectures or elaborate mechanisms to approximate the backpropagation algorithm . Examples include the use of convolutional layers ( Tavanaei & Maida ( 2016 ) ; Lee et al . ( 2018 ) ; Kheradpisheh et al . ( 2018 ) and table therein ) , dendritic computations ( Hussain et al. , 2014 ; Guergiuev et al. , 2016 ; Sacramento et al. , 2017 ) or approximations of the backpropagation algorithm such as feedback alignment ( Lillicrap et al. , 2016 ; Baldi et al. , 2016 ; Nøkland , 2016 ; Samadi et al. , 2017 ; Kohan et al. , 2018 ; Bartunov et al. , 2018 ) equilibrium propagation ( Scellier & Bengio , 2017 ) , membrane potential based backpropagation ( Lee et al. , 2016 ) , restricted Boltzmann machines and deep belief networks ( O ’ Connor et al. , 2013 ; Neftci et al. , 2014 ) , ( localized ) difference target propagation ( Lee et al. , 2015 ; Bartunov et al. , 2018 ) , reinforcementsignal models like AuGMEnT ( Rombouts et al. , 2015 ) or approaches using predictive coding ( Whittington & Bogacz , 2017 ) . Many models implement spiking neurons to stress bio-plausibility ( Liu et al . ( 2016 ) ; Neftci et al . ( 2017 ) ; Kulkarni & Rajendran ( 2018 ) ; Wu et al . ( 2018 ) ; Liu & Yue ( 2018 ) and table therein ) or for coding efficiency ( O ’ Connor et al. , 2017 ) . The conversion of DNNs to spiking neural networks ( SNN ) after training with backpropagation ( Diehl et al. , 2015 ) is a common technique to evade the difficulties of training with spikes . Furthermore , there are models including recurrent activity ( Spoerer et al. , 2017 ; Bellec et al. , 2018 ) or even starting directly from realistic circuits ( Delahunt & Kutz , 2018 ) . We refer to Table 8 for a list of current bio-plausible MNIST benchmark models . 3 RESULTS . We study networks that consist of an input ( l0 ) , one hidden ( l1 ) and an output-layer ( l2 ) connected by weight matrices W1 and W2 ( Figure 1 ) . Training the hidden layer weights W1 with standard supervised training involves ( non-local ) error backpropagation using the transposed weight matrix WT2 ( Figure 1A ) . In the bio-plausible network considered in this paper ( Figure 1B ) , the input-tohidden weights W1 are either learned with an unsupervised method ( Principal Component Analysis or Sparse Coding ) or are fixed random projections . The unsupervised methods assume recurrent inhibitory weights V1 between hidden units to implement competition . 3.1 SPIKING LOCALIZED RANDOM PROJECTIONS . We first present the results with networks of leaky integrate-and-fire ( LIF ) neurons . The network architecture is as in Figure 1B , but without the recurrent connections V1 . For implementing localized Random Projections ( l-RP ) in the hidden layer weights W1 , we first chose the centers of the localized receptive fields at random positions in the input space and then randomly chose the weights therein , see Figure 1C . The receptive field patches span p×p pixels around their center position ( we used p = 10 for the 28×28-pixel MNIST data ) . The output layer weights W2 are trained with a supervised spike timing dependent plasticity ( STDP ) rule . 3.1.1 LIF AND STDP DYNAMICS . The spiking dynamics follow the usual LIF equations ( see methods A.4 ) and the readout weights W2 evolve according to a supervised STDP delta rule using post-synaptic spike-traces tri ( t ) and a post-synaptic target trace tgti ( t ) τtr dtri ( t ) dt = −tri ( t ) + ∑ f δ ( t− tfi ) ∆w2 , ij = α · ( tgtposti ( t ) − tr post i ( t ) ) δ ( t− tfj ) . ( 1 ) Thus , for a specific readout weight w2 , ij , the post-synaptic trace is updated at every post-synaptic spike time tfi and the weight is updated at every pre-synaptic spike time t f j . The target trace is used for feeding in the one-hot coded , supervisory signal for the MNIST classification into the output layer ( l2 ) . For a proof-of-principle and efficient parameter search we first investigate an LIF rate model . This rate model mimics the LIF dynamics by using the LIF activation function ϕLIF as nonlinearity , rate ( u ) = ϕLIF ( u ) = [ ∆abs − τm ln ( 1− ϑ u ) ] −1 , ( 2 ) where u is the membrane potential , ∆abs the refractory period , τm the membrane time constant and ϑ the firing threshold of the LIF model . Furthermore , it employs the rate-version of the STDP delta rule Equation 1 ( see methods section A.4 for details ) ∆wij = α̃ · rateprej · ( tgtrateposti − rate post i ) , ( 3 ) where tgtrateposti is the post-synaptic target rate , corresponding to the post-synaptic target trace tgti ( t ) in Equation 1 . We obtained similar spiking and weight dynamics when the readout weights W2 were either directly trained with STDP or trained with the LIF rate model and then plugged into the spiking LIF network ( as done in e.g . Diehl et al . ( 2015 ) ) . To illustrate the LIF and STDP dynamics , a toy example consisting of one pre- connected to one post-synaptic neuron was integrated for 650 ms . The pre- and post-synaptic membrane potentials show periodic spiking ( Figure 2A ) which induces post-synaptic spike traces and corresponding weight changes ( Figure 2B ) , according to Equation 1 . For the MNIST task , Figure 2C shows a raster plot for an exemplary training and testing protocol . During activity transients after pattern switches , learning is disabled until regular spiking is recovered . This is done , first , to ensure stability during activity transients ( see Naud et al . ( 2008 ) and references therein ) and second , to achieve decorrelation between the activities of subsequent patterns , as needed for stochastic gradient descent ( SGD ) . During the testing period , learning is shut off permanently ( see methods section A.4 for more details ) . 3.1.2 CLASSIFICATION RESULTS FOR LIF l-RP When directly trained with the STDP rule in Equation 1 the spiking LIF l-RP model ( nh = 5000 hidden units and patch size p = 10 ) reaches 98.1 % test accuracy on MNIST . The corresponding LIF rate model reaches 98.5 % test accuracy . Transferring weights learned with the LIF rate model into the spiking LIF model resulted in similar accuracies as the LIF rate model . Table 1 compares the performances of the rate and spiking LIF l-RP models with the reference algorithm l-BP , which is a rate model trained with backpropagation , see subsection 3.2 and subsection 3.3 ( for same hidden layer size nh and patch size p ) . We can see that the spiking LIF model almost reaches the performance of the corresponding rate model . The remaining gap ( 0.4 % ) between rate and spiking LIF model presumably stems from transients and the shorter training time of the spiking model ( only 106 compared to 107 iterations due to long simulation times ) . Both , the rate and spiking LIF model of l-RP achieve accuracies close to the backpropagation reference algorithm l-BP and certainly lie in the range of current bio-plausible MNIST benchmarks , i.e . 97-99 % test accuracy ( see section 2 and Table 8 ) . Based on these numbers we conclude that the spiking LIF model of localized random projections using STDP is capable of learning the MNIST task to a level that is competitive with known benchmarks for spiking networks .

Summary: This article compares different methods to train a two-layer spiking neural network (SNN) in a bio-plausible way on the MNIST dataset, showing that fixed localized random connections that form the hidden layer, in combination with a supervised local learning rule on the output layer can achieve close to state-of-the-art accuracy compared to other SNN architectures. The authors investigate three methods to train the first layer in an unsupervised way: principal component analysis (PCA) on the rates, sparse coding of activations, and fixed random local receptive fields.  Each of the methods is evaluated on the one hand in a time-stepped simulator, using LIF neurons and on the other hand using a rate-approximated model which allows for faster simulations. Results are compared between each other and as reference with  standard backpropagation and feedback alignment.  The main finding is that localized random projections outperform other unsupervised ways of computing first layer features, and with many hidden neurons approaches backpropagation results. These results are summarized in Table 8, which compares results of the paper and other state-of-the-art and bio-plausible SNNs. PCA and sparse coding work worse on MNIST than local random projections, regardless if the network is rate-based, spike-based or a regular ANN trained with the delta rule. Feedback Alignment, although only meant for comparison, performs best of the algorithms investigated in this paper.