Efficient Learning with Spiking Neural Networks

Artificial neural networks constitute a vast field which has reached a level of maturity in some areas. However, a big problem with state-of-the-art methods is the impractical amount of compute power they require when scaled. Spiking neural networks are third generation neural networks with a higher level of biological realism, which typically comes at the cost of even greater computation. Along with their complexity, this makes them impractical for general machine learning. However it may be possible for spiking networks to be more efficient than earlier networks for some problems. This is because they have the capacity to exploit some of the brain’s own efficiency techniques. This thesis explores those techniques in the literature and contrasts them with digital computation. Experiments reproducing some of them in artificial neural networks are then presented. Efficiency of computation is measured using cputime, which is related to how fast an algorithm runs as well as how much energy it consumes. The computational costs of spiking learning are quantified for the first time using the supervised learning algorithm SpikeProp. This was found to be between one and two orders of magnitude slower than the multilayer perceptron, and even worse with a receptive field encoding. Efficiency enhancements to SpikeProp are then presented, including reduced precision, fewer synapses, lookup tables and event-driven computation. The computation required by the enhanced algorithm SpikeProp+ to learn classifiers is again compared to Backprop on the multilayer perceptron, using several standard datasets. The compute time required to train ii Abstract the improved networks was reduced by up to seventeen times, surpassing Backprop on the XOR problem. In that case, the SpikeProp+ network runs in the same time as the perceptron, although it remains over two times slower on other datasets. Finally, in applying SpikeProp+ to a stance-detection application, it was discovered that the spiking algorithm is intrinsically able to learn from data containing many missing values to a high level of accuracy, without any pre-processing. This property was verified with a standard dataset, and is relatively unique amongst machine learning techniques. Thus a case for using spiking neurons as a substitute in more conventional neural network architectures is presented, to both accelerate them and permit the use of lossy data. These properties are desirable in view of the vast sizes and variable quality of modern data. Acknowledgements This dissertation is the culmination of a long and arduous journey of intellectual and emotional self-discovery. I owe a lot of this to the sleepy little town of Bathurst, which first drew me over the mountains from some restless fate. I am deeply indebted to my supervisor Michael Antolovich who has always seen me through the tough times, and without whom this thesis would be impossible. Similarly, my cosupervisor Wayne Moore’s endless enthusiasm and experience helped sculpt the polished article you are about to read. I am also thankful for the late John Carroll, who chose me to realise his new vision for the digital performing arts. The influence of James Crane and Terry Bossomaier on the initial directions of my work is also appreciated. I thank Charles Sturt University for giving me all these opportunities. I must also thank my labmates David, James, Jason, Fenglu and Anthony for the group therapy of bellyaching and procrastination. To all my friends from the university and Bathurst Salsa, especially my good friend Ian, thankyou for helping me stay sane in face of the tribulations. For this I also credit my trusty old motorcycle Duke, who always came through, even in the icy depths of Bathurst winter. Finally, thankyou Mom and Pap for all your help in spite of everything. Certificate of Authorship I hereby declare that this submission is my own work and to the best of my knowledge and belief, understand that it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma at Charles Sturt University or any other educational institution, except where due acknowledgement is made in the thesis. Any contribution made to the research by colleagues with whom I have worked at Charles Sturt University or elsewhere during my candidature is fully acknowledged. I agree that this thesis be accessible for the purpose of study and research in accordance with normal conditions established by the Executive Director, Library Services, Charles Sturt University or nominee, for the care, loan and reproduction of thesis, subject to confidentiality provisions as approved by the University. Name: . Signature: . Date: .