Neuromorphic Computing: WhereHardware Meets AI
Katie SchumanResearch Scientist
Oak Ridge National Laboratory
Contact: schumancd@ornl.gov

ORNL is managed by UT-Battelle, LLC for the US Department of Energy

My Background• Ph.D. in Computer Science from the University of Tennessee– National Science Foundation Graduate Research Fellowship

to study evolutionary algorithms and spiking neural networks• Joined ORNL in 2015 as a Liane Russell Early Career fellow– Project: Programming and Usability of NeuromorphicComputing• 55+ publications in spiking neural networks and neuromorphic

computing, 6 patents– A Survey of Neuromorphic Computing and Neural Networksin Hardware• Joint faculty with the Department of Electrical Engineering &Computer Science at the University of Tennessee• Co-founder of the TENNLab• Department of Energy Early Career Award in 20192

Why should you
care about
hardware?

3

Looming End ofMoore’s Law(And the end of Dennard scaling)

Rise of theInternet of Things

4

Artificial
IntelligenceandMachine Learning?

Neural Hardware and Neuromorphic Computing
Neural Hardware
Accelerates traditional neuralnetwork and deep learning
computation• Google TPU• Intel Movidius NeuralCompute Stick• Well-suited to existing algorithms• Fast computation or low power• Currently deployed in cloud or
mobile devices
5Neural Hardware and Neuromorphic Computing
Neural Hardware

Neuromorphic Computing

Accelerates traditional neuralnetwork and deep learning
computation

Implements spiking recurrent neuralnetwork computation and can besuitable for neuroscience simulation• Google TPU• Intel Movidius NeuralCompute Stick• Well-suited to existing algorithms• Fast computation or low power• Currently deployed in cloud or
mobile devices
6• Intel Loihi• IBM TrueNorth• Significant promise for futurealgorithmic development• Fast computation and low power• Still in development

What is Neuromorphic Computing?
Von Neumann Architecture

CPUMemory•••••
7

Sequential processing
Separated memory and computation
Power intensive
Programmed
High precision

What is Neuromorphic Computing?
Von Neumann Architecture

Neuromorphic Architecture
NeuronCPUMemorySynapse•
••
•
•
8

Sequential processing
Separated memory and computation
Power intensive
Programmed
High precision

••
•
•
•

Massive parallelization
Collocated memory and computation
Very low power
Training or learning
Low precision

How do you program a neuromorphic computer?
CPUCompiledMemory?
9“Compiled”

How do you program a neuromorphic computer?
CPUCompiledMemory“Compiled”

Spiking Neural Network!
10

Traditional Artificial Neural Networks

Input Layer
11

Hidden Layer

Hidden Layer

Output LayerTraditional Artificial Neural Networks

x1x2x312

Traditional Artificial Neural Networks

13

Traditional Artificial Neural Networks
u1

u2

u3

u4f(W(1)x)=u14

Traditional Artificial Neural Networks
v1v2

v3

v4

f(W(2)u)=v
15

Traditional Artificial Neural Networksof(W(3)v)=o16

Spiking Neural Networks• Timecomponent onsynapses• More complex

networkstructures• Temporal input• Temporal output

17

Spiking Neural Networks

18

Spiking Neural Networks

19

Spiking Neural Networks

20

Spiking Neural Networks

21

Spiking Neural Networks

22

Spiking Neural Networks

23

Spiking Neural Networks

24

Spiking Neural Networks

25

Spiking Neural Networks

26

Spiking Neural Networks

27

Spiking Neural Networks

28

Spiking Neural Networks

29

Spiking Neural Networks

30

Spiking Neural Networks

31

Spiking Neural Networks

32

Spiking Neural Networks

33

Spiking Neural Networks

34

Spiking Neural Networks

35

Spiking Neural Networks

36

Spiking Neural Networks

37

Spiking Neural Networks

38

Spiking Neural Networks

39

Spiking Neural Networks

40

Spiking Neural Networks

41

Spiking Neural Networks

42

How do you build (or train) a spiking neural network tosolve a particular problem?
How many neurons?

What should the weights
on each synapse be?

How many synapses?

What should the delays
on each synapse be?

43

What should thethreshold or activation
value be for eachneuron?

How do you build a spiking neural network for aparticular neuromorphic implementation?
• Different neuromorphic implementations have different:– Neuron models (how the neuron functions, how many parameters)– Synapse models (how the synapse functions, how manyparameters)– Levels of connectivity– Devices and materials, which may radically change how thenetworks can function

44

Example Neuromorphic Implementations
DANNA2• Fully digital implementation• Two versions:• DANNA2-dense isprogrammable• DANNA2-sparse isapplication-specific
Mitchell, J. Parker, et al. "DANNA 2: Dynamic adaptive
neural network arrays." Proceedings of theInternational Conference on Neuromorphic Systems.
ACM, 2018.

45mrDANNASOEN• Mixed analog-digital
implementation• Synapses implemented withtwin memristors• Programmable• Optoelectronic• Neurons implemented using
superconductingoptoelectronics• Delays are on neurons, notsynapses

Chakma, Gangotree, et al. "Memristive mixed-signal
neuromorphic systems: Energy-efficient learning at thecircuit-level." IEEE Journal on Emerging and Selected
Topics in Circuits and Systems 8.1 (2018): 125-136.

Buckley, Sonia, et al. "Design of superconducting
optoelectronic networks for neuromorphiccomputing." 2018 IEEE International Conference on
Rebooting Computing (ICRC). IEEE, 2018.

How do you build a spiking neural network for aneuromorphic system for a particular problem?

Not only do we have to come up with the right spiking neuralnetwork structure, that spiking neural network also has to workwithin the hardware constraints: architecture, device, ANDmaterials.

46

Evolutionary Optimization for Neuromorphic Systems(EONS)
RandomInitialization

Ordered Population

ChildPopulation

BestParents

Evaluateand Rank

SelectWorst

47

Reproduce

Why Evolutionary Optimization?
• Applicable to a wide variety of tasks• Applicable to different architectures anddevices• Operates within the characteristics andconstraints of the architecture/device• Can learn topology and parameters (notjust synaptic weights)• Can interact with software simulations ordirectly with hardware• Parallelizable/scalable on HPC
48

Applications of Neuromorphic Computing
SpatioTemporalNoisy
Input

ContinuousLearning• Scientific discovery• Co-processor• Large-scale data analytics

Requiresrobustness

Neuromorphic
Application
Characteristics

Real-TimeProcessing• Cyber security• Autonomous vehicles

Not highprecision

MultimodalLowpower49• Robotics• Internet of things• Smart sensors#4: Asteroids on DANNA 2 (FPGA/ASIC)ORNL is managed by UT-Battelle, LLC for the US Department of Energy

Data from MINERvA (Main Injector Experiment for v-A)
• Neutrino scattering experiment atFermi National Accelerator
Laboratory• The detector is exposed to the NuMI

(Neutrinos at the Main Injector)
neutrino beam

• Millions of simulated neutrino-nucleus scattering events werecreated• Classification task is to classify thehorizontal region where theinteraction originated

MINERvA Detector

Source: A. Terwilliger, et al. Vertex Reconstruction of Neutrino Interactions using Deep Learning. IJCNN 2017.
51

Two Data Inputs Types (Three Views)
Deep Learning: Energy values as interpreted as pixels

Spiking: Time when energy deposition goes over a very low threshold52

Spiking Neural Networks

53

Best Results: Single View
x-view(127x50)

conv pool conv
conv
conv pooldrodro
pool2
pool3
fc1
fc2
fc3
classification
112344pp
(2x1)
(2x1)
(196)
(98)
(11)
(8x3) (2x1) (7x3)(6x3)(6x3) (2x1)
outoutConvolutional Neural Network Result: ~80.42%• 90 neurons, 86 synapses• Estimated energy for a single
classification for mrDANNA
implementation: 1.66 μJ

Spiking Neural Network Result: ~80.63%
Source for CNN results: A. Terwilliger, et al. Vertex Reconstruction of Neutrino Interactions using Deep Learning. IJCNN 2017.
54

Example Application: Autonomous Robot Navigation• Task: Navigate and explore an unfamiliar

environment while avoiding obstacles• Challenges:– No explicit instructions on how to operate– No prior knowledge about the environment– Limited input resolution (LIDAR sensors)– Process all inputs and make control
decisions on-board the robot (nocommunication to/from the robot to anothercomputer system)– Train only in simulation
55

Application: Robotics Control Results

56

Student Application: Parker Mitchell and Grant Bruer (Spring 2017)

Application: Robotics Control Results

57

Student Application: Parker Mitchell and Grant Bruer (Spring 2017)

Summary• The future of AI is likely to include custom hardware like neural

hardware and neuromorphic computing• Neuromorphic computing systems are non-trivial to program• We’ve developed a spiking neural network training methodology

based on evolutionary optimization that has been applied to multiple
implementations and many applications• Now is the time to get involved!

58

Interested in Learning More about Neuromorphic?
“A Survey of Neuromorphic Computing and Neural
Networks in Hardware”
https://arxiv.org/abs/1705.06963

59

Work supported by:
Department of Energy
Air Force Research Lab

neuromorphic.eecs.utk.edu
60

Thank you!
Questions?
Contact:
Email: schumancd@ornl.gov
Website: catherineschuman.com
Twitter: @cdschuman61Non-Traditional Input
Encoding Schemes for
Spiking Neuromorphic
Systems• IJCNN 2019
• In collaboration with Jim Plank, Grant

Bruer, and Jeremy Anatharaj from UTKey Challenge: Input Encoding

Spiking neural networks
require spikes over time
as input63

Key Challenge: Input Encoding

Numerical Input Data0.7

64

0.4-0.3

How do you
convertnumerical datainto spikes?

Common Approaches

0.6

Rate Coding
0.3

0.6

Temporal Coding
0.365

Common Approaches

0.6

Rate Coding
0.3

Key Issue:
Limited input resolution for real-timeprocessing applications
0.6

Temporal Coding
0.366

Proposed Input Encoding Schemes• Motivation: Develop encoding schemes that:– Can be applied to a wide variety of input data types– Can represent single input values over a very short period of time so that they
can be applied to real-time classification or control tasks• Encoding schemes:– Binning– Spike-count– Charge-injection– Complex encoding schemes

Assumption: For a given input k,
we assume that all possible input
values fall in the range mk to Mk• Combining binning, spike-count, and charge-injection encoding to form more complex

calculations

67

Key Observations
The encoding scheme chosen has an impact on application performance.

68Key Observations
The encoding scheme chosen has an impact on application performance.

69

Key Observations
The appropriate encoding
scheme depends
primarily on theapplication, rather thanthe implementation.

70

Multi-ObjectiveOptimization for Size andResilience of Spiking
Neural Networks• IEEE Ubiquitous Computing,• In collaboration with Mihaela Dimovska(University of Minnesota), Travis
Johnston, Parker Mitchell, and Tom
Potok

Size Optimization: Pruning Post-Training• Strategy: Prune internal neurons with low spiking frequency

72

PBRadio

Asteroids

Averagenumber ofinternal
neurons

9.36

77.8

75.09

Averagenumber ofsynapses

32.61

139.19438.18Averageperformance

292.2

0.777

214.9

PBRadio

Asteroids

Averagenumber ofinternal
neurons

2.43

4.97

35.24

Averagenumber ofsynapses

21.53

62.52

35.24

Averageperformance

288.7

0.778

208.3Size Optimization: Multi-Objective for Size
• Strategy: Adjust the training fitness function to include minimizing

size as an objective73PBRadio

Asteroids

Averagenumber ofinternal
neurons

9.36

77.8

75.09

Averagenumber ofsynapses

32.61

139.19438.18Averageperformance

292.2

0.777

214.9

PBRadio

Asteroids

Averagenumber ofinternal
neurons

5.08

0.56

1.3

Averagenumber ofsynapses

23.41

122.9

147.17

Averageperformance

297.5

0.788

235.22

Multi-Objective Optimization for Resiliency

Weighting factor
Score74

Size multiplier

Weighting factor

Average of scores of
perturbed networksMulti-Objective Optimization for Resiliency

75

Island Model for Parallel
Evolutionary Optimizationof Spiking NeuromorphicComputing• GECCO 2019
• In collaboration with Jim Plank (UT),Robert Patton, and Tom PotokScalable Island Model

77

Scalable Island Model••78

Islands with communication
saw consistently better
results in the same amountof time on the same
computational resources asislands withoutcommunication
More resources generally
lead to better results faster(with the righthyperparameters).

Scalable Island Model: Hyperparameters Matter

79

Scalable Island Model: Hyperparameters Matter80