Jülich Aachen Research AllianceMitglied der Helmholtz-Gemeinschaft

The K computer as an instrument tostudy brain-scale neuronal networks at
microscopic resolutionwww.csn.fz-juelich.dewww.nest-initiative.org
Aug 26th 2014 SS-ken HPC Forum, Tokyo

Markus Diesmann
Institute of Neuroscience and Medicine (INM-6)
Institute for Advanced Simulation (IAS-6)
Jülich Research Centre1

multi-disciplinary
national research laboratory.
budget: ~557 Million Euro / year

Mitglied der Helmholtz-Gemeinschaft

as per Dec 31st 2012: Personnel: 5,236 in total
 Scientists: 1,658
 PhDs: 469
 Professors: 93

Aug 26th 2014 SS-ken HPC Forum, Tokyo2

Mitglied der Helmholtz-GemeinschaftRegion

30 km distance to 3 major universities

Aug 26th 2014 SS-ken HPC Forum, Tokyo3

Mitglied der Helmholtz-Gemeinschaft

Research Center Jülich

2.2 km2 campus

Aug 26th 2014 SS-ken HPC Forum, Tokyo

4

Helmholtz Mission

Mitglied der Helmholtz-Gemeinschaft

We contribute to solving grand challenges which face society,science and industry by performing top-rate research in strategic
programmes in the fields of Aeronautics, Space and Transport,
Earth and Environment, Energy, Health, Key Technologies aswell as the Structure of Matter.
We research systems of great complexity with our large-scale
facilities and scientific infrastructure, cooperating closely withnational and international partners.

Aug 26th 2014 SS-ken HPC Forum, Tokyo

5

JUQUEEN supercomputer

Mitglied der Helmholtz-Gemeinschaft

 inaugurated, Feb 14th 2013
 458,752 cores 5.9 peta-flop 28 compared to 72 racks of JUGENE TOP 500, June 2014: 8th (2nd in Europe)

Aug 26th 2014 SS-ken HPC Forum, Tokyo6

The Human Brain Project (HBP)

Mitglied in der Helmholtz-Gemeinschaft

 understanding the human brain
is still a great challenge difficulty: its complexity
 huge number of neurons (1011)
and synapses (104) per neuron
connecting them proper description level unclear

Aug 26th 2014 SS-ken HPC Forum, Tokyo

7

Situation in Neuroscience
120000

Number of Peer Reviewed
Publications on the Brain

100000Aug 26th 2014 SS-ken HPC Forum, Tokyo

80000

60000

40000

20000

82012

2011

2010

2009

2008

2007

2006

2005

2003

2004

2002

2001

2000

1999

1998

1997

1996

1995

1993

1994

1992

0
1991

data flood
data fragmentationno integration planno curation planno plan to bridgelevels
 no plan from animalto human lack of reproducibility
 small groups

1990

Mitglied in der Helmholtz-Gemeinschaft






EU FET (Future and EmergingTechnologies) Flagships


Mitglied in der Helmholtz-Gemeinschaft



grand scientific and societal challenges which require a common
European research effort
science‐driven, large‐scale, multidisciplinary research
initiatives oriented towards a unifying goal
goals are visionary and highly ambitious
require cooperation among a range of disciplines
boosting European innovation
sustained support for a period of up to 10 years
substantial benefits for European competitiveness and society

Aug 26th 2014 SS-ken HPC Forum, Tokyo9

Human Brain ProjectMitglied in der Helmholtz-Gemeinschaft

started in Oct 2013, Kickoff summit in Lausanneramp-up phase (30 months): €54 million from EU Information andCommunication Technologies (ICT) 2013 Work Program
careful monitoring - report: every 6 months
operational phase: finance will come from the Horizon 2020
Research Program
Horizon 2020 proposal presently under review

Aug 26th 2014 SS-ken HPC Forum, Tokyo10

Mitglied in der Helmholtz-Gemeinschaft

Human Brain ProjectAug 26th 2014 SS-ken HPC Forum, Tokyo11

Mitglied in der Helmholtz-Gemeinschaft

The Consortium• 80 Institutions• 150 PIs
• 2000 PhDs
• 20 European Countries
Aug 26th 2014 SS-ken HPC Forum, Tokyo

12

HBP as a large-scale scientific project, model:
Large Hadron Collider(LHC) at CERN
Basic Facts
Construction: 1998-2008

Mitglied in der Helmholtz-Gemeinschaft

Consists of a 27km ring of superconducting
magnets: Thousands of magnets of different
varieties and sizes are used to direct thebeams around the accelerator. These include
1232 dipole magnets 15m in length which
bend the beams, and 392 quadrupole
magnets, each 5–7m long, which focus thebeams.
Built in collaboration with over 10,000
scientists and engineers from over 100
countries, as well as hundreds of universities
and laboratories.

Aug 26th 2014 SS-ken HPC Forum, Tokyo

13

Build, Simulate and ValidateMitglied in der Helmholtz-Gemeinschaft



workbench throughweb portal
integrated
workflow
access toplatforms
provenancetracking

Aug 26th 2014 SS-ken HPC Forum, Tokyo

14

The EU Computational Neuroscience path
towards the Human Brain Project

Mitglied in der Helmholtz-Gemeinschaft

Jülich200120062009

2012

2015

Inauguration
NESTinitiative
Aug 26th 2014 SS-ken HPC Forum, Tokyo15

Overview models of large-scale networks
cortical microcircuit
need for brain-scale models

designing simulation software for the brain scale

Mitglied der Helmholtz-Gemeinschaft

3rd generation simulation kernel[scale of 10,000 compute nodes]
4th generation simulation kernel,
platform in initial phase of HBP[scale of 100,000 compute nodes]
Aug 26th 2014 SS-ken HPC Forum, Tokyo

16

Mitglied der Helmholtz-Gemeinschaft

Interactions between neurons

each neuron receives input from10,000 other neurons

supra-threshold value causes spiketransmitted to post-synaptic neuron

causing large fluctuations ofmembrane potentialpost-synaptic neuron responds with small
excursion of potential after delayinhibitory neurons (20%) cause negative
excursion

emission rate of 1 to 10 spikes
per secondcurrent injection into pre-synaptic neuron
causes excursions of membrane potentialAug 26th 2014 SS-ken HPC Forum, Tokyo17

Mitglied der Helmholtz-Gemeinschaft

Minimal layered cortical network model

1 mm3

1 billion synapses, 100,000 neurons

2 populations of neurons per layer:
E: Excitatory



I: InhibitoryE and I identical neuronal dynamics

laterally homogeneous connectivitylayer- and type-specific 𝐶𝑖𝑗

Aug 26th 2014 SS-ken HPC Forum, Tokyo𝑥𝑦

18

Mitglied der Helmholtz-Gemeinschaft

Anatomical data sets

Aug 26th 2014 SS-ken HPC Forum, Tokyo

19

Mitglied der Helmholtz-Gemeinschaft

Target specificity



Aug 26th 2014 SS-ken HPC Forum, Tokyo

correction for bias inanatomical method20

Mitglied der Helmholtz-Gemeinschaft

Convergence and divergence

Aug 26th 2014 SS-ken HPC Forum, Tokyo

dominated by within-layer
connections

e → e divergence reflects”standard” loope → i divergence reflects
target-specific feedback21

Local cortical microcircuit

Mitglied der Helmholtz-Gemeinschaft

taking into account layer and neuron-type specificconnectivity is sufficient to reproduce
experimentally observed:


asynchronous-irregular spiking ofneurons

higher spike rate of inhibitory neurons

correct distribution of spike rates
across layers



integrates knowledge of more than50 experimental papers

Potjans TC & Diesmann M (2014) The cell-type specific connectivityof the local cortical network explains prominent features of neuronal
activity. Cerebral Cortex 24 (3): 785-806

105 neurons
109 synapses

available at: www.opensourcebrain.org
Aug 26th 2014 SS-ken HPC Forum, Tokyo

22

Mitglied der Helmholtz-Gemeinschaft

Response to transient inputs

Aug 26th 2014 SS-ken HPC Forum, Tokyo

23

Response to transient inputs
 T = -0.4

Mitglied der Helmholtz-Gemeinschaft

 T = +0.4

Aug 26th 2014 SS-ken HPC Forum, Tokyo24

Hypothesis on cortical flow of activity

Mitglied der Helmholtz-Gemeinschaft

handshaking between layers

Potjans TC & Diesmann M (2014) The cell-type specific connectivityof the local cortical network explains prominent features of neuronal
activity. Cerebral Cortex 24 (3): 785-806
Aug 26th 2014 SS-ken HPC Forum, Tokyo25

Mitglied der Helmholtz-Gemeinschaft

Building block for functional studies

Aug 26th 2014 SS-ken HPC Forum, Tokyo26

Building block for mesoscopic studies
 collaboration with Gaute Einevoll (UMB, Norway)

Laminar LFP profile´Morphology

Mitglied der Helmholtz-Gemeinschaft+=Spatial reach ofthe LFP

Linden H, Tetzlaff T, Potjans TC, Pettersen KH, Grün S,Diesmann M, Einevoll GT (2011) Modeling the spatial
reach of the LFP. Neuron 72(5):859-872
Aug 26th 2014 SS-ken HPC Forum, Tokyo

27

Critique of local network models
Lack of constraints in local model criticize local networks

XXXXXXX

Mitglied der Helmholtz-Gemeinschaft

Aug 26th 2014 SS-ken HPC Forum, Tokyo

28

Critique of local network model

Mitglied der Helmholtz-Gemeinschaft

a network of networks with at least three levels of organization:

 neurons in local microcircuit models are missing 50% of synapses
 e.g., power spectrum shows discrepancies, slow oscillations missing
 solution by taking brain-scale anatomy into accountAug 26th 2014 SS-ken HPC Forum, Tokyo29

Meso- and macro-scale measures
brain-scale networks basis for:
 further measures byforward modeling comparison withmean-field models

Mitglied der Helmholtz-Gemeinschaft

mesoscopic measures
 local field potential (LFP) voltage sensitive dyes (VSD)and macroscopic measures
 EEG, MEG
 fMRI resting state networks

Aug 26th 2014 SS-ken HPC Forum, Tokyo30

Mitglied der Helmholtz-Gemeinschaft

Toward a self-consistent model

I. Intra-areal synapses
II. Intra-areal synapses replaced by random input
III. Cortico-cortical synapses
IV. External input represented by random input
V. Thalamic input
Aug 26th 2014 SS-ken HPC Forum, Tokyo Sacha van Albada
 Maximilian Schmidt
 Rembrandt Bakker
31

Multi-area model of macaque visual cortex•••

Mitglied der Helmholtz-Gemeinschaft•rich anatomical data sets available (e.g CoCoMac)close to human
32 areas structured in layers comprising 8∙108 neurons
downscaled model with 3.2∙106 neurons and 3∙1010 synapses

From Dombrowski et al. (2001), Cereb Cortex
architectural types provided by C. Hilgetag (private communication)
Aug 26th 2014 SS-ken HPC Forum, Tokyo

32

Anatomical data from tracing studies

Mitglied der Helmholtz-Gemeinschaft

From Markov et al. (2014),
J. of Comparative Neurology

FLN data from Markov et al.
(2012), Cereb Cortex
Aug 26th 2014 SS-ken HPC Forum, Tokyo33

Cortico-cortical connectivity

Mitglied der Helmholtz-Gemeinschaft

labeling:
y-axis: odd areas
x-axis: even areas

Aug 26th 2014 SS-ken HPC Forum, Tokyo34

Laminar patterns
Sending side

From Markov et al. (2014), J. of Comparative Neurology

Receiving side
synapse layer: CoCoMac databasereceiving synapse type: Computed from
Binzegger et al. (2004)

Mitglied der Helmholtz-Gemeinschaft

Synapse location••

Aug 26th 2014 SS-ken HPC Forum, Tokyo

Fraction of cortico-cortical synapses in each layer
Cell body location
23E
123

4
5623I

0.57

4E

4I

0.18
0.16

5E

5I

6E

0.250.003

0.03

0.09

6I

0.84

0.73

0.16

0.76

0.10

0.14
0.85

From Binzegger et al. (2004), J Neurosci

0.1535

Dynamical results
Heterogeneous laminar firing patterns
Rates in reasonable range (0.2 – 20 spikes/s)
Inhibitory rates > excitatory rates
Broad rate distributions

Mitglied der Helmholtz-Gemeinschaft••••

Aug 26th 2014 SS-ken HPC Forum, Tokyo36

The mesoscopic scale parallel study network of 4mmx4mm covers size of
100 electrode Utah array spike correlation structure, LFP
 experimentally, formation andMitglied der Helmholtz-Gemeinschaft

breakdown of waves observedin relation to behavior

S Kunkel about 1 million neurons with 10 billion synapses
 requires supercomputing but not incredible resources
Aug 26th 2014 SS-ken HPC Forum, Tokyo37

Simulation Technology: the NEST Initiative
collaborative effort and community building
Major goals:
systematically publish
new simulation technology
produce public releases
under GPL

Mitglied der Helmholtz-Gemeinschaft

 origins in 1994, collaboration of several labs (since 2001)
 registered society (since 2012) teaching in international advanced courses:
 Okinawa Computational Neuroscience Course OCNC, Japan
 Advanced Course in Computational Neuroscience ACCN, Europe
 Latin American School on Computational Neuroscience LASCON, South America
www.nest-initiative.org

 core technology ofe.g.: Morrison et al. (2005) Neural Computation

Aug 26th 2014 SS-ken HPC Forum, Tokyo

Zaytsev, Morrison (2013) Frontiers in Neuroinformatics 38

Simulation technology for the brain scale
simulator for large heterogeneous
networks of point-neurons and neurons
with few compartments

 model of memory usage of NEST analysis of memory usage of the 3rd generation simulation
Mitglied der Helmholtz-Gemeinschaft

kernel (released with NEST 2.2)

 new design of data structures in the 4g kernel performance of 4g
www.nest-initiative.org
Aug 26th 2014 SS-ken HPC Forum, Tokyo39

Model of the memory usage of NEST

Mitglied der Helmholtz-Gemeinschaft

 describes the memory usage per MPI process

M total number of MPI processes
T number of threads per MPI process
N total number of neurons
K number of incoming connections per neuron
Aug 26th 2014 SS-ken HPC Forum, Tokyo

Kunkel S et al. (2012)
Front. Neuroinform. 5:35.

403rd generation simulation kernel(released with NEST 2.2 in December 2012) up to 108 neurons on K (andJUQUEEN) 11,250 synapses per neuron(exc-exc STDP)

 using up to MT=196,608

Mitglied der Helmholtz-Gemeinschaft

threads and T=8 threads pernode 16 GB of memory per node
Helias M et al. (2012) Front. Neuroinform. 6:26.

Aug 26th 2014 SS-ken HPC Forum, Tokyo41

Mitglied der Helmholtz-Gemeinschaft

Aug 26th 2014 SS-ken HPC Forum, Tokyo42

Previous connection infrastructure (3g)
 required on each process

Mitglied der Helmholtz-Gemeinschaft

sparse tableinner datastructure from 10,000 nodes on collapse along 2 dimensions
Aug 26th 2014 SS-ken HPC Forum, Tokyo43

3rd generation simulation kernel
analysis of contributions to total memory usage
 in the regime of 10k processes and beyond the inner data structurecauses severe overhead

inner datastructure
sparse table

Mitglied der Helmholtz-Gemeinschaft

synapses

Aug 26th 2014 SS-ken HPC Forum, Tokyo44

3rd generation simulation kernel
analysis of contributions to total memory usage adapt the model to account for short target lists
 potential solution: low-overhead data structure on supercomputers

only one
synapsestoredsparse table

Mitglied der Helmholtz-Gemeinschaft

synapsesmore thanone synapse
storedAug 26th 2014 SS-ken HPC Forum, Tokyo45

New adaptive connection infrastructure (4g)

low overhead persynapse on
supercomputers

Mitglied der Helmholtz-Gemeinschaft

full flexibility on
laptops andmoderately
sized clusters

Aug 26th 2014 SS-ken HPC Forum, Tokyo46

Comparison of 4g to 3g kernel

3g kernel

more than one
synapse storedone synapse
storedinfrastructure

Mitglied der Helmholtz-Gemeinschaft

sparse table
synapses

4g kernel
synapses
Aug 26th 2014 SS-ken HPC Forum, Tokyo47

4th generation simulation kernel
 adaptive connection infrastructure with low overhead in caseof short target lists

Mitglied der Helmholtz-Gemeinschaft

 achieved by metaprogramming no compromise on generality
 reduced memory usage of synapses
 e.g. removed vtable pointers no compromise on precision of synaptic state variables
 reduced setup time reduced simulation time
Aug 26th 2014 SS-ken HPC Forum, Tokyo48

Maximum network size up to 5.73x108 neurons on
229,376 cores of JUQUEEN

 up to 1.27x109 neurons on
663,552 cores of K

 11,250 synapses per neuron

Mitglied der Helmholtz-Gemeinschaft(exc-exc STDP)

 largest general network simulation performed on K
1.86x109 neurons, 6000 synapses per neuron(press release July 2013: 1.73x109)

 on JUQUEEN 1.08x109 neurons, 6000 synapses per neuron
Aug 26th 2014 SS-ken HPC Forum, Tokyo49

Runtime for a simulation of 1s between 8 and 41 min onJUQUEEN

 between 6 and 42 min on theK computer setting up the network takes

Mitglied der Helmholtz-Gemeinschaft

between 3 and 15 min still not fast enough for studies of plasticity
 need to increase multi-threading
Aug 26th 2014 SS-ken HPC Forum, Tokyo50

Mitglied der Helmholtz-Gemeinschaft

 under review …

 last paragraph of introduction:

Aug 26th 2014 SS-ken HPC Forum, Tokyo

51

Further interactions: gap junctions

 at each time point neuron i needs
membrane potential of neuron j

Mitglied der Helmholtz-Gemeinschaft

 gigantic system of differential equations naïve: communication of V in each stepAug 26th 2014 SS-ken HPC Forum, Tokyo52

Jacobi waveform relaxation

 communication of approximation
Mitglied der Helmholtz-Gemeinschaft

only after each min(d) iterate until convergence
Jan Hahne, Matthias Bolten (University Wuppertal), Jun Igarashi (OIST)Aug 26th 2014 SS-ken HPC Forum, Tokyo53

Mitglied der Helmholtz-Gemeinschaft

Speed versus accuracy

 pair of neurons with single gap junction
 for practically usable accuracies RMSE < 0.1 mV:
 waveform relaxation outperforms naïve implementation
 piecewise-cubic interpolation best
 very small network: single step faster than min(d) steps
 reverses for networks larger 1000 neurons
Aug 26th 2014 SS-ken HPC Forum, Tokyo54

Summary full-scale model of local cortical network explains prominent featuresof neuronal activity

 need for brain-scale models
 simulations require memory only available on supercomputers K computer and co-development phase was essential in developing
petascale technology in sequence of 2 journal papers

 NEST is simulation engine used by the Human Brain Project (HBP) at
the resolution of nerve cells and synapses
Mitglied der Helmholtz-Gemeinschaft exascale computers present new challenges for data structures exascale computers offer new opportunities for simulation speed
 MoU between RIKEN-AICS and Juelich establishedAug 26th 2014 SS-ken HPC Forum, Tokyo55