Simulating macroscale brain circuits with microscale resolution
Abigail Morrison1, Tobias C Potjans2,3, Susanne Kunkel4,Hans-Ekkehard Plesser5, Markus Diesmann1,3,4
1 RIKEN Brain Science Institute, Wako City, Saitama, Japan
2 Brain and Neural Systems Team, RIKEN Computational Science Research Program, Wako City,Saitama, Japan
3 Institute of Neurosciences and Medicine, Research Center Juelich, Juelich, Germany
4 Bernstein Center for Computational Neuroscience, Albert-Ludwigs University, Freiburg, Germany
5 Norwegian University of Life Sciences, Aas, Norway
E-mail: abigail@brain.riken.jp
Abstract： In order to understand higher brain functions, it is necessary to simulate macroscale
circuits involving multiple areas of the brain. Such brain-scale networks represent a challenge tocurrent simulation technology, as design solutions that are appropriate for simulating local
cortical networks lead to unreasonable serial memory overhead when the network is scaled up by
one or two orders of magnitude. We quantify these limitations and present strategies forovercoming them on computer architectures with up to 10k processors.

1 Linking microscale to macroscale
brain networks
Thanks to distributed computing techniques,
today it is possible to routinely simulate local
cortical networks of around 105 neurons with up
to 109 synapses on clusters and multiple
processor
sharedmemorymachines.
Simulations of this type carried out with NEST
[1] scale well up to at least 1000 processors.
However, simulations of microscale networks
corresponding to approximately 1mm3 of thecortex are limited in their explanatory power. To
understand the functions of the brain, we needto simulate macroscale circuits involving
multiple interacting areas [2]. One approach is
to develop brain-scale networks in which theindividual nodes are realized by microcircuits at
the resolution of point neurons and synapses
such as the layered local cortical network model
we recently developed [3] (see figure A). These

networks will be one or two orders of magnitudelarger than the previously studied models, notonly in terms of numbers of neurons andsynapses but also in terms of computationalload.

2 Simulation technology forbrain-scale spiking networks
This presents a number of challenges tocurrent simulation technology. Firstly, as thenumber of processors increases, the memory
overhead due to serial data structures
eventually dominates the total memory usage
and so limits the parallelization. An example ofthis is the usage of proxies representing remoteneurons on each machine and providing aninterface to the local neurons. Although such
representations of remote neurons require
much less memory than local neurons, their
proportion of the total neuronal memory usage
approaches one as the number ofmachines increases. Secondly, the size ofthe maximal synaptic delay determines
the size of the spike buffers each neuron
uses to queue incoming spikes [4]. For
local networks these buffers are small,as the synaptic delays are in the order ofmilliseconds. If interacting brain areas
are to be considered, the synaptic delays

increase by an order of magnitude, which
entails a corresponding increase of memory
usage for the spike buffers. We quantify theeffects of these memory limitations in ourapplication up to the order of 10k processors
and present strategies for addressing theseproblems.

Previously only networks in the order of 105
neurons were possible. The simulation shows
good scaling up to 8192 processors and still
improves when using 16384 processors. An evenlarger network with 6 million neurons and fullconnectivity (104 synapses per neuron, redsquares) exhibits good scaling up to 32768
processors. Figure C shows the scaled speed-up:
computing time as a function of the number ofcores when simultaneously increasing the
problem size. In this case, a horizontal line
corresponds to optimal performance. Red
squares show the data for 550 neurons per core,
blue circles for 1100 neurons per core (allsimulations with 104 synapses per neuron).
Both cases show only a slight increase in thecomputing time when simulating a network fivetimes the original problem size on an
equivalently enlarged computer system.
Future work concentrates on the further
reduction of memory consumption.

Acknowledgements
Partiallyfundedby
Next-Generation
Supercomputer Project of MEXT, Japan, EU
Grant 15879 (FACETS), BMBF Grant
01GQ0420 to BCCN Freiburg, and theHelmholtz Alliance on Systems Biology.

3Performance example of NEST on
Blue Gene/P supercomputers

Having implemented these strategies for
reducing memory consumption, we tested theperformance of NEST on a Blue Gene/Psupercomputer. Figure B shows the timerequired to simulate one second of biologicaltime as a function of the number of cores. The
blue circles correspond to a network with 106
neurons and 104 synapses per neuron.

References[1] Gewaltig M-O, Diesmann M: NEST (neuralsimulation tool). Scholarpedia 2007, 2(4),1430[2] Sporns O, Tononi G and Koetter R: The
Human
Connectome:
A
Structural
Description of the Human Brain, PLoS
Comput Biol 2005, 1(4):e42[3] Potjans TC, Diesmann M: Consistency of invitro and in vivo connectivity estimates:
statistical assessment and application tocortical network modeling. 38th Soc for
Neurosci Meeting 2008, 16.1
[4] Morrison A, Mehring C, Geisel T, Aertsen A,Diesmann M: Advancing the Boundaries ofHigh-Connectivity Network Simulation
with Distributed Computing, NeuralComput 2005, 17(8):1776-801