Quantitative Single-Neuron Modeling: Competition 2009
Richard Nauda , Thomas K. Berger a , Brice Bathellierb Wulfram Gerstner aaEcole Polytechnique Fédérale de Lausanne (EPFL), Brain Mind Institute,
Building AAB, Lausanne 1015, SwitzerlandbInstitute of Physiology, University of Bern, Buhlplatz 5, 3012 Bern, Switzerland

1. DESCRIPTION
How well are single-cell properties reproduced by the present-day neuronal models? Recently,
several labs have approached this question by assessing the quality of neuron models withrespect to spike timing prediction or characteristic features of the voltage trace. So far, every
modeler used his own preferred performance measure on his own data set. The Quantitative
Single-Neuron Modeling Competition offers a coherent framework to compare neuronal models
and fitting methods.
1.1. Participation
Participants can submit their prediction to one or more of the challenges A, B, C or D.
Anyone can participate and any type of model is accepted.
1.2. Goal
This competition is an opportunity to bridge the gap between experimentalists and modelers. The Quantitative Single-Neuron Modeling Competition is an invitation to compare yourmethods and models to those of other people in the field. Past competitions remain available
on the INCF website and will serve as benchmarks [ 5, 6]. Prizes are given to outstanding
contributions.
1.3. Prizes
The prizes are given according to the “Rules and Conditions”.
• The INCF1 Prize ( 10 000 CHF) is given to the participant(s) providing a significant win2
in at least 2 of the four challenges.
• The FACETS3 Award (500 CHF) is given to the participant providing a significant win2
in a single challenge, or shared between all participants providing a shared win2 in a single
challenge.
1.4. Important Dates• Submissions via the website will open on June 25 2009.
• August 25 is the submission deadline.
• September 6-8 is the INCF Congress of Neuroinformatics4 where results are presented.
1

http://www.incf.org/
see Rules and Conditions
3
http://facets.kip.uni-heidelberg.de/4
http://www.neuroinformatics2009.org/about/
2

1

2

Richard Naud, Thomas K. Berger , Brice Bathellier Wulfram Gerstner

2. RULES AND CONDITIONS
1. Winning conditions. The prizes are given upon the disclosure of the model. Winningparticipants who fail to provide the source code to the organizers in the time between thesubmission date and the day before the disclosure of the results will be removed from thecompetition. Once the results have been verified by the organizers and the details of thewinning model/method have been published on the competition website the organizers
will communicate with the winners to give the money prize. The INCF Prize (10 000
CHF) is payed as a honorary to the participant(s) providing a significant win in at least
2 of the challenges. The FACETS Award (500 CHF) is payed as a honorary to theparticipant(s) providing a significant win in a single challenge or shared equally between
participants providing a shared win in a single challenge. The participants keep the rightto publish their model in journals and conferences.
2. EPFL employees. EPFL employees may take part in the competition as regular participants but they cannot receive the honorary linked to the FACETS Award or the INCF
Prize. The EPFL employee who wins either INCF Prize or FACETS Award will receive
the certificate associated with his prize, his name will appear in the results, but the money
prize will not be awarded that year.
3. Shared INCF prize. It is possible that two participants win the INCF Prize, in this casethe money prize is shared equally.
4. Significant win. A performance in one of the challenges that surpasses the second best
performance in the same challenge by more than one standard error of the mean is considered a significant win. The value of performance and standard deviation is defined ineach challenge “Evaluation Methods”. There is maximum one significant win per challenge. There can be a significant win in a single challenge only if it receives at least threesubmissions.
5. Shared win. All participants in a challenge having performances that fall within one
standard error of the mean of the best performance are said to achieve a shared win. In
this case, the honorary associated with the prize is shared between the best participant
and these others. the participant with the best performance is also a shared win. In thisway there is either zero or minium two shared win per challenge. The value of performanceand standard deviation is defined in each challenge Evaluation Methods. There can be asignificant win in a single challenge only if it receives at least three submissions.
6. Collective Submission. Multiple people can collaborate to make a submission. In whichcase the first name within the collaborators is the responsible author. The responsible
author of the submission is called the participant.
7. Single Submission. A participant can only submit once per challenge.

Quantitative Single-Neuron Modeling: Competition 20093

3. CHALLENGE A
Predict the spike timing of a regular spiking L5 pyramidal cell
responding to in-vivo-like current injection.
3.1. Experimental Methods
The experiments were performed by Thomas Berger and Richard
Naud in the laboratory of Henry Markram at the EPFL. A 14-day-old
Wistar rats was decapitated and its brain was quickly transferred to aslicing chamber filled with iced artificial cerebrospinal fluid (ACSF).
300 mm thick slices of the primary somatosensory neocortex wereprepared using a HR2 vibratome (Sigmann Elektronik, Heidelberg,
Germany). Slices were incubated at 36◦ C for 45 min and left atroom temperature until recording. The ACSF contained (in mM):
125 NaCl, 2.5 KCl, 25 D-glucose, 25 NaHCO3 , 1.25 NaH2 PO4 , 2
CaCl2 , and 1 MgCl2 .
Somatic recordings were performed at 33-35◦ C with a Axopatch
200B amplifier (Molecular Devices, Union City, CA) in the current
clamp mode. Voltage traces were filtered with a 2.4 kHz Bessel filter.
The amplifier was connected to a ITC-18 acquisition board (Instrutech Co, Port Washington, NY),
which was in turn connected to a PC or Macintosh running a custom written routine under IgorPro(Wavemetrics, Portland, OR). Patch pipettes were pulled with a Flamming/Brown micropipette puller
P-97 (Sutter Instruments Co, Novato, CA) and had an initial resistance of < 4M Ω. Pipettes werefilled with intracellular solution (ICS) containing (in mM): 110 potassium gluconate, 10 KCl, 4 ATPMg, 10 Na-Phosphocreatine, 0 3 Na-GTP,10 Hepes, 30 Mannitol, and 8 Biocytin. Chemicals were
provided by SIGMA or MERCK. The liquid junction potential between the ACSF and the ICS wasaround 12 mV and not corrected for.
The current-clamp stimulus has two parts. The first part is 17.5 seconds of various stimulus waveforms frequently used to calibrate neuron models. It consists of a series of four step current with aduration of 2 seconds and an inter-step rest time of two seconds (one hyperpolarizing and 3 depolarizing steps). The steps are followed by an injection of white noise of two seconds. The white noise
injection can be used to remove the artefact introduced by the electrode while recording the voltage[ 1, 2]. The second part of the stimulus takes the remaining 42.5 seconds and is made of a simulated excitatory and inhibitory spike train. Six spike train were simulated as inhomogeneous poissonprocesses with predefined intensity as a function of time. The first three were convolved with a monoexponential decay of time constant of 2 ms while the last three spike trains were convolved with amono-expontential decay of time constant of 10 ms. The six resulting time-series were then combined
into a single time-series by weighted sum. The weights were chosen such as to drive the neuron to a
firing frequency between 5 and 10 Hz, as seen in experiments in vivo. The time-dependent intensityof the poisson process consisted of a concatenation of blocks of 300-500 ms duration each having a
constant intensity chosen randomly between 0 and 50 Hz. Both the amplitude and the duration of
the blocks were drawn from a uniform distribution.

V(t)

I(t)

3.2. Evaluation Methods
Thirteen repetitions of 60s-stimulation protocol specified above were injected while the voltage wasmeasured at the same electrode. The training set consists of the first 38 s and the test set consist

4Richard Naud, Thomas K. Berger , Brice Bathellier Wulfram Gerstnerof the last 22 s. We provide the injected current waveform in pA for both training and test phases.
For training the participants have access to the voltage trace of each repetition for the first 38 s. For
testing, the participants must predict the spike times of the remaining 22 s, in each repetition. The
spike time is defined as the time at which the voltage recording is crossing 0 mV. The measure of
the performance is based on the Gamma coincidence factor. To evaluate this quantity, we calculate
the number of coincidences Ncoinc between the spikes in the data spike train one repetition at a time
(target) and the spike train of the model submitted by a participant. This number is calculated bycounting the number of target spikes for which we can find at least one model spike within ± 4 ms.
We subtract the expected number of coincidences hNcoinc i that a Poisson spike train with the same
average frequency would give, and we divide by the number of spikes in the two spike trains:
Γi = αNcoinc − hNcoinc i
Ndata + Nmodel

where Ndata and Nmodel denote the number of spikes in the data and model spike trains, and α is afactor that normalizes the coincidence factor Γ to a maximum of 1.
α=2
1 − 2ν∆For details see references [ 4] and [ 5]. Γ = 0 implies that the prediction is not better than chancelevel. Γ = 1 implies that the prediction by the model is optimal.
For each repetition i, we calculate the Gamma coincidence factor Γi between the predicted spiketrain and the observed spike train for the last 22 s. The performance is the mean Γ divided by the
intrinsic reliability:
Nrep
1 X ΓiPA =N i=1 R
where Nrep = 13 is the number of repetitions. The intrinsic reliability, R, is the averaged coincidence
factor obtained by comparing one observed spike train with the spike train from another repetition:
Nrep Nrep

R=

X X
2
Γij
Nrep (Nrep − 1) i=1 j=i+1

The standard error of the mean is computed with the bootstrapping method. A large number(here n =10 000) of samples are generated artificially by re-sampling from the original sample
S0 = {Γ1 /R, Γ2 /R, ..., ΓNrep /R}. Each generated sample Sj is made of Nrep elements chosen randomly from the original sample with replacement. The re-sampled mean, µj , is the average of the
Γ/R within the sample set Sj . The standard error of the mean is then the standard deviation of there-sampled averages:sPn2
j=1 (µj − PA ).
ErrA =
n−1
This bootstrapping method is known to infer the standard error or confidence intervals of measures(such as mean, median, variances, ...) even when the original sample is drawn from a non-gaussian
distribution.
We say that a participant j achieves a significant win if its performance PAj is one standard deviation
(j)
(j)
(i)
above the performance of all other participants (if PA − ErrA > PA ∀ i 6= j then j achieves aQuantitative Single-Neuron Modeling: Competition 20095significant win). When there is no significant win there are multiple participactions who achieve asimilar performance. We say that a particpation makes a shared win if its performance is within one(j)(best)(best)
standard deviation of the best performance (if PA < PA
− ErrA
then j and (best) have ashared win).

3.3. Submission
Submissions are made through the INCF website. The participant must provide his prediction of
spike times for each repetition. The submission consists of a folder containing 13 files, one for each
repetition. Each file contains the predicted spike times in milliseconds (ms) since the beginning of
the repetition. Spike times need to be stored as columns in ASCII file format. The files are labeledfrom 1 to 13 and are called ‘repXX.txt’ (rep1.txt, ... rep13.txt). Submissions must comply with these
specifications to ensure that the automatic evaluation of the results is successful.

4. CHALLENGE B
Predict the spike timing of a L5 fast spiking cell responding toin-vivo-like current injection.

4.1. Experimental Methods
The experimental methods for challenge A and B are identical.

V(t)

I(t)

4.2. Evaluation Methods
The evaluation methods are identical to those of challenge A, withthe only exception that there are 9 repetitions available in challengeB.

4.3. Submission

Submissions are made through the INCF website. The participantmust provide his prediction of spike times for each repetition. The
submission consists of a folder containing 9 files, one for each repetition. Each file contains the predicted spike times in milliseconds(ms) since the beginning of the repetition. Spike times need to be
stored as columns in ASCII file format. The files are labeled from 1
to 9 and are called ‘repXX.txt’ (rep1.txt, ... rep9.txt). Submissions
must comply with these specifications to ensure that the automatic
evaluation of the results is successful.

6Richard Naud, Thomas K. Berger , Brice Bathellier Wulfram Gerstner

5. CHALLENGE C
Predict the timing of somatic spikes of a tufted L5 pyramidal cell
responding to in-vivo-like current injected in the apical dendrites andthe soma simultaneously.

Id (t)

5.1. Experimental Methods
dendrite

The experiments were performed by Brice Bathellier and Richard
Naud in the lab of Matthew Larkum at the university of Bern.
Parasagittal brain slices of the somato-sensory cortex (300-350 m thick) were prepared from 28-35 days-old Wistar rats. Slices were cut in
ice-cold extracellular solution (ACSF), incubated at 34◦ C for 20 min
Vs (t)and stored at room temperature. During experiments, slices were superfused with in ACSF at 34◦ C. The ACSF contained (in mM) 125
NaCl, 25 NaHCO3 , 25 Glucose, 3 KCl, 1.25 NaH2 PO4 , 2 CaCl2 , 1
MgCl2 , pH 7.4, and was continuously bubbled with 5 % CO2 / 95 %Is(t)O2 . The intracellular solution contained (in mM) 115 K+ -gluconate,
20 KCl, 2 Mg-ATP, 2 Na2 -ATP, 10 Na2 -phosphocreatine, 0.3 GTP, 10
HEPES,
0.1, 0.01 Alexa 594 and biocytin (0.2%), pH 7.2.
soma
Recording electrodes were pulled from thick-walled (0.25 mm)borosilicate glass capillaries and used without further modification(pipette tip resistance 5-10 MΩ for soma and 20-30MΩ for dendrites).
Whole-cell voltage recordings were performed at the soma of layer V pyramidal cells. After opening of
the cellular membrane a fluorescent dye, Alexa 594 could diffuse in the entire neuron allowing to perform patch clamp recordings on the apical dendrite 600-700 µm from the soma. Both recordings wereobtained using Axoclamp Dagan BVC-700A amplifiers (Dagan Corporation). Data was acquired withan ITC-16 board (Instrutech) at 10 kHz driven by routines written in the Igor software (Wavemetrics).
The injection waveform consisted of 6 blocks of 12 seconds. Each block is made of three parts:
1) one second of low-variance colored noise injected only in the soma, 2) one second of low-variancecolored noise injected only in the dendritic injection site, 3) ten seconds of high-variance colored noise
whose injection site depends on the block: In the first block, the 10-second stimulus is injected onlyin the dendritic site, the second block delivers a the 10-second stimulus in the soma only, and the four
remaining blocks deliver simultaneous injection in the soma and the dendrites. The colored noise wassimulated with MATLAB as an Ornstein-Uhlenbeck process with a correlation time of 3 ms. The six
blocks make a 72 seconds stimulus that was injected repeatedly without redrawing the colored noise(frozen-noise). Twenty repetitions of the 72-second stimulus were carried out, separated by periodsof 2-120 seconds. Out of the twenty repetitions, a set of seven successive repetitions were selected on
the basis of high intrinsic reliability (see “Evaluation Methods”).

5.2. Evaluation Methods
Seven repetitions of a current waveform of 72 seconds were injected in dendritic and somatic injection
site while the voltage was measured with the same electrodes. The training set consists of the first 38
s and the test set consists of the last 34 s. We provide the somatic and dendritic current waveform in
pA for both training and test phases. For training the participants have access to the voltage trace ofeach repetition in each recording site for the first 38 s. For testing, the participants must predict thesomatic spike times of the remaining 34 s, in each repetition.

Quantitative Single-Neuron Modeling: Competition 20097

The complementary evaluation methods are identical to those described for challenge A.

5.3. Submission
Submissions are made through the INCF website. The participant must provide his prediction of
spike times for each repetition. The submission consists of a folder containing 7 files, one for each
repetition. Each file contains the predicted spike times in milliseconds (ms) since the beginning of
the repetition. Spike times need to be stored as columns in ASCII file format. The files are labeledfrom 1 to 7 and are called ‘repXX.txt’ (rep1.txt, ... rep7.txt). Submissions must comply with these
specifications to ensure that the automatic evaluation of the results is successful.

6. CHALLENGE D
Time-varying
Visual stimulus

Predict the spike timing of a single post-synaptic neuron in thelateral geniculate nucleus (LGN) knowing the spike train on the presynaptic side (i.e. in a retinal ganglion cell (RGC)).

6.1. Experimental Methods

eyeExperiments are described thoroughly in Carandini et al. (2007)
and Sincich et al. (2007) . Briefly, extracellular recordings were performed in-vivo in rhesus monkeys. Epoxylite-coated tungsten electtrode was positioned 5 mm above the LGN. Extracellular potentials
recorded by single tungsten electrodes (Frederick Haer & Co., BowThalamic Neuron
Spike Train
doin ME) were amplified 1000x, band-pass filtered between 300 and
3 kHz, and digitized at 25 kHz (Power 1401, Cambridge Electronic? ?
t Design, Cambridge). Retinal EPSPs and geniculate action potentials
were identified by off-line waveform templating (Spike2; Cambridge
Electronics Design). Only LGN neurons with EPSP that exhibited an absolute refractory period were
included in the data set. Visual stimuli were restricted to the receptive field center, as establishedby manually mapping the receptive field boundaries. The light intensity of a LED was illuminating
only the receptive field center and was black elsewhere. The intensity was varied continously, with anaturalistic temporal frequency power spectrum between 0.2 and 80 Hz. Experiments used procedures
approved by the University of California, San Francisco Institutional Animal Care and Use Committee
in accordance with National Institutes of Health guidelines. The visual stimulus is 10 s long and isrepeated 76 times. The first 5 s was common across trials and the last 5 s segment was unique to thetrial.
Retinal Ganglion Cell
Spike Train

6.2. Evaluation Methods
The training set consists of all the odd repeats and the test set consist of the last 5s in all the evenrepeats. For training the participants have access to both the retinal (pre-synaptic) and geniculate(post-synaptic) spike times. For testing, the participants must predict the geniculate spike times(post-synaptic) knowing only the retinal spike times (pre-synaptic).
For each repetition of the test set, we calculate the Gamma coincidence factor (see challenge A).
The performance is the mean Gamma:
Nrep

1 X
Γi
PD =
N i=1

8

Richard Naud, Thomas K. Berger , Brice Bathellier Wulfram Gerstnerwhere Nrep = 7 is the number of repetitions.
The standard error of the mean is computed with the bootstrapping method. A large number(here n =10 000) of samples are generated artificially by re-sampling from the original sample S0 ={Γ1 /R, Γ2 /R, ..., ΓNrep /R}. Each generated sample Sj is made of Nrep elements chosen randomly fromthe original sample with replacement. The re-sampled mean, µj , is the average of the Γ/R within
the sample set Sj . The standard error of the mean is then the standard deviation of the re-sampledaverages:sPn2
j=1 (µj − PD ).
ErrA =
n−1
This bootstrapping method is known to infer the standard error or confidence intervals of measures(such as mean, median, variances, ...) even when the original sample is drawn from a non-gaussian
distribution.
We say that a participant j achieves a significant win if its performance PDj is one standard deviation
(i)
(j)
(j)
above the performance of all other participants (if PD − ErrD > PD ∀ i 6= j then j achieves asignificant win). When there is no significant win there are multiple participactions who achieve asimilar performance. We say that a particpation makes a shared win if its performance is within one(j)(best)(best)
standard deviation of the best performance (if PD < PD
− ErrD
then j and (best) have ashared win).

6.3. Submission
Submissions are made through the INCF website. The participant must provide his prediction of
the geniculate spike trains for each repetition. The submission consists of a folder containing 38 files,
one for each repetition. Each file contains the predicted spike time in milliseconds (ms) since thebeginning of the repetition. Spike times are stored column wise in ASCII file format. The files arelabeled from 2 to 76 and are called ‘repXX.txt’ (rep2.txt, rep4.txt, ... , rep76.txt). Submissions mustcomply with these specifications to ensure that the automatic evaluation of the results is successful.

REFERENCES
1. Badel, Lefort, Brette, Peterson and Gerstner, J. Neurophys. (2008). PDF
2. Brette, Piwkowska, Monier, Rudolph-Lilith, Fournier, Levy, Frégnac, Bal, Destexhe, Neuron.
(2008). Abstract
3. Carandini, Horton and Sincich, J. Vision. (2007). PDF
4. Gerstner and Kistler, Spiking Neuron Models (2002).
5. Jolivet, Kobayashi, Rauch, Naud, Shinomoto and Gerstner, J. Neurosci. Meth. (2007). PDF.
6. Jolivet, Schurmann, Berger, Naud, Roth and Gerstner, Biol. Cybern. (2008). PDF
7. Sincich, Adams, Economides and Horton, J. Neurosci. (2007).