week ending
1 SEPTEMBER 2006

PHYSICAL REVIEW LETTERSPRL 97, 098102 (2006)

Fisher Information for Spike-Based Population Decoding
Taro Toyoizumi,1,2,* Kazuyuki Aihara,1,3 and Shun-ichi Amari2
1

Institute of Industrial Science, University of Tokyo, 4-6-1 Komaba, Meguro, Tokyo 113-8656, Japan
2
RIKEN Brain Science Institute, Hirosawa 2-1, Wako, Saitama 351-0198, Japan
3
ERATO Complexity Modelling Project, JST, 3-23-5 Uehara, Shibuya, Tokyo 151-0064, Japan(Received 5 January 2006; published 30 August 2006)

We evaluate the Fisher information of a population of model neurons that receive dynamical input andinteract via spikes. With spatially independent threshold noise, the spike-based Fisher information thatsummarizes the information carried by individual spike timings has a particularly simple analytical form.
We calculate the loss of information caused by abandoning spike timing and study the effect of synaptic
connections on the Fisher information. For a simple spatiotemporal input, we derive the optimal recurrent
connectivity that has a local excitation and global inhibition structure. The optimal synaptic connections
depend on the spatial or temporal feature of the input that the system is designed to code.
DOI: 10.1103/PhysRevLett.97.098102

PACS numbers: 87.19.La

Information from the environment is encoded in the
noisy activity of a population of neurons. Reading theneural code is a fundamental problem in neuroscience. In
particular, attention has been paid to the role of precise
spike timing in addition to the spike count information[1,2] and that of correlation between neurons [3–5] on
the information coding. However, a large amount of data
is required to evaluate the amount of information in a real
population of neurons [1,2]. Moreover, it is difficult, withthis approach, to understand the role of neuronal parameters such as recurrent connectivity on the information
coding. In this study, we calculated the Fisher informationof a network of spiking neurons, which limits the accuracyof any unbiased estimate of a stimulus [6,7]. In contrast to
the literature, where Fisher information is evaluated basedon firing rates (rate-based Fisher information) [3–8], we
evaluated the Fisher information when the individual spiketimings of all the neurons were available (spike-based
Fisher information). Interestingly, under the assumptionof independent noise, the expression of the spike-based
information has a simple analytical form. We estimated theamount of information included in the precise timing of
spikes by comparing the information in the two cases andstudied the role of synaptic connectivity on stimulus estimation. We also calculated the spike-based information fora spatiotemporal input, where rate decoding fails, andderived the optimal recurrent connectivity for spike-based
information representation.
Stochastic firing neuron model. From the noisy spiking activity of N recurrently connected neurons in time
duration T, we decode a parameter  of a stimulus, whichis, for example, the orientation of a light bar. Neuron ii 1; . . . ; N receives stimulus-dependent external input potential hi t;  and recurrent spikes from neighboring neufrons. A spike from neuron j at time tj j fj  1; . . . ; nj 
provokes a postsynaptic potential of amplitude wij and
f

fj

f

time course t  tj j   ettj =m t  tj j , where0031-9007=06=97(9)=098102(4)m  10 ms is the leak time constant and  is the
Heaviside step function which takes the value 1 for apositive argument and 0 otherwise. Hence, the total input
potential of the neuron is described by ui t  hi t;  
PN
Pnj
fj
j1 wij fj 1 t  tj . Depending on ui , the neuron
emits a spike with instantaneous firing probability density
i t  gui tRt  t^i ,
wheregu  gM 1 euuc  1 is a sigmoid function with parameters gM t^i
is a
500 Hz,   8, and uc  1, and Rt  t^i   rttt^irefractory factor with the last spike time, t^i , of neuron i andrefractory time constant r . Note that we sometimes use
the abbreviation gi t  gui t. Because u is not reset
after an output spike, the refractory factor may correspondto the integration time until the next spike for a noisy leaky
integrate-and-fire model. The current model is a special
case of the spike response model [9]. We assume conditionally independent instantaneous firings of neurons given
the history of spikes and the input, neglecting common
noise shared among neurons for simplicity.
Spike-based and rate-based information. The output spike train of neuron i is represented by xi t Pnifi
fi 1 t  ti , where  is the Dirac delta function. The
history of the spike activity until time t is described byXt  fxi t0 ji  1; . . . ; N; 0  t0 < tg. Given a realization of history Xt, the spiking probability of the next
very short interval
by using the binary variR t is written
xi t0 dt0 , whereables xi t  ttt
R we formally define
the integral of a delta function by 10 t0  tdt0  1 for
0  t < 1 and 0 otherwise. Because we assume instantaneously independent firing, the spike probability is given asa product of individual
firing probabilities, i.e.,
QN
jXt
Px
Pfxi tgNi tjXt, with the indii1i1vidual binary firing probabilities Pxi tjXt i ttxi t 1  i tt1xi t . However, each conditional firing probability depends on the past spike trains

098102-1© 2006 The American Physical Society

2

0 2

 
2
2
2
2
T Cv
2 for large T with Cv  hs iQ  hsiQ =hsiQ . All
these spike statistics are derived Rfrom the moments
generating
function,   1
0 Qse ds  1 
r eg r g  r 1r g 1  r g; g  r , wheretheincomplete
gammafunction 1 ; 2  
R1
1
s
1
s
e ds is used. Particularly, we find hsiQ 2r er g r g1r g 1  r g; r g and Cv2  21 
r =g  1. Figure 1(a) shows g, , Cv, and information
rates J_ spike and J_ rate of a single neuron. The Fisher information first increases with  (cf. Equation (2)) but de-

(a)

(b)
0.8
0.6
0

100

25

1

[Hz]

150

20
0.5

1

h

J/T

i t;
, g0i t  @gwhere h0i t  @h@
@u juui t , and the mean firing rate is i t  hi tiXt . Equation (2) shows the relations of the Fisher information rate to the number ofneurons, N, to the mean firing rate, , and to the squareof the derivative of the input potential with respect to theparameter . Interestingly, the spike-based information of
Eq. (2) does not explicitly depend on correlations between
neurons. We should note, however, that the mean firing
rates of neurons are modulated by synaptic connections
between them. This kind of spike-based information has
been investigated for a single neuron [1,10]. The amount of
information lost by applying rate decoding and abandoning
individual spike timing is an issue of interest [2]. However,
several previous studies on population coding assumed
only the rate decoding paradigm [3–8]. We restricted
ourselves to a simple case of a constant stimulus for afair comparison and investigated the role of model parameters on the information loss.
Let us introduceRsome notation for rate decoding. We
describe by ri  T1 T0 xi tdt the spike rate in the interval.
Therate-based
Fisherinformation,
Jrate 2
N2
h@ logPfri gi1 =@ ifri gNi1 , is calculated based on theassumption that ri is generated from a Gaussian distribution with mean i  hri i and covariance matrix Cij hri  i rj  j i. Because Cij 1=T for large T,PN@i
0 1 001
Jratei;j1 i Cij j , where i  @ and Cij is the0R[12], where the average means h iQ  Qsds. Hence,
the rate-based Fisher information is written as Jrate 

15

10

50

[Hz]

(2)

0

2  hs iQ hsiQ  for large T
T
hsi2Q

5

11
10
9
8
7
−3

−2

−1

0

1

2

3

25
20
15
10

0
0

0.5

h

1

0
0

50

[Hz]

100

J/T

h0i t2 i t;i12    1=hsiQ and varianceJ/T [Hz]

Except for the average over the spike history, Xt, theabove result is well known as the Fisher information rate ofindependent Poisson neurons [7]. The total spike-based
information from spike trains of duration T is given as anintegration of the information rate, i.e., Jspike R
T _ spike
dt. Particularly, for a low firing rate (if ggM
0 J
always holds), we can approximate the sigmoid g by an
exponential function gM euuc  . This approximation gives


N 
X
g0 t 2
h0i t i
i t
J_ spike t gi t
Xt
i1sinterval (ISI) distribution, Qs  gRseg 0 Rs ds [9],where s represents an ISI and the last spike time is set tos^  0. The ISI distribution is controlled by the input
strength, h, and the refractory time constant, r . For any
well-behaved renewal process, the probability distributionof spike rate Pr is asymptotically Gaussian with meanCv

J_ spike t  lim Jspike t  t  Jspike t=t
t!0

1 @2 logPfxi tgNi1 jXt  limt!0 t@2
XttXN
1 @i t 2

:
(1)
@
Xt
i1 i t

(i, j) component of the inverse of the covariance matrix[11]. Note that the Gaussian ri assumption is generally notvalid for recurrently connected networks, while we do notneed this assumption for the calculation of spike-based
information.
Independent neurons. We consider first a single neuron, or equivalently, a population of neurons without recurrent connections. A neuron receives constant input,h  0:1, and hence has constant g  gh. Because theneuron has the renewal property with constant drive, its
spike train is completely characterized by the
R interspike

Rate [Hz]of all neurons through . The Fisher information, given
spike trains of duration t, is defined by Jspike t 
h@2 logPXt=@2 iXt , where h iXt is the averagewith respect to PXt. Applying PXt  t Pfxi tgNi1 jXtPXt, we define the Fisher information rate byNX

week ending
1 SEPTEMBER 2006

PHYSICAL REVIEW LETTERSPRL 97, 098102 (2006)

5
−1

−0.5

0

0.5

1

w

FIG. 1 (color online). (a) Left: The sigmoid function, g (solidline), its approximation by an exponential function, gM euuc (dashed line), and the firing rate,  (dotted line). The inset shows
the coefficient of variation, Cv. Right: The spike-based and ratebased information rates J_ spike (solid line) and J_ rate (dotted line),for different . The refractory time constant, r , is 10 ms. An
approximation with the exponential g is also shown (dashedline). (b) Top: Comparison of the spike-based information rate,
J_ spike (solid line), and the rate-based one, J_ rate (dotted line), withthe sigmoid g and refractoriness, r  10 ms. Bottom: The same
comparison with the exponential g and without refractoriness.
We set   5.

098102-2

PRL 97, 098102 (2006)

week ending
1 SEPTEMBER 2006

PHYSICAL REVIEW LETTERS

creases for high  due to the sigmoid of g. The two
information rates are similar with constant input but havefinite differences due to the refractory factor, i.e. the observer can fix the value of Rt  t^ by knowing the last
spike time, t^. The deviation of the sigmoid of g from theexponential function and the effect of refractoriness are
negligible at low firing rates. In the following, we assume
r  0 for analytical calculations.
Effect of synaptic connection. To see the effect ofrecurrent connections on the information, we consider two
neurons with a one-way synaptic connection and driven by
constant input. The Gaussian ri assumption is also valid inthis case for large T. For the analytical calculation, we
neglect refractoriness and use the exponential functiongu gM euuc  . We investigate the effect of sigmoid
g and refractoriness by numerical simulation. Two neurons
receive constant inputs h1  0:1 and h2  0:1, and theoutput of the first neuron is fed into the second neuron withsynaptic strength w  w21 . The calculation of spike-based
information is straightforward from Eq. (2). We calculated
the two neurons’s mean firing rates and their covariancematrix to evaluate the rate-based information. Becauser  0, their firing probabilityRdensities are given by 1 gh1  and 2 t  gh2  w t0 t  t0 x1 t0 dt0 . Using
the fact that x1 is a Poisson spike train and g is an exponential function, we can average 1 and 2 over the spikehistory, Xt, and find the mean firing rates 1 
1 Rgh1  and 2  2  gh2  1 K1 , whereKtdt with t  ewt  1. The covariancematrix, C, is calculated as C11  1 =T, C12  C21 1 2 K=T,R and C22R  2 =T  22 L=T, where L 11 wt0 R
t0 dt0 1ejtjt0 dt00
dte 1 jtj
 1. In this way,
we can also calculate the rate-based Fisher information.
Taking the limit as w ! 1, we find Jrate ! Th1 2 1 ,which is the amount of information from neuron 1. Hence,
the output of neuron 2 does not add any additional information at this limit. Signal h2 is buried in the noisy postsynaptic potential of neuron 1 for large w. On the other
hand, within the spike decoding paradigm, neuron 2 canstill transmit information on h2 because the observer
knows when neuron 1 fired. The rate-based and spikebased Fisher information is compared for a range of thesynaptic strength w in Fig. 1(b). The spike-based Fisher
information is significantly larger than the rate-based information for strong synaptic connectivity. This result isqualitatively similar even when we use the sigmoid g andrefractoriness. One difference is that the spike-based information shows unlimited growth with the exponential gbecause g02 2 =g2 is an increasing function of w. However,the spike-based information decreases for large w with thesigmoid g.
Spatiotemporal input. We are most interested in seeing how the correlation among a population of neurons
alters the efficacy of stimulus estimation [3–5]. The effectof recurrent connectivity on stimulus estimation has beenstudied mainly within the rate decoding framework[13,14], where spatially structured synaptic connections
with local excitation and global inhibition are used.
Moreover, calculating the rate-based Fisher information
with recurrent connections is computationally hard because it requires the inverse of the output covariancematrix. In the following, we apply the spike decoding
paradigm by taking account of individual spike timings,
and calculate the spike-based Fisher information to estimate stimulus onsets or locations. The optimal recurrent
connectivity that maximizes the spike-based information isderived under certain conditions on synaptic strength.
We consider a population, N  103 , of recurrently connected neurons evenly distributed with density
N=2 on a circle  < z  , where z refers to apreferred stimulus of a neuron. The synaptic strengthfrom a neuron at z0 to one at z is a function of only thedifference between their preferred stimuli,   z  z0 ,which is described by w. As an example of a spatiotemporal stimulus, we consider a localized pulse centeredat z~ injected at time ~t. The input potential to a neuron at z is
described by a differential equation, m dhz;tdt  hz;t ~~As 1  cosz  z~t  ts  t  t, where the stimulus strength, As , is 1=4 and the stimulus duration, s , is
100 ms. The stimulus location, z~, and the onset time, ~t, arethe parameters to be estimated from the output spikes. The
information, Jzspike and Jtspike , for estimating z~ and ~t iscalculated from Eq. (1) by Rsubstituting  z~ and   ~t,R002
respectively, i.e. Jspike  T0 dt dzhhi z; tgi t =gi tiXT with h0 z; t  @hz;t@ . Assuming a small synaptic
strength, we calculate the optimal synaptic strength, w, thatmaximizes Jspike under constraints on the first two moments of synaptic strength; Ri.e. excitatory and inhibitory synapses are balanced,  w d  0, and thesquarenormofthesynaptic
strengths is restricted,
R
2
1
w

d

N
.
Hence,
thefunction is
R
Robjective
 w2  dL  Jspike  1 
w
d


with
2 

Lagrange multipliers 1 and 2 . We calculate the spikebased Fisher information up to the first order of w. The
zero-order term is given by the spike-based information of
spike
independentneurons,jw0 .R The first-order term isRTR J
dtz;tz;tdw
dt0 t  t0 0 z  ; t0 ,00
where the zero-order instantaneous firing rate,
0 z; t  g0 z; t, is ghz; t, and the coefficientz; t  h0 z; tg00 z; t=g0 z; t2 2g000 z; t=g00 z; t 
g00 z; t=g0 z; t. The optimality condition is represented
L
 0. This leads to theby a functional differentiation, w
optimal synaptic strengthZ
ZT Zw /
dt dt0dzz;tt  t0 0 z;t0 z


0

0

 ;t (3)

up to a constant. The constant and the proportionality
coefficient in Eq. (3) depend on 1 and 2 , which are set

098102-3

PHYSICAL REVIEW LETTERSPRL 97, 098102 (2006)
(a)

(b)

−3
2 x 10C

1.25

Improvement [%]

14

1.5
1.2

1

J tspike

w

0.5

1.15

0

−0.5

1.1

−1
−1.5
−2
−0.5

−0.25

0

ζ [cycle]

0.25

0.5

1.05
305 310 315 320 325 330 335 340

J zspike

12

10

8

6

J zspike

J tspike

FIG. 2 (color online). (a) The optimal synaptic strengths
wlocation (solid line) for the estimation of the stimulus location,z~, and wonset (dashed line) for the stimulus onset, ~t. (b) The spikebased Fisher information, Jzspike for the estimation of z~ and Jtspikefor the estimation of ~t, is plotted for three types of synaptic
connectivity: no synaptic connection (w  0, cross), wlocation(plus sign), and wonset (asterisk). (c) Comparison of the improvement when having wlocation (solid line) and wonset (dashed line)from having no synaptic connectivity. The increases in the spikebased Fisher information, Jzspike and Jtspike , are plotted.

to satisfy the constraints on the synaptic strength.
Moreover, Eq. (3) has the causal Hebbian property.
Equation (3) is proportional to the causal pre- and postcorrelation t  t0 0 t; z0 t0 ; z  , whose learning
coefficient is given by z; t to enhance the stimulus
difference. The optimal recurrent connections to estimatez~ and ~t are described by wlocation and wonset , respectively,
and shown in Fig. 2(a). Both wlocation and wonset have local
excitation and global inhibition structures, which are observed in various neural systems. However, the precise
shape of w depends on the feature of the given stimulusthat the system is designed to code. In Fig. 2(b), theinformation Jzspike and Jtspike is plotted for three types ofsynaptic connectivity: no recurrent connection (w  0),
wlocation , and wonset . The improvement in the information,
Jzspike and Jtspike , due to having synaptic connections
wlocation (solid line) and wonset (dashed line) is plotted inFig. 2(c). The improvements are limited because the synapses are assumed to be small for the analytical calculation. Sharper tuning curves are known to be desirable for
one-dimensional stimulus estimation with independent
Poisson neurons [15]. This sharpening of tuning curves
corresponds to the one by a sharper input, h, without
recurrent connections. On the other hand, sharpening oftuning curves by recurrent connections does not always
lead to a better estimation [13,14]. The result shows thatthe optimal recurrent connections are determined by thesubtle interplay between the input stimulus and the response properties of neurons. The optimal recurrent connections provide, by modulating the recurrent input to eachneuron, an efficient encoding of the stimulus parameterthat minimizes the information loss because of the noisy
spike-generation process. As is in the case of Seriès et al.,we can not increase the information by postprocessing astimulus with a network of recurrently connected neurons[13]. Evaluating the spike-based Fisher information, weweek ending
1 SEPTEMBER 2006

can identify the information in individual spike timing thatcodes temporal features of input, such as stimulus onset.
This kind of temporal feature is difficult to decode fromspike rates with wide temporal windows.
We calculated the spike-based Fisher information of a
population of recurrently connected spiking neurons withindependent threshold noise. Unlike the case of [16], theFisher information is expected to give a reasonable lower
bound of the estimation error because the estimation isbased on a large number of independent spikes.
Generalizing the spike-based Fisher information to estimate a multidimensional parameter is straightforward.
Although we restricted ourselves to weak synapses forthe calculation of the optimal recurrent connections,stronger synapses should also be studied to elucidate therole of nonlinearity in output firing rates and sustained
activity in networks.
This work was supported by the Japan Society for the
Promotion of Science, and a Grant-in-Aid No. 1806772
from JSPS and for Scientific Research on Priority Areas
No. 17022012 from MEXT of Japan. We thank K. Kang,
N. Masuda, and K. Hamaguchi for discussions.

*Electronic address: taro.toyoizumi@brain.riken.jp[1] F. Rieke, D. Warland, R. de Ruyter van Steveninck, and
W. Bialek, Spikes-Exploring the Neural Code (MIT Press,
Cambridge, MA, 1996).
[2] S. Panzeri, R. Peterson, S. Schultz, M. Lebedev, and
M. Diamond, Neuron 29, 769 (2001).
[3] H. Yoon and H. Sompolinsky, in Advances in Neural
Information Processing Systems 12, edited by M. I.
Jordan, M. J. Kearns, and S. A. Solla (MIT-Press,
Cambridge, 1998), pp. 167–173.
[4] L. F. Abbott and P. Dayan, Neural Comput. 11, 91 (1999).
[5] S. Wu, S. Amari, and H. Nakahara, Neural Comput. 14,
999 (2002).
[6] M. A. Paradiso, Biol. Cybern. 58, 35 (1988).
[7] H. Seung and H. Sompolinsky, Proc. Natl. Acad. Sci.
U.S.A. 90, 10 749 (1993).
[8] N. S. Harper and D. McAlpine, Nature (London) 430, 682(2004).
[9] W. Gerstner and W. K. Kistler, Spiking Neuron Models(Cambridge University Press, Cambridge UK, 2002).
[10] C. van Vreeswijk, Neurocomputing; Variable Star Bulletin
38– 40, 417 (2001).
[11] S. M. Kay, Fundamentals of Statistical Signal Processing:
Estimation Theory (Prentice-Hall, Englewood Cliffs, NJ,
1993).
[12] D. R. Cox, Renewal Theory (Methuen, London, 1962).
[13] P. Seriès, P. E. Latham, and A. Pouget, Nat. Neurosci. 7,1129 (2004).
[14] M. Spiridon and W. Gerstner, Network: Computation in
Neural Systems 12, 409 (2001).
[15] K. Zhang and T. J. Sejnowski, Neural Comput. 11, 75(1999).
[16] M. Bethge, D. Rotermund, and K. Pawelzik, NeuralComput. 14, 2317 (2002).

098102-4