Biologically plausible solutions for spiking networks with efficient coding
Authors: Veronika Koren
Stefano Panzeri
Venue: NeurIPS 2022 Accept (2022)
Keywords: spiking neural networks
optimization
loss function
information
population code
E-I network
Generalized leaky integrate-and-fire (LIF) neuron
biological constraints

Abstract: Understanding how the dynamics of neural networks is shaped by the computations they perform is a fundamental question in neuroscience. 
Recently, the framework of efficient coding proposed a theory of how spiking neural networks can compute low-dimensional stimulus signals with high efficiency. Efficient spiking networks are based on time-dependent minimization of a loss function related to information coding with spikes. To inform the understanding of the function and dynamics of biological networks in the brain, however, the mathematical models have to be informed by biology and obey the same constraints as biological networks. Currently, spiking network models of efficient coding have been extended to include some features of biological plausibility, such as architectures with excitatory and inhibitory neurons. However, biological realism of efficient coding theories is still limited to simple cases and does not include  single neuron and network properties that are known to be key in biological circuits. Here, we revisit the theory of efficient coding with spikes to  develop spiking neural networks that are closer to biological circuits. Namely, we find a biologically plausible spiking model realizing efficient coding in the case of a generalized leaky integrate-and-fire network with excitatory and inhibitory units, equipped with fast and slow synaptic currents, local homeostatic currents such as spike-triggered adaptation, hyperpolarization-activated rebound current, heterogeneous firing thresholds and resets, heterogeneous postsynaptic potentials, and structured, low-rank connectivity. We show how the rank of E-E connectivity matrix shapes network responses.

Biologically plausible solutions for spiking networks with efficient coding
Veronika Koren
Department of Excellence for Neural Information Processing
Center for Molecular Neurobiology (ZMNH)
University Medical Center Hamburg-Epppendon (UKE)
Falkenried 94, 20251 Hamburg. Germany korknnukkee de
Stefano Panzeri
Department Excellence for Neural Information Processing
Center for Molecular Neurobiology (ZMNH)
University Medical Center Hamburg-Eppendde (UKE)
Falkenried 94, 20251 Hamburg, Germany
Istituto Italiano di Tecnologia Genoa, Italy
8 ppanzeriuuke.o de
Abstract
Understanding how the dynamics of neural networks is shaped by the computations they perform fundamental question neuroscience. Recently, the framework of efficient coding proposed theory of how spiking neural networks can compute low-dimensional stimulus signals with high efficiency. Efficient spiking networks are based on time-dependent minimization of a loss function related to information coding with spikes. To inform the understanding of the function and dynamics of biological networks in the brain, however, the mathematical models have informed by biology and obey the same constraints as biological networks. Currently, spiking network models of efficient coding have been extended to include some features of biological plausibility, such as architectures with excitatory and inhibitory neurons. However, biological realism of efficient coding theories is still limited to simple cases and does not include single neuron and network properties that are known be key in biological cireuits. Here, revisit the theory efficient coding with spikes to develop spiking neural networks that are find biologically plausible spiking
closer to biological circuits. Namely, model realizing efficient coding in the case of generalized leaky integrate-and fire network with excitatory and inhibitory units, equipped with fast and slow synaptic currents, local homeostatic currents such as spike-triggered adaptation, hyperpolarization-aaciiaaae rebound current, heterogeneous firing thresholds and resets, heterogeneous postsynaptic potentials, and structured, low-rank connectivity. We show how the complexity of E-E connectivity matrix shapes network responses.
Introduction
Complex biological systems are thought to be shaped through evolution to perform optimally of desired functions within the constraints of their biological features. A bulk of previous influential work 34 25
has proposed that the organization of the mammalian cortex, the most sophisticated
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

biological organ, makes no exception and follows too the general principles of evolution towards optimality under biological constraints.
One prominent theory is that the function of the cerebral cortex is organized for efficient coding, in that it optimizes the encoding of information under a number of biological constraints such as the metabolic expenditure, the speed and the timing of information coding. Imposed constraints can be formalized with loss function.
To understand biological neural networks that operate in animal's brains, the mathematical description of networks optimizing functionally relevant loss functions has to include many features of biolog- ical realism as possible, so that the constraints satisfied by the model network will be informative of those obeyed by biological neural circuits. Thus, all elements of the model should have counterpart in biological circuits 15] and be directly relatable to them. It is still unclear how to derive efficient coding theories of biologically plausible neuronal networks.
An influential proposal of efficient coding in cortical circuits has been that visual cortical circuits extract statistical regularities from static natural images with a sparse set of basis functions, which yielded basis functions that resemble receptive fields in primary visual cortex (VI) 34 35). A
related proposal in the domain of auditory processing yielded auditory circuits that extract statistical regularities of natural sounds with a set of basis functions that resemble the tuning properties of auditory nerve fibers (25). In these studies, the goal of efficient coding is to learn sparse spatial or temporal filters that enables optimal information encoding about the stimulus by the activity of each single neuron. Analyses of cortical networks, however, show that neurons are recurrently connected and that information processing tends to he distributed across neurons [8]. In recurrently connected circuits, ceuroonto-neuron interactions typically strongly impact on the information encoded in the spiking activity [43] as well as on how this information is transmitted to and read out by downstream structures produce relevant behavioral outputs [44 48 377. Moreover, analyses of spike trains recorded in the cerebral cortex show that millisecond precision of spike timing of sensory neurons carries information about both static 40 36) and dynamic stimuli (20 32] which suggests that the temporal information in spiking activity is highly relevant for the information processing in the cortex. Thus, analyses of empirical neural data strongly suggest that biologically relevant theories of coding in the cerebral cortex must rely on spiking neuron models and on population codes.
To address this issue, models of how to optimally perform given coding function have been extended to spiking dynamics (3. Models of efficient coding with spikes are developed from a loss function that includes quadratic error between the desired signal and its estimate (see the Supplementary material). Equations for the subthreshold dynamics of membrane potentials and firing thresholds are analytically derived by assuming that the error is minimized with every spike 55. However, the efficient coding with spikes, and the related theory of predictive coding [51), have been so far developed as abstract mathematical frameworks that lack biological realism. Namely, these models are not consistent with the observation that a single neuron can be either excitatory or inhibitory. but not both (Dalc's principle, [TT 19). Very recently, progress has been made in this direction by deriving efficient coding with spiking networks with a more biologically plausible neural circuit architecture with interacting excitatory (E) and inhibitory (I) neurons (26). opening the possibility to model neural computations with biologically relevant models.
Here, we make major further progress by deriving a mathematically well-defined extension of efficient coding for considerably more biologically plausible spiking neural network models of cortical circuits. We develop recurrently connected neural circuit with E and I neurons that performs generic linear transformation between its inputs and its outputs. We impose the E-I architecture and recover a network with low-rank and structured connectivity that maximizes the information transmission about the stimulus with every spike. Here, we derive a solution of the spiking network model from optimality principles that includes many desirable and biologically realistic properties. Our solution takes the form of a biologically plausible generalized leaky integrate-and-fire (LIF) neuron model and satisfies the principle of functional specificity of excitatory and inhibitory neurons (Dale's principle). Our solution also prescribes faster membrane time constant of compared to E neurons, compatible with the biophy properties of pyramidal neurons and fast-spiking inhibitory interneurons. Our derivations prescribe subthreshold dynamics, firing thresholds and resets after spiking, thus giving complete description of a spiking neural network along with network's computation. Importantly, all elements of the model are biologically plausible and directly relatable to empirically measurable currents in neurons.
2

---

The paper is structured as follows. We first introduce the loss functions and explaining how the spike timing conditioned on minimization of loss functions of and neurons in every time step. We then outline the maaltiical treatment of loss functions that leads to expressions of membrane potentials and firing thresholds of a generalized LIF network model with structured, low-rank connectivity. Further, we provide description and biologically plausible interpretation of derived currents. Finally, we simulate the network with biologically plausible parameters and analyze the behavior of the network with respect to the complexity of E-E synaptic connections.
2 Results
2.1 Minimization of the loss function with spiking neurons
From a spiking network of N excitatory and N' inhibitory neurons, we define a linear population readout of the spiking activity of and neurons:
1 +1, + Wr
(1)
-
E {E,1},
(1)
Ty
with Ty the time constant of the population readout. In the population readout, the decoding matrix Wy weights the vector of spikes (t) = (t)...... f%, (t)]T. The M-by-N, matrix Wy, associates each of the NE excitatory (N, inhibitory) neurons with M weights, W,, (w%a)) M, i= I...., Ny. where N₂ is the number of neurons of the cell type y, and M is the number of stimulus features estimated by the network. The population readout in eq. (I) is an estimate of the desired signal (t), that is not directly accessible to the network.
We assume that the desired signal x=() is linear funetion of stimulus features s(t) [s1 (t)......., Sm St. (t)] and is defined as follows:
(t)) Ax(t) + s(t,,
where A = (amn): m. n= 1, I.., M is a square matrix determining the linear transformation between the input features s() and the desired signal æt)). We will refer to the transformation between the features and the desired signal as network's s computation. If A is a diagonal matrix, network's computation operates independently across features, while non-diagonal matrix A implements linear mixing of features.
We hypothesize that the objective of the excitatory network is to minimize the distance between the desired signal and the population readout of neurons, while the objective of the inhibitory population is to minimize the distance between the readout of the E and I populations B 7 Besides the coding error, we also take into account the metabolic cost on spiking. since spiking activity in the brain is energetically expensive (33). These objectives are formalized by two loss functions with quadratic coding error and a quadratic regularizer [16], relative to and cell type:
N
Lt(t)=
Σ m=1
(Im (t) - (t))² + μe i=1
(r E
(t))2
(3a)
Ni
(, Σ (a) i4) + pr > (r())))
(3b)
where constants JE > are weighting the relative contribution of the metabolic cost over the coding error. The low-pass filtered spike train of the neuron is defined
-
y E (E,
with T. the time constant of the low-pass filter. Gathering the variables r (t) across neurons, we define a vector of time-dependent neural activities (t)(t) = (t), ... 'N 1T. Note that an
alternative to the loss function in eq. (3b) would be to minimize the distance between the readout of the population and the signal. However, such a formulation, without further approximations, leads to network where E and I neurons are unconnected.

---

We assume that neuron i of cell type y will fire spike at time t ++ if and only if this decreases the loss function. i.c.
Ly [+]
< (F1
(5)
with the noise term (1) (t). Gaussian random variable (!) has zero mean and covariance (E,(t) (t)) 6t - t',, and o. is the noise intensity. Contrary to previous approaches where the noise is added in the membrane potential 3 23 we here assume noise in the condition for spiking (cq. 5). While the noise in the membrane potential models synaptic inputs that are unrelated to the coding function of the network, the noise as in eq. | captures the noise in spike generation for a given membrane potential, source of noise that characterizes spike generation in biological neurons [2]. In biological neurons, spike is initiated when a sufficient fraction of sodium channels in the neural membrane is opened, and the fraction of open channels for given membrane potential is probabilistic.
Assuming that a spike in neuron at time 1+ is only fired if this decreases the loss function in eq. 5 we apply an analytical treatment of eqs. Ba 3b similar to 23 and obtain the following expressions:
wh (xx(1) - (1)) - pEr, B (1)
1 +E) + t (t)
2
(6)
F (t (b) - (tr
(Iluo 112 + (1) + E
Σ M 1= (w mi) ) Since we assumed threshold crossing, we interpret the left- ceft-hand side of eq. 66 as the membranc potential and the right- hand side as the firing threshold of the neuron i. Note that the firing threshold has deterministic and stochastic part. Collecting membrane potentials across neurons, (t) (t), ................................ "x,
with ||w the squared length of decoding vector of the neuron w
with y {E, 1). we express them with vector notation as follows:
u (t) W (=(t)= Et(()) NETE(()
(7)
uy (t) = W7 (mx(t) - (t)) - (t))(t).
(e,(t)),;(1))
To express the temporal dynamics of the membrane potentials, (1) and (1), we take the time- derivative of all time-dependent terms in eq. (6). Contrary to previous approaches and without loss of generality, we express the mixing matrix as a difference of square matrix B and diagonal matrix Àg MMM.
A = B AgI MzM
with I the identity matrix. Seen that loss functions minimize the distance between the signal and the E estimate (eq. Bab. and between the E and the I estimates (eq. 3b). use the approximations (t)) ss &g (t) and (t) se (t). Moreover, we remove synaptic connections that violate Dale's law (see the Supplementary material). After re-arranging the terms and without further approximations,
interestingly, the network model can be expressed as a generalized LIF model with structured connectivity.
2.2 Generalized siegrate-and-fire network with low-rank structured connectivity
Due to substitutions of the signal and the estimates, we write the membrane potential as u (t) RJ V (t). The subthreshold dynamics of E and neurons is then dependent on the following currents:
TE (7 -v, (t) (1))) + T (1) 1. (1) plocal E (t) + I.
r.cbound (t),
(9a)
Ty (t) = - + 7, - (t) + Hloal (()) (t),
(9b)
and complemented with fire-and-rese rrsett rule: if Vt > (t(t) (+(tt) Vreae For simplicity, we set the resting potential of all neurons to Vrest O, but a biologically plausible resting potential can be introduced in eqs. (b) without affecting network's 's dynamics. From eq.(6), the firing threshold that takes into account the noise in spike generation (eq. 55 is the following:
1 2 (My +
(t)
| (1 + (t),
y (E
(10)
while the reset potentials for E and cell types are

---

(UE - 1100, (
(11)
1 (ui + | (2]
Vi
The firing thresholds in eq. 10 and reset potentials in eq. 1 are thus proportional to the length of decoding vector of the local neuron ||w ||,, and to the regularizer Hy that affects equally all the neurons of the same cell type. Note that if decoding weights w are heterogeneous across neurons, firing thresholds and resets are heterogeneous as well.
The membrane equations in eqs. 9a 9b depend on a number of currents that, interestingly, all have a straightforward counterpart in biological networks. The first currents the right-hand site of eqs. 9a 9b are leak currents, that result from absorbing of several terms, among others the diagonal matrix Ag M from eq. 88 The feedforward current E neurons is given by weighted sum of external inputs (t),
M
(1)))
TE
Σ
w ms
Sm (t)-
(12)
m=
Inhibitory synaptic currents to the postsynaptic neuron are given by weighted sum of spikes of presynaptic neurons,
N (1) - -I Σ rf
(1=(()= --
ET Σ C T(),
(13)
AA
with structured connectivity matrices C where the strength of synaptic connection is proportional the similarity of decoding vectors of the presynaptic neuron j and the postsynaptic neuron
- (w,))
(w 0
- 0,
(14)
otherwise.
Note that synaptic connections between neurons with different selectivity (i.e., neuronal pairs with negative dot product of decoding vectors, (w") < 0). have been set to zero to make the network consistent with Dale's ' law (see the Supplementary material). particular, we had to ensure that particular neuron can only send excitatory or inhibitory currents to other neurons, but not both. Even though there no direct fast synaptic connections between E-to-E neurons, effectively, fast synaptic connectivity implements lateral inhibition between E neurons with similar selectivity. Lateral inhibition (or competition) between E neurons with similar selectivity is a dynamical effect that is essential for efficient neural code [21 and has been demonstrated in biological circuits (8).
While local inhibitory synapses have fast kinetics of spike trains (eq. 13
recurrent excitatory synaptic
currents are slower since they are convolved with the synaptic filter 2 (t):
NR
F.
EE (t) 3j
TE
j=1
j#1
Ng
N Σ D!F
(15)
[[((1) = TF Σ
II (F(() (1)+
TE
1)
TE > TI
TI
j=1
j=1
(t = syn
E ( (t) fF())
TE = TE.
E
with matrices of synaptic interactions:
DuF
(w
Bw if Tw > 0,
B positive semi-def.,
(16)
0,
otherwise.
where y = F. for E-to-E synapses and y = for E-to-T synapses. Same as with fast synaptic currents in eq. 13 the strength of the synapse between the presynaptic neuron and the postsynaptic neuron
5

---

i in eq. (16) depends on the similarity of decoding vectors of the presynaptic and postsynaptic neurons, (wx)) and w.. To make slower synaptic currents consistent with Dale's s law, we removed connections between neurons with different selectivity, i.e., connections for which the following is true: (un²) Tun < O (see eq. 16). In addition, slower synaptic currents depend on the matrix that was expressed from the mixing matrix A (see eq. (). To ensure that presynaptic excitatory neurons always cause an excitatory current in the postsynaptic neuron, we constrained the matrix B to positive semi-definite. Moreover, the excitatory effect of an E-to-I synapse constrains the following relation of time constants: Tу < Tg. Since TE and Ty are membrane time constants of the and cell type, respectively, this relation constrains the membrane time constant in E neurons to be slower than in I neurons, consistent with E cell type modeling pyramidal cells while the I cell type models fast-spiking interneurons (30] Fast-spiking (somatostatin) interneurons are a prevalent class of inhibitory neurons in cortical circuits 47 As pyramidal neurons account for about 80 % of cortical neurons, and fast-spiking interneurons account for at least half of inhibitory neurons in the cortex [49] our model altogether provides a description for about 90 % of cortical neurons. Note that alternative solutions for slow currents can be formulated (see the Supplementary Material). These alternative solutions are, however, of lesser biological relevance since they either result in a network without E-to-E connections, or lead to global imbalance of E and currents in the network. Contrary to fast connections that are expected to implement lateral inhibition among E neurons with similar selectivity (see eq. 13). slower E-to-E synaptic connections are expected to implement cooperation among excitatory neurons with similar selectivity.
The efficient E-I network has low-rank connectivity. The rank of fast synaptic connectivity is, by definition of the connectivity matrices C (eq. [1] and D (eq. [T). equivalent to the maximal number of features encoded by the network, M. We assume that the number of active features (t) is much smaller than the number of neurons in the network, M << N which gives low-rank connectivity matrices [29]. Low-rank connectivity constrains the neural activity to a low-dimensional manifold, and the relevance of the latter for the description of the dynamics of biological neural ensembles has been strongly suggested by empirical neural recordings [ (0). Note also that using heterogeneous and sparse decoding weights w gives heterogeneous and sparse synaptic connectivity.
Next, we have currents that are triggered by spiking of the local neuron,
1 Local & (t) -Hg (1 Ty T.
(t).
y y (E,
(17)
with (t) the single neuron readout with time constant T. (eq. 4). While the kinetics of the local current is given by the single neurons readout, the sign and the strength of the current depend on the relation of time constants between the population readout Tg. and the single neuron readout 7. To allow the population readout अ. (t) to track fast changes of the signal (t),, we consider the case where that the population readout is fast, while the single neuron readout, related to a homeostatic readout of single neuron's 's firing frequency [10], is a slower process. Such a relation of time constants, Ty < T T-M for y E {E, constrains the local current in eq. 17 to spike-triggered adaptation [27]. If, on the contrary, we assume the following relation: Ty < T, T current that might be relevant during learning and restructuring of synaptic connections (39). Besides the difference of time constants between the population and the single neuron readout, the strength of the local current depends the regularizer Py- In loss functions, the regularizer weights the importance of the metabolic cost over the coding error (eq. (3a) (3b)). A strong regularizer My implies that the average firing rate contributes strongly to the loss evaluated by the loss function, situation where the spiking frequency should be kept in check to keep the loss as small as possible. The weighting of the metabolic cost over the coding error in the loss functions is implemented mechanistically as the local current in eq. 17 Its connection with the loss function makes most sense when the local current is spike-triggered adaptation. An increase of the regularizer My increases the importance of the metabolic cost in the loss functions, mechanistically increasing the amplitude of adaptation, which reduces the firing frequency of the spiking neuron.
we get spike-triggered facilitation, a
Finally, we derived a depolarizing current triggered by the spike of the local excitatory neuron, that we termed I.ehound (t). In the efficient spiking network, the diagonal of the E-to-E recurrent connectivity matrix DEE (eq. 15) implements a positive "self-connection" as the neuron spikes, it generates a local depolarizing current with slower kinetics of the low-pass filtered spike train. We write this effect as follows:
1 rehound (t)
DE
E. (t),
(18)
6

---

(t) the synaptic filter as in eq. 15 Since the matrix B is constrained to be positive semi- definite (see eq. [6]. the current in eq. [8 is always depolarizing. After a spike, the local neuron is reset to its reset potential, which, in H. neurons, is necessarily below the resting potential (see eq. 11 the resting potential is here at 0 mV). Right after the spike, the local neuron therefore strongly hyperpolarized, and precisely at this moment, the rebound current activates. The current 1 grebound (t) thus creates a rebound of the membrane potential after a strong hyperpolarization. In cation current (also called h-current (38), a current important for the generation of network oscillations, spontaneous firing and in controlling the excitability of cortical neurons 28
with 2;" (t)
biology, current with these properties is the hyperpolarizatioo-atiiin
2.3 Effect of complexity of E-E connectivity on network responses
In the following, we analyze the effect of the complexity of E-E connections on network responses with analytical considerations and with simulations, We start by considering a simplified network model where we assume the matrix A, that determines the transformation between the external input s(t) and the internal signal xtf) (eq. 8). to be diagonal matrix. A simple way to do so is to set the elements of the matrix B to i.e., B = (bmn): bya = 0 Vm, n ...,, M, which constrains the desired signal to a leaky integration of input features that is independent across features,
(() = x s()).
(19)
The time constant of the desired signal (t() is now equal to the time constant of the population readout of the E cell type in eq. ( and such a network is devoid of E-E and slower component of E-I (cq.15) [5) synaptic currents, well as of the rebound current (eq. 18 We further simplify the network by considering the special case where the time constants of the population and the single neuron readout are equal, ") Ty Vi; E { J, which sets the local currents in eq. (17) to zero.
To simulate the network (the simplified one or a more complex one), we have to set values for decoding weights w which are free parameters. We use unstructured decoding weights that we draw from the normal distribution with zero mean and standard deviation dy w ~ N NO, o with - and the ratio of of 3 1. We chose decoding weights in the I cell type to be stronger than in the E cell type, but the number of I neurons is smaller then the number of E neurons, with a biologically plausible ratio N .² 4 : 1. Decoding weights then determine the strength of recurrent synaptic connections (eqs. 14 and TG).
As we stimulate the simplified network with step input in one of the input features, the network responds with asynchronous spiking of neurons that are aligned to the active feature (Figure BB). The feedforward current to neuron i depends on the alignment of the input s(l) with neuron's decoding
vector : (see eq. 12 When a single feature is activated, e.g. S, (t) * 0, a positive input, (t) > 0, drives neurons with positive decoding weight for the n n-th feature (i.e., neurons with E.E > 0), while negative input, 8n (1) < O, drives neurons with negative decoding weights (neurons with w < 0). If several features are active simultaneously, the feedforward current to E neurons is given by a linear mixture of features. The simplified network is purely feedforward-drrven and the network response never outlasts the stimulus (Figure 3B). The recurrent connectivity of such a network consists of structured fast connections between neurons with similar decoding selectivity (eq. 14), and the stronger the similarity of decoding vectors of neurons and j, the stronger their synapse.
We then considered a network with E-E and slower E-I connections. In general, E-E and slower F-I connections emerge when the matrix A in eq 8 is non-diagonal. The minimal requirement for the emergence of E-E and slower E-I connections is that at least one diagonal element of the matrix A in eq. 8 is different than the inverse membrane time constant of E neurons, Ag. Recurrent excitatory (E-E) connections and slower E-I connections, DE and D'E are then determined by the similarity of decoding vectors wf and :h (same as fast connections). as well as on the matrix B (in contrast to fast connections; see eqs. 14 and 16). Square matrix B is constrained to be positive semi-definite to ensure that the network obeys Dale's law (eq and positive deffniefinsssess turn constrains the matrix B to be symmetric and to have non- non-negative eigenvalues. To ensure that these properties
are satisfied, we write the matrix B as follows:
B= arTT
r= (bi,..., b
(20)
with a
positive constant influencing the strength of E-E and slower E-I con- nections, and bi...., bMr set of linearly independent column vectors, with entries
7

---

and where the number of vectors is constrained as follows: 1 1 M" M. In important to consider that matrix determines the rank of the matrix B, as M' linearly independent column vectors set the rank of B to rank B) M'. The rank of the matrix B in turn determines the rank of connectivity matrices DEE and D'E through eq. (16).
b, 1b.1....... -
The response of the network now critically depends on the complexity of recurrent excitatory (E-E) connections. In the simplest case, beyond the trivial case where all elements of B are set zero that we discussed above, I' is defined with a single column vector, I - b1. This sets the matrix B to the following: = ab1 b T, and determines the rank of the matrix B. as well as the rank of the E-E connectivity matrix F to rank B) = rank D 1. We simulate such network, setting the membrane time constants of E and cell type to biologically plausible values, with Тミ = 10 ms for pyramidal neurons, and Ty = 5 ms for inhibitory interneurons, as measured empirically in the cerebral cortex 47. In addition, we assumed time constants of the single neuron readout to be longer than the time constants of the population readout, as single- be a slower process of homeostatic regulation of the firing rate in single neurons. In particular, we set the time constant of the single neuron read-out to be twice long as the time constant of the population read-out. According to eq. [17) such relation of time constants between the single neuron and population read-out gives spike-triggered adaptation in E and I neurons.
readout is assumed to
As we simulated such a network, we found a population wide oscillation in the network activity (Figure IT)) Such a population-wide wide oscillation is in contrast to the response of the simpler network without E-E connections and without spike- triggered adaptation, where the population firing rate with constant stimulus converges to a constant value after an initial transient (Figure BB). Similarly to the simpler network, however, we still have that only neurons aligned with the input feature are active (e.g., neurons with wy, > O when the active stimulus feature is $1 (t) >> 0).
x(t)
DEE
s(t)
xe(t)
400
400
CE
DIE
40
frott
20
frate
20
8((1)
500
1000
500
1000
CHI
time
time [mne]
Figure 1: (A) Schema of the E-I network. E neurons (gray) receive a linear sum of external inputs s(t). E and I (magenta) neurons with similar selectivity are recurrently connected. Inhibitory synapses C and C have fast kinetics, E-to-I synapses have a fast and slower component (CIE and D'E), while E-to-E synapses only have the slower kinetics C and, as well as through slower synapses DEE and (green). Linear population readout of the spiking activity of E and I cell types define estimates (t) and (t). respectively. (B) Activity of the simplified network without E-E connections and without the adaptation current. Top: Step input in the first feature sy (t). The remaining features are inactive. Middle: Spike trains of E and neurons aligned with the positive input in the first feature (black and magenta, respectively), and aligned with the negative input the first feature (blue and red, respectively). Bottom: Neuron-averager firing rate, with the same color code as in the middle r.
10 ms Vi, bmn = 0 Vm, n = 1,.... M, #gg = μ4 6, of o! = 0.167 Vi, o 1 o : = 1, di = 0.02 ms. (C) Same as in B but showing the activity the network with E-E connections and where E-I synapses have both fast and slow component. The network also has adaptation current in both E and neurons. Parameters: TE 10 ms. Ti 5 ms, T. 20 ms Vi, 7f 10 ms Vi, B = ab, To, with 0.035 and b [0.5, 0, 0.3JT, of of 0.25 Vš. Other parameters as in B.
plot. Parameters: M 3, NE 400, Ng N₂ = 4: 1. TE TI
With higher rank of E-E connectivity, the network response reflects mixing of input fea- tures. We now define the matrix B with M' > 1 linearly independent vectors, where a +... + bm = 0 if and only if on = Vn. Linear independence of column vectors b, implies that the matrix B has rank M' > 1. The rank of the matrix B than determines the rank of
8

---

E-E connectivity matrix due to eq. [6). We find that with rank of E-E connectivity bigger than 1, also neurons that are aligned with the active input feature participate in the network response. As shown on Figure 2 with 81 (t) > O, excitatory neurons with a positive decoding weight for the first feature, wft (black), have similar firing rate than neurons with negative decoding weight for the same feature, wf: < 0 (blue). Since the feedforward current only drives neurons with decoding weights aligned the input feature (eq. 12| here neurons with wff > 0 in black), this means that the response of misaligned neurons (i.e. neurons with w > 0), is driven by the E-E connectivity, In summary, E-E connectivity with rank only drives neurons that are aligned with the active stimulus feature, while higher rank of E-E connectivity also drives misaligned neurons.
For fixed strength of E-E connectivity, increasing the noise results longer duration of network's response to the stimulus, and in spontaneous activation of the network long after the external input turned off (Figure 23). the absence of the external stimulus, such network responds with rhythmic spontaneous activation (Up state), followed by period of silence (Down state; Figure ZC). Up states are triggered by the noise at threshold, that by itself occasionally leads to spike of single neuron (see eq. TO A random spike is amplified by the recurrent excitatory synaptic activity, and spontancous activity is suppressed by the adaptation current.
B
400
of
40-
Leatt
60
20
trate
time [ms]
300
600
800
400
20
300
600
900
time (ms)
Figure 2: (A) Same in Figure DC, but using the matrix B with rank bigger than 1. To construct the matrix B, we impose the rank M' 10.4, 0.1.1.100 0.1]T, b2 = [0.1, 0.4, 0.1]T, by [0.1, -0.4]T Other parameters as in Figure (B) As in A, but with increased strength of the noise. Parameters: of = of 0.3 Vi. Other parameters as in A. (C) Same as in B, but without the external stimulus, 5m (1) = 0 Vm, t. In the network with rank of E-E connectivity matrix larger than 1, all neurons participate the population responses.
= and use the following column vectors: b
Discussion
We developed an E-I spiking network that makes information coding efficient under biologically relevant constraints. Our model is made efficient by minimizing two loss functions comprising quadratic estimation enror and quadratic regularizer, and assuming an arbitrary linear input/output transformation. The general solution of an online minimization of loss functions with spikes is

---

generalized LIF model with specific set of structural and dynamical features. Structural features include low-rank structured connectivity 46 22 24 where fast synapses between E and I neurons implement competition between neurons with similar decoding selectivity and slower synapses implement cooperation. Moreover, structural features resulting from optimization with spikes are external inputs that are shared across neurons, spike-triggered adaptation currents in single neurons rebound current in E neurons. Assuming heterogeneous decoding
and hy opperoolrizatioo-trigeee weights, all currents with the exception of leak currents are heterogeneous across neurons, increasing the biological plausibility of the model.
Similarly to the seminal paper by Boerlin and colleagues [3, the spiking network proposed here assumes that spikes are fired if and only if they minimize the quadratic error between the signal and the population readout. As the network model is analytically developed from such optimization principles, efficient coding a build-in property of the network. In earlier works on efficient coding 5. however, the leak current is not derived exactly but only added to the membrane equation for biological plausibility. Our derivations define an exact expression for the leak current, similarly as in [23]_ Contrary to most previous works on efficient coding with spikes that use single loss function to derive the membrane equations, we here propose two separate loss functions that relate to the activity of E and neurons. With this, we impose the E-I architecture and allow for functional diversity between E and neurons. A previous proposal that already used separate loss functions for E and neurons (7) yielded network similar to our simplified network, but the behavior their network is markedly different than our simplified model, possibly due to very long and unrealistic synaptic time constants they use.
2 assumed specific neural computation, namely a leaky integration of external inputs (but see 1 We here developed the case of the general linear transformation between the external inputs and the internal signals of the network with biologically plausible spiking network. This generalization allowed us to formulate biologically plausible expression for E-E connections and to gain insights on how the structure of the mixing matrix qualitatively changes network responses. Without mixing of features, only neurons that are aligned to the external stimulus, and consequently driven by the feedforward current, participate in network responses. With increased complexity of E-E connections, we also observe activation of neurons that are not driven by the feedforward current, but are instead engaged through recurrent E-E connections. Moreover, in the absence of the external stimulus and with sufficient background noise, such a complex network produces spontaneous Up and Down states that could describe spontaneous Up and Down states observed in biological networks 45
Most previous proposals on efficient coding 6 7 23
Our results demonstrate that the minimization of the quadratic loss function with spikes results in a generalized LIF model, a spiking neuron model that has been shown to provide a good fit to the spiking dynamics of biological networks [4 17 An important advantage of the present computational framework over previously proposed generalized LIF models is the access to functional interpretation of network parameters. In general, a change in a network parameter can have a strong influence network's dynamics. Besides characterizing the effect neural dynamics, it is insightful to also have the interpretation of the role of a specific parameter on the functional/coompuatiinna level. In the present setting, parameters of the spiking network are directly related to the variables minimized in loss functions, which allows to connect their effect on dynamics with their role for network's 's computation.
In the present work, we emphasized the theoretical development of an optimization- approach towards a biologically plausible spiking neural network with complex, non-linear responses. Future work could introduce performance measures, for example by measuring the distance between the signal and the population readout, or by using information theoorti measures 4 41] Moreover, future work could perform an analysis of network parameters and assess how are particular dynamical phenomena, such rhythmic spontaneous Up and Down states (45), related to information processing in efficient spiking networks.
Acknowledgments
This work was supported by the NIH Brain Initiative Grants U19 NS107464, ROI NS109961, and RO1 NS 10104410.
10

---

References
A. Alemi, C. Machens, S. Deneve, and J.-J. Slotine. Learning nonlinear dynamics in efficient, balanced spiking networks using local plasticity rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[2] D. G. Barrett, S. Deneve, and C. K. Machens. Optimal compensation for neuron loss. Elife, 1:412454, 2016.
[3] M. Boerlin, C. K. Machens, and S. Denève, Predictive coding of dynamical variables in balanced spiking networks. PLoS Comput Biol, 9(1 1):ee 1003255, 2013.
[4] A. Borst and F. E. Theunissen. Information theory and neural coding, Nature neuroscience, 2(11):947-957,
1999.
[5] R. Bourdoukan, D. Barrett, S. Deneve, and C. K. Machens. Learning optimal spike- representations. In F. Percira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012. URL httptttaaapppeeeeeeeeeeeeeeeeeee httpeeeednnnnne.enrp...e.. 201.///a/aaa77700600000446666662 Paper pdf
W. Brendel, R. Bourdoukan, Vertechi, C. Machens, and S. Denéve. Learning represent signals spike by spike. PLoS computational biology, 16(3)=1007692, 2020.
[7] Chalk, Gutkin, and S. Deneve, Neural oscillations as a signature efficient coding in the presence of synaptic delays. Elife, 5:e13824, 2016.
[8] S. N. Chettih and C. D. Harvey. Single-neuron perturbations reveal feature-specific competition in vl. Nature, 567(7748) 2019.
[9] S. Denève and C. K. Machens. Efficient codes and balanced networks. Nature neuroscience, 19(3): 975-382, 2016.
[10] N. S. Desai, L. C. Rutherford, and G. G. Turrigiano. Plasticity in the intrinsic excitability of cortical pyramidal neurons. Nature neuroscience, 266))515-520, 1999.
[11] J. C. Eccles. Chemical transmission and dale's principle. Progress in brain research, 68:3-13, 13, 1986.
[12] A. A. Faisal, L. P. Selen, and D. M. Wolpert. Noise in the nervous system. Nature reviews neuroscience, 9 (4):292-303, 2008
[13] J. A. Gallego, M. G. Perich, R. H. Chowdhury, S. A. Solla, and L. E. Miller. Long-term stability of cortical population dynamics underlying consistent behavior. Nature neuroscience, 23(2):260-277 2020.
[14] W. Gerstner and R. Naud. How good are neuron models? Science, 326(5951):339--380 2009.
[15] W. Gerstner, W. M. Kistler, R. Naud, and L. Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[16] G. J. Gutierrez and S. Denève. Population adaptation in efficient balanced networks. ELife, 8:e46926. 2019.
[17] R. Jolivet, F. Schürmann, T. K. Berger, R. Naud, W. Gerstner, and A. Roth. The quantitative single-neuron modeling competition. Biological cybernetics, 99(4)4417422, 2008
[18] M. Kafashan, W. Jaffe, N. Chettih, R. Nogueira, I. Arandia-Romert C. Harvey, Moreno-Bote, and J. Drugowitsch. Scaling of sensory information large neural populations shows signatures of information limiting correlations. Nature communications, 12(1):1-16 2021.
[19] E. R. Kandel. Dale's principle and the functional specificity of neurons. In Prychopharmacologyy Review of Progress, 1957-1967, pages 385-398. US Government Printing Office Washington, DC, 1968.
[20] C. Kayser, N. K. Logothetis, and S. Panzeri. Millisecond encoding precision of auditory cortex neurons. Proceedings of the National Academy of Sciences, 169769) 11116999676611111111111111111111 2010.
[21] P. D. King, J. Zylberberg, and M. R. DeWeese. Inhibitory interneurons decorrelate excitatory cells to drive sparse code formation in a spiking model of v1. Journal of Neuroscience, 33(13):5475-544) 2013.
[22] H. Ko, S. B. Hofer, B. Pichler, K. A. Buchanan, P. J. Sjöström, and T. D. Mrsic-Floo Flogel. Functional specificity of local synaptic connections in neocortical networks. Nature, 473(7345)877799 2011.
11

---

[23] V. Koren and S. Denève. Computational account of spontaneous activity signature of predictive coding. PLoS computational biology, =3( 2017.
[24] V. Koren, A. R. Andrei, M. Hu, V. Dragoi, and K. Obermayer. Pairwise synchrony and correlations depend on the structure of the population code visual cortex. Cell Reports, 33(6):108367, 2020.
[25] M. S. Lewicki. Efficient coding of natural sounds. Nature neuroscience, 5(4):356-363, 2002.
[26] Q. Li and C. Pehlevan. Minimax dynamics of optimally balanced spiking networks of exci- tatory and inhibitory neurons. In H. Larochelle, M. Ranzato, R. Hadsell, Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4894- 4904. Curran Associates. Inc.. 2020. URL htttt:::t/tpteeeeeeeee mew.ridiinnnnnnnnsss.eee..ee.. 33cf42b38bbcf www.:aabccc..aabbbbeooooododd...
Paper.
[27] Y.-H. Liu and X.-J Wang. Spike-frequency adaptation of generalized leaky integrate-and-fire model neuron, Journal of computational neuroscience, 10(1):25-45, 2001.
[28] A. Lüthi and D. A. McCormick. H-current: properties of neuronal and network pacemaker. Neuron, 21 (1):9-12, 1998.
[29] E. Mastrogiuseppe and Ostojic. Linking connectivity, dynamics, and computations low-rank recurrent neural networks. Neuron, 99(3)x60--623, 2018
[30] D. A. McCormick, B. W. Connors, J. W. Lighthall, and A. Prince. Comparative electrophysiology of pyramidal and sparsely spiny stellate neurons of the neocortex. Journal of neurophysiology. 54(4):782-806, 1985.
[31] M. Montemurro and S. Panzeri. Optimal information decoding from neuronal populations with specific stimulus selectivity. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004. URL tt..httt::///// htttp::/pproceeddnnneee wtwpo/eeeinnnn...uuaaaaa...... www.eeepa..ccppppee///0/.. 001286699001103888BBaeee4444ddd0 Paper Paper.pdi
[32] Nemenman, G. D. Lewen, W. Bialek, and R. R. de Ruyter van Steveninck. Neural coding of natural stimuli: information at sub-millisecond resolution. PLoS computational biology, 4(3):e1000025 2008
[33] J. E. Niven. Neuronal energy consumption: biophysics, efficiency and evolution. Current opinion in newrobiology, 41:129-133, 2016.
[34] B. A. Olshausen and D. J. Field. Natural image statistics and efficient coding. Network: computation in neural systems, 7(2):333, 1996.
[35] B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311 1-3325, 1997.
[36] S. Panzeri, R. S. Petersen, S. R. Schultz, M. Lebedev, and M. E. Diamond. The role of spike timing in the coding of stimulus location in rat somatosensory cortex. Neuron, 29(3):769-777, 2001.
[37] S. Panzeri, M. Moroni, H. Safaai, and C. D. Harvey. The structures and functions of correlations in neural population codes. Nature Reviews Neuroscience, 23(9):551-367, 2022.
[38] H.-C Pape. Queer current and pacemaker: the hyperpolarizaationaattie activated cation current in neurons. Annual review of physiology, 58( (8(1):299 327, 1996.
[39] A. Payeur, Guerguiev, F. Zenke, B. A. Richards, and Naud, Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, 2407):1010-1019, 2021.
[40] R. S. Petersen, S. Panzeri, and M. E. Diamond. Population coding of stimulus location in rat somatosensory cortex. Neuron, 32(3):503- -14,505555,..0 2001.
[41] G. Pica, E. Piasini, H. Safaai, C. Runyan, C. Harvey, M. Diamond, C. Kayser, T. Fellin, and S. Panzeri. Quantifying how much sensory information in neural code is relevant for behav- ior. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 30, page 3686-3696. Curran Associates, Inc., 2017. URL httpp::/ppooceediiiieee wtwpeeeeddinnn.eeuureeeeeeeeeeee nwwrips. wwtp:/ew..pppppppppprrreeeeeeeee pcpp:pp//cc/ppppp/////ee//
pdf
[42] A. Pignatelli, M. Borin. A. Fogli Iseppe. C. Gambardella, and O. Belluzzi. The h-current in periglomerular dopaminergic neurons of the mouse olfactory bulb. PLoS One, 8(2):e5677,, 2013.
12

---

[43] J. W. Pillow, J. Shlens, L. Paninski, A. Sher, A. M. Litke. E. Chichilnisky, and E. P. Simoncelli. Spatio-
temporal correlations and visual signalling in complete neuronal population. Nature, 95447207):995-999, 2008.
[44] C. A. Runyan, E. Piasini. S. Panzeri, and C. D. Harvey. Distinet timescales of population coding across cortex. Nature. 548(7665):92 96, 2017.
[45] M. V. Sanchez-Vive and D. A. McCormick. Cellular and network mechanisms of rhythmic recurrent activity neocortex. Nature neuroscience, 3(10): 1034,, 2000.
[46] S. Song, P. J. Sjöström, M. Reigl, S. Nelson, and D. B. Chklovskii. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS biology, 3()xc68, 2005.
[47] R. Tremblay, S. Lee. and B. Rudy. Gabaergic internerrons in the neocortex: from cellular properties circuits. Neuron, 91(2):600-29, 2016.
[48] M. Valente, G. Pica, G. Bondanelli, M. Moroni, C. A. Runyan, A. S. Morcos, C. D. Harvey, and S. Panzeri. Correlations enhance the behavioral readout of neural population activity in association cortex. Nature neuroscience, 24(7):975-986, 2021.
[49] C. P. Wonders and S. A. Anderson. The origin and specification of cortical interneurons. Nature Reviews Neuroscience, 7(9):687-696, 2006.
[50] B. M. Yu, Cunningham, G. Santhanam, S. Ryu, K. Shenoy, and M. Sahani. coussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. D. Koller, D. Schuurmans, Y. Bengio, and I.. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008 URL httptt::///oteeeeeeses http:i/nn..enur.cccccceeeeeeeeee cc/ppppppppp0..////.//pp/p aile/ad972f100 wttp:////0000000000000000003aaa. Paper pdf
[51] M. Zhu and C. J. Rozell. Visual nonclassical receptive field effects emerge from sparse coding in a dynamical system. PLoS computational biology, 1003110 2013.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Present work has societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] see Results and the Supplementary text
(b) Did you include complete proofs of all theoretical results? [Yes] see Results and the Supplementary text
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] see Supplemen- tary material
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] see Figure Captions
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Supplementary material
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
13

---

(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
14


Reviews:
[{"review_id": "i2ms-tfr2Jy", "reviewer": "Reviewer_DCGS", "summary": "The authors present a spiking network model with E/I neurons along with two loss functions which include a squared loss to some stimulus and a quadratic regularizer. From here they show how this model matches a generalized leaky-integrate-and-fire (LIF) network under certain approximations. In this sense the LIF network is ‘optimal’ in the given framework. The authors finally show different network dynamics, first for a simplified network without E-E connections, and later for the full network with varying ranks of the connectivity matrix. The authors highlight their ability to interpret and link different model components to biological processes. ", "questions": "- Why are the loss functions meaningful? While 3a makes sense (if the readout should replicate the signal), 3b is less intuitive. The authors could give some more explanations and context. ", "limitations": "The authors mention their limited evaluation and list further directions for analysis and investigations. They do not comment on negative societal impacts.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe authors give an interesting link of generalized LIF networks to the optimization of specific loss functions. The derivations are extensive and technical sound. The presented model has rich properties and can be used for further analysis. It has the potential to link existing spiking network models with more biological interpretable models and model components. \n\nHowever, there are several weaknesses: \n\n### Weaknesses\n1. While ‘efficient coding’ is one of the main selling points of the authors, it is in the end an optimization of quadratic error term with a quadratic regularizer. As the exact formulation of efficient coding and the made assumption play a major role in efficient coding, the authors should give more context and state clearly the made assumptions.\n2. The main sections 2.1 and 2.2 are written quite technical and miss some clarity. A clear road map could help to guide through the technicalities. \n3. The made assumptions and approximations as well as the derivation from (6) to (9) are hidden and it is hard to grasp their correspondence. \n4. The analysis of the presented network is limited and an interpretation of the results is missing. What is for example the purpose of investigating the simplified network? Why is this interesting? What are the properties of the network? What influence have the different model parameters? What influence has the noise level? \n5. As the whole setup is formulated as an ‘efficient coding problem’, it would be interesting, if not essential, to see the performance of the network for different configurations and/or stimuli. \n\n### Minor comments:\n- Figures are rather small, and could be at least shown with full text width.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M6wVAmZHsCn", "reviewer": "Reviewer_SvZ9", "summary": "As I think the authors do not respect the NeurIPS reviewers, I just copied what they write in the abstract. \n\n\"we find a biologically plausible spiking model realizing efficient coding in the case of a generalized leaky integrate-and-fire network with excitatory and inhibitory units, equipped with fast and slow synaptic currents, local homeostatic currents such as spike-triggered adaptation, hyperpolarization-activated rebound current, heterogeneous firing thresholds and resets, heterogeneous postsynaptic potentials, and structured, low-rank connectivity.\"", "questions": "Typos: \n\n1. line 12: remove the first \"and\"\n2. line 340: remove the first \"how\"\n3. line 79: \"...of the cell type Y and M ...\", missing a \",\" before \"and\"\n\n\n", "limitations": "No insights and no background. The authors even did not change the format from the previous submission. \n\nNo experimental evidence on the application level supports the proposed method. In addition, I did not see the bio-plausible support either. ", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\n\n-1. The authors should reformat the manuscript to NeurIPS format, i.e., introduction, related work, method, results, discussion & conclusion. If the manuscript was for nature, science, etc., please reformat it! It is not respectful for NeurIPS reviewers! \n\n-2. The authors set the objective of the inhibitory population to minimize the distance between E and I. But, no explanation about the objective. Why is it? Then, we can set the objective of the excitatory network to minimize the distance between the desired signal and I? \n\n-3. lines 109-111: it is not accurate. Not only $Na^+$ and $K^+$ are responsible for the potential changes. Which biological model are the authors talking about?\n\n-4. lines 126-128: It is unclear how Eq. (8) can be re-arranged to the LIF model. \n\n-5. I did not see any experimental evidence on the application level to support the effectiveness of the proposed approach.\n\nIn summary, the manuscript lacks insights. The authors only describe the mathematical models without providing any insights and background knowledge. The writing style is poor. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CJfHDyybbOo", "reviewer": "Reviewer_QSDp", "summary": "The authors derive a neural network model that optimizes the information transfer. The optimization leads to a number of biologically relevant features such as generalized leaky integrate-and-fire neuronal dynamics, low-rank connectivity and heterogeneous decoding weights and currents across neurons in the network. ", "questions": "I would like to ask the authors to at least propose experiments that could validate their results. ", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In terms of originality, this work continues a line of research that builds biologically plausible neural networks whose objective is to maximize information coding. In that line of research, this works presents a leap forward in the right direction because it adds a number of biologically realistic features. It is my understanding that this works follows previous work closely so I would argue that the novelty is limited. The paper itself seems solid and there is no major errors I can see in the derivations. The paper is written in a clear enough manner although a section that explains the structure of the paper and/or a section that clearly outlines the contributions of this work would make it easier to read.  In terms of significance, I would argue that it is hard to judge because it is generally hard or impossible to validate how biologically-plausible these features are. It is without a doubt that features such as heterogeneity are biologically plausible, but I miss a way to validate this claims more quantitively. Also it is not completely clear to me from the paper how these different features are contributing to make the coding more efficient.         ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept