Mesoscopic modeling of hidden spiking neurons
Authors: Shuqi Wang
Valentin Schmutz
Guillaume Bellec
Wulfram Gerstner
Venue: NeurIPS 2022 Accept (2022)
Keywords: recurrent spiking neural network
hidden spiking neurons
maximum likelihood
mean-field approximation
finite-size fluctuations
neural population dynamics
latent variable model
expectation-maximization algorithm
metastable dynamics

Abstract: Can we use spiking neural networks (SNN) as generative models of multi-neuronal recordings, while taking into account that most neurons are unobserved? Modeling the unobserved neurons with large pools of hidden spiking neurons leads to severely underconstrained problems that are hard to tackle with maximum likelihood estimation. In this work, we use coarse-graining and mean-field approximations to derive a bottom-up, neuronally-grounded latent variable model (neuLVM), where the activity of the unobserved neurons is reduced to a low-dimensional mesoscopic description. In contrast to previous latent variable models, neuLVM can be explicitly mapped to a recurrent, multi-population SNN, giving it a transparent biological interpretation. We show, on synthetic spike trains, that a few observed neurons are sufficient for neuLVM to perform efficient model inversion of large SNNs, in the sense that it can recover connectivity parameters, infer single-trial latent population activity, reproduce ongoing metastable dynamics, and generalize when subjected to perturbations mimicking optogenetic stimulation.

Mesoscopic modeling of hidden spiking neurons
Shuqi Wang", Valentin Schmutz', Guillaume Bellec, Wulfram Gerstner
Laboratory of Computational Neuroscience
École polytechnique fédérale de Lausanne (EPFI.)
first. lastname@eppl..c
Abstract
Can we spiking neural networks (SNN) as generative models of multi-neuronal recordings, while taking into account that most neurons are unobserved? Modeling the unobserved neurons with large pools of hidden spiking neurons leads to severely underconstrained problems that are hard to tackle with maximum likelihood esti- mation. In this work, we use coarse- grasinngrginngggggg and mean-field approximations derive bottom-up, the activity of the unobserved neurons is reduced to low-dimensional mesoscopic description. In contrast to previous latent variable models, neuLVM can be explic- itly mapped to recurrent, multi-pon multi-popplation SNN, giving a transparent biological interpretation. We show, on synthetic spike trains, that a few observed neurons are sufficient for N to perform efficient model inversion of large SNNs, in the sense that it can recover connectivity parameters, infer single-trial latent population activity. reproduce ongoing metastable dynamics, and generalize when subjected to perturbations mimicking optogenetic stimulation.
wwurronally-gronndee
latent variable model (neuLVM), where
Introduction
The progress of large-scale electrophysiological recording techniques [1 begs the following question: can we reverse engineer the probed neural microcircuit from the recorded data? If so, should we try to design large spiking neural networks (SNN), representing the whole microcircuit, capable of generating the recorded spike trains? Such networks would constitute fine- fine-grained mechanistic models and would make in silico experiments possible. However appealing this endeavor may appear, it faces major obstacle that of unobserved neurons. Indced, despite the large number of neurons that can be simultaneously recorded, they add up to a tiny fraction of the total number of neurons involved in any given task (2). making the problem largely underdetermined. Training SNNs with large numbers of hidden neurons is challenging because a huge number of possible latent spike patterns result in the same recurrent input to the recorded neurons, making training algorithms nontrivial 3
From the perspective of single recorded neuron, the spike activity of all the other neurons can be reduced to single causal variable - the total recurrent input (Figure [ Hence, we argue that fine-grained SNNs not necessary replaced by a coarse-graine grained model of the sea of unobserved neurons. One possible coarse- coarse-graining scheme consists in clustering neurons into homogeneous populations with uniform intra- and inter- population connectivity. With the help of mean-field neuronal population equations 7 T0] this approach enables the reduction of large SNNs of neuronal populations interacting with each other [НН Clusters can reflect the presence of different cell-types |T] [4 (.5 or groups of highly interconnected excitatory neurons 16-2111 From
model the inputs from hidden neurons but can be
www-dinensional mesoscopic models composed
a computational point of view, barse-grainee SNNs offer biologically plausible implementations of
"Equal contributions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A
The challenge:
The approaches: schematic plot of three modeling strategies
modeling the input coming from hidden neurons
Latent activity
SNN
neuLVM
LVM
X
Observed neuron
Hidden
Training SNNS with large numbers of hidden neurons: challenge and approaches. (A) The challenge of modeling the input to the observed neurons (black) coming from the hidden neurons (white) while only small fraction of neurons is observed.
Figure I:
(B) Modeling strategies for the input coming from the hidden neurons: SNNs (left) model the fine-graing spike trains of all hidden neurons; N (middle) uses mesoscopic description the population activity, clustering neurons into homogeneous populations; classic LVMs (right) model the latent activity with low- dimensional phenomenologica variables (the link to SNNs lost).
rate coding by ensembles of neurons and 'computation through neural population dynamies' 24
X
In this paper, we show that, after clustering the unobserved neurons into several homogencous populations, the finite-size neuronal population equation of Schwalger et al. [ can be used to derive a neuronally-grounde
latent variable model (neuLVM) where the activity of the unobserved neurons is summarized in a coarse -aaniie mesoscopic description. The hallmark of neuLVM is its direct correspondence to multi- mouttippppulaton SNN. As a result, both model parameters and latent variables have transparent biological interpretation: the model is parametrized by single-neuron properties and synaptic connectivity; the latent variables are the summed spiking activities of neuronal populations. Coarse-graining by clustering, therefore, turns an underdetermined problem fitting an SNN with large number of hidden neurons into a tractable problem with interpretable solutions.
Switching metastable activity patterns that are not stimulus-locked have attracted large amount of attention in systems neuroscience 25 27 for their putative role in decision maasinnn [2X] attention 22 and sensory processing 30. Since generative SNN models of these metastable dynamics are available [1] 21 31 52), metastable networks constitute ready-to-use testbeds for bottom-up mechanistic latent variable models. Therefore, we propose metastable networks as a new benchmark for mechanistic latent variable models.
2 Relation to prior work
While many latent variable models (LVM), including Poisson Linear Dynamical Systems (PLDS) 331 and Switching Linear Dynamical Systems (SLDS) 34439, have been designed for inferring low-dimensional population dynamics 40.53) their account of the population activity phe- nomenological one. By contrast, the LVM derived in this work is true multiscale model, as latent population dynamics directly stems from neuronal dynamics.
Our method builds on René et al. [54], who showed that the mesoscopic model of Schwalger et al. M enables the inference of neuronal and connectivity parameters of multi-population SNNs via likelihood- based methods when the mesoscopic population activity is fully observable. Here, for the first time, we show that mesoscopic modeling can also be applied unobserved neurons, relating LVMs to mean-field field theories for populations of spiking neurons [2]... Our neuLVM approach towards unobserved neurons differs from the Generalized Linear Model (GLM) approach (55 (and recent extensions (58 60 which either neglects unobserved neurons or replaces unobserved neurons by stimulus-locked inputs. Our approach also avoids microscopic simulations of hidden spiking neurons
which scale poorly with the number of hidden neurons.

---

3 Background: mesoscopic modeling of the population activity
Biophysical neuron models be accurately approximated (neglecting the nonlinearity of dendritic integration) by simple spiking point neurons [3], which can be efficiently fitted to neural data 674 Stochastic spiking neuron models where the neuron's memory of its own spike history is Examples of renewal-type neurons include noisy leaky integrate-and-fire and Spike-Response Model 0' neurons
restricted to the time elapsed since its last spike (its age) are said to be of renewal-type |
TO]. The dynamics of a homogeneous population of interacting renewal-type neurons can be
described, in the mean-field limit, by an exact integral equation 9
69 (see 770 r2 for
rigorous proofs). In the case of homogeneous but finite populations, Schwalger et al. [|] derived stochastic integral equation that provides mesoscopic description (i.e. a description including
finite-size fluctuations) of the population dynamics.
For clarity of exposition, in this section and the next, focus on the case of single homogeneous population with no external input. All the arguments presented below can be readily generalized to the case of multiple interacting populations with external input (Appendices A
Let us consider homogeneous SNN of N recurrently connected renewal-type spiking neurons. For T discrete time steps of length A,, let y {0, 1]NXT (a N x T binary matrix) denote the spike trains generated by the neurons. The fact that the neurons are of renewal-type implies, by definition, that the probability for neuron to emit spike at time t can be written p(yi 1|y1a-1, Q) pat(c" (a', E, 1.yi.-1) where a' is the age of neuron (i.e. the number of time steps elapsed since the last spike of neuron i), the JrJ are the recurrent synaptic weights the network, and are the parameters of neuron The sum Ej Juyi.- represents the past input received by neuron in all time steps up to t 1. The superscript At of the function Pg indicates that we consider here the discrete-time "escape rate' of the neuron but the transition to continuous time is possible [TO]. The explicit expression for pg in the case of leaky integrate-and and-fire neurons with 'escape noise' (LIF) is given Appendix A
A crucial notion in this work is that of a 'homogeneous population'. The SNN described above forms a homogeneous population if the recurrent synaptic weights are identical, that is, J = J/N (mean- field approximation), and if all the neurons share the same parameters, that is, 0 0. In homogeneous population, all the neurons share the same past input Jn1:--1/N, where n1:411 = (n1, 122. ... rle-1) denotes the total number of spikes in the population in time steps 1,2,...,t with ny Σ Ei=1 vi being the total number of spikes in the population at time t'. Then, for any neuron in the population, the probability to emit a spike at time t. given its age is
fire (a, Jni:-1/N))
Pt,a
Importantly, Eq. 1 is independent of the identity of the neuron.
In a microscopic description of the spiking activity, the vector yr depends nonlinearly on the past V1---1 A mesoscopic description aims reduce the nigh-dimensional mieroscopic dynamics to a diver-dimessionll dynamical system involving the population activity n, only (in the case of multiple interacting populations, n, is a vector of dimension K equal to the number of populations. Appendix A. While an exact reduction is not possible in general (neuron models being nonlinear), close approximation in the form of stochastic integral equation was proposed by Schwalger et al. 11 In discrete time, the stochastic integral equation reads
Binomial (N, Re/N),
(2a)
i4 Σ pfrr St.a ne-a + A N N ---Ss Sta ne-a
Et
(2b)
0.11
"finite size correction'
The variable n4 can be interpreted as the expected number of neurons firing at time 1. The survival
Sr,a = -1 - fire +..)) is the probability for a neuron to stay silent between time - a and
'Traditional renewal theory in the mathematical literature 68 is restricted to stationary input whereas we 'renewal-type" in the broader sense that also includes time-dependent input.
3

---

t-1, The finite-size correction term stabilizes the model by enforcing the approximate neuronal mass conservation Σaz1 St, R4-a R N (see D3 for an in-depth mathematical discussion). The 'modulating factor' At has an explicit expression
[ in terms of pfire, S and n (indices are dropped here for simplicity, complete formulas are presented in Appendix A as well as explanations on how to initialize Eq. 2 Importantly, for populations of interacting neurons, p'T S and depend on n, which makes the stochastic equation 2 highly nonlinear. While the mesoscopic model (2) is not mathematically exact, it provides an excellent approximation of the first and second- order statistics of the population activity [T. and is much more tractable than the exact 'field' equation 7 74. Also, the mesoscopic model 2 can be generalized to the case of non-renewal
neurons with ppike-freqenne adaptation [ and short-term synaptic plasticity 77
Formally, Eq is reminiscent of the Point Process Generalized Linear Model (GLM) 55 571 for single neurons, with the notable difference that Eq. 2 contains additional nonlinearities beyond those of the GLM because pfre, S and A all depend on (Appendix A Importantly, Equation Q readily defines an expression for the probability p(n ()) [54 where {J. f denotes the model parameters. Thus, the mesoscopic model 2 allows us to avoid the intractable sum encountered if we naively try to derive y(n( directly from the microscopic description (the intractable sum stems from the fact that the identity of neurons is lost in the observation ny at each time step i', Figure 33).
4 Theoretical result: Neuronally-grounder latent variable model
In this section, we first recall why training SNN with large numbers of hidden neurons via the maximum likelihood estimator is computationally expensive. Then, we show that the mesoscopic description, Eq. P allows us to derive tractable, can be mapped to a multi-populatton SNN.
latent variable model, which
For the sake of simplicity, all the arguments are presented for single homogeneous population, but the generalization to multiple interacting populations is straightforward (Appendices Beatrdddeeennnnnddd Let us assume that we observe, during T' time steps, the spike trains of 9 simultancously recorded neurons that are part of a homogeneous popu of the entire population y {0, 1 xT into the observed spike trains y" (q neurons) and hidden spike trains yh (N neurons). Even for a single population, it is difficult to infer the parameters @ = OJ of the SNN from observation y using the maximum likelihood estimator because, in the presence of hidden neurons, the likelihood L involves a marginalization over the latent spike trains yh:
population of N neurons, with N > q. We split the spike trains
L= -=ply90) = Σ E (
While different variants of the Expeetation-Maxiiizatine (EM) algorithm 766 relying on sampling y have been used to maximize the likelihood S 4 of hidden neurons.
6 these algorithms scale poorly with the number
Instead, we exploit the fact that, for a homogeneous population, the fine-grained knowledge of the latent activity yh is not necessary since all the observed neurons receive time the same input Jng,
where ni = >i= sf is the population activity of Section Hence, we rewrite the likelihood
as
= (= = Σ p(y", n(O),
(4a)
where the probability p(y", ( factorizes in T' terms of the form
ply;; ne 1 9) = plyily 11a-1, O)
pf 11 +)
(4b)
given by neuren model. Eq. approx. by meso. model. Eq.
A comparison of Eqs. B and 4 shows that the digh-diimesional latent activity y"h has been reduced to a low-dimensional mesoscopic description. Importantly, the q observed spike trains are conditionally independent given the population activity n. While the conditional dependence structure implied by Eq. (T typical of standard latent variable models of multi- recordings in our approach, the latent variable explicitly represents the population activity
B 40
78

---

of the generative SNN and the parameters of the model are identical to those of the SNN. As the latent population dynamics directly stems from neuronal dynamics, we call our LVM the neuronally- grounded latent variable model (N)
The nonlinearity and the non-Markovianity of Eq. prevent from using previous EM algorithms
33 40 77 78. Therefore, we fit the neuLVM via the Baum-Viterbi algorithm 79 (also known as which alternates estimation (E) and maximization (M) step
Viterbi training or hard EM [C
E-step. n = argmax, py . n(@"-1),
M-step. @n argmaxg , ()).
The estimated parameters θ and the estimated latent population activity are the result of many iterations of E-step and M-step (Appendix C.. Note that the computational cost of this algorithm does not depend on the number of hidden neurons (it only depends on the number of populations)
5
Experimental results
5.1 Single homogenous population: SNN with metastable cluster states
Although seemingly simple, homogeneous populations of leaky integrate-and-fire (LIF) neurons without external stimulation are SNNs with a rich repertoire of population dynamics, including asynchronous states, synchronous states, and cluster states 7 g In a m-cluster state (with 2 2), the population activity oscillates at frequency m times higher than the average neuronal firing rate: neuron spikes every m cycle on average; conversely, approximately N/m neurons fire in each cycle (N being to the total number of neurons). Cluster states have therefore been described as where all neurons fire in
'higher harmonics' of the synchronous state (or 1-cluster state) 99 synchrony.
In this set of experiments, always consider the same network of 600 LIF neurons (Figure A where only the connectivity parameter varies. When initialized at time 0 in the same unstable asynchronous state, the network can spontaneously settle in a m-cluster state, where m depends on the recurrent connectivity parameter J ( finite-size fluctuations break the symmetry of the asynchronous state and the population splits into m groups of synchronized neurons. The cluster state to which the network converges can be read from the power spectrum of the neuronal spike trains (Figure BB) (the fundamental frequency of the m-cluster state is approximately m times lower than that of the 1-cluster state). Generating spike trains for 6 observed neurons % of the population), we tested whether neuLVM could recover the connectivity parameter J (neuronal parameters were given), for different J's in the 1-, 2-, and 3-cluster states range (Figure 2F. Table S3). The Pearson
correlation between the learned J and the true J was 0.81 with p-value 2.8e 17. showing that, statistically, neuLVM could recover the connectivity parameter the SNN.
To assess how well neul.V V can infer the latent population activity and how neul.M compares with the methods assuming full observability (like René al. (54)). we studied in detail single trial showing a transition from a metastable 4-cluster state to 3-cluster state (Figure D.E). To generate this trial, we chose = 60.32 mV and initialized the network in the 4-cluster state. From the spike trains of only two neurons (red stars in Figure DD. neuLVM could infer the ground truth population activity n" during the 4-cluster state, and during the 3-cluster state, and could approximately detect the transition between the two states (Figure 2). While the summed, smoothed spike trains missed two out of four population activity peaks in the 4-cluster state, and one out of three peaks in the 3-cluster state (purple curve in Figure (E), the strong inductive biases contained in neuLVM enabled
the inference of the "missing' peaks (blue curve in Figure 2). Finally, neuLVM and a method assuming full observability (equivalent to a naive application of René al. [54] were compared through their ability to recover the connectivity parameter J. for varying numbers of observed neurons
(Figure 2 Table S4.. Since naive application of the method of René al. S2! does not take into
account hidden neurons, it led, as expected, wildly inaccurate estimates J when the summed spike train was far from the ground truth population activity (which happened when the number of observed neurons was small, see Figure : for an example). contrast, the neuLVM managed to recover the
Our implementation of the algorithm is openly available athttps: / thub. wwm/EPFF-LCC//eeee
5

---

A One homogeneous population of 600 LIF neurons
Spontaneous spike trains
600
500
Neuron
300
200
100
Spike train power spectrum
100
4-cluster state
3-cluster state
Population activity
Frequency (H2)
50
(ground truth)
smoothied
40
neuLVM
30
20
10
0.0
0.2
0.4
0.6
3- 2- 1-cluster state
Time (s)
C
Parameter recovery = 0.81
Parameter recovery: neuLVM vs. René et al. (2020)
p 2,8 2.8e-17
al. (2020)
j (learned)
- neuLVM
62
20
60
58
60
62
64
66
10 20
100
599
J (true)
Number observed neurons
Figure 2: Single-population SNN with metastable cluster states. (A) Network architecture (for visualization purposes, only a few connections are drawn). (B) Spike train power spectrum for different choices of connectivity parameter J. All simulations start from the same unstable asynchronous state. The corresponding cluster states are indicated below. The blue region around (C) Connectivity recovered by the neuLVM vs ground truth J. The neuLVM was fitted on one-second single-trial recordings of six neurons (1% of the population). For each ground truth J value (seven in total), ten different trials were generated: bars indicate the standard deviations of the recovered J. The Pearson correlation coefficient between the recovered J and J is 0.81 and the associated p-value 2.8e-17 (see Table S3 (D) Spike trains generated by the ground truth SNN for trial showing transition from metastable 4-cluster state to a 3-cluster state. The spike trains of two randomly sampled neurons (red stars) formed the training data (for visualization purposes, only the first 0.6 second of the one-second trial is shown) on which neuLVM was fitted: (E) the inferred population activity fil is compared to the ground truth n" and the summed, smoothed spike trains (Gaussian smoothing window with
58 mV indicates the absence of activity.
1.4 ms, Appendix D of the observed neurons. (F) Absolute difference between the recovered J and the ground truth J for the neuLVM algorithm and the method of René et al. (2020) for varying numbers of observed neurons. Using the same trial as D, for each number of observed neurons, the two methods were tested on 10 different samples of observed neurons (see Table (S4.. The marker "x' indicates that the difference [ - J| is larger than 30 m.. The median samples are linked with dashed lines to show the trends.

---

connectivity parameter, thanks to the fact that the Baum- Baum-Viterbe Viterbi algorithm of Section 4 also infers the population activity (see Figure E for an example).
5.2 Multiple populations: SNN with metastable point attractors
5.2.1 Latent population activity inference and reproduction of metastable dynamics
As a second benchmark, we tested N on synthetic data generated by three interacting pop- ulations with two populations of 400 excitatory LIF neurons and one population of 200 inhibitory neurons (Figure A). Recurrent connections between these population drive winner-take-all dynamics with finite-size inntuaatioo--nnuueedd switches of activity between the two excitatory populations II X4
an example of 'itinerancy between attractor states' [25]. The population activities of this
metastable, three-population SNN constitute the ground truth against which different models will be tested.
To build spiking benchmark dataset, ran- domly selected 9 neurons- - 3 neurons from each of the three populations and considered the spike trains of these neurons as the observed data. For simplicity, the correct partitioning of the 9 neurons into 3 groups is given since it can be reliably obtained by k-means clustering [ using the van- -ee--Rossu----ssss Distance |86] between spike trains. The complete dataset consists of 20 trials of 10 seconds. An example trial is shown in Figure BB
Network architecture
Observed spike trains y
Population activityn'
20
2.5
7.5
In contrast with the experiments of Section 5.1 where the neuronal parameters were given, here, neuronal and connectivity parameters are not given to neuLVM (see Appendix [F. We com- pared the performance of neuLVM with other generative models of spiking data- PLDS [X], SLDS [39], and GLM (55) 57 on single tri- als of the spiking benchmark dataset two
Figure 3: Network architecture and example trial. (A) Architecture of the three-population, metastable, winner-take-all SNN, (B) Example trial from the spiking benchmark dataset: 10 sec- onds recordings of 9 observed neurons (3 neurons from each of the three populations) and (C) cor- responding ground truth latent population activity (for the two excitatory populations).
ways: (i) we measured the Pearson correlation r between the inferred latent population activity ily" and the ground truth population activity
n' (Table first column); (ii) we assessed how well could the fitted models reproduce metastable dynamics by counting the occurrences of stochastic switches in free simulations in other words, samples - of the fitted models (Table I second column). Tests (i) and (ii) on an example trial are shown in Figure 4
The Poisson Linear Dynamical Systems approach (PLDS, assumes that the recorded spikes can be explained by point processes driven by latent linear dynamical system of low dimension. The Poisson Switching Linear Dynamical System (SLDS, kel 69 extends PLDS by allowing the latent variables to switch randomly between two dynamical systems with distinct parameters. We should stress that, in PLDS and SLDS, the latent variables are phenomenological representations of neural population activity which have no direct link with the ground truth population activity n*. In order to still make test (i) possible for PLDS and SLDS, we will consider the best linear transformation of the inferred latent variables which minimizes the mean squared error with the ground truth population activity n..
On test (i), neuLVM gave better estimates fily" of the latent population activity n² (Pearson correla- tion r = 0.81) than the best linear transformation of the latent activity inferred by PLDS and SLDS (r = 0.69 and r = 0.73 respectively) (Table [ first column). The GLM approach cannot be included in test (i) since it ignores unobserved neurons. Interestingly, the example trial in Figure A shows the latent population activity ñly inferred by neul.VM is smoother than the ground truth n before and after the switch (finite-size fluctuations are reduced) but nlyo and n" closely match around the time of the switch. In contrast, fluctuations are exaggerated for PLDS and SLDS. The population activity
7

---

A
Inferred population activity
B
Free simulations of the fitted models: population activity
C
Sampled spike trains corresponding to
0.82
neuLVM
T1.
20
20
0.69
40
PLDS
Neuron
11
0.69
SLDS
112
Nouron
20
20
161100
2.5
5
7.5
10 2.5
5
7.5
2.5
6
7.5
10
Figure 4: Three-populatinn SNN with metastable point attractors. (A) Latent population activity of the two excitatory populations inferred by neuLVM PLDS/ SLDS for one example trial (the same as in Figure D.. The value is the Pearson correlation coefficient between the inferred fily" and the ground truth n" population activities. (B-C) Examples of free simulations of the fitted neuLVM PLDS/ SLDS.
Table 1: Model performance summary (corresponding to Figure
Pearson correlation r between y and n
Number of switches during 100 seconds free simulations
Models
of the fitted models (10.3 = 2.7 for the ground truth SNN)
neuLVM
0.81 + 0.02
7.8 + 4.1 not visible
PLDS
(0.68 + 0.11)
SLDS
(0.73 + 0.02)
11.9 + 8.9
GLM
not visible
Mean and (+t) standard deviation were computed over 20 different trials. Parentheses for PLDS and SLDS indicate that these results are for the best linear transformation of the inferred latent variables.
estimated by simply summing and smoothing the observed spike trains (Appendix D is shown in Figure S6
On test (ii), neuLVM, fitted on a single trial of 10 seconds, was able to reproduce stochastic switches similar to that of the ground truth SNN (Table D second column): free simulations of the fitted neuLVM showed 7.8 switches in 100 seconds on average (10.3 switches on average for the ground truth SNN). To make sure that stochastic switches were the result of parameter learning via the Baum- Viterbi algorithm, we verified that, before learning, N did not show any metastable dynamics (Figure S7). Examples of simulated trials are shown in Figure E3. PLDS failed to reproduce stochastic switches, which is not surprising since wanner-take-all dynamics are typically nonlinear. SLDS could reproduce stochastic switches at the correct mean frequency (11.9 instead of the ground truth 10.3), but the standard deviation of the simulated switch count, 8.9 (2.7 for the ground truth SNN), indicates that a single 10 seconds trial was probably not sufficient for SLDS to learn switching probabilities reliably. Finally, neuronal stochasticity and small network size (9 neurons) did not allow GLM to produce stochastic switches, even when the training trial was prolonged to 500 seconds.
Taken together, only neuLVM could infer the latent population activity and reliably learn the metastable dynamies on single trials of 10 seconds, demonstrating the effectiveness of its neuronally- grounded inductive biases. Of course, these results do not guarantee that the inductive biases of neuLVM would be effective on real data since real data is most certainly out-of-distributtin for Ve While applications on real data are beyond the scope of this paper, in Appendix F we show that M is robust, to certain extent, to aithin-population heterogeneity and -istr--iit data.
8

---

When the active population is stimulated
B
When the silent population is stimulated
SNN (ground truth)
neuLVM
SNN (ground truth)
neuLVM
40
91/100
77/1.00
40
26/100
48/100
No-switch trials
No-switch trials
20
20
active
active
silent
silenl
9/100
23/100
74/100
52/100
40
40
Switch
Switch trials
20
20
trials
0.2
0.4
0.8
0.4 0.8
-0.2
0.4 0.8-0.2
0.4 0.8
Time to stim. (s)
Time to stim. (s)
Figure 5: Network responses to perturbations mimicking optogenetic stimulation. (A) Activities of the excitatory populations when the active population is stimulated (100 trials, ratios indicate the number of No-switch or Switch trials). (B) Same A but the silent population is stimulated.
5.2.2 Generalization: towards experimental predictions with neuronally-groundded modeling
Bottom-up, mechanistic models allow us to perform in silico experiments and generate predictions about neural microcircuits, which can then be tested experimentally. So we wondered: can NeuLVM, fitted on single trial of spontaneous activity (like in Section 5.2.1,, predict the response of the SNN when an external perturbation applied? As a preliminary step in that direction, we tested whether an external stimulation of the fitted model would generate the same response as that of the microscopic SNN when subjected to the same perturbation.
Using the same multi-populatte network as in Section 5.2 (Figure AA and VM fitted on single trial of spontancous activity (Figure BB), compared the response of the ground truth SNN with that of neuLVM when one of the populations was stimulated by current pulse of 4 ms mimicking the stimulation of optogenetically modified population by short light pulse. We simulated 100 trials where the momentarily active excitatory population was stimulated, and 100 where the momentarily silent excitatory population was stimulated (Figure 5JA and B respectively). Each stimulation led to two possible outcomes: stimulation could trigger a state switch (Switch trials) or no state switch (No-switch trials). both the ground truth SNN and the fitted neuLVM. we found that stimulating the silent population triggered more frequent state switches (Figure 5J3 than stimulating the active population (Figure5A), Moreover, in both the ground truth and the fitted neul we could induce "excitatory rebound switches by stimulating the active population (Figure 5 A, lower half).
6 Discussion
Understanding the neural dynamics underlying computation in the brain is one of the main goals We contribute to this effort by proposing here bottom-up. mechanistic LVM the neuronally-groundee latent variable model (neuLVM) which can be mapped to a multi poptilpop SNN. Using baseb generative models, which are more biological than RNN-based models 45). could allow systems neuroscientists to test hypotheses about the architecture of the probed microcircuit, and provide coinmnlllygrounded understanding of computation.
of latent variable modeling of multi-neuronal recordings 45| 47
While this work shows the potential of the neuLVM approach, the application of neuLVM real data faces two methocological challenges. First, there is the problem of identifiability: although neuLVM could recover single unknown connectivity parameter (Section P.I,, our method could not always recover the SNN parameters when many parameters were unknown (Section 5.2) Bayesian inference could circumvent the problem of non-identifiability by estimating the full posterior distribution over model parameters [54] g9 In addition, perturbing the probed network, with optogenetic stimulation, for example, could help model parameter recovery by providing richer data. Second, the case of real data, choosing good generative SNN model is nontrivial task. For example, how many homogeneous populations should the SNN have? Clustering the recorded spike trains could guide the design of possible generative models and Bayesian model comparison, used in biophysical

---

modeling of neuroimaging data 990 could help in selecting the most likely model among several possible models.
The model proposed here is only one particular example of an SNN-based, tractable latent variable model. Whether other such neuronally-groundee models of partially observed spike trains can be formulated and efficiently applied to real data is question left for future work.
Acknowledgments and Disclosure of Funding
We thank Johanni Brea for several discussions and for his comments on an early version of this work. We also thank Tilo Schwalger for the discussions and for sharing his code. Code from Joachim Koerfer was also used. This research was supported by Swiss National Science Foundation (no. 200020_184615).
References
[1] Nicholas A Steinmetz, Christof Koch, Kenneth D Harris, and Matteo Carandini. Challenges and opportunities for large-scale electrophysiology with neuropixels probes. Current Opinion Neurobiology, 50:92-100, 2018.
[2] Peiran Gao and Surya Ganguli. On simplicity and complexity in the brave new world of large- scale neuroscience. Current Opinion in Neurobiololy, 32: 185-155,, 2015.
[3] Jonathan W Pillow and Peter Latham. Neural characterization in partially observed populations of spiking neurons. Advances in Neural Information Processing Systems, 20:116: 10:1161-1168 2008.
[4] Johanni Brea, Walter Senn, and Jean-Pascal Pfister. Sequence learning with hidden units in spiking neural networks. Advances in Neural Information Processing Systems, 24:1422-1430, 2011.
[5] Danilo Rezende, Daan Wierstra, and Wulfram Gerstner. Variational learning for recurrent spiking networks. Advances in Neural Information Processing Systems, 24:136-144 1133-144,22 2011.
[6] Guillaume Bellec, Shuqi Wang, Alireza Modirshanechi, Johanni Brea, and Wulfram Gerstner. Fitting summary statistics of neural data with differentiable spiking network simulator. Advances in Neural Information Processing Systems, 34: 34:18552-18866 2021.
[7] Wulfram Gerstner. Time structure of the activity in neural network models. Physical Review E, 51(1):738, 1995.
[8] Nicolas Brunel and Vincent Hakim. Fast global oscillations in networks of integrate-and-fire neurons with low firing rates. Neural Computation, 11(7):1621-1671, 1999.
[9] Wulfram Gerstner. Population dynamics of spiking neurons: fast transients, asynchronous states, and locking. Neural Compututation, 12(1):43 2000.
[10] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[11] Tilo Schwalger, Moritz Deger, and Wulfram Gerstner. Towards a theory of cortical columns: From spiking neurons to interacting neural populations of finite size. PLoS Computational Biology. 10055070 2017.
[12] Tilo Schwalger and Anton V Chizhov. Mind the last forkee-frrigg rate models for mesoscopic populations of spiking neurons. Current Opinion in Neurobiology, 58: 155-166, 2019.
[13] Valentin Schmutz, Eva Löcherbach, and Tilo Schwalger. On a finite-size neuronal population equation. arXiv preprint arXiv:2106. 14721, 2021.
[14] Sandrine Lefort, Christian Tomm, J-C Floyd Sarri,, and Carl CH Petersen. The excitatory neuronal network of the c2 barrel column mouse primary somatosensory cortex. Neuron, 61 (2):301-316, 2009.
[15] Tobias C Potjans and Markus Diesmann. The cell-type specific cortical microcircuit: relating structure and activity full-scale spiking network model. Cerebral Cortex, 24(3):785-86,, 2014.
10

---

[16] Yumiko Yoshimura, Jami LM Dantzker, and Edward M Callaway. Excitatory cortical neurons form fine-scale functional networks. Nature, 433(7722)):68-- 873, 2005.
[17] Sen Song, Per Jesper Sjöström, Markus Reigl, Sacha Nelson, and Dmitri B Chklovskii. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS Biology, 3(3)=66,, 2005.
[18] Claudia Clopath, Lars Büsing, Eleni Vasilaki, and Wulfram Gerstner. Connectivity reflects coding: a model of voltage-based STDP with homeostasis. Nature Neuroscience, 13(3):344-352, 2010.
[19] Rodrigo Perin, Thomas K Berger, and Henry Markram. A synaptic organizing principle for cortical neuronal groups. Proceedings of the National Academy of Sciences, 108(13):51195544, 2011.
[20] Ho Ko, Sonja B Hofer, Bruno Pichler, Katherine A Buchanan, P P.Jspper Sjöström, and Thomas I) Mrsic-Flogel. Funetional specificity of local synaptic connections in neocortical networks. Nature, 473(7345):87-91, 2011.
[21] Ashok Litwin-Kumal and Brent Doiron. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nature Neuroscience, 15(11):1498-150 2012.
[22] Michael N Shadlen and William T Newsome. Noise, neural codes and cortical organization. Current Opinion in Neurobiology, 4(4):569-579) 1994.
[23] Michael Shadlen and William T Newsome. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding, Journal of Neuroscience, 18 (10):38703886.. 1998.
[24] Saurabh Vyas, Matthew Golub, David Sussillo, and Krishna V Shenoy. Computation through neural population dynamics. Annual Review of Neuroscience, 43:249-275, 2020.
[25] Paul Miller. Itinerancy between attractor states in neural systems. Current Opinion in Neurobi- ology, 40:14-2:: 22, 2016.
[26] Giancarlo La Camera, Alfredo Fontanini, and Luca Mazzucato. Cortical computations via
metastable activity. Current Opinion in Neurobiology, 58:37-45, 2019.
[27] Braden AW Brinkman, Han Yan, Arianna Maffei, I1 Memming Park, Alfredo Fontanini, Jin Wang, and Giancarlo La Camera. Metastable dynamics of neural circuits and networks. Applied Physics Reviews, 9()):01133,, 2022.
[28] Kenneth W Latimer, Jacob Yates, Miriam LR Meister, Alexander C Huk, and Jonathan W Pillow. Single-trial spike trains in parietal cortex reveal discrete steps during decision-making. Science, 349(644))18841887 2015.
[29] Tatiana A Engel, Nicholas Steinmetz, Mare A Gieselmann, Alexander Thiele, Tirin Moore, and Kwabena Boahen. Selective modulation of cortical state during spatial attention. Science, 154(6316)
2016.
[30] Luca Mazzucato, Giancarlo La Camera, and Alfredo Fontanini. Expectation-induced modulation of metastable activity underlies faster coding of sensory stimuli. Nature Neuroscience, 22(5): 787-796. 2019.
[31] Rubén Moreno-Bod Bote. John Rinzel, and Nava Rubin. network model of perceptual bistability. Journal of Neurophysiology. 98(3):1125-1139) 2007.
alternations in attractor
[32] Luca Mazzucato, Alfredo Fontanini, and Giancarlo La Camera. Dynamics of multistable states during ongoing and evoked cortical activity. Journal of Neuroscience, 35(21):8214 (0008821--2233 8231, 2015.
[33] Jakob H Macke, Lars Buesing, John P Cunningham, Byron M Yu, Krishna V Shenoy, and Maneesh Sahani. Empirical models of spiking neural populations. Advances in Neural Information Processing Systems, 24:13501 15510-15538 2011.
[34] Guy Ackerson and K Fu. On state estimation in switching environments. IEEE Transactions on Automatic Control, 15(1):10-17, 1970.
[35] Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models. Neural Computation, 12(4)8311864, 2000.
[36] David Barber. Expectation correction for smoothed inference in switching linear dynamical systems. Journal of Machine Learning Research, 7(11), 2006.
11

---

[37] Emily Fox, Erik Sudderth, Michael Jordan, and Alan Willsky, Nonparametric Bayesian learning of switching linear dynamical systems. Advances in Neural Information Processing Systems, 21: 457-464, 2008.
[38] Biljana Petreska, Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Mancesh Sahani. Dynamical segmentation of single trials from population neural data. Advances in Neural Information Processing Systems, 24:756-764, 2011.
[39] Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial Intelligence and Statistics, pages 914-922. PMLR. 2017.
[40] Jayant E Kulkarni and Liam Paninski. Common-input models for multiple neural spike-train data. Network: Computation in Neural Systems, 18(4):375-477, 2007.
[41] Vernon Lawhern, Wei Wu, Nicholas Hatsopoulos, and Liam Paninski. Population decoding of motor cortical activity using a generalized linear model with hidden states. Journal of Neuroscience Methods, 189(2)2677-280, 2010.
[42] Michael Vidne. Yashar Ahmadian, Jonathon Shlens, Jonathan W Pillow, Jayant Kulkarni, Alan M Litke, EJ Chichilnisky, Eero Simoncelli, and Liam Paninski. Modeling the impact of common noise inputs on the network activity of retinal ganglion cells. Journal of Computational Neuroscience, (3(1))97-121, 2012.
[43] Yuanjun Gao, Evan W Archer, Liam Paninski, and John P Cunningham. Linear dynamical neural population models through nonlinear embeddings. Advances in Neural Information Processing Systems, 29:1 19:163-771, 2016
[44] Anqi Wu, Nicholas A Roy, Stephen Keeley, and Jonathan W Pillow. Gaussian process based non- linear latent structure discovery in multivariate spike train data. Advances Neural Information Processing Systems, 000:3496-5555 2017.
[45] Chethan Pandarinath, Daniel J O' Shea. Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature 2018
Methods, 15(10)8005-88)
[46] Lea Duncker and Maneesh Sahani. Temporal alignment and latent gaussian process factor inference in population spike trains. Advances in neural information processing systems, 31: 10445-10455, 2018.
[47] Lea Duncker, Gergo Bohner, Julien Boussard, and Maneesh Sahani. Learning interpretable continuous-time models of latent stochastic dynamical systems. In International Conference Machine Learning, pages 1726-1734. PMLR, 2019.
[48] Josue Nassar, Scott Linderman, Monica Bugallo, and II Memming Park. Tree-structured recur- rent switching linear dynamical systems for multi-scale modeling. In International Conference on Learning Representattons, 2019.
[49] Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. Advances in Neural Information Processing Systems, 33:1 13:114877-48878 2020.
[50] David Zoltowski, Jonathan Pillow, and Scott Linderman. A general recurrent state space framework for modeling neural dynamics during decision-making. In International Conference on Machine Learning, pages 16800-16611. PMLR, 2020.
[51] Virginia Rutten, Alberto Bernacchia. Maneesh Sahani, and Guillaume Hennequin. Non- reversible Gaussian processes for identifying latent dynamical structure in neural data. Advances in Neural Information Processing Systems, pages 9622-9633 9622-9632, 2020.
[52] Stephen Keeley, Mikio Aoi, Yiyi Yu, Spencer Smith, and Jonathan W Pillow. Identifying signal and noise structure in neural population activity with Gaussian process factor models. Advances in Neural Information Processing Systems, 13:13795-1380)) 2020.
[53] Timothy D Kim, Thomas Z Luo, Jonathan W Pillow, and Carlos Brody. Inferring latent
dynamies underlying neural population activity via neural differential equations. In International Conference on Machine Learning, pages 5551-5561 PMLR, 2021.
[54] Alexandre René, André I Longtin, and Jakob Macke. Inference of a mesoscopic population model from population spike trains. Neural Computation, 32(8):
2020.
12

---

[55] Liam Paninski. Maximum likelihood estimation of cascade point-process neural encoding models. Network: Computation in Neural Systems, 15(4):243-26) 2004.
[56] Wilson Truccolo, Uri Eden, Matthew R Fellows, John P Donoghue, and Emery Brown. point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. Journal of Neurophysiology, 93(2):1074-1089, 2005.
[57] Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M Litke, EJ Chichilnisky, and Eero P Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454(7207):995-99)) 999, 2008.
[58] David Zoltowski and Jonathan W Pillow. Scaling the Poisson GLM to massive neural datasets through polynomial approximations. Advances Neural Information Processing Systems, 31: 3517-3527, 2018.
[59] Regis C Lambert, Christine Tuleau-Malot, Thomas Bessaih, Vincent Rivoirard, Yann Bouret, Nathalie Leresche, and Patricia Reynaud-Bouret. Reconstructing the functional connectivity of multiple spike trains using Hawkes models. Journal of Neuroscience Methods, 297:9-21, 2016 8.
[60] Ryota Kobayashi, Shuhei Kurita, Anno Kurth, Katsunori Kitano, Kenji Mizuseki, Markus Diesmann, Barry J Richmond, and Shigeru Shinomoto. Reconstructing neuronal circuitry from parallel spike trains. Nature Communications, 10(1):1-13, 2019.
[61] Werner M Kistler, Wulfram Gerstner and J Leo van Hemmen. Reduction of the Hodgkin- Huxley equations to single-variable threshold model. Neural Computation, 15): 1997.
[62] Renaud Jolivet, Timothy J Lewis, and Wulfram Gerstner. Generalized integrate- and-finnd-- models of neuronal activity approximate spike trains of detailed model to high degree of accuracy. Journal of Newrophysiology. 92(2):959-976. 2004.
[63] Romain Brette and Wulfram Gerstner. Adaptive exponential integrate-and-fing model as an effective description of neuronal activity. Journal of Neurophysiology, 94(5):3637-3644 2005.
[64] Renaud Jolivet, Alexander Rauch, Hans-Rudolf Lüscher, and Wulfram Gerstner. Predicting spike timing of neocortical pyramidal neurons by simple threshold models. Journal of Computa- tional Neuroscience, (1):3:35 49, 2006.
[65] Ryota Kobayashi, Yasuhiro Tsubo, and Shigeru Shinomoto. Made-to-order spiking neuron model equipped with a multi-timescale adaptive threshold. Frontiers in Computational Neuro- science, 3:9, 2009.
[66] Christian Pozzorini, Skander Mensi, Olivier Hagens, Richard Naud, Christof Koch, and Wulfram Gerstner. Automated high-throughput characterization of single neurons by means of simplified spiking models. PLoS Computational Biology, (004)75.0 2015.
[67] Corinne Teeter, Ramakrishnan Iyer. Vilas Menon, Nathan Gouwens, David Feng. Jim Berg, Aaron Szafer, Nicholas Cain, Hongkui Zeng, Michael Hawrylycz, et al. Generalized leaky integrate-and-fire models classify multiple neuron types. Nature Communications, 9(1):111,, 2018.
[68] David Roxbee Cox. Renewal theory. Methuen, 1962.
[69] Hugh Wilson and Jack D Cowan. Excitatory and inhibitory interactions in localized popula- tions of model neurons. Biophysical Journal, 12(1):1-24, 1972.
[70] Anna De Masi, Antonio Galves, Eva Löcherbach, and Errico Presutti. Hydrodynamic limit for interacting neurons. Journal of Statistical Physics, 158(4):866-902, 2015.
[71] Nicolas Fournier and Eva Löcherbach. On a toy model of interacting neurons. Annales de I'Institut Henri Poincaré, Probabilités et Statistiques, 52: 12:1844-188 1876, 2016.
[72] Julien Chevallier. Mean-field limit of generalized Hawkes processes. Stochastic Processes and their Applications, 127(12)3870-33112 2017.
[73] Julien Chevallier. Fluctuations for mean-field interacting age-dependent Hawkes processes.
Electronic Journal of Probability, 22, 2017.
[74] Grégory Dumont, Alexandre Payeur, and André Longtin. A stochastic-filld description of
finite-size spiking neural networks. PLoS Computational Biology, 13(8)e 2017.
13

---

[75] Valentin Schmutz, Wulfram Gerstner, and Tilo Schwalger. Mesoscopic population equations for spiking neural networks with synaptic short-term plasticity. The Journal of Mathematical Neuroscience, 10(1):1-32, 2020.
[76] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological),
39(1):1-2,, 1977.
[77] Anne C Smith and Emery N Brown. Estimating a state-space model from point process observations. Neural Computation, 15(5):955-991, 2003.
[78] Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen I Ryu, Krishna V Shenoy, and Maneesh Sahani. Gaussian-rrocess factor analysis for low-dimensional single-trial analysis of neural population activity. Journal of Neurophysiology, 102(1):614-633 635, 2009.
[79] Yariv Ephraim and Neri Merhav. Hidden Markov processes. IEEE Transactions on Information
Theory, 48(6):1518-1569 2002.
[80] Armen Allahverdyan and Aram Galstyan. Comparative analysis of Viterbi training and maxi- mum likelihood estimation for HMMs. Advances in Neural Information Processing Systems, 24: 2011. 1674-1682, 2011.
[81] Wulfram Gerstner and J Leo van Hemmen. Coherence and incoherence in a globally coupled ensemble of pulse-emitting units. Physical Review Letters, 71(3):312 1993.
[82] David Golomb and John Rinzel. Clustering in globally coupled inhibitory neurons. Physica D: Nonlinear Phenomena, 72(3):259-282 1994.
[83] U Ernst, K Pawelzik, and T Geisel. Synchronization induced by temporal delays in pulse- coupled oscillators. Physical Review Letters, 74(9):1570, 1995.
[84] Kong-Fatt Wong and Xiao-Jing Wang. A recurrent network mechanism of time integration in perceptual decisions. Journal of Neuroscience, 26(4):1314-1322 2006.
[85] Stuart Lloyd, Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129-137 1982.
[86] Mark CW van Rossum. A novel spike distance. Neural Computation, 13(4):751-763, 2001.
[87] Yuan Zhao and Memming Park. Interpretable nonlinear dynamic modeling of neural trajecto-
ries. Advances in Neural Information Processing Systems, 29:3333-3341, 2016.
[88] Matthew R Whiteway and Daniel A Butts. The quest for interpretable models of neural population activity. Current Opinion in Neurobiology, 58:86-93, 2019.
[89] Jan-Matthis Lueckmann, Pedro Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnen- macher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. Advances in Neural Information Processing Systems, 30:1289-1299, 2017.
[90] William D Penny, Klaas E Stephan, Andrea Mechelli, and Karl Friston. Comparing dynamic causal models. Neuroimage, 157-:111 2004.
[91] Will D Penny. Klaas E Stephan, Jean Daunizeau, Maria Rosa, Karl J Friston, Thomas M Schofield, and Alex P Leff. Comparing families of dynamic causal models. PLoS Computational Biology, 6())ee0007709 2010.
[92] Kay H Brodersen, Thomas M Schofield, Alexander P Leff,, Cheng Soon Ong, Ekaterina I Lomakina, Joachim M Buhmann, and Klaas E. Stephan. Generative embedding for model-based classification of IMRI data, PLoS Computational Biology, 7())e1002079, 2011.
14


Reviews:
[{"review_id": "wzDfpTF8FQF", "reviewer": "Reviewer_1XR1", "summary": "New tool for inferring underlying properties of partially observed spiking neural networks (generalization of the fully observable solution by Rene et al), based on mean field modeling on net effect of unobserved neurons; validated on within model class simulated data.", "questions": "\n- fig 2F: why is the NeuLVM comparison restricted to only up to 10 observed neurons?\n- confused about the logic of 'confirming clustring structure' and the making sure that the downsampling has the same amount of neurons from each subpopulation: please explain. This kind of selection bias does not seem to reflect real world data\n- what does it mean that the 'neural paramaters were given in 5.1'? only J is treated as a parameter, everything else has ground truth values?\n- i do not understand the quantification of state switched, in particular for PLDS and SLDS\n- what does 'free simulations' mean? samples from the graphical model with data inferred parameters?\n- is the poisson noise and/or model mismatch the cause for spike sample differences? i.e. the models (or at least PLDS assumes stationarity and is not designed to handle switches, so model mismatches are accounted for by higher overall noisiness?\n- how fair is the across model comparison given that the ground truth data matched your model assumptions perfectly whereas it is out of model distribution for everything else?\n- can you comment more about model degeneracies when fitting all model parameters rather than just within and across population Js", "limitations": "Technical limitations of the applicability of the procedure to real data should be discussed a lot more. No ethical issues.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- clear setup and motivation\n- novelty: interesting attempt at extracting more interpretable latent models, with a focus on modeling spiking activity of neurons (including unobserved ones) \n- nice mix of traditional comp neuro and ML estimation\n\nWeaknesses:\n-  writing clarity: i would have appreciated a clear spelling out of the graphical model, separation between what counts as observations, inferred latent variable and model parameters (the current text mixes in mean field technicalities which makes it less clear than it should be)\nin particular the link between n and y (Eq.4b back-referring to Eq.1) was difficult to get from the text, but i found both section 3 and 4 meandering and hard to follow in places \n- the interpretability of the parameters may quickly become problematic for out of distribution data, in particular when it's not clear that the observed neural responses can be easily partitioned in a small set of homogeneous subpopulations (especially given the extremely strict notion of homogeneity required here); arguably heterogeneity if a key feature of brain circuits, yet is not clear how sensitive is the estimation to deviations from the strict homogeneity assumption\n- metastable dynamics deviate by construction from the model assumptions of the most commonly used latent dynamical systems models, making them a perhaps unfair choice as the main benchmark for comparison across models.\n- simulations are often somewhat anecdotal (fig 2DE)\n- comparison to the fully observed scenario is rather trivial. if the model assumptions are radically different from the ground truth it is unreasonable to assume its estimated parameters to match the data\n\nMinor:\n-'photo-stimulation' is nonstandard terminology, especially if talking to an experimentalist -- causal manipulation, optogenetics etc would prove more useful\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "swDX8WyGXhZ", "reviewer": "Reviewer_Vyef", "summary": "The paper proposes a latent variable model for modeling observed activity in spiking neural networks while taking into account the activity of unobserved neurons. To do this, the authors propose a mean-field approximation approach to reduce the effect of all unobserved neurons to a lower dimensional summary quantity with simplified parameters. The proposed model, neurLVM, is fit via the approximate “hard EM” algorithm. The model is validated in two simulations, where it provides accurate recovery of net population activity and of switching states. The fitted, reduced neurLVM model was also able to predict the effect of stimulation on the true simulated network. ", "questions": "1. In the conclusion, it is stated that neurLVM cannot always recover the SNN parameters when many parameters are unknown. I appreciate the authors including this detail as a limitation. However, it was not clear to me what the learned parameters were for the metastable point attractors example. Could the authors clarify this and/or make sure those parameters are reported? \n\n2. In the three-population example, it appears that the cluster identities of the nine observed neurons were obtained from clustering all of the neurons (that is, including the unobserved neurons). If that is the case, can you identify the correct clustering just from the 9 observed neurons? \n\n3. As mentioned in the previous section, it is not clear how the method performs on heterogeneous populations. My understanding is that in both sets of simulations, the populations or subpopulations are homogeneous. It would be great to evaluate in a setting where the individual neuron parameters vary within a population, e.g. they are drawn from a distribution over the parameters. \n\n4. I’m curious on what the authors think about the following question. How restrictive is it for the sum of spikes within a population to be the primary variable of interest? That is, the latent population activity is reduced to a one-dimensional variable. I understand that was important for model fitting. But for example, does that limit the method to identifying only one-dimensional summaries of populations? \n\n5. Have the authors considered a different approximate EM algorithm that maintains an approximate distribution over the latent spiking activity? E.g., a variational EM approach? \n", "limitations": "The authors discussed two limitations, parameter recovery and choices of parameters and architecture for applications to real data, in their conclusion. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear and of high-quality. It provides an original contribution for including the effect of unobserved neurons when fitting spiking neural network models. This is a significant step forward in models of neural spiking responses. The proposed methods and experiments were clearly described with sufficient detail. Additionally, the model showed good performance in the simulations. \n\nOne weakness is the homogeneous population assumption seems to limit the generality of the method. It is not clear how the approach would work for heterogeneous populations with higher dimensional activity. Nonetheless, I think the proposed method is an important step forward.\n\n=== Update ===\n\nI appreciate the authors additional experiments and responses to my questions. In particular, I think including the heterogeneous population experiment strengthens the evaluation of the proposed approach. Based on the additional evaluations and clarity, I have increased my score by a point. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gl3qYcAQYHe", "reviewer": "Reviewer_k4DX", "summary": "The authors propose a new type of latent space model for neural spike trains, based on a spiking neural network (SNN). They use mean field approximations to abstract parts of the SNN, resulting in the latent dynamics, but keep the initial formulation for the observed neurons. The authors show, that after pre-defining a single or multiple neural clusters they can recover the connectivity value(s) by an EM algorithm from snippets of 10 sec activity of only some observed neurons. The proposed latent model is able to reproduce some key functionality of the SNN and outperforms other latent variable benchmark models. ", "questions": "1. It is surprising that the number of observed neurons seem to have (almost) no influence on the estimation performance (Fig. 2F / S4). Can the authors give an explanation for this?\n2. A potential application of the method to experimental data poses several questions:\n   - How is the number of clusters obtained?\n   - How are the observed neurons assigned to the clusters?\n   - What would be an appropriate ‘overall connectivity’ $A$ in $J = J^* \\cdot A$ (Section E) for the assumed clusters?\n3. Generalization (l. 285 ff): \n   - (a) It is not clear to me how a “perfectly fitted” neuLVM would behave under the performed additional stimulations. Could the authors comment on this? \n   - (b) The numbers in Fig. 5 suggest quantitatively quite different behaviours from the ground truth. Is this because of different parameter fits, or inherently due to the mesoscopic model?\n4. Could a more general neural model simply be plugged into the neuLVM instead of a LIF? As proposed in the Discussion (l. 312). How would adaptational processes in the SNN / different interactions influence the proposed mesoscopic approach?", "limitations": "The authors mention two shortcomings before applying their method to experimental data. However there are more open questions and limitations which could be commented on (see also questions). The authors did not comment on potential negative societal impact of their work.\n       ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe paper is clearly written and the authors define the posed problem nicely. The work is technical sound and the experiments are worked out thoroughly. The proposed latent model is an interesting approach to bridge the gap between pure statistical models and more biologically interpretable models. It has potentials for further investigations and the applications to experimental data. \n\nHowever, there are several weaknesses: \n\n### Weaknesses\n1. The authors test their setup only on one parameter set, it is not clear that the results are robust to different network configurations. \n2. The comparison which the authors make between the different models is quite unfair as the ground truth data is generated from the microscopic model the neuLVM is based on. They share therefor the same inductive biases whereas the other three models could be advantageous on differently generated data. \n3. A demonstration on real experimental would have been nice. But this is potentially beyond the scope of this manuscript.\n4. There are several open questions (see questions), especially when it comes to an application to experimental data.\n5. The model makes quite some assumptions on the network structure (number of E/I clusters, base connectivity pattern) etc. This is a limitations which the authors should discuss in more detail. \n\n### Minor comments:\n- Fig. 2F: A log scale for the y-axis could be more appropriate.\n\n### Update:\nAfter considering the responses to this review and the reviews of the other reviewers the score was raised by +1. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Vah5UV-tww", "reviewer": "Reviewer_K3UJ", "summary": "In the paper \"Mesoscopic modeling of hidden spiking neurons\" the authors introduce a neuronally-grounded latent variable model for fitting populations of observed and hidden units. The latter are described at the mesoscopic level. The model is evaluated on synthetic data with a single homogeneous population and with multiple populations and compared against other competing methods.", "questions": "The model assumes that the hidden populations are homogeneous with respect to both the recurrent synaptic weights and the single neuron parameters. I wonder how limiting this assumption is. How robust is the model to violations of this assumption? The synthetic data was generated with perfectly homogeneous populations. Would the recovered recurrent synaptic weight be close to the average ground truth parameters if there were some noise on the weights and single neuron parameters? How would neuLVM perform compared to PLDS and SLDS in the case of somewhat inhomogeneous hidden populations?\n\nIn the Introduction, the authors state that from the perspective of a single recorded neuron the spike activity of all the other neurons can be reduced to a single causal variable, further specified as the total recurrent input (line 27). This is true for typical single neurons in SNNs but is not true for real biological neurons or realistic multicompartment models, given all the complex intraneuron dynamics where spikes from particular other neurons can matter. The statement should be made more specific.", "limitations": "The authors did not discuss any societal impact. This is appropriate for this work.\n\nThe authors briefly discuss limitations regarding non-identifiability and potential need for preprocessing. However, given the seemingly strong assumption of homogeneous hidden populations, I would have expected a more thorough discussion of limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors address a very difficult problem, the problem of dealing with unobserved populations in spiking neural networks, which has plagued SNN modeling for decades. They propose a very promising approach, introducing inductive biases from biology to model hidden populations, but still keeping the level of description of these populations coarse. Therefore, significance of this work is excellent.\n\nClarity of the paper and figures is excellent too. The text is very well written and organized focusing on the single hidden population in the main paper and the more general case in the appendix. All equations are clear with symbols well explained.\n\nThe work builds on work on mesoscopic modeling of SNNs by Rene et al. and extends this to a latent variable model, which is trained by the Baum-Viterbi algorithm. This is an interesting and novel approach.\n\nThe evaluation of the model shows some of its strengths when compared to PLDS, SLDS and GLMs but falls short of convincing me that this would hold in more realistic cases where there is some variability in mostly homogeneous hidden populations. See Questions and Limitations below.\n\nThe authors do not see their method as being ready for application to real data yet (c.f. Discussion). Nevertheless, this work is an important stepping stone to deal with unobserved population activity.\n\nThe authors haven't made their source code available! To my understanding, the source code should be available to reviewers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept