Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: New tool for inferring underlying properties of partially observed spiking neural networks (generalization of the fully observable solution by Rene et al), based on mean field modeling on net effect of unobserved neurons; validated on within model class simulated data.
strengths_and_weaknesses: Strengths:
- clear setup and motivation
- novelty: interesting attempt at extracting more interpretable latent models, with a focus on modeling spiking activity of neurons (including unobserved ones) 
- nice mix of traditional comp neuro and ML estimation

Weaknesses:
-  writing clarity: i would have appreciated a clear spelling out of the graphical model, separation between what counts as observations, inferred latent variable and model parameters (the current text mixes in mean field technicalities which makes it less clear than it should be)
in particular the link between n and y (Eq.4b back-referring to Eq.1) was difficult to get from the text, but i found both section 3 and 4 meandering and hard to follow in places 
- the interpretability of the parameters may quickly become problematic for out of distribution data, in particular when it's not clear that the observed neural responses can be easily partitioned in a small set of homogeneous subpopulations (especially given the extremely strict notion of homogeneity required here); arguably heterogeneity if a key feature of brain circuits, yet is not clear how sensitive is the estimation to deviations from the strict homogeneity assumption
- metastable dynamics deviate by construction from the model assumptions of the most commonly used latent dynamical systems models, making them a perhaps unfair choice as the main benchmark for comparison across models.
- simulations are often somewhat anecdotal (fig 2DE)
- comparison to the fully observed scenario is rather trivial. if the model assumptions are radically different from the ground truth it is unreasonable to assume its estimated parameters to match the data

Minor:
-'photo-stimulation' is nonstandard terminology, especially if talking to an experimentalist -- causal manipulation, optogenetics etc would prove more useful

questions: 
- fig 2F: why is the NeuLVM comparison restricted to only up to 10 observed neurons?
- confused about the logic of 'confirming clustring structure' and the making sure that the downsampling has the same amount of neurons from each subpopulation: please explain. This kind of selection bias does not seem to reflect real world data
- what does it mean that the 'neural paramaters were given in 5.1'? only J is treated as a parameter, everything else has ground truth values?
- i do not understand the quantification of state switched, in particular for PLDS and SLDS
- what does 'free simulations' mean? samples from the graphical model with data inferred parameters?
- is the poisson noise and/or model mismatch the cause for spike sample differences? i.e. the models (or at least PLDS assumes stationarity and is not designed to handle switches, so model mismatches are accounted for by higher overall noisiness?
- how fair is the across model comparison given that the ground truth data matched your model assumptions perfectly whereas it is out of model distribution for everything else?
- can you comment more about model degeneracies when fitting all model parameters rather than just within and across population Js
limitations: Technical limitations of the applicability of the procedure to real data should be discussed a lot more. No ethical issues.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new type of latent space model for neural spike trains, based on a spiking neural network (SNN). They use mean field approximations to abstract parts of the SNN, resulting in the latent dynamics, but keep the initial formulation for the observed neurons. The authors show, that after pre-defining a single or multiple neural clusters they can recover the connectivity value(s) by an EM algorithm from snippets of 10 sec activity of only some observed neurons. The proposed latent model is able to reproduce some key functionality of the SNN and outperforms other latent variable benchmark models. 
strengths_and_weaknesses: ### Strengths
The paper is clearly written and the authors define the posed problem nicely. The work is technical sound and the experiments are worked out thoroughly. The proposed latent model is an interesting approach to bridge the gap between pure statistical models and more biologically interpretable models. It has potentials for further investigations and the applications to experimental data. 

However, there are several weaknesses: 

### Weaknesses
1. The authors test their setup only on one parameter set, it is not clear that the results are robust to different network configurations. 
2. The comparison which the authors make between the different models is quite unfair as the ground truth data is generated from the microscopic model the neuLVM is based on. They share therefor the same inductive biases whereas the other three models could be advantageous on differently generated data. 
3. A demonstration on real experimental would have been nice. But this is potentially beyond the scope of this manuscript.
4. There are several open questions (see questions), especially when it comes to an application to experimental data.
5. The model makes quite some assumptions on the network structure (number of E/I clusters, base connectivity pattern) etc. This is a limitations which the authors should discuss in more detail. 

### Minor comments:
- Fig. 2F: A log scale for the y-axis could be more appropriate.

### Update:
After considering the responses to this review and the reviews of the other reviewers the score was raised by +1. 

questions: 1. It is surprising that the number of observed neurons seem to have (almost) no influence on the estimation performance (Fig. 2F / S4). Can the authors give an explanation for this?
2. A potential application of the method to experimental data poses several questions:
   - How is the number of clusters obtained?
   - How are the observed neurons assigned to the clusters?
   - What would be an appropriate ‘overall connectivity’ $A$ in $J = J^* \cdot A$ (Section E) for the assumed clusters?
3. Generalization (l. 285 ff): 
   - (a) It is not clear to me how a “perfectly fitted” neuLVM would behave under the performed additional stimulations. Could the authors comment on this? 
   - (b) The numbers in Fig. 5 suggest quantitatively quite different behaviours from the ground truth. Is this because of different parameter fits, or inherently due to the mesoscopic model?
4. Could a more general neural model simply be plugged into the neuLVM instead of a LIF? As proposed in the Discussion (l. 312). How would adaptational processes in the SNN / different interactions influence the proposed mesoscopic approach?
limitations: The authors mention two shortcomings before applying their method to experimental data. However there are more open questions and limitations which could be commented on (see also questions). The authors did not comment on potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In the paper "Mesoscopic modeling of hidden spiking neurons" the authors introduce a neuronally-grounded latent variable model for fitting populations of observed and hidden units. The latter are described at the mesoscopic level. The model is evaluated on synthetic data with a single homogeneous population and with multiple populations and compared against other competing methods.
strengths_and_weaknesses: The authors address a very difficult problem, the problem of dealing with unobserved populations in spiking neural networks, which has plagued SNN modeling for decades. They propose a very promising approach, introducing inductive biases from biology to model hidden populations, but still keeping the level of description of these populations coarse. Therefore, significance of this work is excellent.

Clarity of the paper and figures is excellent too. The text is very well written and organized focusing on the single hidden population in the main paper and the more general case in the appendix. All equations are clear with symbols well explained.

The work builds on work on mesoscopic modeling of SNNs by Rene et al. and extends this to a latent variable model, which is trained by the Baum-Viterbi algorithm. This is an interesting and novel approach.

The evaluation of the model shows some of its strengths when compared to PLDS, SLDS and GLMs but falls short of convincing me that this would hold in more realistic cases where there is some variability in mostly homogeneous hidden populations. See Questions and Limitations below.

The authors do not see their method as being ready for application to real data yet (c.f. Discussion). Nevertheless, this work is an important stepping stone to deal with unobserved population activity.

The authors haven't made their source code available! To my understanding, the source code should be available to reviewers.
questions: The model assumes that the hidden populations are homogeneous with respect to both the recurrent synaptic weights and the single neuron parameters. I wonder how limiting this assumption is. How robust is the model to violations of this assumption? The synthetic data was generated with perfectly homogeneous populations. Would the recovered recurrent synaptic weight be close to the average ground truth parameters if there were some noise on the weights and single neuron parameters? How would neuLVM perform compared to PLDS and SLDS in the case of somewhat inhomogeneous hidden populations?

In the Introduction, the authors state that from the perspective of a single recorded neuron the spike activity of all the other neurons can be reduced to a single causal variable, further specified as the total recurrent input (line 27). This is true for typical single neurons in SNNs but is not true for real biological neurons or realistic multicompartment models, given all the complex intraneuron dynamics where spikes from particular other neurons can matter. The statement should be made more specific.
limitations: The authors did not discuss any societal impact. This is appropriate for this work.

The authors briefly discuss limitations regarding non-identifiability and potential need for preprocessing. However, given the seemingly strong assumption of homogeneous hidden populations, I would have expected a more thorough discussion of limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a latent variable model for modeling observed activity in spiking neural networks while taking into account the activity of unobserved neurons. To do this, the authors propose a mean-field approximation approach to reduce the effect of all unobserved neurons to a lower dimensional summary quantity with simplified parameters. The proposed model, neurLVM, is fit via the approximate “hard EM” algorithm. The model is validated in two simulations, where it provides accurate recovery of net population activity and of switching states. The fitted, reduced neurLVM model was also able to predict the effect of stimulation on the true simulated network. 
strengths_and_weaknesses: The paper is clear and of high-quality. It provides an original contribution for including the effect of unobserved neurons when fitting spiking neural network models. This is a significant step forward in models of neural spiking responses. The proposed methods and experiments were clearly described with sufficient detail. Additionally, the model showed good performance in the simulations. 

One weakness is the homogeneous population assumption seems to limit the generality of the method. It is not clear how the approach would work for heterogeneous populations with higher dimensional activity. Nonetheless, I think the proposed method is an important step forward.

=== Update ===

I appreciate the authors additional experiments and responses to my questions. In particular, I think including the heterogeneous population experiment strengthens the evaluation of the proposed approach. Based on the additional evaluations and clarity, I have increased my score by a point. 
questions: 1. In the conclusion, it is stated that neurLVM cannot always recover the SNN parameters when many parameters are unknown. I appreciate the authors including this detail as a limitation. However, it was not clear to me what the learned parameters were for the metastable point attractors example. Could the authors clarify this and/or make sure those parameters are reported? 

2. In the three-population example, it appears that the cluster identities of the nine observed neurons were obtained from clustering all of the neurons (that is, including the unobserved neurons). If that is the case, can you identify the correct clustering just from the 9 observed neurons? 

3. As mentioned in the previous section, it is not clear how the method performs on heterogeneous populations. My understanding is that in both sets of simulations, the populations or subpopulations are homogeneous. It would be great to evaluate in a setting where the individual neuron parameters vary within a population, e.g. they are drawn from a distribution over the parameters. 

4. I’m curious on what the authors think about the following question. How restrictive is it for the sum of spikes within a population to be the primary variable of interest? That is, the latent population activity is reduced to a one-dimensional variable. I understand that was important for model fitting. But for example, does that limit the method to identifying only one-dimensional summaries of populations? 

5. Have the authors considered a different approximate EM algorithm that maintains an approximate distribution over the latent spiking activity? E.g., a variational EM approach? 

limitations: The authors discussed two limitations, parameter recovery and choices of parameters and architecture for applications to real data, in their conclusion.
review_title: null

Meta-review: metareview: Dear authors, 

Congratulations on your paper being accepted! The reviewers unanimously recommended acceptance. The reviewers made a number of recommendations on how to improve the paper further, in particular with respect to clarity of writing and explaining the motivation behind different analyses. We strongly encourage you use this feedback to improve the paper— if needs be additional clarifications can be added in the supplement. In addition, it would indeed be highly useful to make your source code publicly available, as you indicated in your response. 

Best, your AC

confidence: Certain
Decision: Accept