Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new method to fit a recurrent spiking neural network (RNN) to neural data, which consists in simultaneously maximizing the likelihood of the spike trains observed (MLE), while fitting other observables such as averaged activity over trials (PSTH) and noise correlations (NC). On real neural data (monkey V1, rat retina), the method is shown to better capture these other observables compared to models only trained on MLE. On semi-synthetic data (artificial network fit to match the V1 dataset with additional hidden neurons), the method is shown to slightly better reflect the underlying connectivity profile compared to a method trained only on MLE.
main_review: __Summary of the review:__ The proposed optimization procedure to fit RNNs to neural data is sophisticated and theoretically well-justified. However the quantitative results of the method are not entirely convincing, the potential of the method to bring new insights to neuroscience is not clearly demonstrated, and the work would greatly benefit from being better tied to prior work. For these reasons, I am not fully convinced that this paper is ready for acceptance at NeurIPS in its current state.

__Strengths__
- The optimization procedure proposed is sophisticated, thoroughly described and well-justified.
- The theoretical insight that fitting these additional observables can be seen as a Bayesian prior on the space of solutions is clear and interesting.

__Limitations__
- The quantitative benefits of the method are not entirely convincing. On real data, the method is shown to indeed improve the fit of the observables compared to a pure MLE model, which is expected, but perhaps disappointingly the MLE fit itself in not improved by the method. On synthetic data, the network connectivity (R2 = 0.64) recovered by fitting PSTH and NC is barely improved compared to a model which does not fit these additional observables (R2 = 0.59).
- The potential of this method to bring new insights to neuroscience is not clearly demonstrated.
- The paper would benefit greatly from being better connected to prior art: 

LFADS is a prior effort of fitting RNN to neural data (it would be nice to understand the difference with this work):

*LFADS - Latent Factor Analysis via Dynamical Systems
David Sussillo, Rafal Jozefowicz, L. F. Abbott, Chethan Pandarinath
https://arxiv.org/abs/1608.06315*

Previous work has shown that the underlying connectivity of a network is very difficult to estimate from the neural activity:

*Systematic errors in connectivity inferred from activity in strongly coupled recurrent circuits
Abhranil Das,  Ila R. Fiete
https://www.biorxiv.org/content/10.1101/512053v1*

There has been a lot of efforts in fitting various observables of population statistics, which are not discussed in the paper. For a review:

*Modeling the correlated activity of neural populations: A review
https://arxiv.org/abs/1806.08167*

For a specific example: the K-pairwise model is the maximum entropy model reproducing the mean spiking activities,
pairwise correlations, population count histogram P(K):

*Tkacik, G., Marre, O., Amodei, D., Schneidman, E., Bialek,
W., and Berry II, M. J. (2014). Searching for Collective
Behavior in a Large Network of Sensory Neurons. PLoS
Comput Biol, 10(1), e1003408.*

*Tkacik, G., Marre, O., Mora, T., Amodei, D., Berry II,
M. J., and Bialek, W. (2013). The simplest maximum entropy model for collective behavior in a neural network.
Journal of Statistical Mechanics: Theory and Experiment,
2013(03), P03011.*

On fitting sequential recordings of overlapping populations (a goal mentioned in the Perspective section):

*Oleksandr Sorochynskyi, Stéphane Deny, Olivier Marre, and Ulisse Ferrari. Predicting synchronous firing of large neural populations from sequential recordings. arXiv preprint arXiv:1904.04544, 2019.*

__Minor__
- It would be interesting to see a fit of the PSTHs in the plots.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel approach to fitting spiking network
models to neural data. The proposed method uses differentiable
recurrent spiking networks, which are fit to the data by minimising a
loss function with the help of recent techniques such as the
straight-through gradient estimate. The key novelty is to generalise
the loss function to be minimized, to include not only the likelihood
of the data, but also a number of regularizing terms that depend on
desired spike train statistics, such as the PSTH or the noise
correlations. These domain-specific terms in the loss function help
with breaking degeneracies in the model, and with improving fit
quality in the face of model misspecification. After a theoretical
analysis of the properties of the proposed loss function(s), the paper
shows examples of applications of the technique to some simulated
data. A comparison to a state-of-the-art generative model for
population spike trains, and an example of usage of the proposed
method on a large database of experimental recordings, are presented
in the supplementary material.
main_review: In my opinion, the paper brings an interesting perspective to a topic
that is of sufficiently broad interest within the NeurIPS community, and
the idea behind the proposed technique is, as far as I am aware, a
novel combination of known methods. The effectiveness of the
sample-and-measure method in improving fit quality, especially in the
face of model misspecification, is convincingly presented, and I
enjoyed the geometrical interpretation of the technique. the authors
do a good job of comparing with another relevant method from the
recent literature, and of giving the reader an idea of how the
technique could scale up to large datasets, which are becoming more
and more common. The paper is clearly written, and I was not able to
spot any technical issue or shortcoming. Overall, I think this is a
solid contribution.

My only critique to the work is that the claims on the usefulness of
this method for connectivity reconstruction seem not to be very
strongly supported by the evidence presented. More specifically, the
claim that "the sample-and-measure method [...] is important to
reconstruct trustworthy connectivity matrices in cortical areas"
(lines 284--288) rests upon the fact that a correctly-specified model
with hidden neurons can recover the connectivity matrix of the visible
population with an R² "almost as high" (line 281) as in the case where
the full population is visible. I have two objections to this:
1. the R² for the proposed method with hidden neurons is 0.64, while
   in the fully-visible case it is 0.66. For the baseline MLE method,
   these numbers are 0.59 vs 0.65. In an absolute sense the numbers
   don't look that far off, and in absence of systematic statistical
   assesments it is hard to tell if the difference between the two
   methods is really up to the difference between them, or it depends
   on the specific dataset/network being simulated.
2. even taking the claim at face value, what the data in figure 4c
   show is that connectivity can be recovered in presence of hidden
   neurons (whether better or not than by simple MLE is up for debate,
   see previous point), *if we work with a perfectly well-specified
   model*. Indeed, and unsurprisingly, the only results we have for a
   misspecified model (figure 4b) show that, even with the
   sample-and-measure method, connectivity estimates should not be
   trusted. But in practice, with real data, our models will be always
   misspecified, so it is not clear how useful it is to show that
   connectivity can be recovered in the well-specified case.
To summarise my two points above, I think that the good performance of
the model at recovering the connectivity can be explained mostly by
the fact that the model is well specified (and indeed, MLE does pretty
well too), rather than as a specific property of the proposed method.


Minor points and typos
- Line 220: "instead of a CNN" (remove "the").


limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes to regularize the MLE with loss on statistics of model-simulated spike train to improve the estimation of recurrent spiking neuron network. It also introduces hidden neurons into the model in hope that it helps inference.
main_review: Originality:  Previous work was discussed in the submission. However, it loses track of some recent work on the same topic.
- [Arribas et al. Neurips 2020] proposed similar idea to the sample-and-measure loss function. They optimize the maximum mean discrepancy (MMD) between data and simulated spike train or MMD-regularized MLE.
- Park et al. Neural computation 2012] proposed a general kernel-based framework that defines dissimilarity and divergence between spike trains.  

Quality: Generally the submission shows theoretical and empirical analysis and evaluation to support the claim. However these properties lack rigorous proof and more evaluations are needed for comprehensive comparison.
- The model in this work looks more like a (autoregressive) GLM to me (It is called GLM in the citations e.g. [1]). Yet I will not dwell on arguing about the definition. 
- MLE of such a model in the submission is a convex problem. However the sample-and-measure loss function is not necessarily convex w.r.t. the parameters in general. Then its existence and uniqueness of the global minimum is questionable. The authors should provide rigorous proof for these properties. Practically,  even if a global minimum exists, it is still unknown if it can be reached by the optimization algorithm used in this work.
- The V1 data used in the experiment is input-driven. It is also necessary to see the result of spontaneous activities, especially the long-term simulation.
- In real data, the recorded neurons always have input from other sources. The hidden neurons in the model could identify them, but it can hardly be enough for single neurons. It is more important to compare the model performance with/without hidden neurons for the same data. 
- The empirical results only compared different losses on the proposed model. Comparisons with other work are needed.

Clarity: Overall the submission is clearly written and well organized. Please clarify the following
- How to determine the number of hidden neurons
- Choice of $\mu_{MLE}$ and $\mu_{SM}$
- Table 1: error of NC and NLL

Significance: One issue of MLE on such models with autoregressive components is that the model is actually different for training and simulation. During training the log-likelihood is conditional on spike history that is considered fixed, but the simulation is conditional on spike history that is generated randomly (random variables). This often results in unrealistic simulation like Fig.1E. The proposed method could help to deal with this issue.
limitations_and_societal_impact: The limitations are not adequately addressed. Comparisons with other work are needed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study proposes an approach to fitting spiking networks to neural population data, by combining a typical maximum likelihood loss (in case of a fully observed network, but an ELBO loss for a network with unobserved neurons) with an additional loss term measuring the distance between summary statistics of recorded data and simulated data. This approach is shown to capture the summary statistics of interest better than a typical maximum likelihood approach (without this additional term) in a few scenarios.

main_review: ### Originality

The novelty of the work resides in the design and analysis of a new loss function for fitting spiking neural networks. Previous work is appropriately cited and it is clear how this work differs from previous contributions.


### Quality

The paper is technically sound, but some of the claims are not sufficiently backed by empirical evidence:

-the empirical results of Figures 3 and 4 seem to be for one run of the model. This is somewhat insufficient to assess the real difference in performance between losses, and increasing this to at least 10 repeats (e.g. different seeds) would be critical;

-the hidden neuron scenario of Figure 4C - where the neural activity is unobserved but the exact number of neurons is known - is quite unrealistic and I would recommend the authors to test a scenario where the exact number of hidden neurons is also unknown.


### Clarity

The manuscript is for the most part clearly written. However, it sometimes contains typos and imprecisions, such as:

-instead of "has been a fundamental tool to progress our understanding", "has been a fundamental tool to progress in our understanding";

-instead of "like the peri-stimulus histogram (PSTH)", "like the peri-stimulus time histogram (PSTH)";

-line 67, z is missing index k in the factorisation of the likelihood;

-line 83, \mu_{MLE} and \mu_{SM} are not defined (similarly, see line 152);

-"the visible spikes zV are clamped to the recorded data zD and the hidden spikes zH are sampled causally using our model". The term "causally" is slightly confusing. I would suggest the authors to either remove the term or rephrase the sentence;

-I would suggest the authors to improve the clarity of the sentence "Although minimizing LMLE recovers the true distribution PD under ideal conditions it does not seem to apply with neural data because the MLE solution often generates unrealistic activity";

-the sentence in lines 159-162 ("This interpretation...ideal for MLE") is not very rigorous and hard to parse, and would benefit from a rephrasing.



### Significance

The results suggest that the developed approach could be a promising step towards scalable methods for fitting spiking networks to neural population data. However, the study generally falls short of empirical evaluations of the method performance (note that the manuscript is one page less than the maximum, and would have the space for additional material).

### After rebuttal

Thank you for the answers to my review. Given the other reviews and the authors' rebuttal, and in particular the willingness of the authors to clarify the main contributions of the paper, I increased my rating to better reflect the quality and significance of the work.
limitations_and_societal_impact: The authors do not sufficiently discuss the limitations of the study. In particular, the authors discuss the computational cost of the method as a function of the dimensionality of the fitted network, but do not include the performance of the method as a function of the dimensionality, which is arguably more important to report and discuss than the computational cost.
review_title: null

Meta-review: Dear authors, 

Congratulations on your paper being accepted at Neurips. Your submission was discussed extensively, (and, at times, controversially) amongst the reviewers. We appreciated that your approach has the potentially to substantially increase the usability of a common analysis tool in neuroscience, by ensuring that GLMs fitted to multi-neuron spike recordings lead more faithful models of neural firing. The current ’state of the art’ is still to use MLE, and there is mounting empirical evidence that this often yields models that do not provide accurate generative samples (e.g. exploding firing rates, or inaccurate representations of PSTHs and correlations). The approach to extend the loss function and to use differential simulators for optimisation seems to provide a clear path for overcoming this general  problem — therefore, the final decision was to accept this paper at Neurips. 

At the same time, the discussion and review process revealed multiple points in which the paper could be strengthed— please see the reviews and comments by the reviewers for specific points. However, a general theme relates to the framing of the paper— it is clear that approach leads to better generative samples, and this is demonstrated empirically. It is not immediately clear why this approach would also lead to better reconstructions of 'functional connectivity’ (aside from the many issues plaguing this concept), and the empirical evidence is similarly not completely convincing. 

We would ask the authors to take the feedback by the reviewers seriously, and adjust the paper accordingly. 

Best, 

Your AC
Decision: Accept (Poster)