Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a model to identify distinct functional clusters of neurons in neural population activity without an an a priori determined number of clusters. Each cluster is characterized by a latent LDS and mapped through an exponential nonlinearity to poisson rates. The prior over number of clusters and cluster assignment is computed efficiently borrowing from recent work on a mixture of finite mixture (MFM) model. The authors use a bespoke procedure for using MCMC to evaluate posteriors for their model, which involves a laplace approximation to estimate the population states, and a poisson-gamma conjugacy to marginalize over of the latent loadings $c_i$ to evaluate the probability of a neuron belonging to a given cluster. The authors show results on simulated data where the true number of clusters is known, and then evaluate the model on neuropixel recordings. They show on the neuropixel data that the clusters identified using their method outperform anatomical clustering or a single cluster on held-out data. This work is an important step in identifying functional groups of neurons in large multi-region neural population recordings.
strengths_and_weaknesses: This paper is technically strong and is an important tool with potential broad use in neuroscience. The addition of a prior over cluster assignment was a non-trivial ML contribution and significant step in understanding increasingly common multi-region datasets with large numbers of neurons. However, there are many pieces to the inference procedure and the appendix has a lot going on. The authors reference many existing approaches and it can be hard to parse the specific approach used in this work.


questions:  Some more detailed clarification on the inference would be helpful. Could you expand on how the laplace approximation is being used in 122-124? More organization in the introductory part of the inference section would I think go a long way. An equation showing the P(x|..) being approximated by Laplace would be helpful. Also it appears there is a typo in the third line of the laplace approximation in the appendix (floating closed parentheses).

Can you also clarify the difference between how the marginal likelihood in (3), evaluated using a conjugacy using Poisson and inverse gamma distributions, relates to the nonconjugacy in the "update $z_i$" section of the appendix. Some reference to equation (3) from the main paper in this section of the appendix would also be helpful for clarification.

A few more sentences providing background on the MFM approach might also help clarify this section. What exactly is the form of H?  

I think giving the specific model here a name would be useful. This would help the authors more easily reference their specific approach in the paper and later allow future researchers to succinctly refer to the work. 

The resolution of the figures is low. Zooming in to see detail is difficult, particularly in 2D. 
limitations: One limitation that I believe is more significant than the author's state is the selection of p being constant across subpopulations. The authors briefly mention ways around this issue in the discussion. I think it is important to more directly point this out as a limitation -- groups of neurons across brain regions very clearly do not have the same dimensionality (low level visual areas and higher level decision making areas, e.g.) This model is only able to isolate subpopulations with this specific constrained feature. 

No potential negative social impact of this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This manuscript proposes a new approach called "Dynamic Poisson Factor Analyzers" (DPFA) that models neural spike trains.  This methodology is developed to split individual neurals into distinct clusters, based upon the hypothesis that neural spike trains from each neural cluster into distinct populations.  This approach uses fast approximations and modern MCMC approaches to implement an inference scheme, which is applied to synthetic data and high-dimensional real datasets.  Results are compared to essentially a PLDS model (no clustering) and a clustering model based upon anatomical location.
strengths_and_weaknesses: The biggest strength and contribution of this manuscript is the inference algorithm.  The inference algorithm seems very well-considered and explored, using fast approximations in the form of well-design Laplacian approximations and approximate conjugate distributions.  Combining these tricks with a Mixture of Finite Mixture (MFM) approach, rather than a Dirichlet Process, seems to mix fast and provide robust inference.  This approach seems fairly original and significant, although I would appreciate additional details and discussion on the efficacy and reliability of individual approximations (e.g., how close is the gamma-poisson distribution to the lognormal?).

The model itself is a reasonable model, but does not seem especially novel or significant given the existence of MixPLDS.  There are changes and advances, but they seem relatively minor.  It would be helpful if the authors could highlight their particular advances more clearly, and explicit discuss how this improves the model.

One of the areas that would benefit from improvement is clarity on how this relates to a clear scientific challenge.  How the neurons cluster is potentially interesting, but I believe that the manuscript would be greatly improved by focusing on a clear scientific question that this methodology can help answer.

The results on real data, though, are one of the biggest weaknesses.  In the real data, the proposed method does not fit the data better based on log likelihood than a clustering based on known anatomy, and is barely better than not using a clustering approach at all.  It is difficult to motivate using this approach based on those results for me, especially as other competing approaches such as deep RNNs (e.g., LFADS) do tend to fit the data better.

questions: What scientific question will this methodology help answer?

In what real-world datasets could this method significantly outperform the baselines?

Assuming discrete clustering of neurons is a strong assumption.  To my knowledge, most neurons participate in multiple subnetworks of the brain.  Why are the authors considering an explicit clustering of neurons rather than an admixture, which may be more biologically realistic?
limitations: No concerns.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors develop a novel model based on a mixture of dynamic Poisson factor analyzers to cluster neurons in large-scale neural recordings. They treat the number of clusters treated as an unknown parameter and propose a novel MCMC algorithm to efficiently sample its posterior distribution. They show in a simulation that their model can accurately recover the true clustering and latent states, and then explore the results of their method on a recording from the Allen Institute Visual Coding Neuropixels dataset.
strengths_and_weaknesses: The strength of the paper lies in the proposed model, that is novel and very well-thought. It is technically sound and well constructed. However, it is a complicated model and the section 2 of the paper could be a little more clear and give more intuition behind the choice of priors and parameters. The results on simulated data are very compelling, but I would have liked to see more experiments on real data (for example, how to choose the parameter of the geometric prior on k depending on the type of probe, or the number of anatomical regions) to gain more intuition. The main weakness of the model lies in the fact that "Low firing rates tend to cause confusions in clustering" and are excluded from the data in the analysis. This does not seem very biologically relevant.  
questions: The model is very interesting and well constructed. However, it is a bit hard to grasp, and I feel like the section 2. could be a little more clear. I find that the graph in Fig. 1, C is not super informative, and could include more information, such as the priors and the additional parameters. H is also never introduced before it is mentioned on line 105. I understand that it corresponds to h, but in this case shouldn't there be an additional line in Equation 1 for mu and H? 

I would have also liked to gain more intuition on why you chose the different priors. For example, I understand the use of a Geometric distribution as the prior on k, and that the mean of this prior should increase when we want to get more clusters, but an experiment showing the different results and recommended parameter with different datasets, probes, tasks, or number of brain region recording would help a lot. 

You propose to use cross-validation for choosing p. I assume you optimize over the log-likelihood. Won't the log-likelihood necessarily increase as you use higher values for p? What criteria is used? 

The latent states are chosen to evolve linearly over time with Gaussian noise. This makes sense when the animal is not doing any tasks. What happens when the animal change state, receives a stimulus or start doing a task? It would be great to add a visualization of the latent states throughout trials. 

In section 2.3, you mention that "Though we may evaluate it by a Laplace approximation, but iterating over all potential clusters for each neuron is computationally intensive. To make faster clustering, we approximate the marginal likelihood by utilizing a Poisson-Gamma conjugacy." and "Another possible idea is to approximate the log-likelihood by second-order polynomials, with coefficients determined by Chebyshev polynomial approximation [Keeley et al., 2019]. However, we find that this approximation doesn’t work well in practice when spike counts have a wide range.". Could you show results and comparisons, in both speed and accuracy for these three approaches? 

The main flaw of the model, to me, is that "Low firing rates tend to cause confusions in clustering". Low-firing rate neurons can encode crucial information in the brain, and artificially removing neurons with rates < 1Hz in the experiments does not seem to have any biological justification, but simply a way to get nicer results ass the firing rates will be more uniform. Some neurons can also one strongly associated to a state of the animal and have overall low firing rate as they fire strongly during specific tasks. 
Could this be due to the choice of prior on lambda (and more specifically on delta/mu?)? The prior on delta is not specified (unless I miss it), but this could maybe help and solve this issue? 
Moreover, the results with low-firing rate units are not even included in the paper. It is essential to include them (at least in the appendix) to have a sense of how it contaminates the clustering results. 
What proportion of the neurons are discarded when keeping only neurons with FR>1Hz?

The low firing rate neurons can also be due to artifacts of spike sorting. It is especially true in Neuropixels datasets, where neurons can be split and thus have their firing rate cut in half (or more). There are many quality metrics developed for the output of spike sorting. (Hill et al. (2011), Harris et al. (2001), Schmitzer-Torbert and Redish. J Neurophy (2004), Chung et al. (2017)...). I think using such metrics to discard "bad" units would be much more appropriate (as is done in Reproducibility of in-vivo electrophysiological measurements in mice, International Brain Laboratory (2022) for example).
it would also be interesting to look at how the clusters are impacted by spike sorting artifacts. If you split a neuron in half, are the two half clustered together? How are the clusters impacted by electrode drift? You need to make sure the clustering algorithm is robust to these sources of noise.  

What exactly is the "anatomical cluster model" mentioned in the result section. Are you running your clustering model on each region independently, using different values for p? (p=[1,11,5,3]). How do you choose the number of clusters per region? Do you use Cross-validation on each region? If you cluster neurons from each region independently, are the neurons clustered together also clustered together when clustering all neurons together? This would be a very interesting experiment to show in the paper.

Finally, MCMC models can be computationally expensive, and your model requires (5-fold) cross-validation. How does it scale with the number of neurons and length of the recording? 
limitations: I don't see any potential negative societal impact of this work.
review_title: null

Meta-review: metareview: The authors present a mixture of dynamic Poisson factor analyzers (sometimes called Poisson linear dynamical systems) model. The model itself is not especially novel (it seems closely related to Poisson switching linear dynamical systems) but the authors make up for it with a well-developed approach to Bayesian inference. They allow for unknown numbers of states with a mixture of finite mixtures model, and they handle the nonconjugacy of the Poisson-Gaussian model with a Metropolis-corrected Polya-gamma augmentation scheme. The empirical results do not show a clear improvement on Neuropixels recordings from the Allen Brain Observatory, but again the assessment is thorough. Overall, I think the paper conveys valuable findings and ideas, and lays a nice foundation for future work. I encourage the authors to incorporate and address the reviewers' feedback when preparing the final manuscript, and to consider expanding the discussion of how the Mix-DPFA model relates to Poisson SLDS.
confidence: Certain
Decision: Accept