Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an unsupervised learning approach to detect and cluster neural spike sequences in neural spike data. To model the data, the authors propose a hierarchical Dirichlet point process model, which employs Hawkes processes to model the temporal dynamics of neural activity within sequences, whereas spike rates within a sequence are modelled via a non-homogeneous Poisson process. The authors derive a particle filter based online inference method to detect sequences and infer their types. They use conjugate priors to derive closed-form updates for obtaining posterior distributions of model parameters. The authors evaluate the performance of their method on both synthetic and real datasets.
strengths_and_weaknesses: Strengths:
- The authors propose a well-grounded probabilistic generative model for modelling neural activity sequences in spike train data
- The model is able to infer the number and types of sequences from the data 
- The authors have devised a tailored particle filter to infer unobserved latent variables for identifying neural activity sequences and inferring their types, which in turn allows for closed-form updates of posterior distributions over model parameters 
- The results on artificial data indicate that the proposed methodology can detect and correctly identify the types of underlying neural sequence activity in the presence of background noise. 
- The results on real data are qualitatively interesting and comparable to the results of another approach. 
- The proposed methodology in principle is scalable to larger datasets with up to 210k spikes.

Weaknesses:
- No analysis is done to show how often the inference procedure leads to suboptimal solutions, which requires reruns of measures like merging and pruning as highlighted by the authors. 
- Experiments on synthetic data involve very low number of sequence types (up to 3), so it is hard to see how the (inference) method would perform for a larger number of sequence types  
- Comparison with the alternative approach was only done on one data set. 



questions: - Have you considered or run experiments where you analysed the convergence behaviour of your methodology by assessing how often you manage to recover ground-truth parameters if you optimise on data generated by the process described in section 4, lines  145-152?

- Why have you not compared PP-seq across the board? How does PP-seq's likelihood compare with the likelihood of your approach?

- Can you elaborate what do you compute ROC curves on in section 6.1? Why to you use just one type of sequence for the experiment?

- How does your method perform if there is a greater (spatial and/or temporal) overlap between sequences than what we can see in synthetic datasets?
limitations: I have highlighted technical limitations and weaknesses above. I have nothing further to add here.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a hierarchical, non-parametric Bayesian model to identify neural sequences from neural spike data and then to categorize the neural sequences, all in an online manner. Specifically, their model consists of a Dirichlet nonhomogeneous Poisson process (DPP) prior for observed spikes and a Dirichlet Hawkes process (DHP) prior for the neural sequences generating those observed spikes. They refer to their model as the Hierarchical Dirichlet Point model (HDPP). They present a particle filtering method to perform inference under their model as well as a scheme for merging and pruning neural sequence categories/types that may have been incorrectly generated early during inference. They compare their method to two other top-down unsupervised methods, ConvNMF and PP-Seq, on synthetic data. They demonstrate that their method is comparable in performance to PP-Seq under increasing background noise rate, while the performance of ConvNMF steadily drops. They then demonstrate their method on two experimentally-collected neural spike datasets, and present results indicating that their method infers the correct number and types of sequences, converge to similar parameter ranges in independent runs, and a constant expected tune cost with increasing number of observations.
strengths_and_weaknesses: **Strengths**
- Neural recordings for neuroscience are increasingly growing in size and length and under fewer experimental constraints. This direction of neuroscience research begets the need for methods that identify neural sequences in a streaming and non-parametric way. The authors' method would provide an important set of contributions to the field.
- The authors' exposition of their model and inference method is clear.

**Weaknesses**
- The authors should improve the clarity of the conclusions that they draw from their results.
- The authors validate their method against two existing methods, in particularly using PP-Seq to validate the number and type of sequences detected. I recommend that the authors expand the depth and rigor of their evaluations and their analysis of their method. For example, the authors do not demonstrate their method on a large dataset or provide evaluation of time and memory costs of their method vs. another method such as PP-Seq. I would also like to see their method perform on a dataset with a larger number of neural sequences, where there may greater uncertainty in sequence types.
questions: - How does the time complexity of this method compare to PP-Seq on a fixed-sized dataset (such as the songbird or rat datasets used)? In other words, what is the baseline time complexity cost imposed by the additions in our method (e.g. the non-parametric formulation)?
- What is the dataset size at which using HDPP faster than PP-Seq?
- Figure 5 shows the time cost of the HDPP method, which the authors imply should remain constant as a function of number of detected sequences. What was the number of detected sequences, as a function of the x-axis?
- In Figure 5, why is there an early higher time cost (~50-100s) before settling in?
limitations: The authors do not mention any limitations of their method nor do they mention potential negative societal impact of their work. Please discuss in greater depth, for example, the tradeoffs that your method's non-parametric formulation and resulting inference algorithm may result in terms of complexity, uncertainty, or power. When, for example, would using PP-Seq be more desirable?
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposed a neural sequence detection method, which is based on a so called ''hierarchical Dirichlet point process''. The model can jointly infer an unbounded number of sequences and the event types from data. The paper develops an online algorithm to process the neural spike data, with the particle filtering framework. The proposed method was evaluated on two real-world datasets. 
strengths_and_weaknesses: Strengths:
1. The application is very interesting and important
2. The experimental data is interesting. From the test log-likelihood, the proposed algorithm seems working. 

Weakness: In summary, there is a big issue in the presentation and the contributions seem weak. 
1. The problem formulation is not clear. A good paper should give a clear definition and motivation of the problem you want to solve, but i did not see it in this paper, making me very confused.  Why do you want to partition the neural spikes into sequences? What is the meaning of doing so?  Aren't the observed events already a sequence by time? For a new sequence (inferred by the model), based on what the events are ordered? The notations confuse me further --- according to line 118, on the spike-train data, you've already had k_s to indicate the neural sequence, why do you want to infer them again with the model (see Eq2)? 

2. The relationship with the classical HDP is not mentioned. The name is confusing in that I thought you are building a model similar to  the HDP of (Teh et. al. 2005). However, it is not. The classical HDP uses the first-level DP to sample an infinite set of bases, and each DP in the second level will share these bases, but generate the data with different mixture weights. It took me a while to find out that this paper is not following the HDP framework. Rather, it is more like a hybrid of two DPs, one is to partition the data into sequences, and the other to partition the events into types. I do not see a clear ``hierarchical'' structure here. The paper should highlight the difference with the classical HDP, to avoid unnecessary misunderstanding. 

2. The contributions seem weak. The proposed model is  an extension of Dirichlet Hawkes process by (Du et. al., 2015). One more DP is added, and it is quite incremental. The inference, based on the particle filtering, is almost the same as the SMC used by (Du et. al., 2015), plus some heuristics to merge and prune. If there are some more significant contributions, the authors should highlight them. Otherwise, I view this is a very incremental extension of   (Du et. al., 2015), applied to spike-train data analysis. 

3. The quantitative metric is only based on test log-likelihood. I have a lot of experience of doing point process modeling. Although many works use test log-likelihood for evaluation, it is not a reliable metric, and is quite misleading. I look forward to seeing some evaluation that can truly reflect the performance, such as future event time prediction. 
questions: see above
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: One hypothesis about neural spike data is that it can be considered as a sequence of 'neural syllables' consisting of the spiking activity of a subset of neurons over a short period of time. Understanding these sequences would then enable relating activity recordings to behaviorally or experimentally relevant observables. The present manuscript proposes a method for unsupervised identification of such sequences.

Recent ML literature, as cited in the manuscript, studied similar problems for point processes in different contexts. In that sense, this manuscript primarily represents an application of existing methodology on hierarchical inference in temporally structured data. Briefly, the manuscript proposes a particle filter based (sampling-based) solution to inferring sequence labels in a hierarchical model where the observables are samples from point processes. The problem is both important and relevant to this community.

The manuscript uses a Dirichlet process prior, which enables the method to decide on the number of sequence labels without relying on prior information. Indeed, such information is not easy to know in practice. Briefly, the sequence types (labels) are modeled as a Dirichlet process so that the same type can appear multiple types in a given recording. Neuronal spiking activity is then modeled as a Hawkes process conditioned on the sequence type. Overall, the model is complicated, but it is fitting to the underlying complicated problem.

Update on Aug. 8: In response to the author rebuttal, I decided to increase my overall score from 5 to 6.
strengths_and_weaknesses: Strengths:
- Unsupervised, adaptive (streaming) inference of the number of sequence types
- Computational scalability

Weaknesses:
- Poor presentation: The manuscript is full of typos and grammatical mistakes. While this obscures the underlying meaning very rarely, reading the manuscript becomes time-consuming and frustrating. Similarly, reasoning should be better and more precisely explained. For instance, the argument in lines 266-267 on using temporal windows to reduce computational load appears applicable to all methods. Please better explain why it would be more beneficial for the proposed method compared to existing methods.

- Other than unsupervised inference of the number of types, one potential advantage of the proposed method could be real-time detection of sequences in a closed-loop experiment if inference can be made faster. (According to Fig. 5, it doesn't appear fast enough with the hardware used in that experiment.) However, this point was never discussed. Is this a desirable capability for neuroscience? What kind of desirable experiments could be performed by such a closed-loop setup? In my opinion, this point determines whether the proposed contribution is timely and impactful or a technical upgrade without practical consequences.
questions: I listed a few questions under Weaknesses above. I'd like to add a few more questions here:

- Could Fig. 4 be turned into a comparative study as well? How much time would previous methods require to process that dataset? I'd be curious to see the results of other methods after temporal windowing as I mentioned under Weaknesses.

- Could you please sort the neurons in Fig. 3 identically across the two compared methods to facilitate comparison?

- Could you please discuss how the different methods would perform when the firing rates are lower?

- Overall, the demonstrations use only a few sequence types. This may be in line with previous literature. However, one might expect to see a much larger set of sequence types in larger, longer recordings. Could you please discuss how the different methods would perform when the number of sequence types is larger?
limitations: The authors should discuss technical limitations of the work. That is, under which conditions can we expect it to fail? What are the roles of different noise sources, firing rates, etc?
review_title: null

Meta-review: metareview: This paper describes a hierarchical Bayesian latent model to identify neural sequences from spike data. Especially in neuroscience, detection of patterns in neural sequences is an important computational problem as the infrared patterns are useful for characterizing brain activity. The key problem is reminiscent of clustering where individual spikes are associated with sequences.
 
The proposed model -- Hierarchical Dirichlet Point model (HDPP) -- consists of a Dirichlet nonhomogeneous Poisson process (DPP) prior for observed spikes and a Dirichlet Hawkes process (DHP) prior for the neural sequences generating those observed spikes. 

Inference is done with sequential Monte Carlo, including a proposal mechanism for merging and pruning neural sequence categories/types that may have been incorrectly generated early during inference. A comparison of the method to two other top-down unsupervised methods (ConvNMF and PP-Seq) on synthetic data is provided.

While the description of the hierarchical model seems to be complete, the reviewers asked for clarifications about the motivations. During the rebuttal, the authors were also able to answer various issues about experimental section and regarding the inference procedure,
They were able to include results of further experiments. As a results, reviewers decided to raise their grades for the paper.

In light of the importance of the problem and the soundness of the methodology, I am inclined to suggest acceptance for this work.

confidence: Certain
Decision: Accept