Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method to efficiently characterize high-dimensional neural dynamics using a soft "tiling" of a lower-dimensionality manifold. Specifically, the method is a combination of two algorithms, (i) "Stable Streaming SVD", a dimensionality reduction method that aims to find the top-k-dimensional subspace instead of the top-k bases; (ii) "BubbleWrap", a Gaussian mixture model of the probability density with Hidden Markov Model transitions, that works on the low-dimensional manifold generated in (i). This is a model-free approach, in the sense that it is not based on a model of the dynamical system, and the authors claim that the flexibility of the method makes it outperform other (model-based) approaches in high-noise regimes. The authors also highlight that the proposed inference is fast, possibly allowing online applications.
main_review: This work addresses an important and interesting problem in the analysis of neural data: a first-pass, low-dimensional characterization of the neural dynamics, especially when the underlying dynamical system is not well understood and/or when it is important to have a rough picture of the dynamics with a limited time budget. With the flexibility of the model formulation and the fast inference, the method has a potential of being a useful tool in the field.

I am enthusiastic about the first part, ssSVD; this could be a very useful method for characterizing low-dimensional dynamics. The idea of finding a stable subspace, instead of the specific basis set, is simple but effective. The demonstration in Fig 1 is also impressive.

I think the "BubbleWrap" algorithm is an interesting take to the problem, with a novel combination of ideas in the context of neural dynamics. However, the presented results are less convincing. In particular, it is not clear what are lessons from the analyses (and what is the potential use case for the method), or how the presented results support the usefulness of the method. My major concerns are below.

Major concerns:

- Log predictive probability is a key quantity in the analysis in the Experiments section, but it is not sufficiently described in the main paper. (It is understandable to keep the details in the supplements, but the main text should be self consistent.) In particular, it is not clear how one should interpret the log predictive probability. What is the message that should be read out of Table 1? Is the absolute value (mean) meaningful, or should one only compare values across different methods within each row? Also in Fig 3d-f and Fig 4b, the values are not really meaningful without having a sense of "how good is good".

- The authors already note in the paper that the proposed method takes flexibility at the cost of reduced accuracy. But in this case, it should be discussed how the (possibly inaccurate) outcome of this method should be used towards a final understanding of the neural data. For example, in Fig 3a, although the highlighted tiles (red) roughly overlap with the trajectory (blue), there are considerable discrepancy, especially for the lower trajectory; if you do any downstream analysis based on this representation, it is inevitable that these analyses will be flawed. If BubbleWrap is meant to be a preliminary analysis, what follow-up analyses could it guide?

- Related to the accuracy issue: the "bubble" structure, due to the Gaussian-mixture nature of the model, introduces an artificial heterogeneity to the probability density. It would be necessary to discuss the effect of these "bubbles" on further analysis based on the resulting probability, and/or ways to remove this artifact by adding a postprocessing step.

- Fig 3b: Text description (L180) says that this dataset was chosen to demonstration a case with rapid transitions between noise clusters. Because the model explicitly infers transition probabilities, it is important that they are shown and discussed in this case, not "omitted for clarity". Speaking of the transition probabilities, it seems that the black lines in Fig 3a do not always correctly capture the trajectory.

Minor comments:

- L107 and Fig 1c: it is not clear how to read out the tradeoff value around n=200 from this figure. In fact, the color-coded points looks like n is transitioning from somewhere around 10 (blue, first point) to around 500 (red, second point) with no intermediate points in between. In general, please indicate the actual values of n in the figure if the value is relevant.


**EDIT**: The author response to my comments, as well as to the comments from the other reviewers, helped clarify my original concerns, which were:

> it is not clear what are lessons from the analyses (and what is the potential use case for the method), or how the presented results support the usefulness of the method.

As for the first point (use case), it was crucial for me to understand that the target use case of this set of methods was close-loop experiments, which I missed in my initial reading of the paper. Now I better appreciate the potential usefulness of the method in real time application.

As for the second point (results), the authors addressed my specific questions in the response. Also importantly, as I was reading the responses to the other reviewers, I realized that I was misunderstanding the "entropy" in the BubbleWrap results as the entropy of the state-space (x) distribution, where it was the entropy of the predictive conditional (z|x). This makes a lot of sense and it allows me to re-evaluate the results in the paper. Clarification on the HMM/GMM terminology was also helpful.

Given this, I am happy to increase my rating from 4 to 6, and now vote for acceptance, under the assumption that the authors will describe these potentially confusing points more clearly in the final version of the manuscript.
limitations_and_societal_impact: The authors discussed limitations of their approach, including the decreased accuracy as a tradeoff to gain flexibility. 

I do not see any potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present Bubblewrap which is a method for dimensionality reduction and discrete approximation of dynamics for high dimensional timeseries datasets. Bubblewrap stabilizes incremental basis estimates into an incremental subspace estimate and the authors show that this converges quickly in practice. The authors then use an HMM with Gaussian emissions to model the projected data along with several heuristics to achieve better performance. The method in benchmarked in terms of likelihood and latency/data rate on synthetic and real neural and behavioral data.
main_review: The manuscript is clearly written and methods that can be run in online during high dimensional neural recordings are important for closed-loop experiments. The reported results are clear and although I have not run the code, it looks complete and uses publicly available datasets.

I'm not an expert in closed-loop neuroscience experiments, but it seems like methods might potentially trade-off
1. latency,
2. the ability to predict the future states of the data Bubblewrap is modeling (e.g., neural states for optogenetic manipulation), or
3. the ability to predict paired timeseries like behavior (e.g., for robotic or task manipulations during neural recordings).

Run-time comparisons for 1. versus refs [30, 32] could be made. 2. is analyzed clearly for this method.

In my opinion, the main area where the manuscript could be strengthened is by including some evaluation of the utility of the inferred discrete states (i.e., 3.). Can the behavioral state of the animal be predicted? The increased log predictive probability compared to other online methods is promising, but some analysis of the ability to predict behavior would make the method more compelling.

General comments/questions
- Lines 126-127: I think many people would consider a HMM with Gaussian emissions to be a state-space model with a noise distribution. Can this sentence be made more precise? The related sentence on lines 214-215 could also be made more clear, I believe the continuous versus discrete dynamics distinction is more salient than moment-to-moment prediction versus tiling. Ultimately, Bubblewrap also learns the tiling through maximizing the data likelihood, which is another way to say prediction. I think the sentence on lines 120-122 has a more clear statement of the comparison.
- Could you provide some interpretation of the plots of entropy in Figs 3 and 4? Is it meaningful that Fig 3 d and f drop closer to zero while Fig 3 e stays closer to the max? Does it tell us something about the neural dynamics?
- In Fig S2, what is the grey in the background of the right column mean?
- Lines 225-226: can you expand on what the most important parameters to tune were, if any?
- The caption for the plots of log predictive probability and entropy don't describe what the black points are? Smoothed average?

Minor comments/questions
- Is there a reason that sparse random projections are used for the initial projection rather than dense random projections? Is there a computational benefit?
- For Fig 4 c, having stacked bars and a log-scale y axis makes it difficult to compare times if they don't happen to be anchored to the same lower point. Having 4 bars (3 parts + total time) grouped side-by-side for each case may help. I would have also expected the "Prediction" time to scale with N, am I misunderstanding something?
limitations_and_societal_impact: I would like to see more analysis or discussion about how the choice of discrete inferred states may impact potential uses for this method. Can the sequence of inferred centroids be used? Are the centroids or networks of centroids related to behavior?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present an online method for estimating low-dimensional neural dynamics that scales well to large observation dimensions and high sampling rates. The first step of their method is an online version of SVD, designed so that it converges to a stable subspace with very little training data. The second step of their method is an online version of an HMM with a large number of states that tile the low-dimensional space provided by SVD. They demonstrate their method on both simulated and real data, demonstrating both its speed and accuracy.
main_review: originality: this work is based on a growing literature of online algorithms designed to capture neural dynamics; the contribution here is a semiparametric model that is not based on a dynamical system (such as LDS), and the authors show their model works better in the high noise regime. 

quality: the model is well-motivated and sources methods from a variety of disciplines (random matrix theory, linear algebra, graphical models). The main feature of the model is its online ability to fit data, and with this in mind the authors do a good job of benchmarking the model, as well as comparing it to relevant baselines. However, I am still left wondering what a potential use-case for this method is - perhaps the authors could give an example in the introduction?

clarity: the manuscript is well written and the model is well-motivated. One point of confusion for me was L123-127: the authors mention both a GMM and an HMM, and I first I thought we were dealing with an HMM with GMM emissions; however, it appears that the emissions are standard Gaussians. Maybe the GMM terminology could be dropped in this case?

significance: online estimation of neural dynamics/state is crucial for causal experiments, and as such successful methods will likely be widely adopted. This method seems well suited to such estimation, though the authors have not shown if their method is accurate enough to enable the targeted interventions required for a specific experiment.
limitations_and_societal_impact: The authors have addressed some of the limitations in the discussion. One thing I'm curious about that did not seem to be addressed was the sheer number of states in the HMM - >1k!! In L49 there is mention of a "sparse transition graph", but I found no further mention of this in the main text or the supplemental. It would be useful if the authors could briefly mention how they handle such a large transition matrix, both during training and inference.
review_title: null

Meta-review: After discussion, the reviewers all converged on accept or weak accept ratings for this paper, once the stated changes are implemented. The methodological novelty is somewhat limited, but the problem is important, the approach is sensible, and the results are good. 
Decision: Accept (Poster)