Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This submission is a very well done application of the probabilistic inference to decoding of odour identity from neural recordings in mice. Interestingly, the latent code is assumed to have temporal dynamics shared across animals. The main benefits (high accuracy, data savings,  robustness to incomplete data) are shown in simulation and neural recordings. The method outperforms the existing (non-probabilistic) neural alignment procedures. 

main_review: To the best of my knowledge, this is a novel method for this important (BMI) application. It should have a large impact on the olfactory field, as it enables an effective combination of neural recordings from many animals.
The relevant work is adequately cited and discussed with two methods implemented for comparison.

The paper is clearly written and comprehensive. 

Minor comments:
- L59: Shouldn't $x_1$ be $\mathbf{z}_1$?
- L119: Equation, first line is missing an integral over $\mathbf{z}_t$.
- Update Figure 4 B caption ("dashed lines...")
limitations_and_societal_impact: I assume the odours used in the experiment were very different. I'd be curious to learn about the temporal latent characteristics over more odours. What are the limits to discriminability? Is the detection of Clostridium difficile even feasible? I'm looking forward to updates into the "space of odours" following up this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Recent advances in recording techniques allow us to record from a large population of neurons - this high dimensional activity can typically be described by low dimensional latent factors. The authors leverage this low dimensional structure to capture the shared dynamical structure across animals in neural recordings from the olfactory bulb as animals are exposed to various odor stimuli. They fit a linear dynamical system (LDS) model with latent parameters and trajectories estimated for each odor that are shared across animals. These latent trajectories are subsequently mapped onto the neural recordings by employing animal specific observation parameters. The LDS model parameters are optimized using expectation maximization (EM). In order to decode the odor identities, the marginal likelihood for all stimulus conditions is computed. They validate the model on simulated data as well as neural recordings from the olfactory bulb. 
main_review: Strengths: Aligning low dimensional structure in neural population recordings across animals is an open problem in neuroscience. The paper provides a novel approach to recover these shared latent dynamics across animals. They demonstrate that their model can recover previously observed low dimensional transient dynamics for odor stimuli in data recorded from the olfactory bulb. They are also able to decode the odor identity across animals by training the model on source animals and a small subset of trials from a target animal. Moreover, the model outperforms CCA and Factor analysis followed by Procrustes alignment on across animal decoding.

Weaknesses: The authors make a strong simplifying assumption of identical dynamics across animals that has not been adequately justified in the paper. 

Neural recordings during each session and across animals might be obtained from different locations in the same area or from a vastly different subset of neurons. It’s highly likely that the recorded neurons are part of different circuits and coding for different dynamical features. 

Animals also have different sniffing cycles that could introduce an added level of complexity in the temporal dynamics that will not be accurately captured by this model. It is not necessary that the dynamics will be exactly temporally aligned. 

Other factors such as varying levels of attention and arousal could further impact the dynamics of odor coding. 

Due to the underlying assumption, this method is limited in scope and its applicability to other tasks and areas remains unclear. Maybe a weaker assumption with constraints on the dynamics parameters; for example, by employing a hierarchical model (Linderman et al., 2019), could make this model more flexible and as a result, more widely applicable.

Originality: To the best of my knowledge, this is the first work that aims to align neural dynamics across animals by having a common dynamics model.

Quality: The paper is well motivated and the writing and figures is easy to follow. 

Clarity:
There are some minor parts that can be improved or might need further explanation.
Line 102: (typo) evaluated on on.
Line 77: It is not necessary to introduce a variational distribution since the authors are using regular EM. 
Fig 4: I’m not sure I understand the source of variation between the latent trajectories for the same stimulus and different animals in panel A. Some further explanation here would be helpful.

Significance:
Aligning low dimensional dynamics across sessions and animals is elemental for identifying common encoding principles in the brain. However, as mentioned above, the strong assumptions in the model put stringent constraints on the dynamics and limit its applicability to other brain areas or contexts. 


limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The report provides a method for decoding stimulus identity (odor) using data from multiple animals. The framework is straightforward and the performance appears respectable. I anticipate that this approach could be of use in neuroscience labs and could help with developing models and BCI. Given its general applicability, it would strengthen the impact if its use was demonstrated on spiking data and/or other types of stimuli or behaviors. This is a well written paper with helpful illustrations. I expect it would support interesting discussions.
main_review: Neural activity is variable, even when sensory stimuli or motor performance are repeated. However, the underlying perceptions or intentions could be quite similar. When analyzing neural activity, it can be good to isolate the conserved components, despite this variability. A straighforward example is averaging over trials, but this only works when the dynamics are relatively time-locked. When the dynamics can play out over a range of speeds, the problem is less trivial. In olfaction, sync'ing to the sniff cycle is one way to register across trials, and in that case, trials are time warped to match a landmarks in a sniff cycle. For this paper, the author(s) developed an approach for solving this problem, and it is more general than timewarping to match the sniff cycle. It can be applied to a range of applications, sensory modalities, and behaviors. The framework allows decoding across animals. The approach is based on the idea that each animal has a set of neural activity manifolds that are each specific for a stimulus or behavior. Manifolds can be combined across animals for decoding. A graphical model, an aligned mixture of latent dynamical systems (amLDS), is computed from the dataset with this idea. They compare performance to CCA and FA+Procrustes. On the one hand, these are standard tools and so it is a reasonable comparison. On the other hand, they are very general tools that are used without taking advantage of the structure of the data, and so it is not so surprising that they provide less accurate results. The performance they present in Fig. 4 looks respectable.

Since they present olfaction data, I am curious to compare this approach to simple sync'ing to a sniff cycle (and they have that data, line 219). But I do not consider this a major weakness. I wish they had presented an application to spiking data. I will not mark that as bad weakness, but I do think it is something that others in the field would like to see. Relatedly, the impact could be greater if the technique were applied to a range of data types (e.g., visual stimuli, motor activity in a reaching task, parietal activity in a two-alternative forced-choice task). Again, it is not a weakness, but I would appreciate seeing it, and I bet I am not alone.
limitations_and_societal_impact: No concerns.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors develop a method for aligning the neural activity of multiple animals. Specifically, they address the challenge of odor detection, where neural activity from multiple animals is recorded in response to many stimuli (odors), and then the odor needs to be decoded from a given animal’s neural activity. They use a hierarchical and probabilistic approach, in which there is a shared low-dimensional space of neural activity that is shared across animals, with different low->high dimensional mappings for each animal. The authors demonstrate the utility of the approach in simulations and on experimental neuroscience data.
main_review: Quality:

The quality of this paper is high. The submission is technically sound. The method was tested out rigorously in simulations and also shown to work well on experimental neuroscience data.


Originality:

This is a new method that builds upon linear dynamical systems / graphical modeling to create an effective model for the given problem.

The relevant literature was cited well.
One additional piece of relevant literature is a hierarchical switching linear dynamical system (which is similar to your approach but has an unknown k during training), as in
https://www.biorxiv.org/content/10.1101/621540v1.full


Significance:

I think the broad topic that the authors are trying to address - aligning neural activity across subjects has broad interest across the neuro/AI communities. However, the specific application of odor detection seems limited in terms of its broad appeal. I think this paper would be much stronger (and in my opinion, a clear accept rather than a borderline accept), if you were able to demonstrate its generality to other applications as well.


Clarity:

While much of the paper was written clearly, I found several aspects difficult to understand. In particular, I think it would be helpful to be more explicit in the beginning about what aspects of the model are learned during the training phase, and then what you are trying to predict during the testing/decoding phase (both for your model and the comparison models). Here are a couple specific points along those lines: 

During training, is it assumed that the odor is always known? It seems like yes, but please clarify up front.

Lines 171-196: I initially found this section confusing since you describe decoding after describing the state-space alignment, so I was confused how you were actually doing decoding w/ CCA and FA+Procrustes. Having a sentence upfront about the overall approach would be helpful for clarity.

On your target animal, are you updating any of the parameters of the model besides learning the C and R for that new animal?

When decoding in the comparison methods, are you only using the target animal labels (odors) for the purpose of aligning the datasets, or are they also used when training the decoder?



Other:

I think multi-set CCA (which is an extension of CCA for >2 subjects) would be another natural comparison.
limitations_and_societal_impact: Limitations are addressed

Potential negative societal impacts are not mentioned in the text
review_title: null

Meta-review: This paper presents a novel probabilistic alignment of stimulus driven neural activity for decoding stimulus identity. Reviewers agree that this is an important contribution that can contribute to advancing neuroscientific understanding. At the same time, reviewers also agree on the assumptions that may limit its applicability outside the olfactory system. Please make sure to add the discussions through the rebuttal in the final version.
Decision: Accept (Spotlight)