Dynamical modeling for real-time inference of nonlinear latent factors in multiscale neural activity

Authors: Eray Erturk, Maryam Shanechi
Primary area: applications to neuroscience & cognitive science
Keywords: Multimodal deep learning, Missing data, Neuroscience, Real-time decoding

Abstract:
Continuous real-time decoding of target variables from time-series data is needed for many applications across various domains including neuroscience. Further, these variables can be encoded across multiple time-series modalities such as discrete spiking activity and continuous field potentials that can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps.  Existing nonlinear models of multimodal neural activity do not support real-time decoding and do not address the different timescales or missing samples across modalities. Here, we develop a learning framework that can nonlinearly aggregate information across multiple time-series modalities with such distinct characteristics, while also enabling real-time decoding. This framework consists of 1) a multiscale encoder that nonlinearly fuses information after learning within-modality dynamics to handle different timescales and missing samples,  2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. We further introduce smoothness regularization objectives on the learned dynamics to better decode smooth target variables such as behavioral variables and employ a dropout technique to increase the robustness for missing samples. We show that our model can aggregate information across modalities to improve target variable decoding in simulations and in a real multiscale brain dataset. Further, our method outperforms prior linear and nonlinear multimodal models.

Decision: reject
Reviewer scores: 6, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730341690604

thread_type:
official_review

<thread_object>
creator: reviewer_9zgw

<content>
summary:
The paper, "MRINE: A Novel Framework for Real-Time Decoding of Target Variables from Multiscale Neural Time-Series Data," presents a new approach aimed at addressing multimodal neural data processing challenges. While the proposed framework could contribute to advancements in real-time decoding, several critical issues and areas for improvement weaken the paper's rigor, interpretability, and reproducibility.

strengths:
The efforts to handle missing data with techniques like smoothness regularization and time-dropout are noteworthy additions to the model.

weaknesses:
-(Lines 64-65): The claim that no existing models address multimodal neural data is inaccurate. Recent works, including those by Rezaei et al. [1] and Coleman et al. [2], have tackled similar challenges.
- (Lines 78-80): The assertion that other models do not address different timescales and missing samples in multimodal data is also incorrect. Models like those by García et al. [3] incorporate stochasticity in dynamical systems, relevant to noise handling in neural data.

-The paper lacks citations and comparisons to relevant models, particularly dynamical VAEs, which bear strong conceptual similarities to MRINE. This oversight diminishes the theoretical foundation of the paper and obscures recent advancements in variational autoencoders (VAEs) applied to dynamical systems.

-The equations in Section 2.1 contain inaccuracies, particularly in the notation used to describe multiscale dynamics. For instance, a single time index t is used, with no discussion on how different time horizons are managed, which undermines clarity regarding multiscale capabilities.

- (Equation 8): The absence of a derivation for the loss function weakens the rigor of the methodology section. Without a clear mathematical path to this objective function, readers are left without the means to evaluate or reproduce the proposed approach accurately.

-The paper lacks a comprehensive comparison with state-of-the-art models in time-series decoding, such as Mamba and S4. The selective comparison hinders the assessment of MRINE’s capabilities relative to these established baselines.

-Figure 1 does not effectively illustrate the multiscale time dynamics central to MRINE’s approach, diminishing its ability to convey the model’s core features.

-The bar plots throughout the paper lack statistical tests and error bars, raising questions about the reliability of reported performance differences. Including these would strengthen the study’s claims and clarify the significance of observed results.

-The theoretical foundation for MRINE’s multiscale capabilities is unclear, particularly in the equations defining these aspects. Some claims made in the introduction, especially those motivating the MRINE framework, appear overstated in light of prior work by Rezaei et al. [1], Coleman et al. [2], and García et al. [3].

-While these techniques are positive additions for handling missing data, they require further explanation, particularly regarding how they compare or improve upon existing methods for handling missing samples in neural data.


To strengthen the paper's contribution to the field, the following improvements are recommended:

Provide accurate citations and address the existence of related multimodal models that tackle similar challenges.
Include mathematical derivations for critical components, such as the loss function, to improve rigor and reproducibility.
Conduct comparative analyses with state-of-the-art models, particularly those in time-series decoding, to provide a balanced evaluation of MRINE’s capabilities.
Add statistical tests and error bars in visualizations to enhance the reliability of reported findings.
Clarify the use and advantages of techniques like smoothness regularization and time-dropout relative to existing methods.
References
[1] Rezaei, Mohammad R., et al. "Inferring cognitive state underlying conflict choices in verbal Stroop task using heterogeneous input discriminative-generative decoder model." Journal of Neural Engineering 20.5 (2023): 056016.

[2] Coleman, Todd P., et al. "A mixed-filter algorithm for dynamically tracking learning from multiple behavioral and neurophysiological measures." The dynamic brain: An exploration of neuronal variability and its functional significance (2011): 3-28.

[3] García, Constantino A., et al. "Stochastic embeddings of dynamical phenomena through variational autoencoders." Journal of Computational Physics 454 (2022): 110970.

questions:
-Could you clarify the novelty of MRINE relative to existing multimodal neural data models? For instance, how does it differ from models like those presented by Rezaei et al. [1] and Coleman et al. [2] that also address multimodal data processing?

- Given that MRINE bears similarities to dynamical VAEs, could you discuss how MRINE’s approach compares to these methods, particularly regarding handling multiscale dynamics?

-To provide a fuller evaluation, would you consider adding comparisons with state-of-the-art time-series decoding models like Mamba and S4?

-The equations in Section 2.1 describe MRINE’s multiscale capabilities, but the use of a single time index t leaves the handling of distinct time horizons unclear. Could you provide additional details or adjustments to the notation to clarify how the model represents multiple time scales?

-The log-likelihood calculation in Figure 3.C is not clearly explained. Could you provide a detailed description of the calculation process and how it relates to model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586327357

thread_type:
official_review

<thread_object>
creator: reviewer_swya

<content>
summary:
The authors propose MRINE,  a dynamical modeling method that nonlinearly aggregates information across multiple modalities with different distributions and distinct timescales,  and missing samples. In addition, it can support inference in real time. The authors apply MRINE to two datasets, one synthetic (stochastic Lorenz attractor simulations) and one publicly available neuroscience dataset (NHP dataset).

strengths:
1) Possible real world applications: Possibility of real time inference support is a great addition to many other multimodal models in neuroscience. This could possibly have positive applications to brain-computer interface research.
2) Robustness: The ability of the model to account for differing sampling rates and missing data is potentially impactful as much neuroscientific data is far from perfect, so models that take this into consideration are useful for real life applications.
3) The presentation is clear.

weaknesses:
- **Poor coverage of the literature**: Despite the discussion of the literature in the paper and comparisons to different models, I find the coverage of the literature to be at a very surface level. There was much work that was not cited in the paper that should be included to accurately show the status of the field now.  For example [2,3] are multi-modal models and [1] despite not being multimodal can handle missing data. As it is now, the paper doesn’t do justice to all the other models that can handle multi-modal data and thus draws an inaccurate/incomplete picture of multi-modal work in neuroscience. To improve this, I suggest incorporating the citations below to the ‘Multimodal Models in Neuroscience’ section in the paper as well as the introduction.
- **Comparisons**: Despite comparing the model to several different ones, I am not sure why it was not compared to some of the novel multimodal neuroscience models [2,3]. I understand there are already 5 comparisons but adding a comparison with more current models like the ones previously stated can be more refreshing, and useful,  and show how MRINE compares to more current models. 
- **Limited experiments**: The application of the model was limited, I wanted a demonstration with at least 1 more real experimental data to really make the model stand out and show its utility. Without validation on multiple real-world neuroscience datasets, it is challenging to assess whether MRINE’s claims generalize across different settings. 


**References**
1. Ramchandran, S., Tikhonov, G., Kujanpää, K., Koskinen, M., & Lähdesmäki, H. (2021, March). Longitudinal variational autoencoder. In International Conference on Artificial Intelligence and Statistics (pp. 3898-3906). PMLR.
2. Gondur, R., Sikandar, U. B., Schaffer, E., Aoi, M. C., & Keeley, S. L. Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data. In The Twelfth International Conference on Learning Representations.
3. Schneider, S., Lee, J. H., & Mathis, M. W. (2023). Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960), 360-368.
4. Zhou, D., & Wei, X. X. (2020). Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE. Advances in Neural Information Processing Systems, 33, 7234-7247.
5. Lee, M., & Pavlovic, V. (2021). Private-shared disentangled multimodal vae for learning of latent representations. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 1692-1700).

questions:
1. Did you try to apply the model into more complex datasets where the behavior was not just 2d reaches? I understand this might be the standard kind of experimentation in BCI but I would be interested to learn about the behavioral decoding ability of MRINE with images or videos. In other words, how does MRINE handle high dimensional datasets?
2. How does the multiscale encoder in MRINE handle differences in noise characteristics between spike and LFP modalities + what impact does this have on the overall performance of behavior decoding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689276179

thread_type:
official_review

<thread_object>
creator: reviewer_gtfz

<content>
summary:
Summary: The paper presents a model named MRINE, which is designed to infer latent factors from multi-scale neural data in real time. The contribution of this model is claimed to generalize the latent variable models to multi-timescales, multi-probabilistic distributions, and allow missing samples across modalities. The model can work on discrete Poisson spiking activity and Gaussian local field potentials (LFPs). The evaluation is worked on simulated and real-world neural datasets, which show performance gain in subject behavior decoding and latent dynamics reconstruction.

strengths:
1. MRINE's key components include a multiscale encoder for fusing modality-specific dynamics, a multiscale dynamical system backbone, and modality-specific decoders. 
2. The writing and presentation of the paper is fairly good and easy for the readers to follow.
3. It's quite interesting for combing and aggregating the infromation from various modalites in neuroscience.

weaknesses:
1. While I am inclined to hold the point that the quesiton/task this paper is focusing on has less scientific meaning, for model generalizaiton and for dealing with abnormal/missing data points.
2. From the modeling and ML perspective, the components in this MRINE's framework seems to be a combination of several existing techniques and empirical regularizers.

questions:
1. What properties do the Poisson observations and Gaussian observations have in common?
2. A have no more questions, other concerns please refer to the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700294210

thread_type:
official_review

<thread_object>
creator: reviewer_44y3

<content>
summary:
This paper presents MRINE, a nonlinear dynamical modeling framework for multiscale neural activity that enables real-time information aggregation across modalities with different timescales and/or missing samples. MRINE consists of a multiscale encoder that fuses modalities after learning within-modality dynamics, a multiscale dynamical backbone for extracting multimodal temporal dynamics and enabling real-time decoding, and modality-specific decoders to account for different probabilistic distributions. The authors also introduce smoothness regularization objectives and a time-dropout technique to improve robustness to missing samples. Through simulations and application to a real multiscale brain dataset, MRINE is shown to outperform prior linear and nonlinear multimodal models in fusing information across modalities to improve decoding, even with different timescales or missing samples, making it valuable for real-time neurotechnology applications.

strengths:
- Modeling neural dynamics across varying modalities.
- The ability to handle different timescales and missing samples.
- Enabling real-time decoding.

weaknesses:
- My main concern is that it’s difficult to find significant scientific insights in this work. It feels more like a combination of various features to achieve an improvement in behavioral decoding accuracy over the baselines, yet the 61% accuracy does not seem particularly high. In other words, the improvement in decoding accuracy alone doesn’t convincingly justify the introduction of different timescales and handling of missing samples, as I think baseline accuracy could potentially be enhanced through additional parameters or optimized training settings, such as changing the factor dimension in LFADS.

questions:
- Why were only 20 spike and LFP channels used? This seems like a small number. Does this mean the input feature dimension of the data is 20+20=40?
- Is it possible to compare the accuracy of baselines with missing samples in Section 3.3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732161525923

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response - 1

comment:
We thank the reviewers for their comments and suggestions on our work. We are pleased that the reviewers find that our problem formulation on combining and aggregating information across neural data modalities is ‘quite interesting’ (reviewer GtFZ), ‘possibility of real time inference support’ is a ‘great addition to many other multimodal models in neuroscience’ which can have ‘positive applications to brain-computer interface research’ (reviewer swya) and our ‘efforts to handle missing data with techniques like smoothness regularization and time-dropout are noteworthy additions’ (reviewer 9Zgw). 

Based on the comments, however, it seems that our main contributions may have been lost in the discussions about performance alone. Thus, before presenting our new results, we would like to emphasize that **the main novelty of MRINE lies in providing two main capabilities for multimodal neural modeling, rather than simply improving performance**: 

1.  Enabling recursive real-time nonlinear inference, which is critical for closed-loop ML/AI systems such as brain-computer interfaces. In contrast, most deep learning models of neural data – including our multimodal baselines and the new models that reviewers asked for comparison to – only perform non-causal non-recursive inference. 
2.  Modeling multiscale neural signals by accounting for their time-scale differences and missing samples, rather than pure performance gains when all samples are available. These are critical capabilities for real-time applications such as brain-computer interfaces (see also Introduction and Related Work). 

Therefore, we would like to highlight the ablation study we performed in Appendix E.4 (which now includes a new baseline we added), which may have been overlooked as we provided this result in our Appendix. In this ablation study, we show that even if the alternative models perform global mean-imputation (i.e., zero-imputation due to z-scoring) and/or masking for the missing LFP samples in their training objective, MRINE still significantly outperforms them.  

**GR Table 1**:
| Model    | Behavior Decoding (CC) |
| -------- | ------- |
| MRINE | 0.611 &pm; 0.012    |
| MRINE w/ Zero Imputation | 0.523 &pm; 0.013 |
| MRINE w/ Zero Imputation and Loss Masking | 0.581 &pm; 0.014    |
| mmPLRNN w/ Zero Imputation| 0.498 &pm; 0.009    |
| mmPLRNN w/ Zero Imputation and Loss Masking |0.539 &pm; 0.011     |
| mmLFADS w/ Zero Imputation | 0.280 &pm; 0.022   |
| mmLFADS w/ Zero Imputation and Loss Masking |0.253 &pm; 0.023    |
| MMGPVAE w/ Zero Imputation | 0.393 &pm; 0.023     |
| MMGPVAE w/ Zero Imputation and Loss Masking | 0.518 &pm; 0.011    |

As expected, all models have improved performance when missing samples are masked from the loss function (except mmLFADS which has poor performance due to high distortion in initial condition summarization), but regardless, MRINE outperforms all models including its own variants that add imputation and/or masking in training. These results clearly show that even adding imputation and/or masking are significantly inferior to the innovations in MRINE’s multiscale encoder to address multiple timescales. Thus, MRINE’s **multiscale modeling is crucial for modalities recorded with different timescales. This clearly shows the advantage of new capability 2 above.**  The new capability 1 is extensively shown in Figures 5 and 6.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces MRINE, a framework for real-time decoding of neural activity across multiple modalities, addressing challenges like missing samples and different timescales. It demonstrates improved decoding performance using both simulated and real-world datasets, with applications in neurotechnology.

Strengths include effective handling of missing data through techniques like smoothness regularization and time-dropout. The presentation is clear, and the real-time decoding capability is a valuable addition to multimodal neural data processing, with potential applications in brain-computer interfaces.

However, the scientific contribution of the paper is unclear, as the reported decoding accuracy (61%) is not significant enough to justify the complexity of the proposed model. Additionally, the paper lacks sufficient comparison with state-of-the-art models in multimodal neural data processing, which makes it difficult to evaluate MRINE's uniqueness and effectiveness. The theoretical foundations, particularly regarding multiscale dynamics, are weak, and the methodology lacks rigorous mathematical derivations, compromising reproducibility. Furthermore, the absence of error bars in performance visualizations raises concerns about the reliability of the reported results. These weaknesses undermine the paper’s scientific impact and rigor, leading to the recommendation for rejection.