Title: Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Deep Learning

Missing Data

Dynamical Models

Neuroscience
Abstract: Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduces Multiscale Real-time Inference of Nonlinear Embeddings (MRINE) that is a novel learning framework designed for real-time, recursive decoding by dynamically and nonlinearly aggregating information from multiple neural time-series modalities.\n\nIt specifically addresses challenges posed by multiscale data, handling modalities with different timescales (e.g., fast spiking activity and slower LFP activity), varying probabilistic distributions (e.g., Poisson for spikes and Gaussian for LFP), and missing samples.\n\nKey to MRINE is its novel multiscale encoder design, which first learns within-modality dynamics using Linear Dynamical Models (LDMs) to account for timescale differences and predict missing data, then performs nonlinear information fusion via a fusion network. The training objective incorporates a multi-horizon k-step-ahead prediction loss, smoothed reconstruction, and smoothness regularization on latent dynamics to encourage learning meaningful representations and prevent overfitting. It also introduces a time-dropout technique to further enhance robustness to missing samples during training.\n\nEmpirically, MRINE has demonstrated its ability to improve real-time target decoding in both stochastic Lorenz attractor simulations and two independent nonhuman primate (NHP) brain datasets. It outperforms various linear and nonlinear multimodal benchmarks, showing particular strength when modalities have different timescales or missing samples, and also exhibits robustness to randomly missing samples during inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n*   The MRINE framework is designed for real-time recursive decoding from multiscale neural modalities. Its architecture, featuring a multiscale encoder with modality-specific Linear Dynamical Models (LDMs) and Kalman filtering, enables nonlinear information aggregation and intrinsically accounts for missing data, avoiding suboptimal zero-padding.\n*    MRINE consistently outperforms various linear and nonlinear multimodal benchmarks (MSID, mmPLRNN, MMGPVAE, MVAE) in both stochastic Lorenz attractor simulations and nonhuman primate (NHP) brain datasets. The model demonstrates strong robustness against randomly missing spike or LFP samples during inference, outperforming baselines. It also achieves competitive performance in neural reconstruction.\n*   Its real-time decoding capabilities and improved accuracy from multimodal fusion show potential for advanced brain-computer interfaces.\n\nWeaknesses\n* The model is based on the assumption of stationary linear dynamical systems, which may limit its capacity to effectively capture the complexity of neural dynamics.\n* Multiscale constraint: The framework requires that one time index be a subset of the other, which may restrict modeling flexibility.\n* Real-time performance: The paper lacks explicit discussion of computational efficiency or scalability. The term \"real-time\" appears to refer to the causal (filtering) nature of the inference procedure. To substantiate claims of real-time capability, the authors should demonstrate that the computational complexity per time step is constant with respect to the number of samples, and that the actual computation time per step does not exceed the duration of the time bin(s).\n* Evaluation methodology: The paper does not clearly specify whether and how k-step prediction was applied to the non-human primate (NHP) datasets. As a result, it is uncertain whether the proposed approach is effective at long-term forecasting or primarily suited for short-term tracking.\n* Comparison of latent factor decoding: The compared methods utilize different encoder and decoder architectures, typically involving nonlinear mappings between latent representations and observations. Therefore, evaluating performance based on linear decodability or Pearson correlation of latent factors may not provide a fair or meaningful comparison.\n* Related work: A technique similar to time-dropout has previously been explored in https://openreview.net/forum?id=Ln8ogihZ2S.\n\nTypos\n- 727:  MMGVAE => MMGPVAE\n- 857: Wilcoxon-signed => Wilcoxon signed-\n- 920: most clear => clearest\n- 981: missing sample => missing samples\n\nQUESTIONS:\n- What is the computational complexity (in Big O notation) per time step, and what is the actual computation time required per time step in practice?\n- How effective is the model in terms of decoding performance when applied to reconstructed observations?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MRINE, a method for real-time decoding of behavior from multimodal neural data with different timescales. By learning within-modality dynamics and fusing information nonlinearly, it improves decoding accuracy on simulations and NHP datasets compared to existing methods, showing potential for real-time BCI applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Good motivation: tackles multimodal fusion of LFP and spikes, missing data, and timescale mismatch.\n- Experiments are detailed and thorough.\n\nWeaknesses\n- Overall method novelty is limited: The MRINE method can be roughly understood as a combination of multimodal inputs (LFP and spike signals), nonlinear dynamics to extract latent variables, fusion of these latent variables, and a self-encoding structure to reconstruct neural signals. While this integration is useful, the overall novelty of the method is limited. The approach is largely based on existing techniques, such as multimodal fusion, nonlinear dynamic systems, and autoencoders, without introducing a fundamentally new concept. Additionally, many spike decoding methods bin spikes into time windows and model the firing rates as Gaussian distributions. This method is widely used and effectively captures the high-frequency nature of spike signals. It would be helpful to discuss the advantages and limitations of this approach and compare it with MRINE’s decoding strategy.\n- Uses too few channels and sessions: Although the paper claims the dataset contains 96 channels of neural data, the original dataset cited actually includes two 96-channel electrode arrays. However, the experiments in the paper only used 20 channels. This reduced number of channels is somewhat limiting for decoding performance, as even channels with low R² values may still carry valuable movement-related information. If the number of channels increases, a single modality signal (such as spike or LFP) might be sufficient to perform the motion decoding task, which makes the use and fusion of multimodal data seem less necessary. Furthermore, the number of sessions used in the study is limited, which raises concerns about the generalizability of the results. A larger dataset with more sessions would provide a more reliable assessment of the model's robustness and its ability to generalize across different experimental conditions.\n- Lacks comparison with some mainstream or latest decoding methods like LFADS, CEBRA, MINT, and BAND.\n- Does not consider non-stationarity of neural signals: Neural signals are inherently non-stationary, meaning their statistical properties can change over time. The relationship between neural signals and movement signals is likely time-dependent, and encoding functions may vary over time. However, MRINE assumes that the encoding model remains stationary, which overlooks the time-varying nature of neural data. This assumption could lead to unstable performance in decoding tasks, particularly in long-duration experiments or varying conditions. Incorporating mechanisms to adapt to the non-stationarity of neural signals would make the model more robust and applicable to real-time scenarios.\n- Real-time performance is not evaluated: Although the paper claims that MRINE is designed for real-time decoding, it does not provide a detailed evaluation of the model's decoding efficiency. Real-time performance, including latency, throughput, and computational demands, is critical for brain-computer interfaces (BCIs) and other real-time applications.\n\nQUESTIONS:\n1. Why did the authors only use 20 channels when the dataset provides two 96-channel Utah arrays? Later channels with lower R² can still carry meaningful information.\n\n2. Why is decoding accuracy evaluated only with CC? A CC around 0.6 is not ideal—what is the decoding R²?\n\n3. How is the real-time performance (latency, throughput) of the decoder evaluated in practice?\n\n4. Why does Table 7 use ranks, and what criteria were used for ranking?\n\n5. In Figure 6, what happens beyond 0.8? The trend has not yet plateaued at that point.\n\n6. How difficult is training this network in practice, and what ensures convergence during training?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method based on dynamical systems for analyzing multi-modal, multi-scale neural data. The proposed method, unlike prior methods, accounts for differences in scales (sampling rates) in the modalities. It first encodes each modality separately and applies non-linear fusion to learn a shared representation.  From this representation, it reconstructs the data across modalities.  Through shared representation, it allows for missing data. Using both synthetic and real-world experiments, the paper demonstrates improved behavioral decoding over both single-modal model and multi-modal models, which do not account for timescale differences.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n \n\nS1: The paper is well-written, with a good literature review and a clear problem statement. The paper provides a good argument for why the non-causal methods fail in multi-scale settings, motivating a causal approach. \n\n \n\nS2: While other methods have proposed using deep learning based dynamical models for single-scale and also multi-scale linear LDS models, this paper considers a newer challenge (at least for deep dynamical models) of non-linearly fusing information across modalities, while still maintaining real-time inference, and handling missing data.  \n\n \n\nS3: The work conducted an interesting set of experiments to show the utility of fusing information across multiple modalities, while still accounting for their timescale differences of the modalities.  The ablation studies demonstrate the effectiveness of different components of the model in improving behavioral decoding in a multi-scale setting.  \n\nWeakness  \n\n \nW1: The number of channels used for the experiment (5-20) is rather small. This deviates from a realistic setting with recordings having 100s of simultaneously recorded neurons (channels). The plots shown in Figure 3 show the diminishing returns in about 10 channels. The current experiments are limited in assessing the impact of the method in more realistic settings.  \n\n\nW2: The evaluation metric for both latent reconstruction (section 4.1) and decoding accuracy (section 4.2) somewhat feels unjustified, since most prior approaches use R2 to directly measure the reconstruction error instead of correlation. How does the correlation inform us about the performance of the model?  Therefore, interpreting percentage improvement in this metric is hard. And plots comparing groundtruth and inferred latents/behavioral quantities could be helpful to see.  \n \n\nW3: The experiments appear to be limited to LFP and spikes, but cited prior methods like MMGPVAE explore wider ranges of multi-modal data, such as (temporal) video recordings of behavior, calcium imaging etc. Absence of other modalities beyond LFP and spikes limits the broader application of the method.\n\nQUESTIONS:\nQuestions on architecture\n\n* On the experiments, including results with more common evaluation metrics such as R2, and considering realistic data settings (see weakness W1) could improve the paper. \n\n* In Figure 1b, the indexing for both modalities appears to be the same. Do you apply prediction for a common set to time indices, before concatenation? \n\n* If Fig 1b, And are the MLPs time invariant (the arrows indicate all inputs (across time) are fed to it at once)?  Maybe adding clarifying text could help. \n\n*  On the smoothness regularizers (Eqn. 10, 3rd term), the choice for fast/slow latent dimensions seems arbitrary. Any insights on the effect of this choice?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission considers real-time bi-modal inference of a discrete and continuous neural modality evolving at different timescales with missing data. The authors combine several ideas in their work, including a multi-scale encoder and recursive variational inference based on Kalman filtering or smoothing. Empirical results on NHP indicate that the method improves the decoding and reconstruction performance compared to previous approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe combination of different regularisers, model architectures and inference models is indeed interesting. \n\nThe zero-imputation for missing values for the baseline methods seems a bit simple. It would be interesting to see how alternative approaches would peform using less naïve imputations, e.g. just a simple Kalman filter for the continuous modality. \n\nThe rationale why one needs the ‘multiscale embedding factors’ a are not fully clear to me. Why can I not just use x instead (and pre-condition with C, and incorporate the noise r into the noise w?).\n\nThe computational complexity of the method relative to the baselines should be made clearer. For example, I would assume that the method can be beneficial compared with the multi-modal GP approach which may require some approximations for many observations.\n\nI found the presentation to be a bit unclear at times. For example, how are the factors and parameters in (7) with ^s and ^y related to the counterparts in (1)? Does this mean that in the generative path (1), there is no independence assumption, while on the inference/encoder path, the two modalities are encoded independently until they are aggregated via an MLP at the final layer?\n\nQUESTIONS:\nTo clarify, (5) means that $a_t$ is deterministic given $s_t$ and $y_t$, so these factors have a point mass distribution. But then in (6) and (7), the modality-specific versions seem to be stochastic with Gaussian noise, and how they are related is not fully clear to me. As far as I understand $x_{t|t}^s$ is then just the mean of the filtering distribution of the latent factor, and does not capture the variance/covariances of the different factor dimensions? \n\nIs T’ subset T a necessary constraint, so that the discrete modality evolves faster?\n\nIn the model description 3.2, it is not clear where the last MLP from Fig.1 comes in?\n\nIt would be useful to analyse how previous work perform in the stochastic Lorenz attractor simulations. Does the relative performance of the suggested approach compared to previous work in this example depend on the specific task/evaluation, such as latent reconstruction accuracy, real-time k-step ahead data predictions? How sensitive are these evaluations with respect to the weighting hyperparameters in the loss function? \n\nWhy do the authors suggest the smoothness regularization term, instead of just using a KL penalty for sequential VAEs? Do the smoothness parameters adapt to the scale of each modality?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper introduces a multi-modal encoder/decoder architecture that is specially designed to work with modalities that have different timescales (e.g. binned spikes, and LFP signals). The architecture is also robust to missing samples in either modality. Empirical evaluation is performed on synthetic, as well as real-world (offline) decoding datasets.

STRENGTHS AND WEAKNESSES:
## Strengths:

- Statistical significance tests have been performed for most results. This is highly appreciated!
    
- The description of the method is relatively clear.
    
- Excellent ablation studies presented in the Appendix.
    

## Weaknesses:

- The method introduces a few hyperparameters. A description for how these were tuned would be beneficial for further use and adaption of the presented method.

QUESTIONS:
- What does “primary” modality mean? From the description of MRINE, i don’t see any reason to consider one of the 2 modalities as primary. Does “primary” mean the modality with the fastest timescale? If this is the case, then how can Gaussian modality be primary?
    
- In Fig 2a and 3a, the points corresponding to number of LFP channels = 0 -- Are these results from single-modality and single-timescale versions of MRINE?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03