Consistent Neural Embeddings through Flow Matching on Attractor-like Neural Manifolds

Authors: Puli Wang, Yu Qi, Yueming Wang, Gang Pan
Primary area: applications to neuroscience & cognitive science
Keywords: Brain-Computer Interface, Neural Decoding, Flow Matching, Dynamical Stability

Abstract:
The primary objective of brain-computer interfaces (BCIs) is to establish a direct connection between neural activity and behavioral actions through neural decoders. 
Consistent neural representation is crucial for achieving high-performance behavioral decoding over time.
Due to the stochastic variability in neural recordings, existing neural representation techniques yield dynamical instability, leading to the failure of behavioral decoders in few-trial scenarios.
In this work, we propose a novel Flow-Based Dynamical Alignment (FDA) framework that leverages attractor-like ensemble dynamics on stable neural manifolds, which facilitate a new source-free alignment through likelihood maximization.
The consistency of latent embeddings obtained through FDA was theoretically verified based on dynamical stability, allowing for rapid adaptation with few trials.
Further experiments on multiple motor cortex datasets validate the superior performance of FDA.
The FDA method establishes a novel framework for consistent neural latent embeddings with few trials. 
Our work offers insights into neural dynamical stability, potentially enhancing the chronic reliability of real-world BCIs.

Decision: reject
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729478406902

thread_type:
official_review

<thread_object>
creator: reviewer_ehkk

<content>
summary:
The authors proposed a flow-based framework to align neural embeddings across days. They benchmarked their model against five existing models, three of which focus on BCI alignment: ERDiff (based on diffusion), NoMAD (based on LFADS), and Cycle-GAN (based on Cycle-GAN). Their proposed model consistently outperformed the others across various benchmarks.

strengths:
1. The alignment of raw neural signals or latent embeddings is crucial and necessary for real-world BCI applications.

2. To the best of my knowledge, this is the first study to apply flow-matching to BCI signal alignment.

3. Extensive benchmarking was conducted against state-of-the-art models.

4. The writing is clear and easy to follow.

weaknesses:
1. I am concerned about the evaluation of NoMAD and Cycle-GAN. In a highly relevant submission (https://openreview.net/forum?id=LNp7KW33Cg), which almost certainly comes from the same group, the R² of NoMAD and Cycle-GAN is nearly six times better than what is reported in this paper.

2. This paper uses the exact same dataset as the Cycle-GAN paper (https://elifesciences.org/articles/84296). In the Cycle-GAN paper, the average R² is above 50% (Figure 3A), consistent with the companion submission I mentioned earlier, but in this paper, the evaluation shows an average R² below 10% (Table 1). Why is there such a significant discrepancy?

3. It is inappropriate to cherry-pick results in the presentation. Main Table 1 and Supplementary Table 8 are nearly identical. However, the authors included nine sessions per monkey in Table 1 but omitted two sessions that had the worst performances (Day 22 and 25 in CO-M; Day 40 and 77 in RT-M). Highlighting the best two or three sessions is one thing, but removing the worst two or three sessions is something entirely different.

questions:
1. I am still confused about the term "attractor-like." The authors mentioned it in the context of "*utilizing attractor-like ensemble dynamics (Gonzalez et al., 2019), a representation mechanism for encoding stimuli in the brain.*" I reviewed the referenced paper, but found that "attractor" is only mentioned once: "*Second, attractor-like mechanisms ensure the persistence of representations over short periods of time (days), even if the animals are not exposed to the task or if the circuit is perturbed by lesions.*" I believe additional explanation of what you mean by "attractor-like" in this context would be helpful.

2. Azabou et al., 2023a and Azabou et al., 2023b are the same paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730403519543

thread_type:
official_review

<thread_object>
creator: reviewer_rpd1

<content>
summary:
This paper proposed a novel Flow-Based Dynamical Alignment framework to obtain consistent neural representation. The FDA approach uses flow matching techniques to extract dynamics on stable manifolds. The FDA work addressing the challenge of dynamical instability offers insights into neural dynamical stability. The latent extracted from FDA has better decoding performance.

strengths:
1.	The proposed FDA method does perform better than baselines on decoding cursor velocity, and the learned latent space has more stability than baselines.
2.	The proposed method has relatively better computational efficiency although may need more parameters.
3.	The authors did multiple ablation studies on main components.

weaknesses:
One weakness (as is already notified by the authors) is the lack of cross-subject validation, for example, between Monkey C and M.

questions:
1.	The results on Monkey C seem very different from Monkey M, could you explain the reason?
2.	How many trials are in the source domain?
3.	The authors computed average MLE, what is the average across? Then what are all MLEs instead of the average one?
4.	Is fine tuning the reason that FDA performs better on few-trials scenarios?
5.	How do you choose the hyper parameters? Especially the dimensionality of your embedded latent space. Also, when you compare all the different models, do they have the same latent dimension?
6.	I am just curious, in Table 1, for each method, the worst r2 is in a different day, e.g., in CO-M, LSTM has the worst r2 in day29, but in FDA-MLA, it is just day8. Could you explain the reason?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684635360

thread_type:
official_review

<thread_object>
creator: reviewer_eh95

<content>
summary:
The paper proposes FDA - an alignment method to maintain performance of BCI decoder across recording sessions. The method is based on flow matching to achieve consistent neural embeddings which is theoretically shown to be dynamically stable and facilitates alignment in few-trial scenarios. Alignment performance is validated on multi-day datasets of monkeys performing motor tasks, showing competitive results against other baselines.

strengths:
* The method based on flow matching is original, tackling an important and long standing issue in BCI field.
* Problem formulation makes sense. Quantitative results are quite thorough with comparison using several baselines and datasets.
* The paper is well written for the most part. Figures are clear and well presented.

weaknesses:
* The paper cited a recent related work [1] but did not compare with it. Incorporating this additional baseline would provide a more comprehensive evaluation of the proposed method.
* Some texts are unclear and need more elaboration (details in Questions).

[1] Ayesha Vermani, Il Memming Park, and Josue Nassar. Leveraging generative models for unsupervised alignment of neural time series data. In The Twelfth International Conference on Learning Representations, 2024.

questions:
* What aspect in the proposed method uses "attractor-like ensemble dynamics”? Can the authors provide more clarification on what “attractor-like ensemble dynamics” mean and how it is relevant to certain part of FDA? The term is used a lot in the Introduction but was never referred to again in the Methodology or Experiments & Results. 
* Line 124: how was the signal window sampled? how many windows are sampled per trial? Are the windows overlap with each other? 
* Line 125: why behavior label $y_i$ is only taken at the w-th timestep of $x_i$? Does this mean the method decodes the downsampled behavior time series rather than the original one? The other baselines didn’t seem to have the behavior target downsampled. Decoding the downsampled behavior might make it an easier task than decoding the original behavior.
* Figure 1: in the first block, shouldn’t $c^S$ be at the top and $c^T$ be at the bottom?
* Also figure 1: according to description in Methodology section, $x^T$ and $z^T$ should not be used during Pre-training phase (left and middle blocks)?
* Line 185: does the transformer utilize positional embeddings? If so, what kind of positional embeddings was used?
* Line 206: how was $\eta$ pre-defined? Is $\eta$ kept the same across days? 
* Figure 2c: Is each point on the plot the average of all test sessions or average of different choices of samples with the same ratio $r$? Providing this clarification and also adding errorbars for each point would make it more informative.
* Also figure 2c: will performance of other baselines improve and reach the same performance of FDA if $r$ increases? If so, at how many trials will they become comparable to FDA? This is important to gauge the helpfulness of FDA in cases where scarcity of target samples is not a problem. 
* Figure 2d: why there are 9 days in Table 1 but 11 rows/columns in the shown matrices?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689218610

thread_type:
official_review

<thread_object>
creator: reviewer_fr7z

<content>
summary:
This paper proposes an consistent neural embeddings using flow matching, and leverages attractor-like ensemble dynamics. The numerical experiments showed consistent alignment results and better results than existing algorithms. This paper also theoretically showed the stability on the alignment using the algorithm.

strengths:
This paper uses the source-free alignment via likelihood maximization (FDA-MLA) and uses pre-training and fine-tuning to achieve consistent neural embeddings from non-stationary neural signals. The set up of latent extraction is via conditional feature extraction based on neural dynamics.

weaknesses:
1. The dynamical stability verification is not clearly explained for the theoretical support.

2. Some results are puzzling (see Questions below).

questions:
1. In part 3.2.2, can you explain the reason of using maximum mean discrepancy, why it is better than other matrix, can you show it ?

2. For table 1, can you explain why the results of ERDiff is extremely different and worse than the others, that is a relatively new paper published in NeurIPS 2023.

3. For table 1, why the R2(%) decreases to 23.79(FDA-MLA) and 45.23(FDA-MMD) at Day 8, but increases to 50.15 and 55.9? In the Cycle--GAN paper, the R2 decreases continuously. My understanding is that the alignment should be worser with longer time-gap relative to Day 0.

4. For table 1, why the R2(%) of CEBRA is much better in RT-M dataset than CO-M dataset?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Flow-Based Dynamical Alignment (FDA), a method for aligning neural embeddings across recording sessions using flow matching techniques. The approach aims to address the challenge of dynamical instability and achieve consistent neural representations, validated through experiments that demonstrate improved decoding performance and alignment compared to several existing methods.

The paper's strengths lie in its innovative use of flow matching, which tackles a critical issue in brain-computer interface research. The authors conducted extensive benchmarking against multiple baselines, showing competitive improvements in decoding performance and stability. The methodology is computationally efficient and includes ablation studies to validate key components of the proposed framework.

However, several weaknesses detract from the paper's overall impact, as outlined below. I appreciate the reviewers' comments, which have significantly helped improve the paper, as well as the authors' rebuttal. However, I believe there are some fundamental issues, as mentioned below, that cannot be fully addressed within the brief rebuttal period. I recommend that the authors continue refining the paper to enhance its quality for future submissions.