Generalizable, real-time neural decoding with hybrid state-space models
Authors: Avery Hee-Woon Ryoo
Nanda H Krishna
Ximeng Mao
Mehdi Azabou
Eva L Dyer
Matthew G Perich
Guillaume Lajoie
Venue: NeurIPS 2025 poster ()
Keywords: neural decoding
brain-computer interfaces
neural spiking data
state-space models
real–time
Primary area: neuroscience_and_cognitive_science
TL;DR: We present POSSM, a novel architecture that combines input cross-attention with a recurrent state-space model, achieving competitive accuracy, fast inference, and efficient generalization for real-time neural decoding applications.

Abstract: Real-time decoding of neural activity is central to neuroscience and neurotechnology applications, from closed-loop experiments to brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low-resource or real-time settings. To address these shortcomings, we present POSSM, a novel hybrid architecture that combines individual spike tokenization via a cross-attention module with a recurrent state-space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pretraining. We evaluate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at a fraction of the inference cost (up to 9x faster on GPU). These results suggest that hybrid SSMs are a promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.

Generalizable, real-time neural decoding with hybrid state-space models
Avery Hee-Woon Ryoo 12,,
Nanda H Krishna 1,2, Ximeng Mao 1,2,,,00
Mehdi 2zabou Eva L Dyer*
Matthew G Perich 122.t
Guillaume
Mila Quebec AI Institute
Université de Montréal CColumbia University
A University of Pennsylvania *Canada CIFAR Al Chair *Co-first authors *Co-senior authors
[hee-woon. ryooo, rooo, nanda. harishankar-krisshma ximeng. mao, gui laumee law.aaaeeeaaieeeealaa lajoie quebec nanda. https://possm-rraal to
Abstract
Real-time decoding neural activity is central to neuroscience and neurotech- nology applications. from closed-loop experiments to brain prain-computer interfaces, where models are subject strict latency constraints. Traditional methods, includ- ing simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-be based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low- resource or real-time settings. To address these shortcomings, we present POSSM, novel hybrid architecture that combines individual spike tokenization via a cross- attention module with recurrent space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to sessions, individuals, and tasks through ate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at fraction of the inference cost (up 9x faster GPU). These results suggest that hybrid SSMs are promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.
pretraining. We evalu-
Introduction
Neural decoding the process of mapping neural activity to behavioural cognitive variables is a core component of modern neuroscience and neurotechnology. As neural recording techniques evolve and datasets grow in size, there is increasing interest in building generalist decoders that scale and flexibly adapt across subjects and experiments. Several important downstream applications including closed-loop neuroscience experiments and brain-computer interfaces (BCIs) require fine- grained, low-latency decoding for real-time control [1]. Advances these technologies would enable next-generation clinical interventions in motor decoding [2], speech prostheses [3], and closed-loop neuromodulation [4, 5]. Building towards these applications will require neural decoders that meet three requirements: (I) robust and accurate predictions, (2) causal, low-latency inference that is viable in an online setting, and (3) flexible generalization to new subjects, tasks, and experimental settings. Although recent developments in machine learning (ML) have enabled significant strides in each of these axes, building neural decoder that achieves all three remains an open challenge.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recurrent neural networks (RNNs) [6] and attention- based models such as Transformers [7] have shown significant promise for neural decoding tasks [8-10]. RNNs (Figure la) offer fast, low-latency inference on sequential data and strong performance when trained on specific tasks [10]. However, their ability to gen- eralize to new subjects is limited due to their rigid input format. Specifically, their reliance on fixed-size, time-binned inputs means that they typically cannot learn from new sessions with different neuron iden- tities sampling rates without full re- re-training and/or modifying the architecture. In contrast, Transformer- based architectures (Figure 1b) offer greater flexibility thanks approaches [11-13]. Nonetheless, they struggle with applications involving real-time processing due their quadratic computational complexity, in addition to the overall computational load of the attention mechanism. Recent efforts in sequence modelling with large language models have explored hybrid architectures that combine recurrent layers, such as gated recurrent units (GRUs) [14] or state-space models (SSMs) [15-18], with attention layers. These models show encouraging gains in long-context understanding and computational efficiency [19-21]. While hybrid attention-SSS SSM approaches offer a promising solution for real-time neural decoding, to the best of our knowledge, they remain unexplored in this area.
b
Readout
RNN
Readout
Attention
binned spikes
OOOO
Figure I: Existing deep learning models for neural decoding. (a) Recurrent neural net-
works (RNNs). (b) Attention- baased models such as Transformers.
adaptable neural tokenization
We address this gap with POSSM', a hybrid model that combines the flexible input-processing of Transformers [22] with the efficient, online inference capabilities of recurrent SSM backbone. Unlike traditional methods that rely on rigid time-binning, POSSM operates at a millisecond-lewe resolution by tokenizing individual spikes. In essence, POSSM builds on POYO-style cross-altention encoder [13] that projects a variable number of spike tokens to fixed-size latent space. The resulting output is then fed an SSM that updates its hidden state across consecutive chunks in time. This architecture, as illustrated in Figure 2, offers two key benefits: (1) the recurrent backbone allows for lightweight, constant-tim time predictions over consecutive chunks of time and (2) by adopting POYO's spike tokenization, encoding, and decoding schemes, POSSM can effectively generalize to different sessions, tasks, and subjects.
In this paper, we introduce the POSSM architecture and evaluate its performance on intracortical recordings of spiking activity from experiments in both non-human primates (NHPs) and humans. Although our current evaluations are conducted offline, POSSM is designed for real-time inference and can be readily implemented for online experiments. Finally, while this paper is concerned with invasive electrophysiology recordings, this method could be extended further to other modalities using Ptyo-style tokenization (see Section for discussion). Our contributions are summarized as follows:
Performance and efficiency: We evaluate POSSM against other popular models using NHP datasets that contain multiple sessions, subjects, and different reaching tasks (centre-out, random target, maze, etc.). We find that POSSM matches or outperforms other models all these tests, doing so with greater speed and significantly reduced computational cost.
Multi-dataset pretraining improves performance: Through large-scale pretraining, we find that POSSM delivers improved performance on NHP datasets across sessions, subjects, and even tasks.
Cross- species transfer learning: Pretraining on diverse NHP datasets and then finetuning POSSM leads to state-of-the-ard performance when decoding imagined handwritten letters from human cortical activity [23]. This cross-species transfer not only outlines the remark- able transfcrability of neural dynamics across different primates, but also shows the potential for leveraging abundant non-human data augment limited human datasets for improved decoding performance.
Long sequence complex decoding: When trained on human motor-cortical data during attempted speech [3], POSSM achieves strong decoding performance. In contrast, attention- based models struggle with the task's ' long-context demands, making them computationally prohibitive.
POSSMM stands for POYO-SSM, and is pronounced like the animal ("possum") IPA: / pusom/).

---

Linear Readout
Linear Readout
Linear Readout
(SessEmb(s) bad)
(SessEmb(s) lex))
Cross Attention
Q Cross Attention
o Cross Attention
State-Space Model
State- State-Space Model
State-Space Model
Cross Attention KW
0 pm
Cross Attention
010
cm0
a Cross Attention
KV
KV
Buira
Suvn
upas UnitEmb(i
Uni
UnitEmb()
units
units
SeBS,
unils
BeE8,
units
I1I
11
Figure 2: An architecture for generalizable, real-time neural decoding, POSSM combines individual spike tokenization [12] and input-output cross-attention [22] with recurrent SSM backbone. this paper, we typically consider k = 3 and Te = 50 ms.
Methods
We focus on decoding neuronal spiking activity, which consists of discrete events triggered when excitatory input to neuron exceeds certain threshold. The timing and frequency of spikes encode information conveyed to other neurons throughout the brain, underlying communication system that is central to all neural function [24, 25J. However, their sparse nature [26] requires an effective input representation that can handle their temporal irregularity. Furthermore, there exists no direct mapping between the neurons of one living organism another, highlighting the non-triviality of the alignment problem when training across multiple individuals. this section, we describe how POSSM designed to sequentially process sub-second windows of these spikes for the online prediction of behaviour, while maintaining flexible input framework that allows it to efficiently generalize to an entirely new set of neurons during finetuning.
2.1 Input processing
Streaming neural activity as input. Our focus is on real-time performance, which requires minimal latency between the moment that neural activity is observed and when corresponding prediction is generated. This constraint significantly limits the time duration, and consequently the amount of data, that model can use for each prediction. To this end, POSSM maintains hidden state as data is streamed in, allowing it to incorporate past information without reprocessing previous inputs. each variable-length sequence
chunk, the number of spikes varies, meaning each input represented of spikes. While POSSM generally uses contiguous 50 ms time chunks here, we also demonstrate strong performance with 20 chunks (see Section D. I). In theory, these windows could even be shorter or longer (and even overlapping) depending on the task, with the understanding that there would be some trade-off between temporal resolution and computational complexity.
Spike tokenization. We adopt the tokenization scheme from the original POYO model, where each neuronal spike is represented using two pieces of information: (1) the identity of the neural unit which came from and (2) the timestamp which it occurred (see Figure 2). The former corresponds to a unique learnable unit embedding for each neuron, while the latter is encoded with rotary position embedding (RoPE) [27] that allows the tokens be processed based on their relative timing rather than their absolute timestamps. For example, spike from some neuron with integer ID i that occurs
3

---

at time Espike would be represented as D-dimensional token given by:
x (UnitEmb(i), (spike),
where UnitEmb 2 - RD is the unit embedding map and D is a hyperparameter. As tokenization is an element-wise operation, a time chunk with N spikes will yield N spike tokens. We opt to use the general term "neural unit", as spikes could be assigned at a coarser specificity than an individual neuron (e.g., multi- unit activity on a single electrode channel) depending on the dataset and task at hand. This flexibility, coupled with the model's 's ability handle variable number of units, facilitates both training and efficient finetuning (see Section 2.3) on multiple sessions and even across datasets.
2.2 Architecture
Input cross-attention. We employ the original POYO [12] encoder, where a cross-attention module inspired by the PerceiverIO architecture [22] compresses a variable-length sequence of input spike tokens into a fixed-size latent representation. Unlike POYO, however, the encoder is applied short 50 ms time chunks, each of which is mapped to a single latent vector. This is achieved by setting the spike tokens the attention keys and values, and using learnable query vector q R where M is a hyperparameter, Given a sequence X, = [Xx, X1. Xy] E R N'x of N spike tokens from some chunk of time indexed by I, the latent output of the cross-attention module is computed as such:
qK,
2 (t) = softmax
V
where K, = X, W. and V. = X,W, u. with W.. W. E RDXM, are the projections of the input token sequence, following standard Transformer notation. Following POYO, our implementation also uses the standard Transformer block with pre-normalization layers and feed-forward networks.
Recurrent backbone. The output of the cross-attention (9)) is then fed to an SSM (or another variety of RNN), which we refer to as the recurrent backbone. The hidden state of the recurrent backbone is updated as follows:
(+) = (z) ( ((-)))
While the input cross-attention captures local temporal structure (i.e., within the 50 ms chunk), the SSM integrates this information with historical context through its hidden state, allowing POSSM to process information at both local and global timescales. We run experiments with three different backbone architectures: diagonal structured state-space models (S4D) [15], GRU [14], and Mamba [16]. However, we wish to note that this method is compatible with any other type of recurrent model. Specifics regarding each of these backbone architectures can be found in Section B.2.
Output cross-attention and readout. To decode behaviour, we select the k most recent hidden states (h(t=++)x<0)) (k 3 in our experiments), and use a cross-attention module query them for behavioural predictions. For a given time chunk t, we generate P queries, one for each timestamp at which we wish to predict behaviour. Each query encodes the associated timestamp (using RoPE), along with a learnable session embedding that captures latent factors of the recording session. The design of our output module enables flexibility in several ways: (1) we can predict multiple outputs per time chunk, enabling a denser and richer supervision signal, (2) we are not required to precisely align behaviour to chunk boundaries, and (3) we can predict behaviour beyond the current chunk, allowing us to account for lags between neural activity and behavioural action (see Section D.5).
2.3 Generalizing to unseen data with pretrained models
Given pretrained model, we outline two strategies for adapting it to new set of neural units, with trade-off between efficiency and decoding accuracy.
Unit identification. To enable efficient generalization to previously unseen neural units, we adopt unit identification (UI), finetuning strategy enabled by the spike tokenization scheme adopted from POYO [12]. In UI, new neural units can be processed by a model simply by learning new embeddings. Using this approach, we freeze the model weights, initialize new unit and session embeddings, and then train them on the data from the new session while the rest of the model is kept unchanged. This allows us to preserve the neural dynamics learned during pretraining, resulting in an efficient generalization strategy that typically updates less than 1% of the model's total parameters.

---

b
o U P
Centre-out reach (CO)
Random target (RT)
Random target (RT) Controlier: Touchscreen
Maze centre-out reach Controller: Touchscreen
Controller: Menipulandum
Controller: Manipulandum
Monkey 2016/10/13
Monkey 201 2/18//2/2
hel
helo
hello
hellc
Ground tuuh C-P0SSN-RRJ
Ground c-PSSSN-RUU
0
2
3
0
2
3
Handwriting decoding
Speech decoding
time
Figure 3: Task schematics and outputs. (a) Centre-out (CO) task with manipulandum. (b) Random larget (RT) task with a manipulandum. (c) RT task with a touchscreen. (d) Maze task with touchscreen. (e) Ground truth vs. predicted behaviour outputs from a held-out CO session. (f) Same as (e) but for an RT session. (g) Human handwriting decoding task. (h) Human speech decoding task.
Full finetuning, While unit identification is an efficient finetuning method, it does not reliably match the performance of models trained end-to-end on individual sessions. To address this, also explored full finetuning (FT), a complementary approach which uses a gradual unfreezing strategy. We begin by only doing UI for some number of epochs before unfreezing the entire model and training end-to-end. This allows us to retain the benefits of pretraining while gradually adapting to the new session. As shown below, full finetuning consistently outperforms both single-session training and unit identification across all tasks explored, demonstrating effective transfer to new animals, tasks, and remarkably, across species.
Experiments
We evaluate POSSM across three categories of cortical activity datasets: NHP reaching tasks, human imagined handwriting, and human attempted speech. For the NHP reaching tasks, we highlight the benefits of scale by introducing o-POSSM, a POSSM variant pretrained on multiple datasets. We compare to single-session models on held-out sessions with varying similarity with the training data, demonstrating improved decoding accuracy with pretraining. Next, we show that o-POSSM achieves powerful downstream decoding performance on human handwriting task, illustrating the POSSM framework's capacity for cross- cross-species transfer. Finally, we demonstrate that POSSM effectively leverages its recurrent architecture to efficiently decode human speech a long-context task that can become computationally expensive for standard Transformer-baseee models with quadratic complexity. In each of these tasks, we see that POSSM consistently matches or outperforms other architectures in causal evaluation setting, with performance improving model size and pretraining scale increase.
3.1 Non-human primate reaching
We first evaluate POSSM on the task decoding two-dimensional hand velocities monkeys perform- ing various reaching tasks (shown in Figure 3a-f). Each training sample consists of s of spiking activity paired with the corresponding 2D hand velocity time series over that same interval. This window is split up into 20 non non-overlapping chunks of 50 ms, which are fed sequentially POSSM (see Section D.I for results on 20 ms chunks). This streaming setup enables efficient real-time decoding. where only the most recent 50 ms chunk is processed at each step. In sharp contrast, the original POYO model reprocesses an entire s window of activity with each new input, resulting in significantly higher computational cost.
Experimental Setup. We use similar experimental setup to POYO [12]. The pretraining dataset includes four NHP reaching datasets collected by different laboratories [28-31], covering three types
5

---

Table 1: Behavioural decoding results on NHP reaching tasks. Values are reported as mean R (f) + SD over sessions. The number of sessions in each dataset is contained in (-). C, T, and H are subject identifiers. SS: Single-session UI: Unit identification; FT: Full finetuning. The top performing models in each category are indicated in boldface (1st) and underlined (2nd).
Same animal, other days
New animal
New dataset
Method
C-CO (2)
C-CO 2010 (5)
T- (6)
T RT (6)
(1)
MLP
0,9210 + 0.0010
0.6946 1 0.0912
0.7976 + 0.0220
0,7007 = 0.0774
0.4179
FROM SCRATCH
S4D
0,9381 + 0.0083
0.6619 1 0,0844
0.8526 + 0.0243
0.7145 0,0671
0.3942
Mamha
0,9287 0.0034
0.6708 + 0.1154
0.7692 + 0.0235
0.6694
0.1220
0.7231
GRU
0,9376 0.0036
0.7308 + 0.1089
0.8453 + 0.0200
0.7279 0.0679
0.8128
POYO-SS
0,9427 + 0.0019
0.7381 + 0,0887
0.8705 + 0.0193
0.7156
0,0966
0,4974
POSSM-S4D-SS
0,9515 0.0021
0.7768 + 0.0993
0.8838 + 0.0171
0.7505
0.0735
0.8163
POSSH-Mamba-S5
0.9550 0.0003
0.7989 + 0.0772
0.8747 + 0.0173
0.7418
0.0790
0.8350
POSSH-GRU-SS
0.9549 + 0.0012
0.7930 + 0.1028
0.8863 + 0.0222
0.7687
0.0669
0.8161
NDT-2 (FT)
0,9258 0.0031
0.7846 : 0.1167
0.7173 1: 0.0443
0.6323
0.1339
0.8233
POYO-1 (UT)
0,9580 = 0.0033
0.7883 # 0.0846
0,8181 = 0.0428
0,7073 + 0.0960
0.7861
POYO-1 (FT)
0,9580 + 0.0038
0,8300 + 0.0788
0.8859 + 0.0317
0.7718
0,0785
0.8601
PRETRAINED
D-POSSM-S4D (UT)
0,9603 0.0029
0.7935 1 0.0924
0,8763 +: 0.0259
0.7333 0.0717
0.8394
u-POsSH-Mambu (UI)
0.9607 + 0.0036
0.7504 1 0.0959
0.8729 + 0.0178
0.7388
0.0702
0.7533
U-POSSM-GRU (UT)
0,9595 + 0.0035
0,8022 1 0.0818
0.8921 +: 0.0174
0,7464
0,0692
0.8220
O-POSSM-S4D (FT)
0,9609 0.0042
0.8216 + 0.0945
0.9068 + 0.0170
0.7605
0.0619
0.8941
o-POsSM-Mamba (FT)
0.9615 0.0056
0.8195 + 0.0750
0,9001 = 0.0131
0.7612 0.0722
0.8838
U-POSSM-GRU (FT)
0,9614 + 0.0029
0.8161 + 0.0912
0.9024 + 0.0159
0,7741 0.0683
0.8743
of reaching tasks: centre-out (CO), random target (RT), and Maze. CO is a highly-structured task involving movements from the centre of the screen to one of eight targets (Figure 3a). Conversely, the RT (Figure 3b-c) and Maze (Figure 3d) tasks are behaviourally more complex, requiring movement to randomly placed targets and navigating through a maze, respectively. The testing sessions include: (1) new sessions from Monkey C which was seen during pretraining, (2) new sessions from Monkey not seen during pretraining but collected in the same lab as Monkey C, and (3) a new session from a dataset unseen during pretraining. We call our model pretrained on this dataset o-POSSM (see Section B.3 for details). In total, o-POSSM was trained on 148 sessions comprising more than 670 million spikes from 26,032 neural units recorded across the primary motor (M1), dorsal premotor (PMd), and primary somatosensory (S1) cortices (see Section A for details). We also pretrain two baseline models, NDT-22 [11] and POYO-1 [12]. Additionally, we report the results of single-session models trained from scratch on individual sessions. This includes single-session variants of POSSM (across all backbone architectures) and POYO, well as other baselines such as a multi-layer perceptron (MLP) [32], S4D [15], GRU [14], and Mamba [16].
Causal evaluation. To simulate real-world decoding scenarios, we adopt causal evaluation strategy for all models. This is straightforward for POSSM and the recurrent baselines we consider -sequences are split into 50 ms time chunks and fed sequentially to the model. For the MLP and POYO, we provide a fixed 1 s history of neural activity at each inference timestep, sliding forward in small increments of 50 ms to collect predictions for all behavioural timestamps. For POYO, we only recorded predictions for timestamps in the final 50 ms of each 1 s context window.
During training, the models are presented with contiguous and non-overlapping are not trial-aligned. However, we evaluate our models on entire trials, which are typically much longer than second. For example, in sessions from the Perich et al. [28] dataset, trials in the validation and testing sets are at least 3x and up to 5x longer than the training sequences. This means that a recurrent model must generalize to sequences longer than the ones seen during training.
sequences, which
Transfer to new sessions. In Table 1, we evaluate the transferability of our pretrained models to new sessions, animals, and datasets, and compare them to single-session models. When trained on single session, POSSM is on par with or outperforms POYO on most sessions. When using pretrained models, o- o-POSSM-S4D shows the best overall performance. Finally, we observe that FT is noticeably better than UI when transferring to new animals or datasets. However, UI performs on par with or better than several single-session models, and requires far fewer parameters to be trained.

---

Monkey
CO
2016/10//21
Monkey T RT
2013/08/22
1.0
1.00
0.8
0.8
0.75
R*
valdd R2
0.6
0.6
PUSSM-SS
0.25
Scratch Unit
POSSM-SS
Scratch
P-POSSM
PPSSM
Fut FT
0.4
0.00
0.2
0.2
32
64
128
1000
2000
3000
8
32
1000
2000
3000
# sample triils
training steps
sample trials
# training steps
C
Monkey C- CO
2016/10/13
Monkey CO
2016/10/13
Monkey T CO
POSSM-SS
POYO-1
valid. 1t..
0.90
PRSSIIN- SS
POYO-SS
0.85
POYD-SS
GRU
0.80
MLP
MIP
0.75
30
parameters
inference time (ms)
inference time (ms)
Figure 4: Sample and compute efficcency benchmarking, (a) Results held-out CO session from Monkey [28]. On the left, we show the sample efficiency of adapting a pretrained model versus training from scratch. On the right, compare training compute efficiency between these two approaches. (b) Same as (a) but for a held-out RT session from Monkey [28] - new subject not seen during training. (c) Comparing model performance and compute efficiency baseline models.
Inference times are computed on workstation-clatt GPU (NVIDIA RTX8000). For all these results, used GRU backbone for POSSM.
Sample and training compute efficiency. key motivation behind training neural decoders on large, heterogeneous datasets is to enable efficient transfer to new individuals, tasks, species with minimal finetuning. To this end, we evaluated effectiveness of our finetuning strategies through few-shot experiments. Our results, shown in Figure 4a-b, demonstrate that o-POSSM outperforms single-session models trained from scratch in low-data regimes. Notably, we observe that pretraining results in a considerable initial boost in performance when adapting only unit and session embeddings are updated. In some cases, single-session models fail to match the performance of finetuned model, even after extensive training. Overall, our results in line with observations from Azabou et al. [12], supporting the idea that with the right tokenization and data aggregation schemes, pretrained models can amortize both data and training costs, leading to efficient adaptation to downstream applications.
session, even when
Inference efficiency. In addition to being efficient to train and finetune, we also evaluated whether POSSM efficient inference time. In Figure 4c, we report parameter counts and inference time per time chunk on workstation-class GPU (NVIDIA RTX8000) for POSSM and several state-of-the-arr neural decoders. We find that our single and multi-session models achieve inference speeds that are comparable to lightweight models like GRUs and ML.s and significantly lower than complex models like NDD--2 and POYO, while retaining competitive performance. Single-session POSSM models (POSSM- SS) contained the fewest parameters, and even O-POSSM, with about 8M parameters, maintained low latency. These results held a CPU environment (AMD EPYC 7502 32-Core) as well, with POSSM- SS and o-POSSM achieving inference speeds of ~2.4 ms/chunk and 55.6 ms/chunk, respectively. Overall, our results show that POSSM'S inference time is well within the optimal real-time BCI decoding latency of < 10 ms [8], making it viable option for real-time BCI decoding applications.
3.2 Human handwriting
Next, we evaluated POSSM on a human handwriting dataset [23]. This dataset contains 11 sessions recorded from single individual, where they imagined writing individual characters and drawing straight lines (Figure 3g). Spike counts from multi-unit threshold crossings were recorded from two 96-channel microclectrode arrays implanted in the participant's motor cortex [23]. These were then
7

---

binned at 10 ms intervals. The models were trained to classify the intended characters or lines based on this neural activity. For evaluation, we used 9 sessions, each containing 10 trials per character class. Each individual trial consisted of 1.6 time-window centred around the "go" cue.
Results. We compared POSSM with five baselines: the pre- viously published statistical method PCA-KNN [23], GRU [14]. S4D [15], Mamba [16] and POYO. For POSSM and all baselines except. POYO, we adopted the causal evaluation strat- egy described in Section 3.1, training on s intervals and evaluating on full 1.6 s trials. For POYO, we followed the evaluation scheme from the original paper, using fixed context windows for both training and testing.
Table Human handwriting decod- ing results. Values are mean accuracy + SD for classifying the intended character or line, over 9 sessions and seeds. *,4*: p Σ 0.05, 0.01 for paired t-test vs. POSSM-GRU.
Method
Acc. (%
As shown in Table 2, POSSM-GRU outperforms all baseline models when trained from scratch on the 9 sessions. Re- markably, finetuning o-POSSM, which was only pretrained on NHP data, led significant performance gains: 2% for POSSM-GRU and more than 5% for both POSSM-S4D and
PCA-KNN (SS)
81.36 7:53
S4D
95.46 5.02
FROM SCRATCH
Mamba
93.55 4.53
GRU
93.57 4.22
POYO
94.86 3.53
POSSM S41M
92.11 6.66
POSsM-Mamba. All of the POSSM models achieve state-of-
POSSM-Mamba
92.95 + 4.96
are performance on this task, with the finetuned o-POSSM variants considerably outperforming the baseline PCA-KNN, achieving test accuracies that are about 16% greater.
POSSH-GRU
95.82 +: 3.41
PRETRANED
POYO-1 (FT)
95.82 3.12
NTEP
D-POSSH (FT)
97.25 2,88*
These results establish a critical finding: neural dynamics learned from NHP datasets can generalize across different species performing distinct tasks. This especially impactful
D-POSSM-Mambs (FT)
97.73 231**
O-POSSM-CRU (FT)
97.37 2.32*
given the challenges of collecting large-scale human clectrophysiology datasets, suggesting that the abundance of NHP datasets can used to effectively improve human BCI decoders.
3.3 Human speech
Finally, we evaluated POSSM on the task of human speech decod- ing. Unlike the reaching and handwriting tasks, which involved fixed-length context windows, speech decoding involves mod- elling variable-length phoneme sequences that depend on both the length of the sentence and the individual's speaking pace. We used public dataset [3, 33] consisting of 24 sessions in which a human participant with speech deficits attempted to speak sentences that appeared on a screen (Figure 3h). Neural activity was recorded from four 64-channel microelectrode arrays that covered the pre- motor cortex and Broca's 's area. Multi-unit spiking activity extracted and binned at a resolution of 20 ms, where the length of each trial ranged from 2 to 18 seconds. This poses problem for Transformers like POYO that were designed for s contexts, as the quadratic complexity of the attention mechanism would result in a substantial increase in computation for longer sentences.
Table 3: Human speech decod- ing results. We report the av- erage phoneme error rate (PER) over seeds. All SDs < 0. %.
Method
PER (%)
GRU (no aug.)
39.16
POSSM-GRU (no aug.)
29.70
GRU
30.06
S4D
35.99
Mamba
32.19
POSSM-GRU
27.32
GRU (mult.)
21.74
POSSM-GRU (mult.)
19.80
Although both uni- and bi-directional GRUs were used in the original study [3, 33], we focused primarily on the causal, uni-directional case, as is more relevant for real-time decoding. In line with Willett et al. [3], we 7-scored the neural data and added Gaussian noise as an augmentation. We used the phoneme error rate (PER) as our primary evaluation metric. While prior work has successfully incorporated language models to leverage textual priors, we leave this as a future research direction, instead focusing here on POSSM'S capabilities.
Previous work [33] has shown that Transformer-based decoders perform poorly on this task compared to GRU baselines. Here, we demonstrate the value instead integrating attention with a recurrent model by using POSSM, specifically with GRU backbone. However, since only normalized spike counts (and not spike times) were available the dataset, we were unable to use the POYO-style tokenization as-is. Instead, we treated each multi-unit channel as neural unit and encoded the normalized spike counts with value embeddings. Furthermore, we replaced the output cross-attention module with ID strided convolution layer to control the length of the output sequence. This
8

---

approach significantly reduced the number of model parameters compared to the GRU baseline, which used strided sliding windows of neural activity as inputs instead.
We found that a two-phase training procedure yielded the best results. In the first phase, we trained the input cross-attention module along with the latent and unit embeddings by reconstructing the spike counts at each individual time bin. In the second phase, we trained the entire POSSM model on the target phoneme sequences using Connectionist Temporal Classification (CTC) [34] loss.
Results. POSSM achieved a significant improvement over all other baselines, as shown in Table 3. Notably, POSSM maintained comparable performance even without the Gaussian noise augmentation. while the performance of the baseline GRU was greatly impaired under the same conditions. Further- more, we show in preliminary experiments with multiple input modalities (i.c., both spike counts and spiking-band powers) that POSSM yet again outperforms the baseline. These results illustrate the robustness of the POSSM architecture to variability in data preprocessing and its flexibility with respect to input modalities, further strengthening its case as feasible real-world decoder.
4 Related Work
Neural decoding. Neural decoding for continuous tasks such as motor control in BCIs was tra- ditionally accomplished using statistical models such the Kalman filter [35-38]. While these models are reliable and perform well for specific users and sessions, they require careful adaptation to generalize to new days, users, and even tasks. Such adaptation is typically accomplished using model fitting approaches to estimate the Kalman filter parameters [38-40], a process that requires considerable new training data for each application. Traditional approaches to multi-session neural decoding often consist of learning the decoder's s parameters on a specific day or session (e.g., the first day). followed by learning models align the neural activity on subsequent days to facilitate generalization [41-43]. Given the recent availability of large, public neural recordings datasets [3, 23, 28-30J, modern neural decoding approaches have attempted to leverage advances in large-scale deep learning to build data-driven BCI decoders [11-13, 44-49]. For example, in the context of decoders for neuronal spiking activity, NDT [44] jointly embeds spikes from a neural population into a single token per time bin, spatiotemporal NDT (STNDT) [50] separately tokenizes across units and time and learns a joint representation across these two contexts, NDT-2 [11] tokenizes spatiotemporal patches of neural data akin to ViT [51], and POYO [12] eschews bin-based tokenization, opting to tokenize individual spikes and using a PerceiverlO [22] Transformer backbone to query behaviours from within specific context windows. While several of these works excel at neural decoding, they do not focus on enabling generalizable, online decoding in spike-based BCIs and closed-loop protocols. The BRAND platform [8] enables the deployment of specialized deep learning models in real-time closed-loop brain-computer interface experiments with invasive recordings, demonstrating suitable low-latency neural decoding in other models such as LFADS [52]. Finally, note other ML methods for neural data processing other than direct behaviour decoding. Contrastive learning methods that aim to identify joint latent variables between neural activity and behaviour can be useful for decoding but work is needed for online use [53]. Diffusion-based approaches are promising for jointly forecasting neural activity and behaviour, but again are not readily suited for online use [54].
Hybrid attention-recurrence models. Several works have attempted to combine self- and cross- attention layers with recurrent architectures, usually with the goal of combining the expressivity of attention over shorter timescales with the long-term context modelling abilities of recurrent models such structured SSMs [15-17]. While traditional SSMs have been used for several neuroscience applications [55-58], modern SSMs and hybrid models remain underexplored in the field. Didolkar al. [59] propose an architecture comprising Transformer blocks which process information at faster, shorter timescales while a recurrent backbone integrates information from these blocks over longer timescales for long-term contextual modeling. A similar approach the
Transformer [60], wherein recurrent cell operates on a block of tokens in parallel, thus propagating a block of state vectors through timesteps. Pilault al. [61] propose the Block- Stack- Transformer architecture, which introduces a layer consisting of an SSM to process long input sequences, the outputs of which are sent in blocks to several Transformers that process them in parallel to produce output tokens. Furthermore, several recent works on high-throughput language modelling [19-21] have leveraged hybrid models, where self-attention layers are replaced with SSM blocks to take advantage of their subquadratic computational complexity.

---

5 Discussion
Summary. We introduced POSSM, a scalable and generalizable hybrid architecture that pairs spike tokenization and input-output cross-attention with SSMs. This architecture enables efficient online decoding applications, achieving state-of-the-a performance for several neural decoding tasks while having fewer parameters and faster inference compared to fully Transformer-baas -baeer approaches. Our model achieves high performance with millisecond-scale inference times on standard workstations, even without GPUs, making it suitable for real-time deployment in clinical settings.
A key contribution of this work is demonstrating, to our knowledge, the first successful cross-species transfer of learned neural dynamics for a deep learning- based decoder- - from NHP motor cortex to human clinical data (see [11, 62] for related efforts). This outlines a solution to a major clinical hurdle, where obtaining sufficient data for large-scale modelling is challenging or impossible in patient populations. We demonstrate that we can leverage the large corpus of existing non-human experimental data to improve personalized clinical outcomes by finetuning pretrained models.
Future directions and applications. The POSSM architecture is applied here for motor neural decoding, but it can be adapted to achieve a variety of outcomes. Our current model focuses on processing spiking data from implanted arrays in the motor cortex of monkeys and human clinical trial participants. However, our hybrid architecture is flexible and could readily accommodate data of other neural data modalities through variety of proven tokenization schemes (e.g.. Calcium imaging [13]. EEG [45, 47, 63]). While in the present work we focus on decoding of behavioural timestamps immediately following our input time chunks, our hybrid architecture well-suited towards forecasting over longer timescales. Further, in the future we plan to explore the ability of POSSM to learn and generalize across different regions beyond the motor cortex.
Ultimately, we envision POSSM as a first step towards fast, generalizable neural foundation model for various neural interfacing tasks, with downstream applications such as clinical diagnosties and the development of smart, closed-loop neuromodulation techniques that link predicted states to optimized neural stimulators (e.g.. [4, 5)). Future steps include multimodal pretraining and decoding [22, 49, 64, 65] as well as principled self-supervised pretraining scheme. By enabling efficient inference and flexible generalization through transfer learning, POSSM marks a new direction for general-purpose neural decoders with real-world practicality.
Broader Impact
This research could potentially contribute to the development of neural decoders that are not only accurate but also amenable to deployment in online systems, such those found in neuroprosthetics and other brain-computer interfaces. In addition to POSSM'S general performance, it reduces the need for extensive individual calibration due the pretraining/fieetning scheme. Additionally, the cross- species transfer results on the handwriting task suggest that patients with limited availability of data could benefit from models pretrained with larger datasets from different species. While the potential downstream applications of POSSM are exciting, it is important to consider the ethical concerns that exist for any medical technology, including but not limited to data privacy and humane data collection from animals. Strict testing should be implemented before deployment in any human-related setting.
Acknowledgments and Disclosure of Funding
The authors would like to thank Patrick Mineault, Blake Richards, Ayesha Vermani, and Alexandre André for support and feedback. MA acknowledges support from the NSF AI Institute for Artificial and Natural Intelligence, MGP acknowledges support Future Leaders award from the Brain Canada Foundation and a J1 Chercheurs-boursiers en intelligence artificielle from the Fonds de recherche du Québec- - Santé. GL acknowledges support from NSERC Discovery Grant RGPIN- 2018 8-8821441 the Canada Research Chair in Neural Computations and Interfacing, a Canada-CIFs AI Chair, IVADO, and the Canada First Research Excellence Fund. The authors also acknowledge the support of computational resources provided by Mila, the Digital Research Alliance of Canada, and NVIDIA that enabled this research.
10

---

References
[1] A. Vermani, M. Dowling, Jeon, Jordan, J. Nassar, Bernaerts. Zhao, S. Van Vaerenbergh, and M. Park. (Real-Time Machine Learning Strategies for a New Kind of Neuroscience Experiments" 2024 32nd European Signal Processing Conference (EUSIPCO). 2024, pp. 1127-1131.
[2] S. N. Flesher, J. E. Downey, J. M. Weiss, C. L. Hughes, A. Herrera, E. C. Tyler-Kabara, M. L.. Boninger,
J. L. Collinger, and R. A. Gaunt. "A brain-computer interface that evokes tactile sensations improves robotic control". Science 372.6544 (2021), pp. 831-836.
[3] F. R. Willen, E. M. Kunz, C. Fan, D. T. Avansino, G. H. Wilson, E. Y. Choi, Kamdar, M. F. Glasser, R. Hochberg, S. Druckmann, al. "A high-performance speech neuroprostleesi". Nature 620.7976 (2023), pp. 1031-1036.
[4] H. Lorach, A. Galvez, V. Spagnolo, Martel, S. Karakas, N. Intering. M. Vat, O. Faivre, C. Harte, Komi, et al. "Walking naturally after spinal cord injury using brain brain-spine spine interface". Nature 618.7963
(2023), pp. 126-133.
[5] M. Bonizzato, R. Guay Hottin, S. L. Côté, E. Massai, L. Choinière, U. Macar, S. Laferrière, P. Sirpal, S. Quessy, G. Lajoie, al. "Autonomous optimization of neuroprosthetic stimulation parameters that drive the motor cortex and spinal cord outputs in rats and monkeys". Cell Reports Medicine 4.4 (2023), 101008.
[6] J. L. Elman. "Finding Structure in Time". Cognitive Science 14.2 (1990), pp. 179-211.
[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. "Attention is All you Need". Advances in Neural Information Processing Systems, Vol. 30. 2017.
[8] Y. H. Ali, K. Bodkin, M. Digotti-Thompson K. Patel, N. S. Card, B. Bhaduri, S. R. Sison-Tomaszewsii
D. M. Mifsud, X. Hou, C. Nicolas, al. "BRAND: platform for closed-loop experiments with deep network models". Journal of Neural Engineering 21.2 (2024), p. 026046.
Yang. P. Ahmadipour, and M. Shanechi. "Adaptive latent state modcling of brain network dynamies with real-time learning rate optimization", Journal of Neural Engineering 18.3 (2021), p. 036013.
[10] D. Sussillo, S. D. Stavisky, J. C. Kao. S. Ryu, and K. Shenoy. "Making brain-machine interfaces robust to future neural variability". Nature Communications 7.1 (2016).
[11] J. Ye, J. Collinger, L. Wehbe, and R. Gaunt. "Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity". Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 80352- 80374.
[12] M. Azabou, V. Arora, V. Ganesh, X. Mao, S. Nachimuthu, M. Mendelson, B. Richards, M. Perich, G. Lajoie, and E. Dyer. "A Unified, Scalable Framework for Neural Population Decoding" Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 44937-44956.
[13] M. Azabou, K. X. Pan, V. Arora, I. J. Knight, L., Dyer, and B. Richards. "Multi-session, multi-task neural decoding from distinct cell-types and brain regions". The Thirteenth International Conference on Learning Representations. 2025.
[14] K. Cho. B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk. and Y. Bengio. "Learning Phrase Representations using RNN Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar, 2014, pp. 1724-1734.
Decoder for Statistical Machine Translation".
[15] A. Gu, K. Goel, and C. Re. "Efficiently Modeling Long Sequences with Structured State Spaces". International Conference on Learning Representations. 2022.
[16] A. Gu and T. Dao. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". 2024. arXiv: 2312. 00752.
[17]
J. T. Smith, A. Warrington, and S. Linderman. "Simplified State Space Layers for Sequence Modeling". The Eleventh International Conference Learning Representations. 2023.
[18] T. Dao and A. Gu. "Transformers SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality". Proceedings of the 41st International Conference on Machine Learning. Vol. 235. Proceedings of Machine Learning Research. 2024, pp.
[19] B. Lenz, O. Lieber, A. Arazi, A. Bergman, A. Manevich, B. Pelog, B. Aviram, C. Almagor, C. Fridman, D. Padnos, et al. "Jamba: Hybrid Transformer-Mamba Language Models". The Thirteenth International Conference on Learning Representations. 2025.

---

[20] S. De. S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, et al. "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models". 2024, arXiv: 2402. 19427.
[21] B. N. Patro, Ranganath, P. Namboodiri, and V. Agneeswaran. "Heracles: Hybrid SSM- Transformer Model for gigh-Resouution Image and Time- Series Analysis", 2024, arXiv: 2403. 18063.
Time-Series
[22] A. Jacgle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. lonescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. "Perceiver IO: A General Architecture for Structured Inputs & Outputs". International Conference on Learning Representations. 2022.
[23] F. R. Willett, D. T. Avansino, L. R. Hochberg, J. M. Henderson, and K. V. Shenoy. "Hligh-performance brain-to-text communication via handwriting". Nature 593.7858 (2021), pp. 249-254.
[24] A. E. Urai, B. Doiron, A. M. Leifer, and A. K. Churchland. "Large-scale neural recordings call for new insights to link brain and behavior". Nature Neuroscience 25.1 (2022), pp. 11-19.
[25] M. D. Humphries. "Spike-Traie Train Communities: Finding Groups of Similar Spike Trains". Journal of Neuroscience 31.6 (2011), pp. 2321-2336.
[26] H. G. Rey, C. Pedreira, and R. Quian Quiroga. "Past, present and future of spike sorting techniques". Broin Research Bulletin 1 119 (2015). Advances in electrophysiological data analysis, pp. 106-1 17.
[27] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. "RoFormer: Enhanced transformer with Rotary Position Emhedding". Neurocomputing 568 (2024), p. 127063.
[28] M. G. Perich, I.. F. Miller, M. Azabou, and F. L.. Dyer. "Long-term recordings of motor and premotor cortical spiking activity during reaching in monkeys". 2025.
[29] J. E. O'Doherty, M. M. B. Cardoso, J. G. Makin, and P. N. Sabes. "Nonhuman Primate Reaching with Multichannel Sensorimotor Cortex Electrophtsiologyy" Zenodo: 10. 5281 22281/ 3854034, 2020.
[30] M. M. Churchland, P. Cunningham, M. T. Kaufmmnn, J. D. Foster, P. Nuyujukian, S. I. Ryu, and K. Shenoy. "Neural population dynamics during reaching". Nature 487.7405 (2012), pp. 51-56.
[31] Pei, J. Ye, D. Zoltowski, A. Wu, R. Chowdhury, H. Sohn, J. O' O'Dohrty, K. V. Shenoy, M. Kaufman, M. Churchland, et al. "Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity". Proceedings of the Neural Information Processing Syxtems Track Datasets and Benchmarks. Vol. 2021.
[32] J. I. Glaser, A. S. Benjamin, R. H. Chowdhury, M. G. Perich, L. E. Miller, and K. P. Kording. "Machine Learning for Neural Decoding" " eNeuro 7.4 (2020).
[33] R. Willett J. Li, T. Le, C. Fan, M. Chen, Shlizerman, Chen, X. Zheng, S. Okubo, T. Benster, et al. Brain-to-Texx Benchmark 24: Lessons Learned". 2024. arXiv: 2412. 17227.
[34] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks". Proceedings of the 23rd International Conference on Machine Learning. ICMI. '06. Pittsuurg,, Pennsylvania, USA, 2006, pp- 369-376.
[35] R. F. Kalman. "A New Approach Linear Filtering and Prediction Problems". Journal of Basic Engineering 82.1 (1960), pp. 35-45.
[36] W. Wu, M. Black, Y. Gao, M. Semuua,, A. Shaikhouni, Donoghue, and E. Bienenstock. "Neural Advances Filterr,.dvvncess Neural Information Processing Systems.
Decoding Cursor Motion Using a Kalman Filter", Vol. 15. 2002.
[37] S. Koyama, S. M. Chase, A. S. Whitford, M. Velliste, A. B. Schwartz, and R. E. Kass. "Comparison of brain compute interface decoding algorithms in open-loop and closed-loop control". Journal of Computational Neuroscience 29, 19.1-2 (2010), pp. 73 73,
[38] F. R. Willett, D. R. Young, B. A. Murphy, W. D. Memberg, C. H. Blabe, C. Pandarinath, S. D. Stavisky, Rezaii, J. Saab, B. L. Walter, Feedback Control Model",, Scientific Reports 9.1 (2019).
"Principled BCI Decoder Design and Parameter Selection Using
[39] B. Jarosiewicz, Y. Masse, Bacher, S. Cash, Eskandar, G. Frichs, P. Donoghue, and L. R. Hochberg. " "Advantages of closed-loop calibration in intracortical brain-computer interfaces for people with tetraplegia". Journal of Neural Engineering 10.4 (2013), 046012.
[40] V. Gilja, C. Pandarinath, C. H. Blabe, P. Nuyujukian, J. D. Simeral, A. A. Sarma, B. L. Sorice, J. A. Perge, B. Jarosiewicz. L. R. Hochberg, et al. "Clinical translation of high performance neural prosthesis". Nature Medicine 21.10 (2015), pp. 1142-1145.
12

---

[41] A. Farshchian, J. A. Gallego. J. P. Cohen, Y. Bengio, L. E. Miller, and S. A. Solla. "Adversarial Domain Adaptation for Stable Brain-Maching Interfaces". 2019. arXiv:
[42] A. D. Degenhart, W. E. Bishop, E. R. Oby, E. C. Tyler-Kabara, S. M. Chase, A. P. Batista, and B. M. Yu. "Sttabllization of brain computer interface via the alignment low-dimensional spaces of neural activity". Nature Biomedical Engineering 4.7 (2020), pp. 672-685.
[43] X. Ma. F. Rizzoglio, K. L. Bodkin, E. Perreault, L. E. Miller, and A. Kennedy, "Using adversarial networks to extend brain computer interface decoding accuracy over time". eLife 12 (2023), c84296.
[44] J. Ye and C. Pandarinath. "Representation learning for neural population activity with Neural Data Transformers". Neurons, Behavior, Data analysis, and Theory 5.3 (2021), pp. 1-18.
[45] W. Jiang, L. Zhao, and B.-l. Lu. "Large Brain Model for Learning Generic Representations with Tremen- dous EEG Data in BCT". The Twelfth International Conference Learning Representations. 2024.
[46] Y. Zhang. Y. Wang. D. M. Liménez-Beneté Z. Wang. M. Azabou, B. Richards, R. Tung. O. Winter, I. B. Laboratory, E. Dyer, al. "Towards a "Universal Translator" for Neural Dynamics at Single-Cell. Single-Spike Resolution". Advances in Neural Information Processing Systems. Vol. 37. 2024, pp. 80495- 80521.
"ussuution".Advvncess
[47] G. Chau, C. Wang, S. J. Talukder, V. Subramaniam, S. Socdarmadji, Y. Yue, B. Katz, and A. Barbu. "Population Transformer: I Learning Population-Ievel Representations of Neural Activity". The Thirteenth International Conference on Learning Representations. 2025.
[48] B. Candelori. G. Bardella, I. Spinelli,, S. Ramawat, Pani, S. Ferraina, and S. Scardapane. "Spatio- temporal transformers for decoding neural movement control". Journal of Neural Engineering 22.1 (2025), 016023.
[49] Y. Zhang, Y. Wang, M. Azabou, A. Andre, Z. Wang, H. Lyu, I. B. Laboratory, E. L. Dyer, L. Paninski, and C. L. Hurwitz. "Neural Encoding and Decoding at Scale", Proceedings of the 42nd International Conference Machine Learning. Vol. 267. Proceedings of Machine Learning Research. 2025, pp. 76175- 76192.
[50] T. Le and E. Shlizerman. "STNDT: Modeling Neural Population Activity with Spatiotemporal Transform- ers". Advances Neural Information Processing Systems. Vol. 35. 2022, pp. 17926-17939.
[51] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", International Conference on Learning Representations. 2021.
[52] C. Pandarinath, D. J. D'Shea, J. Collins. R. Jozefowicz, S. D. Stavisky. J. C. Kao, E. M. Trautmann, M. T. Kaufman, S. L. Ryu, L. R. Hochberg, al. "Inferring single-trial trial neural population dynamics using scquential auto-encoders" Nature Methods 15.10 (2018), pp. 805-815.
[53] Schneider, J. H. Lee, and M. W. Mathis. "Learnalle latent emheddings for joint behavioural and neural analysis". Nature 61 7.7.960 (2023), pp. 360-368.
[54] A. C. Filipe and M. Park. "One Model to Train Them All: A Unified Diffusion Framework for Multi-Context Neural Population Forecasting". OpenReview. 2025.
[55] S. Linderman, A. Nichols, D. Blei, M. Zimmer, and L. Paninski. "Hierarchical recurrent state space models reveal discrete and contiquous dynamics of neural activity in C. elegans". bioRxiv: 621540. 2019.
[56] D. Zoltowski, J. Pillow, and S. Linderman. "A general recurrent state space framework for modeling neural dynamics during decision- making". Proceedings of the 37th International Conference Machine Learning. Vol. 119, Proceedings of Machine Learning Research. 2020, pp. 11680-11691
[57] A. Vinograd, A. Nair, H. Kim, W. Linderman, and D. Anderson. "Causal evidence of line attractor encoding an affective state". Nature 634.8035 (2024), pp. 910-911 918.
[58] A. Vermani, J. Nassar, H. Jeon, M. Dowling, and I. M. Park. "Meta-Dynamical State Space Models for Integrative Neural Data Analysis". The Thirteenth International Conference on Learning Representations. 2025.
[59] A. Didolkar, K. Gupta, A. Goyal, N. B. Gundavarapu, A. M. Lamb, N. R. Ke, and Y. Bengio. "Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning". Advances Neural Information Processing Systems. Vol. 35. 2022, pp. (0505-10520.
[60] D. Hutchins, Schlag. Wu, E. Dyer, and B. Neyshabur. Feeuur-Re Transformers". Advances in Neural Information Processing Systems. Vol. 35. 2022, pp. 33248-33261.
[61] J. Pilault, M. Fathi, O. Firat, C. Pal, P.-... Bacon, and R. Goroshin. "Block-State Transformers". Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 7311-7329.
13

---

[62] F. Rizzoglio, E. Altan, X. Ma, K. L. Bodkin, B. M. Dekleva, S. A. Solla, A. Kennedy, and L. E. Miller. "From monkeys humans: observation-based EMG brain-computer interface decoders for humans with paralysis". Journal of Neural Engineering 20.5 (2023), p. 056040.
[63] C. Wang, V. Subramaniam, U. Yaari, G. Kreiman, B. Katz, Cases, and Barbu. "BrainBERT: Self- supervised representation learning for intracranial recordings". The Eleventh International Conference on Learning Representations. 2023.
[64] M. Ramezani, X. Liu, C. Ren, T. Komiyama, and D. Kuzum. "Processing multimodal neural data and decoding neural dynamics using cross-modality inference". 2023 11th International IEEE/EMBS
Conference Neural Engineering (NER). 2023, pp. 1-4.
[65] Yang, C. Feng, D. Wang, T. Wang. Z. Zeng. Z. Xu, H. Park, Ji Ji, H. Zhao, Y. Li, al. "NeuroBind: Towards Unified Multimodal Representations Neural Signals". 2024, arXiv: 2407. 14020.
[66] D. Flint, E. Lindberg, Jordan, Miller, and Slutzky. "Accurate decoding of reaching movements from field potentials in the absence of spikes". Journal of Neural Engineering 9.4 (2012), 046006.
[67] A. Gu, Johnson, A. Timalsina, A. Rudra, and C. Rc. "How to Train your HiPPO: State Space Models with Generalized Orthogonal Basis Projections". International Conference Learning Representations. 2023.
[68] I. Loshchilov and F. Hutter. "Decoupled Weight Decay Regularization". International Conference on Learning Representations. 2019.
[69] You, Li, Reddi, J. Hseu, Kumar, S. Bhojanapalli, X. Song, Demmel, Keutzer, C.-J. Hsieh. "Large Batch Optimization for Deep Learning: Training BERT in 76 minutes". 2020. arXiv: 1904.00962.
[70] M. Azabou, M. G. Azar, R. Liu, C.H.. Lin, E. C. Johnson, K. Bhaskaran- M. Dabagia, B. Avila- Pires, L. Kitchell, K. B. Hengen, al. "Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction". 2021, arXiv; 2102. 10106.
[71] E. de Vries, J. H. Siegle, and C. Koch. "Sharing neurophysiology data from the Allen Brain Observa- tory". eLife 12 (2023), e85550.
[72] J. Hu, Shen, P. Wallis, Z. Allen-Zhu, Y. L. S. Wang. Wang, and Chen. "LoRA: Low-Rank Adaptation of Large Language Models". International Conference on Learning Representations. 2022.
14

---

Supplementary Material
A Dataset Details
Table 4: Summary of neural datasets used for pretraining and evaluation. Values represent totals across training, validation, and testing subsets of the pretraining and evaluation datasets.
Dataset
Regions
Tasks
# Indiv.
# Sess.
# Units
# Spikes
# Bhvr.
Perich al. [28]
MI, PMd
CO,
9040
96.8M
12.7M
TraInIng
D'oherty et al. [29]
M1.
RT
2
87.8M
10.4M
Churchland al. [30]
MI, PMd
CO
2
10
1911
739M
85.0M
NLB Maze [31]
M1, PMd
Maze
1
182
3.64M
6.81M
Perich al. [28] Monkey C
MI, PMd
CO
520
4.65M
221K
EVALUATION
Perich al. [28] Monkey T
MI,
CO
336
1,53M
589K
Perich [28] Monkey T
MI. PMd
RT
349
1.59M
551K
Flint al. [66] Monkey C
M1
CO
5
957
7.88M
318K
NLB Area2 [31]
S1
CO
1
1
65
222K
50.7K
train
validation
test
f 0
session S
trials
=
Figure 5: Session-wise data splits for training, validation, and testing, Per session (across all datasets), 10% of the trials were used for validation and 20% were used for testing. The remaining data, including inter-trial segments, used for training.
Dataset details. For all our experiments, we attempted to minimize dataset preprocessing, in line with POYO [12]. The various datasets used during pretraining were collected from different labs using different experimental setups, effectors, etc. However, we did not standardize the datasets in any way and only excluded behavioural data based on a simple thresholding of acceleration values to remove outliers. Thus, like POYO: (1) we did not filter units reject multi-units; (2) we did not apply identical spike sorting algorithms for all datasets (our training data consists of a mix of threshold crossing units and spike sorted single units); and (3) did not resample process the velocity, ie.. the behavioural output to be decoded. Details on the datasets used and bession-wisse data splits are provided in Table 4 and Figure 5 respectively. Note that for the NLB Area2 dataset [31], we did not include perturbed trials.
B
Architectural Details
B.1
Tokenization
For all our models, we used POYO-style tokenization [12], albeit with one key difference: unlike POYO, did not provide delimiter tokens (i.e., [START] and [END] tokens) to the model. This reduced the total number of tokens passed as input to the model, thus increasing efficiency, but did not negatively impact decoding performance in our experiments.
B.2 Recurrent backbones
In this section, we provide additional details on each of the recurrent backbones used by POSSM. is important to note that POSSM is theoretically agnostic to the type of recurrent architecture.
Gated recurrent unit (GRU). Like traditional recurrent neural networks, the GRU [14] maintains a hidden state h, E &d that is governed by the input at the current timestep x, E IR and the previous hidden state h At each timestep, the network calculates update gate and reset gate rt-
15

---

defined as follows:
Zt 0 W + - r, xW, + - b,),
where W., W, E Rdxn, U. U, Raxd, and b., b, Rd are the input weights, hidden state weights, and biases, respectively. a the sigmoid activation function. A candidate hidden state is then generated:
h, = tanh W + Un @ h₂-1) + bn),
where W E IRdx" U E gdxd by E R are learnable parameters and @ is the Hadamard product.
The final hidden state is calculated by weighing the candidate and previous hidden states by the update and reset gates:
h = (z)) zz) 3 h + 2e O h..
Diagonal structured state-space models (S4D). Structured state-space models (S4) [15] are built on the theory of linear iime-invariant SSMs. Specifically, the hidden state x(t) E [R and output (t)) E R. are calculated as follows:
x(1) = Ax(()) + Bu((). v()) Cx((),
where A Rddd B E R and C E R 1xd are weight matrices and uut) E R is the input at time t. additional parameter, A. denotes the step size or the time-resolution of the input. For inputs with multiple channels, the SSM is applied independently to each channel. To be applied to discrete input sequence (No. ......) rather than a continuous function u(t), the above continuous-tinn can be discretized through transformation of its parameters (A, B, C, A), yielding (A, B, C) [15]. Unlike traditional RNNs that can only process sequences one element at time, S4's linear nature allows it to formulate its output as discrete convolution post discretization:
SSM
K
A=
where L is the length of the sequence and K is convolutional kernel parameterized such:
K2
BB.
This powerful property allows the computation be parallelized across elements of the sequence during training, resulting in faster training than non-linear RNNs of the same size. Finally, this work, use the Diagonal $4 version (S4D), which replaces the traditional HiPPO initialization [67] with diagonal-only version which is faster yet has been shown to retain similar performance.
Mamba. Building the SSM neural network introduced by S4, Mamba [16] also aims to be recurrent architecture that allows for parallelizable inference during training. However, unlike $4, the model learns dynamic weight matrices B, and C, that are now parameterized by position in the sequence (and are functions of the input). This allows Mamba to selectively compress information into the hidden state, essentially discarding information that is deemed unimportant.
B.3 Non-human primate reaching
The hyperparameters changed between the single-session model and the pretrained (o-POSSM) models are the input dimensionality (i.e., size of the spike and latent tokens), RNN hidden dimensionality. and the number of RNN layers. While a of hyperparameters was kept constant across single-session and multi-session models (one set for each), the number of parameters that the model has varies based on the choice of recurrent backbone. We find that S4D is the smallest and Mamba the largest when these three hyperparameters are kept constant. Details are provided in Table 5,
B.4 Human handwriting
The statistical baseline PCA-KNN was implemented following Willett et al. [23]. A Gaussian kernel was initially used to smooth the spikes, followed by PCA reduce the dimensionality to 15. Then, 7 time-warp factors were used to resample the PC-projected neural activity, and Euclidean distances were calculated between the results of each time-warp factor and all other points in the dataset. Finally, the minimum time-warp distances over all 7 factors were used to classify characters using a k-nearest neighbours (KNN) classifier. The only difference here that we considered 1.6 intervals of neural activity. For more details on this method, please refer to the original publication.
Baselines.
16

---

Table 5: Hyperparameters and model sizes of POSSM variants used for NHP reaching tasks.
Backbone
SS / FT
Input Dim.
RNN Hidden Dim.
# RNN Layers
# Params.
SS
64
256
0.41M
S4D
FT
256
512
4.56M
SS
64
256
0.47M
GRU
FT
256
512
7.96M
SS
64
256
0.68M
Mamba
FT
256
512
8.96M
POSSM.
We used the same POSSM architecture the NHP reaching tasks.
B.5 Human speech
We adopted the same hyperparameters as mentioned in Willett et al. [3] for the baseline GRU, including block-wis, normalization and two sources of Gaussian noise injection. Sliding windows of 32 bins of normalized spike counts with a stride of 4 bins were used at the input for both uni-directional and bi-directional models. For uni-directional and bi-directional GRUs, the number of parameters were 55M and 133M, respectively. We also introduce two more binning-based baselines by replacing the GRU backbone of the model with Mamba and S4D.
Baselines.
POSSM. We used POSSM with GRU as the recurrent backbone (uni-directionn and bi-directional according to the problem setup) for this task. The encoder was a cross- coss-attention module with single head, followed by a self-attention module with 2 heads, operating strictly within each 20 ms bin. We used an input dimension of 64 and 4 latents per bin. At the output of the encoder, we concatenated the latents from the same bin, resulting in 256-dimensional vector that served as the input to the GRU. As mentioned in the previous section, the baseline method processed sliding windows of neural activity inputs, consisting of & bins (usually 14 or 32) with stride (usually 4). Values of l and were chosen to control the length the output sequence, with larger values of & being shown to help performance [3]. In this paper, we instead set both k and as I and utilized convolutional layer the output to control the length of output sequences. This ensured that the emission frequency of output phonemes from POSSM was identical to that of the baselines. The baseline method also introduced a linear layer to map the concatenation of all & bins to a fixed hidden dimension, which quickly became computationally expensive especially when & was large. For POSSM, avoiding the use of sliding windows led a sizable parameter reduction (c.g., 24M fewer parameters than a GRU with the above setup). For both uni-directional and bi-directional POSSM, the parameter counts were 32M and 86M, respectively.
C Training Details
C.1 Non-human primate reaching
As the prediction target is two-dimensional time series of either hand or cursor velocity, the loss function was chosen be mean squared error. We increased the weight of the loss for centre-out reaching segments by a factor of 5, following POYO. A maximum of 100 behaviour values were provided as targets for each s training sample. If the dataset had a behavioural sampling rate greater than 100 Hz, 100 behaviour samples were randomly chosen within the second.
Baselines. We considered four single-session binning-based baselines for the reaching tasks: MLP. GRU, S4, and Mamba. We used 50 ms bins to extract spike counts for these models, with step sizes of 50 ms and 10 ms for the MLP and the other models, respectively. All baselines were trained using the AdamW W optimizer [68] with batch size of 128, base learning rate of 0.004, and a cosine scheduler for a total of 500 epochs.
In addition the aforementioned models, both single-session and pretrained versions of POYO were used as baselines. Following Azabou et [12], we used context window of s for POYO (and a step size of 50 ms during evaluation, described in Section 3.1). The model was trained using the LAMB
17

---

optimizer [69] with a batch size of 128, base learning rates of 0.004 and 0.002 for single session and pretrained models respectively, and a cosine scheduler for total of 500 epochs. During training, we also applied a data augmentation scheme called unit dropout [12, 70], where we randomly drop out subset of units from the input along with all their spikes for each iteration. Both unit identification and full finetuning used the same training hyperparameters as the single-session models, except with a constant learning rate. For full finetuning, unit identification was first performed for 100 epochs before the rest of the model was unfrozen and trained for another 400 epochs. All models were trained for total of 500 epochs.
Finally, for NDT-2, we used the code provided by Ye et al. [11] and their default hyperparameters. However, we pretrained and evaluated NDT-2 on the same datasets and splits used for POSSM.
POSSM. Single-session models were trained on a single NVIDIA RTX8000 GPU using LAMB with batch size of 128, a base learning rate of 0.004, and a cosine scheduler. Multi-dataset pretraining was done on four NVIDIA H100 GPUs using LAMB with a batch size of 256, a base learning rate of 0.002, and a cosine scheduler. The unit dropout data augmentation scheme mentioned previously was employed during training. Both unit identification and full finetuning used the same training hyperparameters as the single-session models, except with a constant learning rate. For full finetuning, unit identification was first performed for 100 epochs before the rest of the model was unfrozen and trained for another 400 epochs. All models were trained for a total of 500 epochs. Single-session models took less than 30 minutes to train, while multi-dataset pretraining took around 36 hours.
C.2 Human handwriting
Baselines. For the PCA-KNN baseline, we performed hyperparameter tuning on validation set for each session, selecting the number of neighbours from a range of 2 to 14. For the GRU, S4, and Mamba, we provided non-overlapping 50 ms bins of spike counts as inputs.
POYO and POSSM. In this task, we used the unit dropout data augmentation scheme for POYO and POSSM. As for POYO, aside from unit dropout, we applied an additional augmentation scheme called random time-scaling, where the timestamps are randomly scaled at each iteration.
A gradual unfreezing strategy was used during finetuning for the handwriting task, where only the unit embeddings, session embeddings, and the linear readout layer for the task were trained at the beginning. Note that unfreezing the linear readout layer differentiates the finetuning strategy for the handwriting task from the one used for the reaching tasks described previously. Training the linear layer was necessary handwriting decoding was not present in the pretraining datasets. We finetuned POYO-1 for 600 epochs (unfrozen at epoch 40), while o-POSSM was finetuned for total of 1000 epochs (unfrozen at epoch 50). All training hyperparameters for POYO followed the suggestions of its original authors.
For POYO and POSSM trained from scratch, well finetuned POYO-1, training was performed on a single NVIDIA RTX8000 GPU using LAMB with a batch size of 256, a base learning rate of 0.00256, and cosine scheduler. When finetuning O-POSSM, the base learning rate was 0.008 instead. We trained the models with three random seeds and reported the mean accuracy.
C.3 Human speech
Baselines. All baselines for speech decoding were trained on a single NVIDIA RTX8000 GPU using AdamW with a batch size of 64 and a base learning rate of 0.02. For mni-directional models, a linear decaying scheduler was applied to the base learning rate, while for bi-directional models, the learning rate was kept constant.
POSSM. Both the uni-directional and bi-directional versions of POSSM were trained on single NVIDIA A100 GPU (80GB) with a batch size of 16. We conducted a hyperparameter sweep on the learning rate schedule via cross-validation on the training split. In the uni-directional case, adopted a base learning rate of 0.0001 with a cosine scheduler. the bi-directional case, we used a base learning rate of 0.00004 with two-phase one-cyele scheduler where the learning rate was first increased to 0.00008 at epoch 50 then reduced with cosine annealing.
We trained all models for 100 epochs and with three random seeds, and reported the mean accuracy.
18

---

D Additional Results
D.1 Performance on NHP reaching with 20 ms time chunks
We ran additional experiments comparing single-session models working with contiguous time chunks of 20 opposed to 50 ms. As shown Table we do not drops in POSSM's performance with these smaller time chunks, and it continues to perform better than or on par with baseline methods. In practice, given POSSM's low inference times, possible to run POSSM in real-time with 50 ms time chunks and step size of 20 ms, if a quicker prediction frequency is desired.
Table Behavioural decoding results on NHP reaching tasks with 20 ms time chunks. Best performing models are in boldface (1st) and underlined (2nd).
Same animal, other days
New animal
New dataset
Method
C-CO 2016
C CO 2010 (5)
T- CO (6)
T- RT (6)
H (1)
MLP
0,9269 + 0.0044
0,5842 1 0.2052
0.7940 + 0.0341
0.6082 0.3014
0.7602
FROM SCRATCH
S4D
0,9313 + 0.0038
0.6842 + 0.1017
0.8597 + 0.0275
0.7120 0,0764
0.4554
Mamha
0,9283 0.0096
0.6840 0.0936
0.7318 0.0426
0.6653
0.0978
0.7045
GRU
0.9435 0.0069
0.7742 + 0.0964
0.8389 + 0.0248
0.7414
0.0426
0.8101
POSSH-S4D-SS
0,9470 + 0.0016
0.7611 + 0.1179
0.8777 + 0.0228
0.7369
0.0691
0.7958
POSsH-Manba-SS
0,9490 0.0059
0.7691 +: 0.0786
0.8613 + 0.0121
0.7300
0.0719
0.7065
POSSM-GRU-SS
0.9514 0.0010
0.7780 + 0.0980
0.8724 + 0.0190
0.7429
0.0708
0.7816
u-POSSM-S4D (UT)
0.9589 0.0025
0.7835 1: 0.0839
0.8661 +: 0.0201
0.7256
0.0740
0.8402
PRETRA
D-POSSM-Mamba (UI)
0.9588
0.7175 1 0.1373
0.8483 +: 0.0247
0.7226
0.8102
O-POSSM-GRU (UT)
0,9551 = 0.0070
0.7837 # 0.0916
0.8619 = 0.0304
0.7323 = 0.0812
0.8087
O-POSSM-S4D (FT)
0,9601 0.0062
0,8052 + 0.0887
0.8991 + 0.0229
0.7543 0.0672
0,8834
O-PossM-Mamba (FT)
0.9605 0.0016
0.8034 0.0971
0.8946 + 0.0168
0.7508
0.0816
0.8639
u-POSSM-GRU (FT)
0,9580 + 0.0040
0.8110 1 0.0753
0.8908 1 0.0214
0.7561 0.0727
0.8588
D.2 Effects of context length on handwriting decoding
In Figure we plot imagined handwriting decoding accuracy against the context length that POSSM receives. This shows that POSSM can make accurate predictions in a causal manner, given sufficient context. More importantly. these results showcase POSSM's applicability real-time applications. with time series data fed into the neural decoder in a streamlined fashion.
0-PO5SM-S4D
M-POssM-Mamba
O-POSSM-GRU
1.00
0.75
accuracy
0.50
0.25
0.00
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
-0.2
0.0 0.2 0.4
0.6
0.8 1.0 1.2
-0.2 0.0
0.2
0.4
1.0 1.2 PCA-KNN
0.6
0.8
go cue
timestamp relative to go cue time
Figure 6: Handwriting decoding accuracy given various context lengths. We plot mean accuracies and 99% confidence intervals for each finetuned o-POSSM model, given an increasing amount of neural activity context. For each context length, POSSM makes prediction the last timestamp. The plot's x-axis is aligned with the "go" cue time.
D.3 Speech decoding with bi-directional models
We ran the bi-directional versions of the top competitors identified in the uni-directional case. As shown in Table both with and without noise augmentation, bi-directional models demonstrated consistent improvement of 2-3% over their uni-directional counterparts. This is expected as
19

---

bi-directional models can utilize the full temporal sequence of neural activity for each phoneme prediction. Similarly the uni-directional case, we observed that POSSM outperformed the baseline even without noise augmentation.
Table 7: Human speech decoding results with bi-directional model backbones. We report the phoneme error rate (PER) averaged over 3 seeds. All SDs < 0.1% 1.
Method
PER (%) +
BiGRU (no aug.)
37.18
POSSM-BiGRU (no aug.)
26.62
BiGRU
27.86
DOSSM-BiGRU
25.80
D.4 Decoding behaviour from non-motor neural activity
To validate the applicability of POSSM to decoding behaviour from other, non-motor brain regions, preliminary experiment with POSSM on 5 randomly selected sessions from the Allen Visual Behaviour Neuropixels dataset [71], which tracked spiking activity mice. Here, we attempted to decode running speed from multi-region neural activity (visual cortex and several subcortical areas, including visual thalamic areas, hippocampus, and midbrain). We found that a single-session POSSM-GRU achieved R of 0.9295 + 0.0326, averaged over 5 sessions.
D.5 Predicting future behaviour at each time chunk
order to demonstrate the flexibility of our output cross-attention module, we trained single-session POSSM models to predict cursor velocities at timestamps that were 50 ms or 100 ms beyond the end of each time chunk (denoted as "lags"). Thus, we queried the output module with timestamps beyond those associated with the input time chunk. Our results in this setting were par with, only slightly lower than, single-session POSSM performance with no lags (Table 8).
Table 8: Results decoding cursor velocities at timestamps beyond the end of each time chunk. We consider lags of 50 ms and 100 ms, and report the mean R² + SD over sessions.
Method
C CO 2016 (2)
T- RT (6)
POSSM-GRU-SS (50 ms lag)
0.9495 + 0.0045
0.7498 + 0.0874
POSSM-GRU-SS (100 ms lag)
0.9368 = 0.0027
0.7252 + 0.0813
D.6 Ablating spike timing information
A key advantage of POYO-style tokenization, apart from facilitating pretraining and generalization, is that it retains spike timing information unlike binning-bart based methods. To specifically test whether this scheme yielded performance advantages over binning, trained single-session POSSM models without precise spike timing information, instead providing bin-level timestamps for all spikes. We found that models receiving spike timing information consistently outperformed those receiving only bin-level timestamps (Table 9), showing the advantage of precise spike times for neural decoding.
Table 9: Behavioural decoding results with and without precise spike times. We report the mean R² SD over sessions.
Method
C CO 2010 (5)
T RT (6)
- H-CO (1)
POSsM-Mamba-SS (w/ spike times)
0.7959 + 0.0772
0.7418 + 0.0790
0.8350
POssM-Mamba-SS (w/ bin-level times)
0.7040 + 0.0903
0.7275 + 0.0734
0.5568
20

---

D.7
Using feature mixer MLPs
While we did not use feature mixer MLPs after cross- aesss-attentoon and SSM layers in our experiments, we conducted an experiment to test whether using them improved decoding performance. As shown in Table 10, performance with feature mixers was comparable or slightly better than our existing single-session models. However, these improvements come at the cost of increased model size and inference times. We leave more detailed investigation on this for future work.
Table 10: Behavioural decoding results with and without feature mixer MLPs. We report the mean R2 + SD over sessions.
Method
C C-CO 2016 (2)
T RT (6)
POssN-Mamba-SS (w/ mixers)
0.9560 = 0.0018
0.7487 = 0.0651
PossN-Mamba-SS (no mixers) 0.9550 = 0.0003
0.7418 = 0.0790
D.8 Using a recurrent encoder instead of cross-attention
We conducted an experiment with single-session POSSM models using a recurrent encoder (GRU) to encode spike tokens instead of the input cross-attention module. We maintained the same input configuration as with the cross-attention, providing contiguous, non-overlapping 50 ms time chunks as inputs the model. The encoder's hidden state reinitialized for each chunk, although in practice a state could be maintained across chunks for streaming neural activity. We did not use any positional embeddings for the spikes, but they were ordered by spike times. Our results showed that models with a recurrent encoder underperformed the standard POSSM models, but were superior to those not receiving precise spike-timing information (Table 11).
Table 11:
Behavioural decoding results with a recurrent encoder vs. a cross-attention encoder with and without precise spike timing information. We report the mean R + SD over sessions.
Method
C CO 2016 (2)
T RT (6)
POSSM-GRU-SS (w/ GRU encoder)
0.9396 + 0.0066
0.7334 0.0611
POSSM-GRU-SS (w/ spike times)
0.9549 + 0.0012
0.7687 + 0.0669
POSSM-GRU-SS (w/o spike times)
0.9026 + 0.0135
0.7207 + 0.0632
D.9 LoRA finetuning as an effective alternative to unit identification
In preliminary experiment with O-POSSM-GRU, we showed that finetuning embeddings on held-out sessions with low-rank adapters (LoRA) [72] could be an effective alternative unit identification (Table 12). This finetuning scheme trained fewer parameters and was quicker compared to unit identification. more detailed investigation is left for future work.
Table 12: Behavioural decoding results with LoRA finetuning compared to UI. We report the mean R² + SD over sessions.
Method
T RT (6)
Avg. # params, finetuned
O-POSSM-GRU (LoRA, 16, 16)
0.7478 = 0.0634
9.16K
O-POSSM-GRU (UT)
0.7464 0.0692
15.6K
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper includes both claims backed up by the experiments in the paper and also aspirational claims that stated as such.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Discussion, we state that most of our tasks and data are associated with the motor areas of the brain and that we currently only explore supervised training. We also state that our model cannot process a diverse set of input modalities and that multimodal neural decoding an interesting direction for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There are no theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our code will be available publicly through the torch_brais Python package. Additional details and configurations will be linked on our project page (ht t/tp:///pssm/ penssm-bbainn. n. git.u io). Details on our experiments and hyperparameter configurations are also provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
23

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Our code and associated documentation will be available publicly through the torch_brain Python package. Additional details and configurations will be linked our project page /ttp::://possmbbraaii -iiinnngnnnn gitb.b. io). All data used in this research is publicly available and the source materials repositories have been cited.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. jublic/guides/Cort souuddeess/oddeuummsssoooooooolo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips.. cttpsscccpbblll dtcee/udde///eoeeuubeeeeeee stiobmmmssoooocccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details have been provided in the main paper and the Supplementary Material. Additional details and configurations
will also be linked on our project page
(https:/ppossm-br) github..on
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All bar plots and line plots include error bars or shading appropriate. Standard deviations and results of statistical significance tests, where required, have been reported in tables.
24

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: These details have been provided the main paper and the Supplementary Material. Additional details and configurations will also be linked on project page bnttps:/ppossm-braill gi thub.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed and affirm that the research conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
25

---

Answer: [Yes]
Justification: Both potential positive and negative societal impacts have been discussed in the main text and in the Broader Impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have referenced the publications or source repositories for all datasets we have used. Our code based on and will be released publicly through the torch_brain Python package, and the associated publication has been cited. Details the datasets used have been provided the Supplementary Material and additional information on datasets and code will be linked through our project page (https: http://///sm//aaiiiee gi thub. io).
26

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and associated documentation will be available publicly through the torch_bre bacann Python package. Additional details and configurations will be linked on our project page ptttts::/ppsssmbraiii gi thub. io).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
27

---

Answer:
[NA]
Justification:
This paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not a component of the methods introduced or experiments con- ducted as part of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non- non-standard components. Please refer to our LLM policy (https: hoss://neeee //tprr/seuup///e/ee/eeeeeeee.e httppp///e/eerncnnee/ooeeeeeeeee L00 for what should should not be described.
28


Reviews:
[{"review_id": "jXgf6drovI", "reviewer": "Reviewer_oDCz", "summary": "Paper: Generalizable, real-time neural decoding with hybrid state-space models.\nThe paper addresses an unmet need in neural decoding: how to maintain fast, real-time causal inference while supporting generalization across subjects, sessions, and tasks. This is a longstanding challenge in closed-loop BCIs, where models have to adapt to new neural distributions without retraining from scratch, and where computational latency has to remain within strict limits (<10ms). Most existing models either generalize poorly or are too slow for real-time deployment. However, the improvements are marginal.", "questions": "see above", "limitations": "see above", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Novelty of methods/approach\nThe state-of-the-art for neural decoders rely on either binned spike counts with RNNs or long-context Transformers, both of which struggle with either generalization or real-time constraints. This paper introduces a hybrid architecture that processes individual spikes as tokens, rather than binned aggregates, and compresses them into a latent representation suitable for efficient recurrent modeling. What’s new is the combination of spike-level tokenization with chunk-wise cross-attention and a recurrent backbone, enabling both fast, causal inference and generalization across tasks, sessions, and even species.\n\n\nSignificance of results\nPOSSM matches or exceeds performance of state-of-the-art models across three domains: monkey reaching, human handwriting, and human speech decoding. Of particular note is the successful transfer of dynamics from pretraining on NHP datasets to a human subject handwriting task—an important result for neuroprosthetic applications where human data is limited. The architecture also achieves 5–9× faster inference than transformer baselines, including on CPU, making it suitable for embedded or clinical BCI systems.\n\nClarity of presentation\nThe paper is clearly structured and easy to follow. The figures effectively convey the model architecture and benchmarking results. The experimental comparisons are extensive and well-labeled. One minor suggestion would be to include a small schematic highlighting differences in causal vs. non-causal evaluation modes.\n\nTechnical correctness\n​​The methodological details are rigorous and complete. The authors compare against appropriate baselines, including both from-scratch and pretrained models, and provide detailed breakdowns of performance across subjects, tasks, and fine-tuning strategies. The use of causal decoding and inference benchmarks suggests the claims are well-supported.", "was_revised": true, "final_justification": "My concerns have been addressed and I revised up my evaluation.", "rebuttal": "We thank you for recognizing the several significant contributions of our work, including our fast, real-time inference, generalization capabilities, and successful transfer of dynamics from NHP datasets to a human handwriting task. Below, we address the points from your review:\n\n> However, the improvements are marginal.\n\nWe would like to respectfully note that this point appears to be in contradiction with other points in the review, e.g., \"POSSM matches or exceeds performance of state-of-the-art models across three domains...\", \"Of particular note is the successful transfer of dynamics from pretraining on NHP datasets to a human subject handwriting task—an important result for neuroprosthetic applications where human data is limited\", and \"The architecture also achieves 5–9× faster inference than transformer baselines...\".\n\nFirst, we would like to emphasize that the primary goal of our work is not just to improve decoding performance, but to do so in a way that maintains strong performance **while generalizing easily to new subjects and being efficient for real-time use**. We have also endeavoured to make our baselines as competitive as possible. Nevertheless, we have demonstrated that our method consistently matches or outperforms several popular and state-of-the-art baselines. In new experiments, we also pretrained our large-scale, multi-session o-POSSM models on an expanded dataset that now includes all 10 sessions from the Churchland et al. [1] dataset. With this, we significantly improved the performance of our unit identification finetuning to be comparable to or better than single-session performance, while maintaining or improving full finetuning performance. We provide these results in the table below.\n\n|Dataset|Best POSSM-SS|Best o-POSSM-UI|Best o-POSSM-FT|o-POSSM vs. POSSM improvement|\n|-|-|-|-|-|\n|C–CO 2016|0.9550 $\\pm$ 0.0003|0.9607 $\\pm$ 0.0036|0.9615 $\\pm$ 0.0056|0.68%|\n|C–CO 2010|0.7959 $\\pm$ 0.0772|0.8022 $\\pm$ 0.0818|0.8216 $\\pm$ 0.0945|3.23%|\n|T–CO|0.8863 $\\pm$ 0.0222|0.8921 $\\pm$ 0.0174|0.9068 $\\pm$ 0.0170|2.31%|\n|T–RT|0.7687 $\\pm$ 0.0669|0.7464 $\\pm$ 0.0692|0.7741 $\\pm$ 0.0683|0.70%|\n|NLB Area2|0.8350|0.8394|0.8941|7.08%|\n\nWith these new models, we have further improved our results on data efficiency (few-shot finetuning). We provide these results in the tables below, on a CO session:\n\n|Model|4 Trials|8 Trials|16 Trials|32 Trials|64 Trials|128 Trials|\n|-|-|-|-|-|-|-|\n|POSSM-SS|0.4607 $\\pm$ 0.0474|0.6010 $\\pm$ 0.0997|0.8390 $\\pm$ 0.0375|0.8929 $\\pm$ 0.0267|0.9410 $\\pm$ 0.0057|0.9463 $\\pm$ 0.0029|\n|o-POSSM-FT|0.5330 $\\pm$ 0.1374|0.7410 $\\pm$ 0.1097|0.9122 $\\pm$ 0.0163|0.9410 $\\pm$ 0.0055|0.9508 $\\pm$ 0.004|0.9582 $\\pm$ 0.001|\n\nand an RT session:\n\n|Model|4 Trials|8 Trials|16 Trials|32 Trials|64 Trials|\n|-|-|-|-|-|-|\n|POSSM-SS|0.2979 $\\pm$ 0.0324|0.3892 $\\pm$ 0.0152|0.5291 $\\pm$ 0.0097|0.6523 $\\pm$ 0.0121|0.7064 $\\pm$ 0.0138|\n|o-POSSM-FT|0.3613 $\\pm$ 0.0628|0.5372 $\\pm$ 0.0103|0.6157 $\\pm$ 0.0143|0.6709 $\\pm$ 0.0268|0.7087 $\\pm$ 0.0148|\n\nFinally, we have also integrated an additional modality – spike-band powers – in our speech decoding model, allowing us to further improve the performance over our previous results and the highly performant GRU baseline [2].\n\n|Model|Validation PER ($\\downarrow$)|\n|-|-|\n|Unidirectional GRU|21.74%|\n|Unidirectional POSSM-GRU|19.80%|\n\nOverall, we believe that our results not only perform comparably with or advance the state-of-the-art on several tasks, but also maintain efficiency both in terms of finetuning data and inference times. If you have any other questions or concerns about our results, we would be happy to address them during the discussion phase!\n\n> One minor suggestion would be to include a small schematic highlighting differences in causal vs. non-causal evaluation modes.\n\nThank you for the suggestion! While NeurIPS does not allow us to add figures or links in the rebuttal, we will include these in our final paper.\n\n---\n\nReferences:\n\n1. Churchland et al. \"Neural population dynamics during reaching.\" Nature (2012).\n2. Willett et al. \"Brain-to-Text Benchmark'24: Lessons Learned.\" arXiv:2412.17227 (2024).\n\n---\n\nOverall, while we are thankful for your positive comments, we are quite confused by the \"borderline reject\" rating of our paper as the review does not list specific weaknesses or questions for us to address. We would be grateful for any clarifications you might be willing to offer regarding your assessment, as this would help us better understand the recommendation. If our responses here and to other reviewers have addressed your concerns, we kindly ask if you would be willing to reconsider your score. We are also happy to address any concerns and questions you might have during the discussion phase, and eagerly look forward to your response."}, {"review_id": "NV0q82FFu1", "reviewer": "Reviewer_Eajm", "summary": "The paper introduces POSSM, an hybrid (SSM + Transformer) model to accelerate neural decoding inference time by leveraging the leaner SSM models while retaining a previously developed spike tokenization scheme. The new architecture blueprint allows for several SSM components instantiations (GRU, S4, Mamba). The authors have benchmarked the new method on monkey and human spike neural data recorded while performing motor tasks (plus a speech decoding task).", "questions": "- Is the gain of pre-training meaningful in a real-world setting, i.e. how many recording hours are needed for the single-session version to catch up with pretrained model?\n- Val $R^2$ performance seems to be close between the single-session and the o-POSSM version, despite the latter being significantly bigger in size (8M vs <1M). Is scaling of the single-session to match the size of the o-POSSM version making an impact?\n-  The o-POSSM has the same inference time as the POSSM-SS despite being ~x10 larger, how is this achieved?\n- How does the particular tokenization scheme used compare with a more ViT-like (used by NDT-2 for example)? What are its benefits?", "limitations": "The authors have not provided an explicit section showcasing the limitations of the current technique.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\nThe authors have successfully combined previous SSM models with a spike-based tokenization scheme and develop a novel architecture which delivers similar performance while significantly reducing inference time, which is important for practical BMI applications. The paper is in general well written with clear figures.\n\n## Weaknesses\n- The overall evaluation suite report limited comparisons against alternative models. Beyond the similar POYO-1 model (which the current architecture is inspired to) Table 1 only reports comparisons against NDT-2 (with caveats on hyperparameters tuning). All other listed results represents variants of the proposed architecture.\n- The claim that full fine-tuning yields better results than unit identification (Line 234-235) is fully expected.\n- \"[...] We see that pre-training results in a considerable initial boost in performance when fine-tuning to unseen data or new subjects.\" (Line 241-242). This claim seems to clash with what reported in either Figure 3a or 3b where o-POSSM and POSSM-SS have either comparable initial $R^2$ values as a function of training step (where o-POSSM starts actually lower, 3a) or as a function of number of samples (3b). Furthermore it seems that the gap seems to close after only ~32/64 trials, which sounds reasonable for calibrating a BCI.\n- Unclear what data displayed in Figure 3c, right refers to. Probably results are from training on Monkey C-CO and testing on T-CO (based on comparing with Table 1), but the caption is unclear.\n- The author evaluated their new family of models on the human speech decoding task, however they did not report any meaningful comparison for their results. As such it is very hard to assess whether the proposed model represents any improvements on the sota. In fact, the original publication (Willet et al. 2023) reported PER well below the authors own results (~20% vs. 27.32% for their best GRU model).\n- Little to no details are provided for the o-POSSM variant which Figure 3c shows to be ~8M vs. <1M model, however it is unclear where this scaling takes place.\n\nTypos & Minor:\n- (Line 100) Ml --> ML\n- (Line 125) retrain --> retain\n- (Line 275-276) Text line width is incorrect", "was_revised": true, "final_justification": "The authors have fully addressed my original concerns in their in-depth rebuttal, providing new results that showed: (1) that with multimodal features their model can indeed improve upon previously published results, (2) training with more data further improved their systems, (3) spike-times information during the tokenization scheme contributes significantly to the overall decoding performances, (4) all minor clarity issues/statements are addressed.", "rebuttal": "We sincerely thank you for your detailed review. We address your comments below:\n\n> W1. The overall evaluation suite report limited comparisons ...\n\nIn Table 1, we provide performance metrics for several non-POYO-style methods, including MLP, S4D, Mamba, GRU, and NDT-2, which can be used in a causal manner. These are popular, SOTA models for neural decoding from spikes [1], and unlike previous works, we have endeavoured to make these baseline models as performant as they can get. Notably, we have improved the results of the NDT-2 baseline through hyperparameter tuning and pretraining on all our data (see table). However, we demonstrate in Table 1 that POSSM handily outperforms all non-POYO baselines. If you have any additional suggestions on baseline methods, we would be happy to compare against them.\n\n|Model|C–CO 2010|T–CO|T–RT|\n|-|-|-|-|\n|NDT-2|0.7527 $\\pm$ 0.0926|0.6897 $\\pm$ 0.0657|0.6895 $\\pm$ 0.0662|\n|Best non-POYO baseline (single session)|0.7308 $\\pm$ 0.1089|0.8526 $\\pm$ 0.0243|0.7279 $\\pm$ 0.0679|\n|POSSM-SS best|0.7959 $\\pm$ 0.0772|0.8863 $\\pm$ 0.0222|0.7687 $\\pm$ 0.0669|\n|o-POSSM-FT best|0.8216 $\\pm$ 0.0945|0.9068 $\\pm$ 0.0170|0.7741 $\\pm$ 0.0683|\n\n> W2. The claim that full fine-tuning yields better results ...\n\nWe would like to clarify that this statement should not be seen as a new claim, but as an interpretation of our results. We are emphasizing that - especially on new subjects - full finetuning (FT) has better performance overall, although it remains notable that unit identification (UI) is close to or better than single session performance (see next response). We will rephrase this statement to clearly indicate this. Again, we would like to kindly point out that this is not our main claim or contribution.\n\n> W3. This claim seems to clash with what reported in either ...\n\nThe initial results at training step 0 are random because unit and session embeddings are randomly initialized. Thus, the variance in performance at the initial steps is expected to be high – we do have runs where o-POSSM starts higher than POSSM-SS. We would like to emphasize that the key takeaway of Fig. 3a is that UI leads to much faster convergence and higher performance than single session models (see table below), in less than 1/3rd of the training steps and by finetuning far fewer parameters (~10K vs ~600K).\n\n|Dataset|Best POSSM-SS|Best o-POSSM-UI|Best o-POSSM-FT|o-POSSM vs. POSSM improvement|\n|-|-|-|-|-|\n|C–CO 2010|0.7959 $\\pm$ 0.0772|0.8022 $\\pm$ 0.0818|0.8216 $\\pm$ 0.0945|3.23%|\n|T–CO|0.8863 $\\pm$ 0.0222|0.8921 $\\pm$ 0.0174|0.9068 $\\pm$ 0.0170|2.31%|\n|T–RT|0.7687 $\\pm$ 0.0669|0.7464 $\\pm$ 0.0692|0.7741 $\\pm$ 0.0683|0.70%|\n|NLB Area2|0.8350|0.8394|0.8941|7.08%|\n\nThese improved metrics are a result of pretraining on more data, now including all sessions from the Churchland et al. dataset [2]. With this, we have improved the performance of our few-shot finetuning as well:\n\n|Model/Trials|4|8|16|32|64|128|\n|-|-|-|-|-|-|-|\n|POSSM-SS|0.4607 $\\pm$ 0.0474|0.6010 $\\pm$ 0.0997|0.8390 $\\pm$ 0.0375|0.8929 $\\pm$ 0.0267|0.9410 $\\pm$ 0.0057|0.9463 $\\pm$ 0.0029|\n|o-POSSM-FT|0.5330 $\\pm$ 0.1374|0.7410 $\\pm$ 0.1097|0.9122 $\\pm$ 0.0163|0.9410 $\\pm$ 0.0055|0.9508 $\\pm$ 0.004|0.9582 $\\pm$ 0.001|\n\n|Model/Trials|4|8|16|32|64|\n|-|-|-|-|-|-|\n|POSSM-SS|0.2979 $\\pm$ 0.0324|0.3892 $\\pm$ 0.0152|0.5291 $\\pm$ 0.0097|0.6523 $\\pm$ 0.0121|0.7064 $\\pm$ 0.0138|\n|o-POSSM-FT|0.3613 $\\pm$ 0.0628|0.5372 $\\pm$ 0.0103|0.6157 $\\pm$ 0.0143|0.6709 $\\pm$ 0.0268|0.7087 $\\pm$ 0.0148|\n\nThe performance improvements from pretraining are now more apparent both for fewer and more trials. Furthermore, we would argue that in the \"real world\", calibration should be as data- and compute-efficient as possible, so the fewer trials needed to calibrate the BCI, the better (64 trials = >5 minutes of active task performance, which can get cumbersome if done everyday).\n\n> W4. Unclear what data displayed in Figure 3c ... training on Monkey C-CO and testing on T-CO ...\n\nYes, you are correct – we will clarify this in the caption. Thank you for pointing out this clarity issue. While NeurIPS does not allow us to provide figures with the rebuttal, we have updated the caption to be clearer and modified all relevant figures to indicate the subjects and sessions clearly in our draft manuscript.\n\n> W5. The author evaluated their new family of models on the human speech decoding task ...\n\nIn our paper, we initially considered the problem of decoding speech from neural spikes alone. Importantly, the GRU you are referring to is achieved with spike-band powers included (we reproduced PER = 21.74%). This is a **multimodal** approach that is outside the scope of this paper, but we are actively exploring such approaches in ongoing work. As a preliminary experiment, we provide performance comparisons between the baseline and POSSM. We achieve competitive performance, even before full exploration of the design space. We believe with an improved tokenization scheme and access to more data, a significant bump in performance can be attained.\n\n|Model|Val PER ($\\downarrow$)|\n|-|-|\n|Unidirectional GRU|21.74%|\n|Unidirectional POSSM-GRU|19.80%|\n\nOur work focuses on PER, not downstream language modeling, which many prior works optimize via ensembling and bidirectional GRUs (most other methods such as transformers and SSMs do not outperform the GRU, and most top teams use biGRU decoders with training tricks [1]). These methods also lack public code and do not report PER scores [1], making direct comparisons difficult. In contrast, our unidirectional model does not have these speech-specific optimizations and is designed to be general-purpose, yet still attains competitive performance. While the current dataset is small, we expect our latent representations to become more expressive when trained with larger datasets, such as the upcoming Brain2Text-2025 benchmark. We are also exploring finetuning a model pre-trained on motor tasks, similar to what we did for handwriting.\n\n> W6. Little to no details ...\n\nApologies that this was not clear. Currently, these details are provided in the Appendix (Section S.5.2, Table 5), but we have now updated our draft to include these details in the main paper. Scaling is primarily determined by the number of SSM/RNN layers, and also by an increase in the dimensionality of the input and RNN hidden state.\n\n> Q1. Is the gain of pre-training meaningful ...\n\nThis proposed study is made difficult by the fact that sessions in existing datasets have limited amounts of data. However, we show that single-session training is far more expensive, as it requires training ~600K params while UI only needs to tune ~10K. While being parameter-efficient, we also know that UI achieves higher performance more quickly than training from scratch. These benefits are especially felt when there are multiple downstream subjects to whom the model must be finetuned, as the reduced training times and parameter counts are compounded. Finally, UI outperforms single-session especially in low data regimes (c.f. finetuning results above), which are reflective of BCI use cases, indicating that UI is a viable and useful approach in real-world settings.\n\n> Q2. Val $R^2$ performance seems to be close between ...\n\nIn our updated results shared above, we show gains from additional pretraining for o-POSSM. We decided to scale up the size of the model only if there was a performance benefit from doing so, and we believe that with even more pretraining data, the gains would be higher. We would also like to note that the variance in sizes for o-POSSM variants is quite high (S4D 4.5M, GRU 7.9M, Mamba 9M).\n\n> Q3. The o-POSSM has the same inference time ...\n\nThe o-POSSM models are slightly slower on average, especially on CPU. The similarity in inference times is simply the result of PyTorch having efficient, parallelizable implementations of various architectural components (attention and GRU), we do nothing in addition to ensure the closeness in inference times. We are, however, working on using Flash Attention for our initial cross-attention layer to further speed up computations, especially during training.\n\n> Q4. How does the particular tokenization scheme ...\n\nThe ViT-like patching scheme used in NDT-2 groups together arbitrary neurons within a patch, does not have a mechanism to identify separate units, and is not permutation-invariant like our scheme. Meanwhile, we explicitly learn embeddings for each unit, and we believe that this scheme both generalizes better than and enables a more straightforward and efficient finetuning procedure than NDT-2 by relearning just the unit/session embeddings.\n\nFurthermore, we believe that using spike timing information could be instrumental in improving decoding performance. In the table below, we show results with and without access to raw spike times for each spike, where the latter would be identical to binning-based approaches. We see a considerable performance drop in the latter case, thus further motivating our choice of tokenization scheme.\n\n|Dataset|POSSM-Mamba-SS w/ spike times|w/o spike times|\n|-|-|-|\n|C–CO 2010|0.7959 $\\pm$ 0.0772|0.7040 $\\pm$ 0.0903|\n|T–RT|0.7418 $\\pm$ 0.0790|0.7275 $\\pm$ 0.0734|\n|NLB Area2|0.8350|0.5568|\n\nWe believe that these differences in tokenization could be the key factor behind the gap in performance between POSSM/POYO-style models and NDT-2. In ongoing work, we are also investigating how well the unit embeddings encode neuronal properties when trained with SSL objectives.\n\nReferences:\n\n1. Willett et al. \"Brain-to-Text Benchmark '24: Lessons Learned.\" arXiv:2412.17227 (2024)\n2. Churchland et al. \"Neural population dynamics during reaching.\" Nature (2012).\n\nWe thank you once again for your review and attention to detail, and believe your comments have helped improve our work. We hope we have addressed your concerns, and if so, we kindly ask if you would be willing to reconsider your score. We look forward to discussing further with you!"}, {"review_id": "B5CJlML8Q8", "reviewer": "Reviewer_eRCh", "summary": "In this paper, the authors propose a novel hybrid method, POSSM,  designed for generalizable, real-time neural decoding addressing limitations of traditional recurrent neural networks (RNNs) and transformer-based approaches. POSSM combines individual spike tokenization via a cross-attention module with a recurrent state-space model, enabling fast and causal online prediction on neural activity and efficient generalization to new sessions, individuals, and tasks through multi-dataset pre-training. It processes neural activity in 50ms chunks, tokenizing individual spikes.\n\nThe paper demonstrates that POSSM achieves comparable decoding accuracy with state-of-the-art Transformers at a fraction of the inference cost, making it suitable for real-time brain-computer interfaces as its inference time is well within the optimal latency. A notable contribution is its potential for cross-species transfer learning, shown by improved human handwriting decoding performance after pre-training on monkey motor-cortical recordings. POSSM also effectively decodes long-context human speech, a task computationally expensive for standard transformer models. These results position hybrid SSMs as a promising approach for practical, general neural interfaces.", "questions": "- Could the authors comment on how POSSM's applicability might be extended beyond motor areas? For example, how could it be adapted to cognitive or sensory domains where clear time-to-time correspondence is lacking? Addressing this would help establish its broader utility as a generalizable neural interface.\n- Could the authors provide further insights into the interpretability of POSSM and how it may facilitates understanding of neural dynamics?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n*   Superior decoding accuracy comparable to state-of-the-art transformers, yet it is significantly faster during inference. This speed is crucial for real-time BCIs, as its inference time is well within the optimal latency. \n*   Efficient Generalization and Flexible Input Processing: Unlike traditional RNNs that struggle to generalize to new sessions or subjects due to rigid, fixed-size, time-binned inputs, POSSM leverages individual spike tokenization. This flexible input processing, combined with multi-dataset pre-training, allows POSSM to efficiently generalize to new sessions, individuals, and tasks.\n*   Demonstrated Cross-Species Transfer Learning: A unique and impactful strength is POSSM’s capability for cross-species transfer learning. This provides a solution to the challenge of limited human data by leveraging a larger corpus of existing animal model data to enhance clinical outcomes.\n\nWeaknesses\n* Limited Evaluated Scope to Motor Areas: the tasks and data used for POSSM's evaluation are largely associated with the motor areas. Its performance and applicability outside of motor-related neural activity are not yet fully demonstrated.\n* Reliance on Supervised Training: The current training approach for POSSM is supervised, meaning it requires labeled neural activity paired with behavioral or cognitive variables. This also pertains to the applicability outside of motor-related neural activity that might have no explicit behavioral labels.\n* Lack of Intrinsic Multimodality: POSSM, in its current form, lacks inherent multimodality. While its architecture is flexible enough to accommodate different neural data modalities through proven tokenization schemes (e.g., Calcium Imaging, LFP), it is not intrinsically designed to seamlessly integrate and process multiple data types simultaneously, a potential area for future development.", "was_revised": true, "final_justification": "The paper introduces POSSM, a novel hybrid architecture for real-time neural decoding that successfully bridges the gap between accuracy, inference speed, and generalization. The demonstrated cross-species transfer learning, from non-human primates to human handwriting tasks, is particularly exciting and clinically relevant, addressing a major data acquisition challenge. The reported faster inference time compared to state-of-the-art Transformers is a critical advantage for real-time Brain-Computer Interfaces (BCIs).\n\nThe authors' rebuttal effectively addressed key concerns. Applicability beyond motor areas: The preliminary results decoding running speed from multi-region (visual cortex + subcortical) neural activity and the clarification on handling tasks without dense time-to-time correspondence significantly bolster the model's claimed generalizability.\nIntrinsic Multimodality: New results showing improved performance when integrating binned spike counts and spike-band powers for speech decoding demonstrate the architecture's flexibility.\n\nRemaining unresolved issues, such as the reliance on purely supervised training and in-depth interpretability, are reasonably acknowledged by the authors as important future work and not the primary focus of this paper. These limitations are also transparently discussed within the paper.", "rebuttal": "We thank you sincerely for your positive review of our paper and for outlining important research directions that we are actively investigating for future work. Below we respond to your questions and concerns:\n\n> W1. Limited Evaluated Scope to Motor Areas\n\nWe acknowledge that in this work, we have mostly considered datasets with motor-cortical neurons and some motor-related behaviour. However, these datasets also contain neurons from Broca's area (speech task [1]) and S1 (primary somatosensory cortex; random target task [2]) – demonstrating that the method can process neurons from a broader set of neural regions.\n\nAn investigation of POSSM's performance on data from other brain regions is an interesting direction for future work, and we have good reason to believe that it would work well beyond the motor domain. Notably, POYO+ [3], which adapts the POYO tokenization for regular time series data such as Calcium recordings (similarly to our tokenization for binned spike counts in the speech and handwriting experiments), demonstrates strong behavioural decoding performance on visual cortex recordings from the Allen Brain Observatory dataset [4]. As POSSM shares the same multitask readout framework and tokenization scheme as POYO while exhibiting similar performance, we expect that it will generalize well to other brain regions and tasks.\n\nTo support this, we provide preliminary results on 5 random sessions from the Allen Visual Behavior Neuropixels dataset [4], where we decode running speed from multi-region neural activity (visual cortex + several subcortical areas including visual thalamic areas, hippocampus, and midbrain).\n\n|Model|Running Speed $R^2$|\n|-|-|\n|POSSM-SS-GRU|0.9295 $\\pm$ 0.0326|\n\nThis result provides early evidence that POSSM is effective across a variety of brain regions. We also anticipate that even better performance can be obtained with a mixture of more careful experimentation, training on multiple tasks [3], backbone changes, and hyperparameter tuning. Note that the sequence lengths at evaluation are 10x the training sequence lengths, again demonstrating strong length generalization.\n\n> W2. Reliance on Supervised Training\n\nWe agree with the reviewer on the limitations of using a purely supervised learning paradigm. Indeed, requiring labeled data restricts both the amount and diversity of training data, potentially resulting in models that are biased towards certain tasks with denser or easier-to-curate lables. Moving away from supervised training and towards self-supervised learning (SSL) objectives is a key direction that we are investigating in ongoing work. With SSL, we aim to leverage unlabeled neural data to further improve downstream decoding performance, especially when finetuning to new sessions or subjects. We are also investigating the interpretability of the learned parameters, such as the unit embeddings [3], trained through SSL – specifically, whether we can identify finer details about the neural population (e.g., cell types, brain regions, etc.). Finally, we are also interested in how best to train an SSM using SSL objectives, compared to Transformer-based masked autoencoders [5]. As this is an ongoing effort, we leave discussion of our results for future work.\n\n> W3. Lack of Intrinsic Multimodality\n\nThank you for mentioning multimodality, which is another important direction in our ongoing research. While we have only considered neural spiking data in this paper, we would like to point out that several of these datasets are recorded with multiple distinct \"modalities\", such as threshold crossings, sorted or unsorted spikes (a unit could be either a single neuron or a \"multiunit\" or \"channel\" depending on the dataset), and binned spike counts from public datasets [1,6]. To this end, we have recently trained POSSM on the speech task [1] **on both the binned spike counts and spike-band powers**. Excitingly, we observe strong performance when integrating both modalities (see table below).\n\n\n|Model|Validation PER ($\\downarrow$)|\n|-|-|\n|Unidirectional GRU|21.74%|\n|Unidirectional POSSM-GRU|19.80% |\n\nFinally, in ongoing work, we are training our models on both spikes and local field potentials (LFPs) in order to investigate whether the same cross-attention layer can process both spikes and LFPs (with an additional modality embedding), or if other forms of early/late modality fusion work better. The tokenization scheme we use is identical in both these cases (LFPs are tokenized similarly to binned spike counts and Calcium imaging in POYO+ [3]), so we would argue that our approach is quite general and involves minimal preprocessing. We also thank you for acknowledging in your review that our architecture is \"flexible enough to accommodate different neural data modalities through proven tokenization schemes\".\n\n> Q1. Could the authors comment on how POSSM's applicability might be extended beyond motor areas? For example, how could it be adapted to cognitive or sensory domains where clear time-to-time correspondence is lacking? Addressing this would help establish its broader utility as a generalizable neural interface.\n\nThis is an important question, and we have provided promising evidence (running speed decoding from Neuropixels data [4]) and clarifications (having neurons from S1 and Broca's area in our data mix) that POSSM extends beyond just motor areas in the response to W1. Furthermore, we already explore domains where time-to-time correspondence is lacking – the speech decoding task [1] does not have dense labels for each time chunk, so we use connectionist temporal classification (CTC) [7] to predict the best sentence response at the whole-trial level for the given neural activity sequence. We have also carried out an experiment on predicting cursor velocity 50ms or 100ms in the future, thus querying the output module with timestamps beyond the current time chunk. Results are provided in the table below. We believe that such flexibility will enable the use of POSSM in tasks such as those you have outlined.\n\n|Setup|C–CO 2016|T–RT|\n|-|-|-|\n|POSSM-GRU-SS 50ms future|0.9495 $\\pm$ 0.0045|0.7498 $\\pm$ 0.0874|\n|POSSM-GRU-SS 100ms future|0.9368 $\\pm$ 0.0027|0.7252 $\\pm$ 0.0813|\n\n> Q2. Could the authors provide further insights into the interpretability of POSSM and how it may facilitates understanding of neural dynamics?\n\nThank you for the question. While the our work is focused on strong predictive performance and fast, real-time inference for neural decoding, we agree that interpreting how deep learning models arrive at their predictions is an important line of work.\n\nFirst, simple methods such as vanilla RNN/SSM models already help with understanding neural dynamics very well through dynamical systems analysis techniques, so we do not think POSSM can provide much additional benefit here. The spike tokenization scheme, on the other hand, can enable classification of brain regions and cell types [3] – the similarity of learned unit embeddings for neurons could potentially correlate with functional similarity of neurons across sessions or animals.\n\nAs for the interpretability of POSSM itself, we looked at the evolution of hidden states from the POSSM-Mamba and POSSM-GRU models over test trials. Unfortunately, NeurIPS does not allow us to share figures or links for the rebuttal – nevertheless, we aim to describe our findings here. We found that the top 2 principal components (PCs) of the hidden states explained 68% and 51% of the variance for the Mamba and GRU respectively. For both models, the first PC resembled the magnitude of the cursor speed in upcoming time chunks, while the second PC seemed to represent change in cursor velocity (~acceleration, direction-sensitive). We leave more detailed analysis of the hidden states and various architectural components for future work. We would like to note, however, that enforcing learned latents to be human-interpretable rarely leads to improved downstream performance [8], while the objective of our work was to obtain strong performance under real-time inference constraints.\n\n---\n\nReferences:\n\n1. Willett et al. \"A high-performance speech neuroprosthesis.\" Nature (2023).\n2. O’Doherty et al. \"Nonhuman primate reaching with multichannel sensorimotor cortex electrophysiology.\" Zenodo (2017).\n3. Azabou et al. \"Multi-session, multi-task neural decoding from distinct cell-types and brain regions.\" ICLR (2025).\n4. de Vries et al. \"Sharing neurophysiology data from the Allen Brain Observatory.\" Elife 12 (2023).\n5. Ye et al. \"Neural data transformer 2: multi-context pretraining for neural spiking activity.\" NeurIPS (2023).\n6. Willett et al. \"High-performance brain-to-text communication via handwriting.\" Nature (2021).\n7. Graves et al. \"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.\" ICML 2006.\n8. Mittal et al. \"Does learning the right latent variables necessarily improve in-context learning?\" ICML (2025).\n\n---\n\nOnce again, we thank you for your time and positive review, and hope that our response has addressed your concerns. We will endeavour to include all these additional results in our paper and supplementary material. We would be grateful if you would consider supporting our paper more strongly as a result. We are also happy to answer any other questions you may have and look forward to discussing further!"}, {"review_id": "CW6KEM0Xjy", "reviewer": "Reviewer_Xwt5", "summary": "The paper introduces POSSM, a hybrid model architecture for spike-based neural decoding that builds upon the POYO-1 framework. While maintaining state-of-the-art (SOTA) performance, POSSM is designed to operate causally and with low latency, making it well-suited for real-time brain-computer interface (BCI) applications. Evaluations across diverse datasets highlight the model's robustness and generalization capabilities, with particularly notable results in cross-species knowledge transfer. This suggests that POSSM could represent an early step toward foundational models in neural decoding.", "questions": "Questions: \n- In the first cross attention layer, why you don't use again a SSM so you don't need to process spikes by 50ms windows. You could also do that with a sliding window attention architecture. Furthermore, you could use a sliding window attention instead of an SSM, which could be an interesting extra comparison (I mention that because it seems that they perform very similarly as it was shown in MesaNet[von Oswald et al 2025]).\n- In line 154, in the equation it would be nice to have the dimenssionality of the $W_k$ and $W_v$ matrices, I assume they are MxN. \n- Something that was a bit unclear to me is whether you have feature mixers (a small MLP, or a multiplication with a gate like it happens in Mamba) after the attention or SSM layer. Feature mixers although a bit computationally expensive, they can improve the performance considerably.\n- From Table 1, some models trained on single sessions perform better than pretrained models with UI (like POSSM-GRU) for the new dataset, do i read it, right? It's also interesting POSSM-SS models perform so much better than POYO-SS, do you have any idea why that is the case? It seems that the UI finetuning is not that useful for the SSM models. Maybe a possibility could be to do a LORA fine-tuning, which could be more efficient?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Achieves SOTA performance with low latency, which is crucial in neural decoding and BCI applications.\n- They manange to extend the POYO-1 architecture, with the smart spike tokenization, allowing it to run causally and at the same time well below 10ms.\n\nWeaknesses:\n- The results in Table 2 are not well presented, because from the numbers except from PCA-KNN all the other result do not look to differ significantly. given that the plus-minus is the std (where if i am not wrong the $\\pm$ is not reported).\n- In section 4.3, if I understood correctly you first transform the spikes with some preprocessing and then in the first step you retrieve again spikes, which looks odd.", "was_revised": true, "final_justification": "As I wrote from my initial review, I found that this paper is technically solid and is very interesting for the field. The authors clarified all my questions, and I am pleased to keep my initial score, as I explained in my official comment.", "rebuttal": "We sincerely thank you for your positive assessment of our paper and insightful review! Below, we respond to your concerns and questions:\n\n> W1. The results in Table 2 are not well presented, because from the numbers except from PCA-KNN all the other result do not look to differ significantly. given that the plus-minus is the std (where if i am not wrong the $\\pm$ is not reported).\n\nWe apologize for the lack of clarity, we reported the arithmetic mean of mean performance across sessions $\\pm$ the geometric mean of standard deviations across sessions, aggregated across seeds. We will update the caption to state this clearly. We also conducted a paired t-test between session-wise performances across seeds, between the best NHP-pretrained + finetuned o-POSSM and the best non-pretrained POSSM, where we found the results to be significant with p < 0.05 (p = 0.0152). Furthermore, comparing the mean performances across sessions for different seeds between all NHP-pretrained and non-pretrained models yielded p < 0.05 for 7/9 valid comparisons, p <= 0.1 for 2/9 cases (lower performing o-POSSM variants vs. non-pretrained POGRU). Thus, we show that the transfer result is statistically significant overall. We will update our paper to reflect this.\n\n> W2. In section 4.3, if I understood correctly you first transform the spikes with some preprocessing and then in the first step you retrieve again spikes, which looks odd.\n\nApologies that this was not clear. We do not transform or preprocess the spikes; the data from the original source is already preprocessed through 20ms binning [1]. Therefore, we do not have access to raw spike timings. From preliminary experiments on other datasets, we found that having access to these raw spike timings can significantly improve performance (see table below). Additionally, given that the dataset is small, we found that our two-stage training curriculum yielded the best performance. Given access to more data and raw spikes, we believe this scheme would become unnecessary.\n\n|Dataset|POSSM-Mamba-SS w/ spike times|w/o spike times|\n|-|-|-|\n|C–CO 2010|0.7959 $\\pm$ 0.0772|0.7040 $\\pm$ 0.0903|\n|T–RT|0.7418 $\\pm$ 0.0790|0.7275 $\\pm$ 0.0734|\n|NLB Area2|0.8350|0.5568|\n\n> Q1. In the first cross attention layer, why you don't use again a SSM so you don't need to process spikes by 50ms windows. You could also do that with a sliding window attention architecture. Furthermore, you could use a sliding window attention instead of an SSM, which could be an interesting extra comparison (I mention that because it seems that they perform very similarly as it was shown in MesaNet).\n\nIf we understand you correctly, using an SSM instead of the first cross-attention (CA) layer would mean that each token is processed by the model as each spike is recorded, i.e., one forward pass per token. This could perhaps allow for more fine-grained prediction timescales, but might be wasteful in terms of computational efficiency and is overall a harder problem because: (1) the sequences will become far longer and (2) the timing between spike tokens is highly irregular. Observing a window of activity for the whole population is important for the model to predict behaviour in dynamics tasks, which is why we use CA to process chunks of activity. Please let us know if we have misunderstood something or if you have an alternate implementation in mind, we would be happy to explore this.\n\nSliding window attention (SWA) seems better suited to our tasks, and we have experimented with giving the model overlapping time chunks of a certain width and stride (e.g., 50ms and 20ms respectively) – performance is comparable to the contiguous non-overlapping window case.\n\nReplacing the recurrent backbone with SWA makes for an interesting additional comparison. We provide some results in the table below with one SWA layer in single session models. We would like to note that von Oswald et al. [2] show that smaller sliding windows do not perform very well. In our case, the performance of SWA does not match the best recurrent backbone – and we interestingly observe that larger window sizes perform slightly worse than smaller ones. This would have to be studied further, with hyperparameter tuning of the number of attention layers or heads, their dimensionality, dropout, etc. – which we will explore in the future.\n\n|Model|C–CO 2016|T–RT|\n|-|-|-|\n|Best POSSM-SS|0.9550 $\\pm$ 0.0003|0.7687 $\\pm$ 0.0669|\n|POSSM-SS-SWA(5)|0.9492 $\\pm$ 0.0038|0.7188 $\\pm$ 0.0966|\n|POSSM-SS-SWA(15)|0.9469 $\\pm$ 0.0011|0.7043 $\\pm$ 0.1117|\n|POSSM-SS-SWA(20)|0.9449 $\\pm$ 0.0005|0.6736 $\\pm$ 0.1282|\n\n> Q2. In line 154, in the equation it would be nice to have the dimenssionality of the $W_k$ and $W_v$ matrices, I assume they are MxN.\n\nYes, thank you for the suggestion and apologies for not including this earlier – we have updated the paper to state the dimensionality explicitly.\n\n> Q3. Something that was a bit unclear to me is whether you have feature mixers (a small MLP, or a multiplication with a gate like it happens in Mamba) after the attention or SSM layer. Feature mixers although a bit computationally expensive, they can improve the performance considerably.\n\nWe did not use mixer MLPs after the attention or SSM layers, mostly due to efficiency and strong performance even without them. While mixer MLPs provided marginal improvements in POSSM-SS models, they may be necessary at greater pretraining scale, so we will explore these in more detail in future work. A comparison with POSSM-Mamba-SS on the Perich et al. dataset [3] is given below, showing a slight increase in performance with mixer MLPs. We will add these results to our paper.\n\n|Type|C–CO 2016|T–CO|T–RT|\n|-|-|-|-|\n|No MLP|0.9550 $\\pm$ 0.0003|0.8747 $\\pm$ 0.0173|0.7418 $\\pm$ 0.0790|\n|With MLP|0.9560 $\\pm$ 0.0018|0.8828 $\\pm$ 0.0164|0.7487 $\\pm$ 0.0651|\n\n> Q4. From Table 1, some models trained on single sessions perform better than pretrained models with UI (like POSSM-GRU) for the new dataset, do i read it, right? It's also interesting POSSM-SS models perform so much better than POYO-SS, do you have any idea why that is the case? It seems that the UI finetuning is not that useful for the SSM models. Maybe a possibility could be to do a LORA fine-tuning, which could be more efficient?\n\nThank you for this question, this was the case before. However, we now pretrained models on more data, including all 10 sessions from the Churchland et al. dataset [4] as opposed to just 2 earlier. With this, we significantly improved our UI results, and now they perform on par with or sometimes even better than POSSM-SS models (see table) on data from Perich et al. [3]. With these, we are able to finetune very few parameters overall for far less time and still achieve good performance.\n\n|Model|C–CO 2016|T–CO|T–RT|\n|-|-|-|-|\n|Best POSSM-SS|0.9550 $\\pm$ 0.0003|0.8863 $\\pm$ 0.0222|0.7687 $\\pm$ 0.0669|\n|Best POSSM-UI|0.9607 $\\pm$ 0.0036|0.8921 $\\pm$ 0.0174|0.7464 $\\pm$ 0.0692|\n\nLoRA is a great suggestion: we are exploring this in ongoing work involving SSL pretraining. For the sake of further compute efficiency and few-shot benefits on top of LoRA, it might be worth bootstrapping unit embeddings using existing pretrained ones (rather than starting from random for a new session or animal) – this direction requires further exploration. We believe this kind of LoRA-based UI pipeline is best suited to adaptation across days for the same animal, and especially when embedding multiunits (channels) rather than single neurons.\n\nWe present some preliminary results below, where we assign existing trained unit embeddings for new units (in a new animal heldout from training), and finetune a LoRA for the embedding layer. We anticipate improvements with further hyperparameter tuning.\n\n|o-POSSM-GRU|T–RT|avg # params finetuned (range)|\n|-|-|-|\n|UI|0.7464 $\\pm$ 0.0692|15.66K (10-19.5K)|\n|LoRA (r=16, $\\alpha$=16)|0.7478 $\\pm$ 0.0634|9.16K (8.8-9.4K)|\n\nIt is unclear exactly why POYO-SS slightly underperforms POSSM-SS, although potential explanations include:\n\n* Prior work [5] finds that recurrent architectures are better-suited to neural decoding than transformers (caveat: smaller dataset scale).\n* Other work [6] shows that the inductive biases of SSMs allow them to learn delay embeddings better than transformers, thus enabling better performance on dynamics tasks (especially with less data and fewer parameters).\n\nWe do find that POYO performs on par with or marginally better than o-POSSM on some multi-session experiments (e.g., Flint et al. dataset [7]). Deeper investigation into the latent representations learnt by these models is required to understand these differences in performance, and we are keen on exploring this in future work. For a preliminary study, please refer to our response to Reviewer eRCh, where we interpret the SSM hidden states.\n\n---\n\nReferences:\n\n1. Willett et al. \"A high-performance speech neuroprosthesis.\" Nature (2023).\n2. von Oswald et al. \"MesaNet: Sequence Modeling by Locally Optimal Test-Time Training.\" arXiv:2506.05233 (2025).\n3. Perich et al. \"A Neural Population Mechanism for Rapid Learning.\" Neuron (2018).\n4. Churchland et al. \"Neural population dynamics during reaching.\" Nature (2012).\n5. Willett et al. \"Brain-to-Text Benchmark'24: Lessons Learned.\" arXiv:2412.17227 (2024).\n6. Ostrow et al. \"Delay embedding theory of neural sequence models.\" arXiv:2406.11993 (2024).\n7. Flint et al. \"Accurate decoding of reaching movements from field potentials in the absence of spikes.\" Journal of Neural Engineering (2012).\n\n---\n\nOnce again, thank you for your time, insightful suggestions, and kind assessment of our paper. We will endavour to include all these additional results in our paper and supplementary material. Overall, we hope this rebuttal addresses your concerns, and would be grateful if you would consider supporting our paper more strongly. We would be happy to answer any other questions during the discussion phase."}]

Author rebuttal:

Meta-review: This work introduces POSSM, a brain-computer interface (BCI) model that uses a hybrid transformer-state space model architecture to achieve state-of-the-art decoding performance in real time. POSSM uses spike-based tokenization (à la POYO), input cross-attention, and an SSM recurrent backbone to produce embeddings that allow for accurate decoding while lowering model inference times. POSSM is then applied to a variety of benchmark tasks (monkey reach, human handwriting, speech decoding) with state of the art results.

Reviewers were enthusiastic about the novel architecture, rigorous benchmarking, and careful experiments that show the value of more data and pretraining in generalizing across sessions and participants. Reviewers noted some small weaknesses in performance improvements and breadth of model comparisons, which the authors addressed during rebuttal with new results showing improved performance with additional data.

In all, a strong contribution to the literature on large-scale brain decoding models.
Decision: Accept (poster)