Neural decoding from stereotactic EEG: accounting for electrode variability across subjects
Authors: Georgios Mentzelopoulos
Evangelos Chatzipantazis
Ashwin G Ramayya
Michelle Hedlund
Vivek Buch
Kostas Daniilidis
Konrad Kording
Flavia Vitale
Venue: NeurIPS 2024 poster (2024)
Keywords: sEEG
Neural Decoding
Transformers
Multi-Subject Training
Primary area: neuroscience_and_cognitive_science
TL;DR: Seegnificant: a framework and architecture for multi-subject neural decoding based on sEEG

Abstract: Deep learning based neural decoding from stereotactic electroencephalography (sEEG) would likely benefit from scaling up both dataset and model size. To achieve this, combining data across multiple subjects is crucial. However, in sEEG cohorts, each subject has a variable number of electrodes placed at distinct locations in their brain, solely based on clinical needs. Such heterogeneity in electrode number/placement poses a significant challenge for data integration, since there is no clear correspondence of the neural activity recorded at distinct sites between individuals. Here we introduce seegnificant: a training framework and architecture that can be used to decode behavior across subjects using sEEG data. We tokenize the neural activity within electrodes using convolutions and extract long-term temporal dependencies between tokens using self-attention in the time dimension. The 3D location of each electrode is then mixed with the tokens, followed by another self-attention in the electrode dimension to extract effective spatiotemporal neural representations. Subject-specific heads are then used for downstream decoding tasks. Using this approach, we construct a multi-subject model trained on the combined data from 21 subjects performing a behavioral task. We demonstrate that our model is able to decode the trial-wise response time of the subjects during the behavioral task solely from neural data. We also show that the neural representations learned by pretraining our model across individuals can be transferred in a few-shot manner to new subjects. This work introduces a scalable approach towards sEEG data integration for multi-subject model training, paving the way for cross-subject generalization for sEEG decoding.

Neural decoding from stereotactic EEG: accounting for electrode variability across subjects
Georgios Mentzelopoulos Evangelos Chatzipantazis' Ashwin G. Ramayya², Michelle J. Hedlund", Vivek Buch?, Kostas Daniiliiss. Konrad P. Kording'. and Flavia Vitale
University of Pennsylvania, Stanford University, Archimedes, Athena RC
Abstract
Deep learning based neural decoding from stereotactic electroencephallgraphy (sEEG) would likely benefit from scaling up both dataset and model size. To achieve this, combining data across multiple subjects is crucial. However, in sEEG cohorts, each subject has a variable number of electrodes placed at distinct locations in their brain, solely based on clinical needs. Such heterogeneity electrode number/placement poses a significant challenge for data integration, since there is no clear correspondence of the neural activity recorded at distinct sites between individuals. Here we introduce seegnificant: training framework and architecture that can be used to decode behavior across subjects using sEEG data. We tokenize the neural activity within electrodes using convolutions and extract long-tern temporal dependencies between tokens using self-attention in the time dimension. The 3D location of each electrode is then mixed with the tokens, followed by another self-attention in the electrode dimension to extract effective spatiotemporal neural representations. Subject-specific heads then used for downstream decoding tasks. Using this approach, we construct a multi-subject model trained on the combined data from 21 subjects performing behavioral task. We demonstrate that our model is able to decode the trial-wise response time of the subjects during the behavioral task solely from neural data. We also show that the neural representations learned by pretraining our model across individuals can be transferred in few-shot manner to new subjects. This work introduces a scalable approach towards sEEG data integration for multi-subject model training, paving the way for cross-subject generalization for sEEG decoding.
1 Introduction
Deep learning has revolutionized many fields ranging from natural language processing VVaswani al. 2017], computer vision (Dosovitskiy al. 2020), and neural decoding [Azabou al.) 2023
Advances in these fields have shown that performance and generalization benefit from scaling both datasets and model size, underscoring the importance of integrating data across multiple individuals. Recent work by Azabou et al, [203) showed that such approaches can achieve excellent generalization in the motor decoding domain, by combining invasive microelectrode array recordings from multiple non-humn non-human primates. Despite their promise, however, such approaches have not been used to build scalable decoders for human brain recordings collected with stereotactic electneenchehaalogrppyy (sEEG). Compared to other approaches (e.g., ECoG, microelectrodes). sEEG is minimally invasive and is currently the gold-standard clinical tool invasive electrophysiology humans. Therefore, building strong neural decoding models using sEEG promises to be medically relevant.
"Contact: gment@upenn.ccm edu, vinnler@@eammmmciiie......m Project page: http:///mmenzzziiiuubbbiiioogggo
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

There are several challenges associated with decoding from SEEG data. A major barrier is the hetero- geneity of the sEEG cohorts. SEEG subjects are most often patients suffering from pharmacoresistant epilepsy that are implanted with sEEG electrodes as part of their medical care. Fach subject gets implanted with a variable number of electrodes in specific locations in their brain, solely based on clinical needs [Kovac et al., 2017]. This heterogeneity makes it hard to identify any correspondence between the neural activity recorded across different subjects. This lack of correspondence poses significant challenge to combining data across subjects in meaningful way, resulting in most research groups resorting to within-subject models [Wu et al. 2024 Angrick et al., 202T] Petrosyan et al. 2022, Meng et al. 2021]. While somewhat effective, single-subject approaches neither scale nor generalize, suggesting a need to effectively integrate multi multi-suujectt data.
Here we introduce seegnificant: a scalable framework that can be used to decode behavior from sEEG recordings using sestios,ssion, multi-suhject data. Our approach uses a unified feature extraction backhone, designed to extract global behaviorally relevant neural representations that are shared across subjects. uses personalized task- tailored to the idiosyncrasies of individual participants, that can be used for downstream decoding tasks. To account for the heterogeneous electrode placement across individuals, we use a convolutional tokenizer that operates within electrodes individually. The tokenized latents are then processed by self-attention within the time dimension to capture long-tent temporal dependencies within electrodes. Using a novel positional encoding scheme, we then imprint the latents of each electrode with information about their 3D location in the brain, using their MNI coordinates, a standarized 3D coordinate system used to localize brain regions (Taliircch and Tournoux 1988 Collins et al. 1994]. The spatially aware latents then processed by another self-attention that operates within the electrode dimension to capture long-range dependencies across electrodes. The resulting latents, then compressed into neural representations that capture the spatiotemporal dependencies within and between electrodes that are used for behavioral decoding.
To evaluate our approach, we combine data from 21 subjects, across 29 recording sessions, that performed behavioral reaction time task, while their SEEG was simultaneously recorded, totaling more that 3600 behavioral trials and 100 electrode- hours) of sEEG recording. We demonstrate that using our approach, we are able to extract global neural representations, capable of decoding the response time of the participants during the behavioral task from neural data. Using leave-one-ous cross validation scheme, we also show that by pretraining model large amounts of data, can transfer the learned neural representations new subjects with minimal training examples. Our work introduces a novel framework designed to train models on multi-session, multi-subject sEEG data for behavioral decoding applications in scalable way.
(number of electrodes x number of recording
Our contributions can be summarized as:
A frumework for multi-subject truining based on sEEG. We present a novel approach to training transformer based models for neural decoding based on the combined data of multi-subject, multi-session sEEG datasets.
Pre-trained models for behavioral decoding based on sEEG. We trained a multi-session, multi-subject model for response time decoding based on sEEG that can be finetuned to new subjects. We will make the model and code publicly available as a resource for the community.
2 Related Work
2.1 Neural decoding using sEEG
Over the past decade, significant progress has been made in the field of neural decoding using sEEG, Angrick et al. [2021] demonstrated effective real-time speech synthesis based on sEEG using a linear discriminant analysis approach, and Petrosyan et al. [2022) showed effective speech decoding using a convolutional architecture. [Megg et al. [021] identified discriminative features for decoding speech perception and overt and imagined speech production from sEEG using non-deep learning model. Wu et al. [2022) showed that grasp force can be successfully decoded from sEEG signals using a CNN+RNN decoder and subsequently performed a comparative study where they explored the efficacy of using a variety of convolutional architectures to classify different movement based on the sEEG neural activity [Wu et al. 2023 2024). Unfortunately, all aforementioned works conduct
experiments on datasets containing only few subjects (12 or less, except Wu et al. 2023) and build

---

within subject models, with no clear vision towards generalization across subjects. Diverging from within-subjec approaches, in this work we combine data from many subjects build models capable of extracting global neural representations that generalize across subjects. Contrary to previous works which used CNNs and/or RNNs, we build our models using the transformer architecture, which has been shown to excel in capturing long-range dependencies when trained on large and diverse datasets.
2.2 Transformer architectures for neural data
Transformers have attracted lot of attention in the analysis of non-invasive neural signals. In the context of non-invasive electroencephalograply (EEG), multiple works use the attention mechanism for various decoding tasks such as motor-imagery visual decoding, and emotional recognition
2023 Liu et al. 2022 Lan et al. 2020]. Song et al. Tang et al. 2024 Kan et al., 2023 Li et al.
2023 achieved SOTA performance in visual decoding task using convolutional tokenizer that
operates within both time and electrode dimensions and is then processed using self self-attention. Using the same tokenization approach, Cui al. [2023] built neuro-GPT, a foundational model for non- invasive EEG based on transformers. However, non-invasive EEG uses a predefined number of electrodes placed at consistent locations of the scalp across subjects. This approach provides well-defined correspondence across the neural activity recorded between subjects. In sEEG, the number and placement of electrodes across subjects is highly variable and inconsistent between subjects, complicating efforts to translate approaches from EEG to SEEG data.
In the context of invasive microelectrode recordings, prior works have attempted to transformers for neural decoding with success. Ye and Pandarinath [202]] introduced the Neural Data Transformers alternative to RNN models GGlaser et al.] 2020
(NDT) model neural population activity
Sussillo et al. 2016 Le and Shlizerman [202]] extended the NDT model to process data in both the electrode and time dimensions. However, all aforementioned works trained single-subjects models.
2023 diverged from single subject approaches by combining data from multiple non-human primates and achieved excellent decoding performance in variety of different motor decoding tasks. However microclectrode recordings have a well-defined tokenization scheme based on extracellular neuronal spikes, which are detectable by mm-scale sEEG electrodes. Thus, is unclear how spike-based tokenization approaches can be meaningfully translated to SEEG.
Azabou et al.
2.3 Shared trunk architectures
Shared trunk architectures are a popular choice for multi task learning applications. They follow a simple outline: a global feature extractor, whose parameters are shared across tasks, followed by
task-specific branches, with parameters specific to each task [Crawshaw] 2020]_ They have been successfully used in multiple domains including facial recognition [Zhang et al. 2014), instance- aware semantic segmentation (Dai et al.) 2015), image retrieval (Zhao et al. 2018] and classification [Liu et al., 2018). Their effectiveness stems from their ability to extract shared representations that are common to all tasks. Combining data from multiple different tasks also helps mitigate the large scale data requirements for effective training of deep networks. In this work, while we are not dealing with multiple different tasks, we use shared trunk architecture to model inter-person differences between subjects in our cohort. Our network is composed of a backbone, common to all subjects, that extracts global representations and subject-specifie heads that tailor the model's output to the unique statistical profile of each subject.
3 Methodology
SEEG is neural recording modality capable of recording local field potentials (LFPs) that reflect the coordinated activation of hundreds of thousands of neurons in the vicinity of an electrode, providing mesoscale measurement of brain activity (Saez and Gu 2023 Due to the distributed nature of sEEG, subjects typically get implanted with tens to hundreds of electrodes in widespread locations across their brain. The resulting data multi-variate time series that typically epoched around behavioral stimulus of interest (e.g. a visual stimulus presented during each trial of behavioral task).
Neural activity recorded via SEEG across the electrodes of an individual are, of course, not indepen- dent. The captured recordings represent a conversation between thousands of neurons in distributed brain networks. The challenge is to find way to interpret those recordings in the context of this
3

---

broader conversation, instead of interpreting them in isolation [Azabou et al., 2023). This challenge exacerbated when trying to integrate neural recordings across different subjects, each of which has variable number of electrodes distributed across different brain regions. This is equivalent to monitoring the neural conversation with several microphones placed in different brain locations unique to each subject. For each monitored neural population, we do not know the identity or the functional tuning (what stimuli they respond to).
BEEG
Seognificant
Behavioral Prediction
E, www.wraanee www.holwwwwwwwwooooooo
Vector Embeddings
Subject Specific Regression Heads
Tomporal
CNN
MLP
MLP
MLP
B.
Neural Representations (F)
Electrode Timepoint
Token
Spatial Tokens
Temporal Tokens
(Electrodes)
Self Self-Attention (Time)
Attention
MLP
Positional Encoding
Time & Space Transformer Encoder Blocks
Figure 1: Outline of our network architecture. The sEEG signals converted vector embeddings through temporal convolutions and then processed by sequential self-attention operations in the time and electrode dimensions in an alternating fashion. The latents compressed and projected through subject-specific task heads to obtain behavioral predictions.
3.1 Signal processing
Motivated by the lack of correspondence between the neural activity recorded between subjects, we sought to identify the subset of electrodes across subjects that respond behavioral stimulus of interest (i.e., share the same functional tuning in the neural conversation) while subjects perform behavioral task. This can be achieved using the methodology introduced by Coon and Schalk (2016) and refined by Paraskevopoulou et al. [2021). This methodology identifies electrodes whose high- band activity (narrowband neural activity with frequency content between 70-150 Hz) is modulated by the behavioral stimulus of interest. We note that high-y band activity is used because it has been associated with the instantancous firing of neuronal populations the vicinity an electrode [Miller et al.) 2009) and, thus, it likely represents neural computation that is relevant the behavioral stimulus interest.
As described by Paraskevopoulou et al. 2021 separately for each electrode, the neural signals are filtered to the high-y band and the envelope extracted using the Hilbert transform. The high-y envelopes are then epoched around the behavioral stimulus of interest and separated into baseline period (short time window prior to stimulus) and task period (short time window after the stimulus). The median high-r envelope of the task (Medianta and baseline (M (Medianaasrtine) periods are computed across the trials of the behavioral task. Then for each electrode, the signal-to-noise ratio (SNR) is calculated
SNR 2 (Medinntakk)
where (-) refers to the variance of (.). Electrodes whose high-y band activity is significantly modulated by the behavioral stimulus can then be identified using bootstrap randomization test and subsequently be used for decoding applications.
The implementation details of this analysis can be found in Appendix A.I

---

3.2 Network architecture
3.2.1 Tokenization
To combat the inhomogeneity of the number/placement of electrodes across different individuals, we designed convolutional tokenizer that operates on the voltage traces of each electrode separately, Instead of performing a two-dimensional convolution across both the time and electrode dimension, which would blend information between the dissimilar electrodes that are randomly placed across individuals, we perform one- mee-dimnniionl convolution across the temporal dimension only. This way, each electrode gets assigned K- -dimensional learnable embedding for chunks of time, defined by the length of the convolutional kernel.
Specifically, let Te E RT denote the LFP recorded by electrode for given trial of length Trrial- Let k; denote the jMh convolutional kernel, from collection K kernels. Our approach returns learnalle vector embedding &e E RTeriat x by performing convolutions of the form,
Ze.t Ie ki, for i L....,
on xh (convolutional kernel weights are shared across the electrodes of all subjects). After temporal convolutions, batch normalization is applied to the K features extracted by the convolution followed by average pooling along the time dimension (which reduces the time dimension from Torsal to T), returning E R R
Stacking the results of these operations across electrodes, we obtain the latent E R where E denotes the number of electrodes, T is the number of time samples, and K is the number of features extracted by the convolutions. By representing our data in this way, our network capable of handling inputs with varying numbers of electrodes and time lengths. We do note, however, that the time resolution (i.e., sampling rate) needs be fixed for the convolutions to be meaningful across subjects.
3.2.2 Capturing long-range spatiotemporal dependencies
The latent E RExTxK semantically represents the cumulative neural activity of all electrodes of a subject, for given trial. It can also be conceptualized as a separable dual tokenization across the time and electrode dimensions, with electrode- lattedsselttetssseee Ze RT*K and time-latents 2 E F This separability, enables us to process the vector embeddings separately in the time and electrode dimension, giving us significant computational advantage to subsequent self-attention layers, whose computational complexity scales quadratically with sequence length.
Attention in the time dimension. We first interrogate data for long-term temporal dependencies within electrodes. To do so, for each electrode separately, we arrange the latents ze R into a sequence of tokens, where each token has dimensionality K. The tokens are then projected into equally shaped queries: Q = Wyzes keys: K Wg and values: V Wa and processed using
self-attention
Attention (Q, K, V) softmax QKT
Vd V
For this operation we used the standard transformer block Vaswani et al. 2017] preceding the attention operation with normalization layers and following with a feed-forward network. This operation is parallelized across electrodes, and we stack the results to obtain latents of the form Zint R
Spatial positional encoding. To account for the inconsistency of electrode locations across individu- als, there is need inject information about the spatial location of each electrode into model.
To do so, we used radial basis functions to capture the information about each electrode location in the brain based on its MNI coordinates [MacDonald al. 2000 Talairach and Tournoux 1988 We discretize the space of each coordinate, into n bins, with midpoints fo- P1,...; Pn-1- For each midpoint, centered univariate gaussians of the form,
1
(s 14:)2
(t)
exp
(4)
1 2m
20
5

---

with variances oj for i 1 and = 0,1..... .................... m m- Separately for the y, and coordinates, we compute fi, [t)(v). and (2) j. which we then concatenate into a vector E RR which represents the positional encoding of an electrode. The positional encodings for each electrode are then projected K dimensions (using linear layer) and added the latents Zint E -REXIXX X
Attention in the space dimension. Having captured temporal dependencies within electrodes in our data, and having "stamped" the latents with their spatial positional encodings, we then use self-attention again, this time in the electrode dimension capture long-range dependencies between the neural activity across electrodes in our data. The latents Zint E REXTxR are arranged, separately for each timepoint, into 2 RE*K
sequence of E tokens of dimension K, which are projected into equally shaped queries: Q = Waz,, keys: K = Wi and values: V = W The tokens are processed with self-attention using equation 3 Again, the standard transformer block is used, with appropriate pre-normalization followed by a feed-forward network. This time, the operation is parallelized across timepoints, and the results are stacked obtain latents of the form यो REXTXX x
Multiple layers of self-attention in time and self-attention in space can be stacked back-to-bace back to enhance the fitting ability of the network, if necessary. Leveraging the versatility of transformers to accept inputs of varying lengths, the attention operation in the both time and space, as described above, does not constrain our model to accept inputs of fixed length in terms of the number of electrodes, or timepoints. In the case where inputs do have unequal number of electrodes or timepoints, computations can be efficiently parallelized by masking tokens that correspond to padded electrodes and timepoints during the self-attention operations.
Extracting global features. In behavioral neuroscience, experimenters are usually interested in decoding few behavioral variables, such as a response time or hit rate to a given stimulus [Posner and Petersen 1990 Caaaao---oaaaaaaa et al.) 2014). With this in mind, the information contained in the latent 2 E x will ultimately need to be compressed into a single value. Therefore, following the self-attention operations, unroll the latents 2 and, using feed-forward network, we project them to a low dimensional neural representation F E R. where d denotes the final number of the extracted features, with d < E T K, that will be used for the downstream decoding tasks.
3.2.3 Personalizing to individual subjects
Our architecture is built around the idea of extracting neural representations that common across sEEG subjects. We also, however, need to take into account that performance to behavioral tasks is intrinsically different between individuals [Fozard et al., 1994 Davranche et al. 2006 Green and Bavelier 2003] Der and Deary (006). Therefore, we designed a separate task head for each individual, composed of shallow feed-forward network, that maps the extracted neural representations F to the behavioral outcome of a given trial.
Experiments
In this section, we test and validate our approach for multi-session across-subject behavioral decoding using sEEG on a large and diverse cohort of sEEG subjects performing a behavioral task.
4.1 Experimental setup
Dataset. The core design constraint of our approach is to enable the unified training of our architecture (see Fig. [T) on diverse, multi-session, multi-subject sEEG dataset despite the inhomogeneity of electrode number/placement in each subject. Towards that goal, we built a large and diverse dataset of subjects that performed repetitive reaction time task, while their sEEG was simultaneously recorded. Contrary to most studies, which collected few hundred behavioral trials from few subjects, we collected more than 3600 behavioral trials and more than 100 electrode- -ours of recording from 2/ subjects over 29 recording sessions. Our dataset is composed of 13 females and males, with ages ranging from 16 to 57 years old, with unique electrode nunber/placement for each subject, solely based on clinical needs. Across subjects electrodes span white and grey matter; cortical, subcortical, and deep structures (see Fig. 2

---

B
Stimulus
Color-Change
Response
Pre-trial
Delay
Response time
Left View
Top View
Right View
Figure 2: Overview of behavioral experiment. A. Schematic of the color change detection task. B. Electrode placement projected onto the MNI brain template for four example subjects in our cohort. Red dots show electrodes used for model training; grey dots show electrodes excluded from model training (see section 3.1). C. Electrodes used for model training across all subjects in our cohort projected on an MNI brain template.
Behavioral task. Subjects performed the color-change detection task (see Fig. A & Appendix A.1 In each trial, a visual stimulus was presented. After a variable foreperiod delay, stimulus changed color. At that time, the participant responded by pressing button as fast as they could. Our goal was to decode the trial-wise response time of the subjects using their sEEG.
Design choices. Throughout all experiments, we use the neural data from a window of [0, 1500] msec after the stimulus color change train models and make behavioral predictions. All models were trained with N₂ = 105 temporal tokens within each electrode and Ne E [3, 28) electrode tokens within each timepoint, which varied based on the number of electrodes for each participant. The dimensionality of each token was fixed to K 2. All training details are provided in Appendix A.2
4.2 Training within-subject models
To test whether modeling approach would be able to decode the trial-wise response time from sEEG data, we began with a lithin--ubject approach and tested whether our architecture was capable of decoding the response times for each subject. We trained separate model for each subject in cohort, combining data across sessions for participants that had multiple behavioral sessions. The experimental setup and the preprocessing of the sEEG data was kept the for all subject. Across the 2/ single-subject models, the average test set R2 was 0.30 + 0.05 (mean + sem).
4.3 Training multi-session, multi-subject model
To investigate whether training more data, despite the heterogeneity (see Fig. 2B. C). would improve response time decoding performance, we trained a unified model on the combined neural data of all the behavioral trials across all participants in our dataset. This effectively increased the number training samples 2old compared to the data available for the single-subject models.
This model achieved test set R² of 0.54 + 0.01 (mean + sem) on the combined data of all participants. The root mean square error (RMSE) for all predicted response times the test set, across subjects, was RMSE 82 = 2 (mean + sem) msec. For reference, the mean response time in the test set, across subjects, 410 (mean + sem) msec, which is order of magnitude greater than the RMSE. Since the response time profiles of each subject are distinct, we also evaluated model performance within-subjects. Across subjects, the averuge per-subject test set R2 was 0.39 + 0.05 (mewn + sem). Compared to training single-subjects, the multi-subject training approach boosted decoding performance for our proposed architecture by AR² 0.09, on average. We then compared the performance of the sunglc-subject models to that of the multi-subject model,
7

---

head to head (see Fig. BA). For the majority of subjects, there was a clear performance boost from multi-subject training.
We then explored whether fine- tuning the multi-session, multi- subject model to individual sub- jects would further boost the
Multi Subjjet
Finetuning (i29
Multi-Subject
performance gains, compare to
the sinle-subject models. The
finetuned models for each sub-
60
30
ject achieved an average test set
-32
R² of 0.41 = 0.05, across sub- jects, indicating that finetuning the multi-session, multi-subject model single subjects can fur- ther boost the performance gains, Importantly, finetuning the uni- fied model boosted the perfor- mance gains the multi-subject model compared the single-
-644
-22
00
0.
64
0.6
0.8
-04
-02
0.0 06
08
Sinjle-Subject (R²)
Single-Subject (R)
Figure 3: Comparing decoding performance between the single- subject and multi-subject models for each subject. A. Single- subject vs multi-subject model. Single-subject vs finetuned, multi-subject model. Circle size denotes the number of trials and color the number of electrodes (cyan to magenta represents ascending order)
subject models by R = 0.11,
average (sce Fig. 3B for pur-subject head to head comparisons). These results suggest that sEEG- based neural decoding benefits from multi-subjeet joint model training, despite the heterogeneity of electrode number/placement across subjects, demonstrating the power of multi-subject approaches compared to single-subject ones.
4.4 Transferring to new subjects
Having shown that training our model across diverse, multi-subject data boosts decoding performance, we were interested in identifying whether the representations learned by pretraining our models on multiple subjects can efficiently transferred new subjects, unseen from the model during training. This is important in any real-world clinical scenario, where only a few number of behavioral trials can be collected from subject. To test this, we employed a lcave-one-out cross validation approach. We trained 21 models, each of which was trained on the combined data of all subjects but one. The average test R² of the multi-subject models was 0.48 0.006 (mean + sem) across all trials of all subject. This indicates that our training framework is robust to data variations and that our model did not depend on data of any specific subject to train effectively.
The weights of the pretrained models were then used as the basis for finetuning on the data of each left-out participant, un-
Transfer (RR)
Single Subject Transfer
Single-Subject
seen from the models during training. Across subjects, the
models trained on other subjects
c.0
and transferred to a new one achieved average test set N² of 0.38 0.05 (mean + sem). This is clear improvement to training single-subject models from scratch and comparable to training multi-session, multi- subject models. Specifically, the transferred single-subject models
-64
-04
-02
02
0.4
0.5
0.4
-22
0.0
02
04
Sungle-Subject (R*)
Multi-Subject (R)
Figure 4: Comparing decoding performance between (A) Trans- ferred single-subject finetuned models single-subject models trained from scratch, and (B) Transferred single-subject finetuned models the multi-session, multi-subject model.
showed a per-subject decoding performance increase AR 0.08 compared to single-subjedt models trained from scratch (see Fig. 4TA) and a pur-subject decoding performance decrease AR -0.01, compared the multi-session, multi-subject model (Fig. 4)).
This suggests that pretraining models on diverse, multi-subject data and using their weights as finetune to transfer the learned representations to subjects is almost effective as training

---

the combined data of all subjects. Given the significant computational benefit of the transferring approach, its implementation in real world medical scenarios might be more likely.
4.5 Comparison with baselines
We were also interested in quantifying the performance benefit of our modeling approach compared to other traditional and state-of-the-art neural decoding approaches. To do so, trained single-subject baseline models (described in detail in A.3). and compared those with seegnificant. We observed that seegnificant outperformed the baseline models when trained on single subject, multiple subjects. and when transferred (see Fig. (5). Specifically, our single-subject models (section (42) outperformed all baselines by an average per-subject test set AR > 0.03. Our multi-subject models (section 4.3 outperformed the baselines by a per-subject test set A 2 0.12, which could be further increased to A RR 2 0.14 by finetuning single subjects. Importantly, our transfer-learned single-subject models (section 4. 4.4) also outperformed all baseline models by per-subject test set AR? 2 0.11, on average.
Model Performance (PP)
PCA Witeor
PCR Bidge
PCA XGB
MLP
GNN MLP
PCA Lasoo
Single-iurppe
Mutl Sufject
Motii Sudbect
Pineture
Single Subyer
Tranator
Baselines
Seegnificant (ours)
Figure 5: Decoding performance (mean = sem) various baselines and our proposed models.
4.6 Ablations
Last, we sought identify the contribution of each building block of our proposed architecture to the decoding performance of our models. To investigate this, we performed an ablation study on proposed architecture when trained on multiple subject (same in section 4.3 To ensure fair comparison, all training hyperparameters were kept identical across all training runs.
4.6.1 Model components
In this subsection, we trained our proposed model while ablating its components one by one. The model is composed of: 1. con- volutional tokenizer (CT), at- tention in time dimension (AT),
Performance (R9)
A......... a..
Multi-uulg Subject
3. spatial positional encoding (PE), 4. attention in space dimen- sion (AS), multi-layer percep- tron (MLP), 6. subject-specific regression heads (RH) (see also Fig. [. Since there is no straight- forward way to process the raw sEEG data with AT without first extracting features, we did not ablate CT. Additionally, since there is straightforward way to project the output of the AS to the regression heads without a linear layer, we also did not ablate the MLP.
Full
-AT
-PE
-AS
-RH
Variant with Alin (R')
Figure 6: Summary of ablation results. (A) Decoding performance
(mean + sem) across different model variants. (B) Head-to-head comparison of the decoding performance of our multi-session, multi-subject model its 2D attention variant.
The ablation study results are summarized in Fig. 6 A. Inspection of the results suggests that the
subject-specifie regression heads are key to the model's decoding performance (A R² = 18), likely because the response time profiles across subjects quite diverse. Ablating the attention mechanism

---

in the space dimension also substantially degrades decoding performance (AR² = 0.10) suggesting that it might be the main mechanism by which the model capable of dealing with the heterogenous number/placement of electrodes across subjects.
4.6.2 Model variants
We also compared our architecture with a close variant. Our architecture is designed with two separate attention mechanisms over the time and space dimensions of our data (see section 3.2.2). mainly to reduce computational burden since the computational complexity of attention scales quadratically with sequence length. To en- sure that this choice did not negatively affect the model's 'ss decoding performance, we compared our architecture against a variant where the two separate attention mechanisms are replaced by single 2D attention mechanism over the combined time and space dimensions of the data (to enable this, the positional encoding its 2D attention variant by a per-subject, test set AR² = 0.06, on average (see Table and Fig. 633) We also identified that our proposed architecture trained ~ 5.5x faster.
Table 1: Decoding performance (mean = sem) for the different models.
Model
Per-Subject R²
Train time (mins)
Variant
0.33 + 0.05
141.6 +: 3.23
Ours 0.39 + 0.05
25.7 + 0.03
ablated). We idenrified that our proposed architecture outperformed
5 Discussion
In this work, we introduce seegnificant: novel framework and architecture that can be used for multi- session, across subject decoding based on SEEG. We show that training on diverse and large cohorts of sEEG data, despite inter-subject differences in terms of electrode number/placement, is not only possible but also leads to better decoding performance compared to single-subject approaches. We also show that the neural representations learned by pretrained models can be efficiently transferred to new subjects, despite the small number of available trials rithin-subjeets (which is the case in any real-world clinical setting). In fact, this few-shot, transfer lcarning approach gives a clear benefit to training from scratch and can reach decoding performance levels that very close to those of multi-subject pretrained models, with a significant computational benefit.
Our results suggest that training on diverse multi-subject sEEG datasets boosts decoding performance compared to single-subject approaches. While our dataset is large compared to other works [Angrick] et al. [222 Petrosyan et al. [022]] Meng et al., 2021 Wu et al. 2022 2024), scaling up further will inevitably require combining data collected while subjects perform a variety of different behavioral tasks. Advances in other fields have shown that training multi-task data can improve performance and generalization [Azabou et al. 2023 Ruder et al. 2017). Therefore, investigating whether those ceteommanncegennalizaatiii
gains will hold sEEG decoding is worth exploring. Training multi-task datasets would also give insight as to which aspects of neural computations are shared across tasks and which are not. We leave the investigation of multi-task model training future work.
Given the clinical circumstances under which sEEG is recorded, performance to downstream decoding tasks would likely benefit from pretraining using self-supervised objectives. When patients are admitted in the epilepsy monitoring unit, their sEEG recordings are collected continuously over 5-10 days. In contrast, sEEG study participants perform only a few minutes of a behavioral task. Evidently, the amount of unsupervised data collected is orders of magnitude more than the supervised data. Strategies such as autoencoding, masked modeling, or next token prediction could enable training on the vasts amounts of data that are collected but unused by supervised learning approaches.
Overall, this work advances clinical and human neuroscience research by introducing a framework that enables multi-subject model training based on sEEG. We show that models pretrained on large and diverse sEEG datasets can easily transferred to new subjects and therefore can be used to accelerate progress in the neural decoding domain. Given that sEEG is currently the gold standard invasive neural recording modality used in humans, our work advances progress towards sEEG-based neural decoding therapeutic interventions, ultimately bringing them closer to clinical translation.
10

---

Acknowledgments and Disclosure of Funding
We would like thank the following funding sources for their generous support: National Institutes of Health grant ROINS121219 (FV); Office of Naval Research grant N00014-22-1-2677 (KD); National Institutes of Health grant 6732N5091006 (AGR); Onassis Foundation graduate student scholarship and A. G. Leventis Foundation graduate student scholarship (GM). Authors declare no competing interests.
References
M. Angrick, M. C. Ottenhoff, L. Diener, D. Ivucic, G. Ivucic. S. Goulis, J. Saal, A. J. Colon,
L. Wagner, D. J. Krusienski, L. Kubben, T. Schultz, and C. Herff. Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity. Communications Biology, 4(1, Sept. 2021. ISSN 2399-3642. doi: 10. 10.10395-4203-00--0377----- URL http
7/dx org.rrg////... 10. 1038/ 442003 021 02578-0
M. Azabou, V. Arora, V. Ganesh, X. Mao, S. Nachimuthu, M. J. Mendelson, B. Richards, M. G. Perich, G. Lajoie, and E. L. Dyer. unified, scalable framework for neural population decoding. 2023. URL/tttps: http///axxvvvvvaa orgoorggga abs ..tp.//......00.. 16046
Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: A practical and powerful approach multiple testing. Journal of the Royal Statistical Sociery: Series B (Methodological), 57(1):289-300, Jan. 1995. doi: 10. 1111111111 (1122277-666111995.0222222 10. 1111/j 101111/..2111--11 2517-6661 1995. tb02031.x
URL httpss/t:://do org/
V. P. Buch, J. M. Bernabei, G. Ng. A. G. Richardson, A. Ramayya, C. Brandon, J. Stiso, D. S. Bassett, and T. H. Lucas. "primed to perform:" dynamic white matter graph communicability may drive metastable network representations of enhanced preparatory cognitive control. Sept. 2022. doi: 111100/2222.09255505355
URL http: //tp://dx.. doi. org/ 10. 1 1101/20222.221 09. 25. 509351
F. Burke, K. A. Zaghloul, J. Jacobs, R. B. Williams, M. R. Sperling. A. D. Sharan, and M. Kahana. Synchronous and asynchronous theta and gamma activity during episodic memory formation. The Journal of Neuroscience, 33(1):2922 1:222300,4 Jan. 2013. ISSN 1529-2401. doi: 10. 1523/jnuroos.... 2057- 12.2013.
(ttp::tt///// best:::/ase //tx doi. org/ 10. 1523/ JNEUROSNE 2057-12.00 2013
D. L. Collins, P. Neelin, T. M. Peters, and A. C. Evans. Automatic 3d intersubject registration of mr volumetric data in standardized talairach space. Journal of computer assisted tomography, 18(2): 192- 205, 1994.
W. Coon and G. Schalk. A method to establish the spatiotemporal evolution of task-related cortical activity from electrocortccoraapiic signals in single trials. Journal of Neuroscience Methods, 271:76-85, Sept. 2016. ISSN 0165-0270. doi: 10. i0016//nccmeehhh21166600000
URLhttp
//dx wrg. 10. 1oee./..nntte.eemeeetttt 2016. 024
M. Crawshaw. Multi-task learning with deep neural networks: A survey, ArXiv, abs/2009.09796. 2020. httt:ttss::///at www.pppp.saaaaaacnanaaacaaaaamaa .rg/ Cttpg/CrrpusID... 221819295
W. Cui, W. Jeong, P. Thölke, T. Medani, K. Jerbi, A. A. Joshi, and R. M. Leahy. Neuro-gpt: Towards foundation model for eeg, 2023. http:ttts://arziit org abs/2311
J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3150-3158, 2015. URL[https http:////pppsemmmccccccchohaaaa org/ Slegg/CorpusID... 85116677
K. Davranche, B. Burle, M. Audiffren, and T. Hasbroucq. Physical exercise facilitates motor processes in simple reaction time performance: An electromyographhic analysis. Neuroscience Letters, 396(1):54-56, Mar. 2006. ISSN 0304-3940. doi: 10.10166f.ncuuee2200......0 URL http: //dx doi. org/10. 1016/j newlet........ee 2005. 11. 008
G. Der and I. Deary. Age and sex differences in reaction time adulthood: Results from the united kingdom health and lifestyle survey. Psychology and Aging, (1 Mar. 2006. ISSN 0882- 7974. doi: 10.1037/0882-7774.1... 62
httpttpp: doi 10170/000007000822777777

---

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. URL bttpstts:://aaa bttps://axxxi org/ abs/ 2010. 11929
J. L. Fozard, M. Vereruyssen, S. L. Reynolds, P. A. Hancock, and R. E. Quilter. Age differences and changes in reaction time: The baltimore longitudinal study of aging. Journal of Gerontology, P9(4))P179-P189, July 1994. ISSN 0022-1422 1422. doi: 00.1093/geronj/4444.1... URL http
//dx.doi.coo 10. 1093/ ger.:///eroo//4gg/ 4, p179
A. Galvao-Carmona, J. J. González-Rosa, A. R. Hidalgo-Muñoz, D. Páramo, M. L. Benítez, G. Izquierdo, and M. Vázquez-Marrufo Disentangling the attention network test: behavioral, event related potentials, and neural source analyses. Frontiers in Human Neuroscience, 8:813, h/wps:://wwww./ frontiersin
2014. ISSN 1662-5161. doi: 0ttt9//fnnum.2144008888.. org hrg/articee/10..o 338./3389//000 /ffffffnuummm 2014.00813 00813
I. Glaser, A. S. Benjamin, H. Chowdhury, M. G. Perich, E. Miller, and K. Kording. Machine learning for neural decoding. eneuro, 7(4::NNEUOO.0500 1aseo0005066111 1900066-9..0000 2020. July 2020. ISSN 2373-2822. doi: 10.1 1523//meuro.05001 19.2020.
URI. http::/dx.... doi org/10. 10. 1523/E22/E ENEURD 0506-
2020
C. S. Green and D. Bavelier. Action video game modifies visual selective attention. Nature, 423(6939) 537, May 2003. ISSN 1476-4687. doi: 10.1033/mature011647. URL http //dx. doix.doi..e wrg/10. 10w0//atttutu nat./ee6uure
H. Kan, Yu, Huang, Z. Liu, H. Wang, and H. Zhou. Self-supervised group meiosis contrastive learning for eeg-based emotion recognition. Applied Intelligence, 25220022255,02222222227222,,, Sept. 2023. ISSN 1573-7497. doi: 10.
/ /ttp://dxx.doe http://xx.oi.or//////.///
s10489-022 023---71---000-00-------------0-
S. Kovac, V. N. Vakharia, C. Scott, and B. Diehl. Invasive epilepsy surgery evaluation. Seizure, 13:12,-136, Jan. 2017. ISSN 1059-1311. //xx.doi.ogg/10
doi: 00016f/.eezurre221611..0000 10.016. URL http
10trg//10.. 10. 1016/j seizure...eeee...e.eeeee 2016.10.016
Y.TT. Lan, W. Liu, and B.-L. Lu. Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, July 2020. doi: 10. 101100/jcmm486055222000 20w.:/005..22222222222. URL
http://áx. doi. orgo 10. 1119/1100IIH/A 1199/IICC148605 2020. 9207625
Le and E. Shlizerman. Stndt: Modeling neural population activity with a spatiotemporal transformer, 2022. htttps::/arxiv.orrg..
04727
M. Leszczyński, A. Barczak, Kajikawa, Ulbert, A. Y. Falchier, I. Tal, S. Haegens, L. Melloni, R. T. Knight, and C. E. Schroeder. Dissociation of broadband high-frequency activity and neuronal firing in the neocortex. Science Advances, 6(33), Aug. 2020. ISSN 2375-2548. doi: 10.1126/ sciadv.abb0977. URL http://// doi. org/ 10. 1126/sc sss/sscaadss aociddv.abb9977
C. Li, Z. Zhang, X. Zhang, G. Huang, Y. Liu, and X. Chen. Eeg-based emotion recognition via transformer neural architecture search. IEEE Transactions on Industrial Informatics, 19
(4):6016-6025, Apr. 2023. ISSN 1941-0050. doi: 10.1109/ti.20022.317072 URL http! //dx.doi. www. 1109/T191TI 2022.3170422
S. Liu, H. Johns, and A. Davison. End-to-end multi-task learning with attention, 2018. URL https://arxiv wwwpxxiv.oorg///oo abs////b//// 100as//883.1111 10704
W. Liu, J.-L. Qiu, W.-L. Zheng, and B.-L. Lu. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Transactions on Cognitive and Developmental Systems. 14(2):715-729, June 2022. ISSN 2379-8939, doi: 10. 19..1009/ddd..22..3000
URL htRpht/p:///as doi wtt/////////// 1109 TCDS 2021.3071111
D. MacDonald, N. Kabani, D. Avis, and A. C. Evans. Automated 3-d extraction of inner and outer surfaces of cerebral cortex from mri. Neurolmage, 12(3):34-3566, Sept. 2000. ISSN 1053-811 19. doi: 10.1006/nimg.111 1999.09999900 0534. URL //dx: doi org/ / www.000//mmmmm.m.. 0999905344
12

---

K. Meng, D. B. Grayden, M. J. Cook, S. Vogrin, and F. Goodarzy. Identification of discriminative features for decoding overt and imagined speech using stereotactic electroencephalograhy In 2021 9th International Winter Conference Bruin-Computer Interfuce (BCI). IEEE, Feb. 2021. doi: 10. 1190/hhh551222.2221...99 93885555. URL ssttp://dx. www. www.o.oo......... 1ww111110010...0000 2021 9385355
K. Miller, Sorensen, G. Ojemann, and M. den Nijs. Power-law scaling in the brain surface electric potential. PLoS Computational Biology, 1000609, Dec. 2009. ISSN 1553-7358. doi: 10.1371/joumal.pchhi 1c0.0..009.. 1000609. http: doi 1000606
S. E. Paraskevopoulou, W. G. Coon, P. Brunner,
Miller. and G. Schalk. Within-subject reaction time variability: Role of cortical networks and underlying neurophysiological mechanisms. Neurolmage, 237:118127, Aug. 2021. ISSN 1053-8119. doi: 10. wwwp://.eeuoommae2222..... URL http://dss doi. org/ 10. 1016/] neuroimage... 2021 118127
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Köpf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL httth:tttttiiiiii wttpa/wwwww.gg//bb//11100000.... 01703
A. Petrosyan, A. Voskoboinikov, D. Sukhinin, A. Makarova, A. Skalnaya, N. Arkhipova, M. Sinkin, and A. Ossadtchi. Speech decoding from a small set of spatially segregated minimally invasive intracranial eeg electrodes with compact and interpretable neural network. Journal of Neural Engineering, 1966)666016,, Nov. 2022. ISSN 1741-2552. doi: 10.1088/1741-255222 http://dx doi. org/ 100.0.....8111 10010088111 0011/1111111112///100 acalel
URL
M. I. Posner and S. E. Petersen. The attention system of the human brain. 13(1):25 42, Mar. 1990. doi: hta.1/////eereee.1..31... 13. 23.000190.0m 0301900000335
http://merreeee....3331100.0..00 1t.....9.3311000022.... URL https: org/10. 1146/ /ee//nnnuvee...m .n
A. G. Ramayya, V. Buch, A. Richardson, T. Lucas, and J. I. Gold. Simple human response times are governed by dual anticipatory processes with distinct and distributed neural signatures. June 2022. doi: 10. 10.1101/2022.06 011.:222..0.1..492222 13.496029. URL hattp::/dxx: doi. org/ 10. 1111/20220002. 06. 13. 496029
S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard. Latent multi-task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4324-4331, 2017. URL https: //arxiiv org/ abs/ 0abbs//10550111
I. Saez and X. Gu. Invasive computational psychiatry. Biological Psychiatry, 93(6):661-670, Apr. 2023. ISSN 0006- 3223. doi: 10.1016/j wiwpsyyh..022.09..ooo http://///dxx.oo 1016/j .wwpsschh.......c.....ooo....... 2022.022222 09.0022
Y. Song, Q. Zheng, B. Liu, and X. Gao. Eeg conformer: Convolutional transformer for eeg decoding and visualization. JEEE Transactions on Neural Systems and Rehabilitation Engineering. 31: 710-719, 2023. ISSN 1558-0210. doi: 10. 101100//see2222.23300000 10. 1.SSE/.TRR........ 3230220
URL http://dx doi org/
J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, Feb. 2024. ISSN 0925-2312. doi: 10. 10.101 2ttcomm2022.1220665
WRRLhttp:/// //xx doi wttpd//....oooooo.ooooo// 1016/j www.om.mmmmmmmmm peeoucom.22........ 127063
D. Sussillo, R. Jozefowicz, L. F. Abbott, and C. Pandarinath. Lfads latent factor analysis via dynamical systems, 2016.
http:::///a/v////a org/
06315
J. Talairach and P. Tournoux. Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: An approach to cerebral imaging. Thieme Medical Publishers, 1988.
Tang, Z. Ma, K. Gan, Zhang, and Z. Yin. Hierarchical multimodallfusian of physiological signals for emotion recognition with scenario adaption and contrastive alignment. Information Fusion, 103:102122 Mar. 2024. ISSN 1566-2535. doi: 10. 10. 002 URL
http:// dx. doi orw.orgg//11 1016/j inffus 2023. 102129
13

---

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need, 2017. /tttps::://ari wttp://ww..gggb/////////// 1700.03766222
X. Wu, G. Li, S. Jiang, Wellington, Liu, Z. Wu, B. Metcalfe, L. Chen, and D. Zhang. Decoding continuous kinetic information of grasp from steroo-electreencephlllggggg
(seeg) recordings.
Journal of Neural Engineering, 19(2):026047, Apr. 2022. ISSN 1741-2552. doi: 10.1088/
1741-2552/ac65b1. URL http::/ddz doi org 21t1:/0.0000/11122222/22////3500 accbbbc
X. Wu, S. Jiang, G. Li, S. Liu, B. Metcalfe, L. Chen, and D. Zhang. Deep learning with convolutional neural networks for motor brain-computer interfaces based on stereoeellcctoencchhhaloggyyyy (seeg). IEEE Journal of Biomedical and Health Informatics, 27 May 2023. ISSN 2168-2208. doi: 3242262
URL httphttpp/dx dwi./xx.ioooooooooooooooooo.oo.. http:/......gg/..111////////// 2023
X. Wu, G. Li, X. Gao, B. Metcalfe, and D. Zhang. Channel selection for stereo- electroencephalogra- phy (seeg) beaee) invasive brain-computer interfaces using deep learning methods. IEEE Transac- tions on Neural Systems and Rehabilitation Engineering, 32:800 8110 2024. ISSN 1558-0210. doi: 10.1
URL bttp://x doi org/ 1109/TNSRE 2024 3364752,
J. Ye and C. Pandarinath. Representation learning for neural population activity with neural data transformers. Neurons, Behavior, Data analysis, and Theory, 5(3), Aug. 2021. ISSN 2690-2664. doi: 10.51628/001c.273888 URL http: http:://dxx dow::://xxdoooooo/oo/oooo//o. org/ 10. 51628/0011ee 27358
Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Facial Landmark Detection by Deep Multi-task Learning, page 94-108, Springer International Publishing, 2014. ISBN 9783319105994. doi: 1001/ 978-3-319-10599 10599-447. URL
X. Zhao, H. Li, Shen, X. Liang, and Y. Wu. A Modulation Module for Multi-task Learning with Applications in Image Retrieval, page 415-432. Springer International Publishing, 2018. ISBN 9783030012465. doi: 00.1007/978---000-012222
URL CRRLhttp::// //dx. /t//xxxxi..... or./10. 1007
14

---

A Appendix/ / supplemental material
A.1 Data collection and signal processing details
Experimental setup. Stereotactic EEG was recorded from 23 participants diagnosed with medically refractory epilepsy who underwent surgical implantation of intracranial electrodes for seizure local- ization. Patients were implanted with intraparencymal depth electrodes ("stereo EEG," " Ad-tech, 1.1 diameter, 4 contacts spaced 5 mm apart), except in one patient who also had subdural grid electrodes (Ad-tech, 4 mm contacts, spaced 10 mm apart). sEEG was recorded using Natus recording system. Based on the amplifier and the discretion of the clinical team, signals were sampled at either 512 or 1024 Hz. Clinical circumstances alone determined the number and placement of the implanted electrodes. To participate this study, participants provided written informed consent in accordance with the IRB of the University of Pennsylvanis vania and were compensated for their time.
a stimulus-detcttion task with a variable foreperiod delay [Buch et al., 2022 Ramayya et al. [022) while their SEEG activity was simultaneously recorded. Participants viewed visual stimuli on laptop and responded by pressing a button on game controller. The behavioral task is composed of sequences of trials described here. Each trial began with the presentation of a visual stimulus (small white box) at a randomized location on the screen as a fixation target (one of nine locations on a 3 x 3 grid). The stimulus changed color (from white to yellow) after one of two randomly selected foreperiod delays: (i) 500 ms, or (ii) 1,500 ms. The response time (time between color change and button press) for each trial was recorded.
Behavioral task. Participants performed the color-change-detection-te
Signal preprocessing. The SEEG signals were first converted to a bipolar montage by taking the difference between pairs of immediately adjacent contacts on the same electrode shank. The resulting bipolar signals were treated as new virtual electrodes (henceforth, electrodes) whose virtual location was the midpoint between the adjacent contacts [Ramayya al. 2022 Burke al. 2013 Within electrodes, we computed the line noise SNR, as the ratio of each electrode 's spectral power in the frequency range of 58-62 Hz over the spectral power in the range 18-22 Hz. Electrodes for which the SNR was greater than one were excluded from further analysis. Additionally, disconnected electrodes, defined as electrodes whose voltage had a standard deviation of zero were also excluded from further study. For the remaining electrodes, we investigated for noise contaminated trials defined as trials where (i) the voltage was not recorded due to saturation of the amplifier, or (ii) the mean + SD of the voltage of the trial was greater than 10 times the mean of all trials of that electrode. Trials that were determined as noise contaminated were then removed.
Signal processing details. To combat the heterogeneity of electrode placement between subjects, we identified the subset of electrodes across subjects that responded to the stimulus color change while participants performed the color-change-dee litanne-detcctio-taiee -tak. To do so, we followed the methodology used by |Paaaskevopoulouu al. (202T]. Specifically, we identified electrode locations where high-y band activity was significantly modulated by the stimulus color change.
We first filtered the data, separately for each electrode, to the high-y band (70-150 Hz) using 4th order Butterworth filter and subsequently extracted the high-y envelope using the Hilbert Transform. Then, we epoched the high-y envelopes around the color-change (l-500, 1 1500] ms) and separated each trial into a baseline period (I-500, 0J ms, before the color-change) and a task period ((O, 1500] ms, after the color-change). For each electrode, we computed the median high-y amplitude time course of the task (M ediintass)) and baseline (Medianbaselinn periods across trials. Then for each electrode, we calculated the signal-to-ne -o-ise ratio (SNR) as
02 (Meedaaraak) SNR (2 )
(5)
where o² (z) refers to the variance of x. To identify electrodes with statistically significant SNR values, we applied a bootstrap randomization test in which we randomly shuffled the task and baseline labels from all locations 10,000 times and computed one random SNR value for each such iteration. We then calculated p-values for each electrode location as the fraction of randomized SNR values that were larger than the computed SNR. We determined responsive electrodes as those with a p-value < 0.05, which corresponds to a confidence level a = 0.05. Prior to determining significance, we adjusted the p-values calculated across all electrodes of all subjects using false discovery rate [Bnnammini and Hochberg 1995).
15

---

The broadband voltage traces of all responsive electrodes were then z-scored (within electrodes), and downsampled to 400 Hz. Those electrodes were then used for decoding response times by our models. We note that using this procedure, we also rejected two participants from the study for whom only zero and one electrodes were returned as significant (lowering our participant count from 23 to 21). This electrode selection procedure lowered our computational cost, and because it merely uses high-y to select electrodes, it avoids problems of statistical double dipping.
A.2 Model training details
Model complexity. The total number of trainable parameters for the multi-session, multi-subject model is 797,095. From those, 753,394 are shared across subjects (shared trunk) and the rest 43,70) 701 parameters are subject-specific (parameters of the regression heads with 2,081 parameters per subject).
Training hyperparameters. All models were implemented and trained using Pytorch 2. 2.1.0+cu121
(Paszke al. 2019). AdamW was used as the optimizer [Leszczyński et al. 2020] (with b1 = 0.5 and b2 = 0.999). All models were trained for 1000 epochs. A step learning rate scheduler was used with initial learning rate set to 10 3 and decayed by a factor of 0.5 every 200 epochs for single-subject models and by a factor of 0.9 every 100 epochs for multi-subject models. Batch size was fixed to 64 and 1024 for all single-subject and multi-subject models, respectively. All models were optimized using Huber loss, except for when finetuning the multi-session, multi-subject model (see section 4.3) to individual subjects, where MSE loss used.
Hyperparameters of the spatial positional encoding. For all models, the gaussian kernels used in the spatial positional encoding were centered at μ E (-90,- 70, -5,, .0,.,....,,...... 50, 70} and had variances o² E {1, ,,,,,....64,, 64).
Controlling for random data splits. For all models, the train/validationnee split was 70/15/15 %. To ensure that our results would not be biased in any way by the data splits, experiments performed in sections 4.2
4.6.2 A.4.2 A...3. were run using 5 different data splits and the results were averaged across the different splits. For experiments described in section 4.4 multi-subject models were trained for data split (since training 21 multi-session, multi-subject models x 5 times for the different data split would be very computationally expensive). Finetuning to single-subjects was performed across 5 different data splits and the results across the splits were averaged. For the ablation study described in section 4.6. I models were trained for 3 data splits, to reduce computational burden.
4.3 4.5
Finetuning, For finetuning multi-session, multi-subject models to single subjects in section 4.3 all model weights were updated throughout the single-subipp finetuning. For transferring multi-session. multi-subject models to single subjects in section 4.4] since the regression heads of the left-out participant were completely untrained, we gradually unfroze model weights. For the first 400 training epochs, only the regression head (2,081 parameters per subject) of the left-out participant was trained (the remaining 753,394 model parameters were frozen). For the remaining 600 epochs, all model parameters were trained. All training parameters were identical to those described in paragraph "Training hyperparameters" above.
Computational resources. All models were trained on a machine with an AMD EPYC 7502P 32-Core Processor and 1 Nvidia A40 GPU with 44.99 GiB of memory. Single-subject models trained on average within 5 mins and multi-subject models trained within an hour. The memory requirement to train the multi-session, multi-subject model with a batch size of 1024 was GiB.
A.3 Baseline model details
Non deep learning models. The following non-deep learning models were trained on single-subjects, using scikit-learn (version 1.2.2): 1. Wiener Filter, 2. Ridge Regression, 3. Lasso Regression, 4. Gradient-Boosting (XGB). All models were used as part of a pipeline composed of: standard scaling + PCA - model. For Ridge and Lasso Regression, we optimized over the hyperparameter alpha {0.1, 1, 10). For XGB, we optimized over the hyperparameter estimators E (50, 100, 200). All other hyperparameters were kept at their default values.
Deep learning models. The following deep learning models were trained on single-subjects, using the training procedure described in section A.2 (same as that used for our proposed architecture): 1.
MLP (4 layers), 2. CNN MLP (1 + layers, respectively).
16

---

A.4 Additional experiments
A.4.1 Model real-time applicability
To understand whether our multi-session, multi-subjecc model (see section 4.3 could be used in real time, which is essential for any real-world neural decoding system, measured our model's inference time on two machines: a commercial laptop and a server. The results, summarized in Table
2 show that our model runs in 10 msec both machines, on either CPU or GPU. This indicates that our proposed architecture can be easily used in real time, since real-time systems run on the order of 100 msec.
Table 2: Model inference time on different hardware. Units are in msec.
Machine
CPU
GPU
AMD EPYC 7502P + Nvidia A40
9.1
5.1
Intel Core i9 + Nvidia A2000
4.0
7.9
A.4.2 Contribution of spatial positional encoding
Following our ablation study (see section 4.6.1). J. we sought to better understand the contribution of our spatial positional encoding to the model's performance. Contrary to other positional encoding techniques, constructed to encode the order of tokens a sequence (see Vaswani et al. [2017] for
Fourier and (Su et al. 2024) for rotation based approaches), our approach encodes tokens with their 3D location the brain, based on each electrode's MNI coordinates.
To quantify the performance gains, we trained variant of our multi-session, multi-subject model, which was iden-
tical to that described in section 4.3 with the positional
0.6
encoding ablated. We observed that ablating the the po-
With Positional Encoding (RP)
sitional encoding leads to per-subject decoding perfor-
mance decrease of A A -0.02, on average. We show within-subject head to head comparison of the perfor-
mance of the two models in Fig.
-02-
We also investigated whether the decoding performance of the model with and without positional encoding is signif- icantly different. Specifically, we performed a Wilcoxon rank-sum test between the groups: (1) test set R² scores of all subjects obtained by training the multi multi--ubject model with spatial positional encoding, and (2) test set R2 scores of all subjects obtained by training the multi-subject model without spatial positional encoding. The test returned test-statistic = 0.34 and p-value = 0.73, indicating that there no significant difference between the performance of the model with and without spatial positional encoding.
-04
-04
-0.2
0.0
0.2
0.4
0.6
0.8
Without Positional Encoding (R")
Figure 7: Per subject decoding perfor- mance comparison between the model with and without spatial positional coding.
Those results, while not significant, suggests that informing electrode-tokens with their 3D location in the brain prior to self-attention in the space dimension might slightly benefit decoding performance. Intuitively, this would makes sense, since the functional computations happening within roughly the same brain regions are similar across humans.
A.4.3 Comparison of our proposed spatial positional encoding against other approaches.
The results of section A.4.2 suggest that the decoding performance gains introduced by the positional encoding scheme used in this work are only modest. Therefore, we explored whether other positional encoding schemes would boost decoding performance further. To do so, we trained variants of our proposed architecture on the combined data of all subjects (same training scheme as the one used in section 4.3 with the positional encoding replaced by the: 1. positional encoding scheme introduced
17

---

by Vaswani et al. [2017]. 2. a variant of the positional encoding scheme used in this work where RBFs are replaced by fourier functions. While the explored schemes are far from comprehensive. we identified that our proposed positional encoding scheme performed as well as or better than other approaches (see 3). However, the decoding performance gains provided by all positional encodings explored in this work are modest. Therefore, it would be worth exploring other positional encoding schemes, perhaps based on whole brain MRI images and/or brain atlases other than MNI Talairach and Tournoux, 1988).
Table 3: Model performance using different positional encoding schemes.
Positinal Encoding
Per-Subject R
Vaswani et al. [0017
0.16 + 0.04
MNI-Fouride
0.39 = 0.05
MNI-RBF (Ours)
0.39 + 0.05
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: To the best of our knowledge, all claims made in the abstract and introduction are supported by our experiments described in section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of proposed approach in sections 3 and
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
No theoretical results are introduced in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the methodology introduced in this work in section The details to reproduce the behavioral experiment used this work are provided in section A. The details of how to train the models introduced in this work are provided section A.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Our code available at http:///iiibbb.oooommmmeee//sggs The dataset used in this work contains electrophysiological data collected from human patients. To respect their privacy and comply with HIPAA regulations, we cannot make the dataset public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sublic/guidess httpe//guiiideeeeeeuubbeeeeee
(CodeSubmissioooool11
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cilips.cc/publle httpe//wwlccggguddeeeeeeeeeeeeee
http:/ddeeeeoeeeSuumaassiisooloo ssinioolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We refer the reader to section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All figures with barplots include error bars and the factors of variability capture by them are included in captions of the figure. One statistical test (corrected appropriately for multiple comparisons) was used in this work, described in section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We refer the reader to A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ate::/nneurispp http.c/ccbllc/tthcccccc...eeedad
Answer: [Yes]
Justification: To the best of our knowledge, this manuscript conforms in every aspect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer and
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data used in this work are not shared to protect the privacy of the human participants whose clectrophysiologice data was used in this study. To the best of our knowledge, the code shared with this work does not pose a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide proper citations for all the works upon which this work is built.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All details of the code are discused in section A.2 Code is publicly available
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification:
We do not have access the full text of instructions given participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Approval was obtained by the IRB of the University of Pennsylvania prior to data collection.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "AKMC6D3ggf", "reviewer": "Reviewer_VKBz", "summary": "The paper presents a novel approach to sEEG decoding. Authors highlight the benefit of using data from various subjects for training. However, due to the nature of the sEEG technique, collection of such data is difficult. Authors provide a new deep learning based decoding approach which utilizes spatial positional encoding (to provide a model with information on electrodes’ locations), temporal and spatial attention, MLP and subject-specific regression heads. Then, authors train their model in multiple frameworks and highlight the efficacy of multi-subject training.", "questions": "1. How many trainable and frozen parameters there were during training and finetuning of the model?\n2. How large is the variance in the response time within each subject?\n3. Are all K convolution kernels in Convolution tokenizer the same for all electrodes?\n4. How much recording hours of data was used overall and on average per subject (only the electrode-hours are provided)?\n5. What is the statistical significance of the metrics increase while using positional encoding compared to not using it?", "limitations": "Yes, authors provide a good description of the limitations and address potential topics for future research (larger datasets, self-supervised pre-training).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper highlights a novel approach to sEEG decoding which enables training of their model on multiple subjects. Generalization across subjects is overall a very important and difficult task to achieve in many modalities and applications of neuroimaging. Authors present a method which has a value in itself and provides ideas for future research in this direction. A paper clearly presents the ideas and performed experiments. Visualizations help to understand both the data collection process and the deep learning architecture.", "weaknesses": "1. No comparison with other methods during within-subject training was done. Authors only provide the results for their own architecture for within-subject experiments. Therefore, they only show that their multi-subject trained model is stronger compared to their single-subject model. However, it might be possible that some of the other existing single-subject State-of-the-Art models will be more effective than authors’ multi-subject model on authors’ data. The importance of these comparisons are magnified by the fact that authors’ dataset will stay private (therefore only the authors of this paper can provide metrics for other models applied to this dataset); and by the reasonably large size of authors’ model (it might be too big to be efficiently trained on a single person data, while other models, which were developed for single-subject tasks, might be lighter and train better on small datasets).\n2. Decoding performance with and without spatial positional encoding seems very similar. For such a difference it is interesting to see if it has a statistical significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and suggestions. We are excited to read that our \"method has a value in itself and provides ideas for future research\"! Here, we provide our response to your comments/questions and the results from new experiments that we ran to address your concerns. We hope that our answers will clear any lingering concerns. We are also available to provide further clarification, if needed.\n\n## Weaknesses\n\n> 1. No comparison with other methods during within-subject training was done. Authors only provide the results for their own architecture for within-subject experiments. Therefore, they only show that their multi-subject trained model is stronger compared to their single-subject model. However, it might be possible that some of the other existing single-subject State-of-the-Art models will be more effective than authors’ multi-subject model on authors’ data. The importance of these comparisons are magnified by the fact that authors’ dataset will stay private (therefore only the authors of this paper can provide metrics for other models applied to this dataset); and by the reasonably large size of authors’ model (it might be too big to be efficiently trained on a single person data, while other models, which were developed for single-subject tasks, might be lighter and train better on small datasets). \n    \nWe thank the reviewer for this constructive criticism. To address this, we trained classic and state-of-the-art single-subject models to compare against our model. The results are summarized in Table 1 of our General Response (see also Figure 1 of the attached pdf) and suggest that our architecture outperforms the baselines when trained on either single-subject or multi-subject data. Specifically, our single-subject models outperformed all baselines by $\\Delta R^{2} >= 0.03$, while our multi-subject models outperformed all baseline by $\\Delta R^{2} >= 0.12$. Importantly, our transfer learned single-subject models also beat the baselines by $\\Delta R^{2} >= 0.10$. We hope that these comparisons are sufficient to address your concerns. We plan to include this analysis in section 4 of our revised manuscript.\n\n> 2. Decoding performance with and without spatial positional encoding seems very similar. For such a difference it is interesting to see if it has a statistical significance. \n\nTo identify whether the difference in the decoding performance with and without spatial positional encoding is statistically significant, we performed a Wilcoxon rank-sum test between the groups: (1) $R^{2}$ of all subjects obtained by training the multi-subject model with spatial positional encoding, and (2) $R^{2}$ of all subjects obtained by training the multi-subject model without spatial positional encoding. The test returned a test-statistic = 0.34 and a p-value = 0.73, indicating that there is no significant difference between the performance of the model with and without spatial positional encoding. We plan to include this result in section 4.5 of our revised manuscript.\n    \n## Questions\n\n> 1. How many trainable and frozen parameters there were during training and finetuning of the model? \n\nThe total number of trainable parameters for the multi-subject model is 797,095. From those parameters 753,394 are shared across subjects and the rest 43,701 parameters are subject-specific (2,081 parameters per subject). When training single-subject models (section 4.2) and multi-session, multi-subject models (section 4.3), all parameters of the model (shared + subject specific) were trained. When transferring the pretrained, multi-subject models to new subjects (section 4.4), for the first 400 epochs the shared parameters were kept frozen (753,394 parameters) while the subject-specific parameters were being trained (2,081 parameters per subject). For the remaining 600 epochs, all parameters were unfrozen and trained. We plan to include those numbers in section A.2 of our revised manuscript.\n\n> 2. How large is the variance in the response time within each subject? \n\nFor convenience, the mean of the variance of the response times across all 21 subjects of our dataset is $\\sigma^{2} = 0.011 \\pm 0.0017$ (mean $\\pm$ sem) sec$^{2}$. The variance of the response times for each subject is:  (18, 9, 5, 6, 7, 3, 4, 5, 17, 25, 4, 9, 9, 13, 21, 19, 8, 31, 4, 11, 5) $\\times 10^{-3}$ sec$^{2}$. \n    \n> 3. Are all K convolution kernels in Convolution tokenizer the same for all electrodes? \n\nYes. The K convolutional kernels that our tokenizer uses are the same for all electrodes of all subjects. This design choice increases the effective number of training samples available to the tokenizer, making it less prone to overfitting and increasing the model's robustness. We will make sure to ephasize this in section 3.2.1 of our revised manuscript.\n\n> 4. How much recording hours of data was used overall and on average per subject (only the electrode-hours are provided)? \n\nThe total recording hours of data used during model training were 1.54 hrs. The average per-subject recording time that was used for model training was 4.39 $\\pm$ 0.39 (mean $\\pm$ sem) mins. We will make sure to add those is section A.2 of our manuscript.\n    \n> 5. What is the statistical significance of the metrics increase while using positional encoding compared to not using it? \n\nThe increase in the performance due to spatial positional encoding compared to not using it is not statistically significant. Please refer to our response to your comment under _Weaknesses (bullet point No. 2)_ for a detailed description of how we obtained this result.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qljr3mTFYI", "reviewer": "Reviewer_pdN2", "summary": "This paper presents a framework and architecture for decoding behavior across subjects using stereotactic electroencephalography (sEEG) data, addressing the challenge of electrode variability. By tokenizing neural activity with convolutions and employing self-attention mechanisms along with a positional encoding scheme based on MNI coordinates, the model extracts effective spatiotemporal neural representations. The study demonstrates successful decoding of behavioral response times from 21 subjects' data and shows that the pretrained neural representations can be transferred to new subjects with minimal training data. This work offers a scalable approach for integrating and decoding sEEG data across multiple subjects.", "questions": "1. Scale and diversity of the dataset: The paper mentions using data from 21 subjects, totaling 3600 behavioral trials and 100 hours of sEEG recordings. Is this amount of data sufficient to represent the neural activity patterns of all subjects? Could the diversity of the dataset be insufficient to support the model's generalization capabilities?\n\n2.  Effectiveness of spatial positional encoding: The paper proposes a spatial positional encoding method based on MNI coordinates to handle electrode placement variability across subjects. Has the effectiveness of this method been thoroughly validated? Are there comparative experiments showing that this encoding method is superior to other possible encoding methods?\n\n3. Individualized decoder heads: The paper mentions that each subject has a personalized task head for downstream decoding tasks. How effective is this approach when dealing with new subjects? Are there detailed experimental results demonstrating the performance of this method on new subjects?\n\n4. Computational complexity and scalability: The paper outlines a complex model architecture involving convolutional tokenization, self-attention in both time and electrode dimensions, and individualized regression heads. What are the computational requirements for training and running this model? Is the approach scalable to larger datasets or real-time applications?", "limitations": "Plz go and check questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Multi-Subject Generalization: The framework's ability to generalize across subjects by combining data from multiple individuals and training a unified model is a substantial step forward compared to traditional single-subject approaches.\n\n2. Methodology: The detailed methodology, including signal preprocessing, and bootstrap randomization test for identifying significant electrodes, ensures the robustness of the proposed approach.\n\n3. Clarity: The paper is well-organized and clearly written, making it accessible to readers from both neuroscience and machine learning backgrounds", "weaknesses": "Plz go and check questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your though provoking feedback. We appreciate reading that \"combining data from multiple individuals and training a unified model is a substantial step forward compared to traditional single-subject approaches\"! Here, we provide our response to your questions and the results from new experiments that we ran to address your concerns. We are available to provide further clarification, if needed.\n\n> 1. Scale and diversity of the dataset: The paper mentions using data from 21 subjects, totaling 3600 behavioral trials and 100 hours of sEEG recordings. Is this amount of data sufficient to represent the neural activity patterns of all subjects? Could the diversity of the dataset be insufficient to support the model's generalization capabilities? \n    \nThis is a very thoughtful concern. It is likely that the amount of data in our dataset is insufficient to represent the neural activity patterns of all humans in a global sense. However, our dataset is larger than that of other recent sEEG datasets (Angrick (2021), Petrosyan (2022), Meng (2021), Wu (2022), Wu (2024) have data from 12 subjects or less), suggesting that it is more likely to represent general neural activity patterns compared to most other sEEG datasets.\n\nIn terms of our dataset's diversity: (1) our cohort is composed of 13 females and 8 males, (2) subjects' age ranges from 16 to 57 years old, and (3) electrode number & placement for each subject is unique (solely based on clinical needs). Across subjects electrodes span white and grey matter; cortical, subcortical, and deep structures; and every major structural region of the brain (see Figure 2 of our General Response pdf). Our dataset is very diverse, suggesting that our model, trained on the combined data of all subjects, is likely to generalize. \n\nThanks for the food for thought, we will include this discussion in our revised manuscript.\n\n> 2. Effectiveness of spatial positional encoding: The paper proposes a spatial positional encoding method based on MNI coordinates to handle electrode placement variability across subjects. Has the effectiveness of this method been thoroughly validated? Are there comparative experiments showing that this encoding method is superior to other possible encoding methods?\n\nThanks for pointing out this limitation. In response to this concern, we validated our spatial positional encoding against other positional encodings. The results are summarized in Table 2 of our General Response and suggest that our proposed positional encoding, based on MNI coordinates, works as well as or better than other approaches.\n\nWe believe it is also important to note that the MNI-Fourier and MNI-RBF positional encoding schemes outperform other approaches, indicating that informing models with sEEG electrode location boosts their performance. However, the performance gains are small, indicating that there is still a lot of margin for improving spatial positional encoding mechanisms. We will make sure to discuss these findings in section 4.5 of our revised manuscript.\n\n> 3. Individualized decoder heads: The paper mentions that each subject has a personalized task head for downstream decoding tasks. How effective is this approach when dealing with new subjects? Are there detailed experimental results demonstrating the performance of this method on new subjects?\n\nThis is a great question. We echo that the ability of dealing with new subjects is very important to support our model's generalization capabilities. \n\nIn section 4.4 of our submitted manuscript we demonstrate that our approach is effective when dealing with new subjects. Specifically, in that section, we take a leave-one-out cross validation approach in which we train 21 models (one for each subject in our cohort) each of which is trained on the combined data of all subjects except one. Then, we use the weights of the pre-trained models as a basis of finetune, upon which we train a new single-subject model with the data of the left-out subject. Across subjects, the  models trained on other subjects and finetuned to a new one achieved an average test set $R^{2}$ score of $0.37 \\pm 0.06$ (mean $\\pm$ sem). Importantly, the performance of those models is superior to that of models trained from scratch and outperforms all baseline models (see Table 1 of our General Response) by $\\Delta R^{2} >= 0.10$.\n\n> 4. Computational complexity and scalability: The paper outlines a complex model architecture involving convolutional tokenization, self-attention in both time and electrode dimensions, and individualized regression heads. What are the computational requirements for training and running this model? Is the approach scalable to larger datasets or real-time applications?\n\nThe computational resources used to train our model are provided in lines 617-619 of our submitted manuscript. Single- and multi-subject models train within 5 mins and 1 hour, respectively, on a machine with an AMD EPYC 7502P Processor and one Nvidia A40 GPU, which is well within the resources of most computational research labs. The memory requirement to train the multi-subject model with a batch size of 1024 is $\\sim$ 8 GiB, making model training tractable using less powerful GPUs as well. \n\nIn terms of scalability, assuming that the batch size remains fixed, irrespective of the dataset size, the aforementioned computational resources should be enough to train the model. Naturally, the time requirement for training would scale linearly with the number available trials. However, even a dataset of $\\sim$ 80K trials would train in roughly 24 hrs, which is very manageable considering the dataset size.\n\nTo investigate whether our model can be used in real time, we measured the model's inference time on a sever and a laptop (see Table 3 of our General Response). All inference times are on the order of a few msec, which easily allows for integrating our model with real-time systems. We will make sure to add this result to our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lmjsuPXf7u", "reviewer": "Reviewer_o6UF", "summary": "The authors propose a novel training framework and architecture to predict response time for a color change behavior task from stereotactic electroencephalography (sEEG) data, focusing on integrating data across multiple subjects despite the variability in electrode placement and count. The model tokenizes neural activity within electrodes using convolutions, extracts temporal dependencies with self-attention, incorporates electrode location with a positional encoding scheme followed by another spatial self-attention layer, and a subject-specific prediction layer. The model is trained on data from 21 subjects, using different procedures: single-subject training (baseline), multi-subject training, and multi-subject training plus single-subject finetuning. The proposed method demonstrates an improved ability to decode trial-wise response times compared to the baseline, and transferability of learned representations to new subjects.", "questions": "1. With the number of electrodes differing across subjects, how did you handle the different input sizes?\n2. Instead of plotting all the sEEG channels, it would be helpful to show the selected channels actually used in training.\n3. What is the (relative) root mean square error for the prediction?", "limitations": "The method could provide valuable insights for various BCI studies if the authors can demonstrate its performance on more complex tasks. In the Discussion, the authors mention that developing a multi-task model is planned for future work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* **Innovative Framework**: The proposed framework effectively addresses the heterogeneity across subjects, a significant challenge in sEEG data processing, by using electrode placement and subject-specific prediction layers. The ability to pretrain the model on a larger multi-subject dataset and fine-tune it for new subjects with minimal training is a valuable feature, enhancing the model's practicality and applicability.\n* **Comprehensive Evaluation**: The model's performance is validated using a substantial dataset (21 subjects), showing consistent improvement in performance for most subjects. An ablation study is also performed.", "weaknesses": "The paper discusses the impact of the positional decoding in the ablation study, but this is the least impactful structure compared to other layers (Fig 6). It would be beneficial to explore other position decoding mechanisms besides using MNI locations for future work, perhaps based on different brain atlases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and question. We were really excited to read that \"The proposed framework effectively addresses the heterogeneity across subjects, a significant challenge in sEEG data processing\". Here, we provide our response to your questions and the results from new experiments that we ran to address your concerns. If you would like any further clarifications, please let us know.\n\n## Weaknesses\n\n> The paper discusses the impact of the positional decoding in the ablation study, but this is the least impactful structure compared to other layers (Fig 6). It would be beneficial to explore other position decoding mechanisms besides using MNI locations for future work, perhaps based on different brain atlases.\n\nThank you for the insightful suggestion! We echo your observation that our proposed spatial positional encoding only improves decoding performance slightly and agree that future work should identify better ways to encode the location of sEEG electrodes in the brain. Using different brain atlases is a great suggestion that we will definitely look into. Unfortunately, for this dataset, we only have the MNI coordinates of the sEEG electrodes and therefore we are not able to test whether encoding position with some other atlases would achieve better decoding results. However, we certainly look forward to exploring such ideas in the future.\n\nThe best we could do to adress your feedback was to compare our spatial positional encoding approach against other positional encoding approaches. The results are summarized in Table 2 of our General Response and suggest that our proposed positional encoding, works as well as, or better than other approaches. However, as you pointed out, the explored positional encoding schemes are far from comprehensive, indicating that there is still a lot of margin for improving spatial positional encoding mechanisms. We will make sure to discuss the idea of encoding electrode location using other brain atlases in section 4.5 of our revised manuscript.\n\n## Questions\n\n> 1. With the number of electrodes differing across subjects, how did you handle the different input sizes?\n\nThis is a great question. Our models is capable of handling different input sizes due to the ability of convolutional neural networks and transformers to accept inputs of varying lengths. Prior to procesing the latents with our architecture's MLP, we pad the latents of all subjects to a fixed length. For the mathematical details, we refer you to section 3.2 of our manuscript. Here, we try to provide a more intuitive explanation of why our network is capable of handling inputs of different sizes.\n\nThe input to our network, $X \\in R^{E \\times T}$, is a 2D array of electrodes $\\times$ timepoints, where $E$ refers to the electrode dimension and $T$ to the time dimension. The tokenizer performs $K$ 1-D convolutions on the input along the time dimension $T$, while parallelizing the computation across the $E$ dimension (electrodes). This operation returns a latent $z \\in R^{E \\times T \\times K}$, where $K$ refers to the number of convolutional kernels. Then, self-attention is performed on $z$ along the time dimension, while computation is parallelized across the $E$ dimension. Self-attention can accept sequences of arbitrary lengths and therefore no matter the number of timepoints of the latent $z$, this operation can be performed successfully. The output of the self-attention has the same size as the input, returning a latent of the form $z \\in R^{E \\times T \\times K}$. Then positional encodings are added to the latents, which does not alter the shape of $z$. Another self-attention operation is performed, this time along the $E$ dimension of $z$, parallelized across the $T$ dimension (timepoints), which again preserves the size of the latent $z \\in R^{E \\times T \\times K}$. At this point, the latents are unrolled to get a latent of the form $z \\in R^{E \\cdot T \\cdot K}$. $T$ and $K$ have the same dimensionality across subjects; $E$ however (the number of electrodes) does not. Therefore, each subject is going to have a latent $z \\in R^{E \\cdot T \\cdot K}$ that has a unique dimensionality. This latent will be projected to a lower dimensional space through a multi-layer perceptron, which requires a fixed input length. Therefore, at this stage the latents $z \\in R^{E \\cdot T \\cdot K}$ are padded with zeros to obtain a fixed length $E_{max} \\cdot T \\cdot K$ that is common to all subjects. For convenience, $E_{max}$ is set to the maximum number of electrodes that a subject has in our cohort. The MLP projects the latent to $z \\in R^{F}$, where $F$ represents a small number of features that will then be projected through the subject-specific MLPs to a single number corresponding to the response time of a subject for a given trial. We hope that this explanation answers your question. \n\n> 2. Instead of plotting all the sEEG channels, it would be helpful to show the selected channels actually used in training.\n\nThis is a great suggestion. Please take a look at Figure 2 of the pdf we submitted with our General Response and let us know if you have any further suggestions. We will appreciate your input on this.\n\n> 3. What is the (relative) root mean square error for the prediction?\n\nFor the multi-subject model, the root mean square error for all predicted response times in the test set, across subjects is RMSE = 0.082 $\\pm$ 0.002 (mean $\\pm$ sem) sec. For reference, the mean response time in the test set across subjects is $0.41$ $\\pm$ 0.005 sec (mean $\\pm$ sem). We will make sure to include this in section 4.3 of our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rzcEsicu6i", "reviewer": "Reviewer_j3GS", "summary": "The work proposes a transformer architecture for decoding behavior using sEEG neural recording which can account for inter-subject variability that exist due to different placements of sEEG electrodes, different SNRs, inherent biological differences etc. The ability of the proposed transformer architecture to deal with inter-subject variability allows the transformer architecture to leverage a much larger multi-subject dataset as compare to existing works which rely on subject-specific models. The proposed architecture consists of six main components: (i) a CNN temporal tokenizer; (ii) A temporal attention mechanism; (iii) A hand-designed RBF-based spatial encoding of different electrode locations; (iv) A spatial attention mechanism; (iv) a feed-forward MLP to extract small dimension neural representations; and (iv) a multi-head regression layer to deal with inherent patient-to-patient variability. The authors test their proposed architecture on a dataset collected from 21 subjects performing a visual color-change task, and show that their proposed architecture significantly boosts its decoding performance when using multi-subject datasets as compared to single-subject dataset.", "questions": "1. Line 230: What is C? Should it be E?\n2. Section 3.2.2: Spatial Positional Encoding: A couple of questions: \n- I am assuming that an MRI for each subject in the study was available, from which the MNI coordinates of the sEEG electrodes were calculated. In the spirit of being fully data-driven, why not use the whole MRI (with the sEEG shanks) for encoding the spatial latents? A simple CNN or Graph Convolutional Net could be used to directly encode the spatial information into a latent which can be used instead of  the hand-designed coordinate-based spatial feature used in this work. A minor issue might be that the spatial latents might also contain information about the electrodes discarded in the preprocessing but that can possibly be remedied by appropriate masking or removing the sEEG coordinated from the MRI while pre-processing (whichever is easier). A possible advantage of directly using the MRI data would be that it would also account for differences in the brain structure along with the difference in sEEG probe placement.\n- I do not understand the point of using $m$-different 1-D RBFs for modeling the positional encoding. Is the goal to model positional encodings at different spatial scales. Why is information at different spatial scales relevant? How are the different $\\sigma_j$ chosen (are they learnt from the data or chosen as hyperparameters)? The choice seems somewhat arbitrary, especially since the gain in the model's performance by using the spatial encoding is marginal (a 0.02 increase in R$^2$) as reported by the authors in Section 4.3.       \n- Is the rationale for using 1-D RBFs, i.e., the RBF for each coordinate (eq (4)) instead of using 3-D RBFs (e.g., something like  $C\\exp(-D((x-\\mu_x)^2+(y-\\mu_y)^2+(z-\\mu_z)^2$)) is to also incorporate the information of the direction of the coordinate. \n- Lines 205-207: The authors state that the projected positional encodings are added to the latents. I am having trouble fully understanding what does this operation entail? Does this mean that the projected positional encodings, denoted by $p\\in\\mathbb{R}^K$, and the latents, denoted as $z_{int}^{3} = z_{int}[i,j,:]$ (using the pythonic notation) for some $0 \\leq i \\leq E-1$ and $0 \\leq j \\leq T-1$, are added to produce the output $o\\in\\mathbb{R}^{E\\times T\\times K}$ of the operation in the following manner: $o^{3}=z_{int}^3+p$ followed by appropriate stacking. If that is the case, then what are the relative magnitudes of $z_{int}^{3}$ and $p$? Since $p$ is calculated using the 1-D RBFs, depending upon the magnitude of $\\sigma_j$'s, the corresponding projected $p$ could be really numerically small, and the corresponding output $o^3=z_{int}^3+p$ might be dominated by the term $z_{int}^3$. I think checking the relative magnitude of  $z_{int}^3$ and $p$, and ensuring that they roughly have comparable magnitudes is important to ensure that the positional encoding impact the final outcome. \n3. Why is Huber loss used for training multi-subject model whereas MSE loss used for training single-subject loss? Ideally both multi-subject and single-subject models should have been trained using both losses and for each model, the Huber or MSE loss should have been chosen based on validation error. The choice is surprising as I would have expected to use the more robust Huber loss on the single-subject model (which would be more noisy and prone to effects of outliers) and MSE on the multi-subject data where the much larger size of the dataset can regularize the training process.\n4. Another point of potential unfair comparison is that single-subject models are trained for much smaller number of total (gradient) descent steps compared to multi-subject models. On average, the multi-subject models have 21x larger dataset compared to a single-subject model. Hence, a single epoch while training multi-subject models will go through 21 x more gradient descent steps compared to an epoch of a single-subject model. So, multi-subject model trains for 21x longer than the single-subject model. I think authors should provide a justification for this large discrepancy in the training periods of multi-subject and single-subject models. For example, try training the multi-subject model for 1000/20 = 50 epochs and see its $R^2$ coefficient or try training a couple of best-performing single-subject model and train them for 21000 epochs and see their performance. **Edit**: This comment is no longer applicable. I missed the part about different batch-sizes.", "limitations": "Yes, the authors provide a discussion on some limitations in the discussion section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The paper addresses an important issue of data heterogeneity in sEEG recordings among participants which makes constructing large dataset challenging as data from multiple participants cannot be easily pooled together. This particularly affects the applicability of deep learning models as they typically rely on large datasets to get low generalization error.\n- The proposed architecture is clearly defined and uses a multi-head regression layer with a shared trunk (similar to some multi-task architecture) to model inter-patient variability. The authors also employ spatial and temporal attention layers to model temporal dependence in the neural activity and the spatial dependence between different electrodes, respectively. and  \n- The experimental details are clearly provided and the results shown in the work clearly show that training transformers on multi-subject dataset significantly boosts their decoding performance. \n- The ablation study is insightful and provides important insights into the functioning of different components of the proposed architecture.", "weaknesses": "1. A main weakness of this paper is the spatial encoding (which is framed as a main contribution of the work) only seems to marginally improve the performance of the transformer architecture. The results shown in Section 4.5 only show that average decrease of 0.02 in $R^2$ coefficient across the subjects which is well within the standard error of 0.05 (line 279) reported by authors for the average $R^2$ coefficient with the transformer model incorporating spatial encoding. I am surprised why the authors do not use this result to conclude that the proposed spatial encoding does not have a significant effect on model's accuracy and, consequently, the heterogeneity in the *actual* positions of sEEG electrode does not seem to impede the training of multi-subject models. Looking at authors' result of ablation study in A.3.1, it seems more plausible that attention mechanism for space (without which the $R^2$ coefficient decreases by a more significant amount of ~0.1, figure 6) is the main mechanism by which the proposed architecture deals with electrode heterogeneity.  \n\n2. Furthermore, the ablation study detailed in A.3.1 seems to suggest that it is the multi-head regression component of the proposed architecture is the main component enabling the better performance of multi-subject model. I am surprised as to why this observation is not mentioned in the main manuscript. \n\n3. Due to the marginal performance boost offered by positional encoding, the argument that heterogeneity of electrode placement needs to be explicitly accounted for by tokenizing space and time separately seems weak (lines 161-164). Consequently, the separation of time and space attention mechanism does not seem to be adequately justified. I think comparison against 2-D attention mechanism will benefit the study.  \n \n4. Comparison against other methodology is limited. Authors only compare their architecture on single-subject dataset with multi-subject dataset and show a significant increase in the performance of their transformer architecture when training with the larger multi-subject data. Naively training transformer architecture without heavy regularization on small single subject dataset seems to provide poor performance as already discussed in Ye & Pandarinath'21. Hence, the gains of using multi-subject dataset reported in this study might be artificially larger than the ones obtained with proper regularization (such as dropout, $L_2,$ and $L_1$ regularization) of the transformer architecture. Furthermore, authors should also use some classical decoding models such as (PCA with linear regression, Unscented Kalman Filters, see Xu et al. JNE'22) to give a sense of how much gain is obtained in the first place by using the more complicated transformers architecture and provide a baseline accuracy for comparison. Authors should also compare against existing techniques such as SEEG-Net (Wang et al. Computers in Biology and Medicine'22) which used domain generalization techniques for dealing with heterogeneous sEEG data distribution among different subjects. As a side note, I think the work would benefit from a discussion on multi-task literature, as the multi-head regression structure is quite similar to many multi-task architecture proposed earlier (shared trunk architecture, see the review Crawshaw'20 on arvix).\n\nMinor:\n- Line 210: \"long-*range* dependency\" might be more appropriate than \"long-*term* dependency\" as the authors are talking about variables representing space.\n- Please provide the test/train/validation splits used in this work. It would facilitate in understanding the reported test $R^2$ coefficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a very detailed review. Here, we provide our response to your questions. Due to space constraints, for each question we provide only the beginning of your prompt. We apologize for the inconvenience.\n\n_Throughout this rebuttal we refer to single-subject as SS and multi-subject as MS._\n\n## Weaknesses\n\n> 1. A main weakness of this paper is the spatial encoding...\n\nThanks for the constructive criticism. Indeed, our proposed PE only marginally boosts decoding performance, suggesting the attention mechanism in space is the main mechanism by which our model deals with electrode heterogeneity. We will highlight this in our revised manuscript.\n\n> 2. Furthermore, the ablation study detailed in A.3.1...\n\nThis is a valid point. We will highlight this in revisions.\n\n> 3. Due to the marginal performance boost offered by positional encoding...\n\nThanks for suggesting this. We compared the performance of our architecture against a variant where 2D attention is performed over time & space. We include the results of this analysis in Table 4 of our General Response (see also Figure 1E of the pdf). The results suggest that our proposed architecture outperforms its 2D attention variant by $\\Delta R^{2} = 0.06$. \n\n> 4. Comparison against other methodology is limited...\n\nThanks for the constructive criticism. To address this, we compared our model with classic and SOTA SS models. The results, shown in Table 1 of our General Response, indicate that our architecture outperforms all baselines across the board.\n\nUnfortunately, UKF and SEEG-Net are missing from those baselines. We have been unable to get good results using UKF, likely due to the nature of our data, since the trial response times (included in UKF's state) are not stationary, making state estimation unreliable. We also looked into sEEG-Net, but the code is not public and the method requires substantial modification before use, since it operates on the electrode level while our problems requires operation on the subject level. We are still trying to make those models work and hope to include them as baselines in revisions. \n\nThank you for suggesting we discuss multi-task literature in revisions. We plan to do so, including Crawshaw (2020), in the related work section of our manuscript. \n\n## Minor\n> * Line 210: \"long-range dependency\" might be more appropriate...\n\nWe will amend this in revisions.\n\n> * Please provide the test/train/validation splits...\n\nThe training/validation/test split is 70/15/15. We will include this in revisions.\n\n## Questions\n\n> 1. Line 230: What is C? Should it be E?\n\nYes. We will fix the typo in revisions.\n\n> * I am assuming that an MRI for each subject in the study was available...\n\nThis is a great idea. Unfortunately, for this dataset we only have the MNI coordinates of electrodes and not whole MRIs or other imaging data. Therefore, we are unable to test this out. We look forward to exploring this option in the future.\n\n> * I do not understand the point of using m-different 1-D RBFs for modeling the positional encoding...\n\nYour intuition is correct; we used RBFs with different variances to encode position at different spatial scales. This is relevant because studies (Meunier (2009)) show that the brain has a hierarchical modular structure. Therefore, encoding both short and long electrode connections could be useful. The scales $\\sigma_{j}$ for each RBF were hyperparameters with values: [1, 2, 4, ..., 64]. \n\n> * Is the rationale for using 1-D RBFs, i.e., the RBF for each coordinate...\n\nWe used 1D RBFs instead of 3D because 1D RBFs: (1) inform the model with direction (as you correctly identified) and (2) introduce less parameters to the model compared to 3D RBFs. Specifically, assuming that sampling an interval in 1D requires m points, sampling a 3D interval, separately in each dimension, at the same scale requires 3m points. Sampling the same 3D interval in 3 dimensions combined requires m$^{3}$ points. \n\n> * Lines 205-207: The authors state that the projected positional encodings are added to the latents...\n\nThank you for this insightful suggestion. Your description of the operation correct. To ensure that $z^{3}$ & $p$ have comparable magnitudes, we computed $||z^{3}||$ and $||p||$ for 512 random samples from our test set. The norms were $||z^{3}|| = 0.790 \\pm 0.53$ & $||p|| = 0.013 \\pm 0.01$ (mean + sdev), suggesting that $||z^{3}||$ is greater than $||p||$ but not to the extent that the model's output would be unaffected by this operation.\n\n> 3. Why is Huber loss used for training multi-subject model whereas MSE loss used for training single-subject...\n\nWe optimized SS models using MSE Loss because $R^{2}$ is directly proportional to the MSE Loss. \nFollowing your suggestion we trained both SS & MS models using both MSE & Huber Loss. The results show that training with Huber Loss boosts performance for all models. We have update the SS model results to reflect this. \n\n_Table 5. Comparison of model performance trained with Huber vs MSE Loss_\nModel | MSE | Huber\n--- | --- | ---\nSS | 0.28 $\\pm$ 0.05 | 0.30 $\\pm$ 0.05\nMS | 0.35 $\\pm$ 0.05 | 0.39 $\\pm$ 0.05\n\n> 4. Another point of potential unfair comparison...\n\nThank you for this thoughtful concern. Your intuition is correct. However, the MS and SS models were trained with batch size 1024 and 64, respectively (see line 599 of manuscript). Our dataset contains 3685 trials (175 trials per subject). Therefore, during 1K training epochs MS and SS models underwent 4K and 3K gradient descent steps (GDS), respectively. \n\nTo ensure that this did not artificially inflate the performance difference between the MS and SS models, we retrained all SS models for 1.5K epochs (4.5K GDS). The average per-subject test set R$^2$ of the SS models was 0.29 $\\pm$ 0.05 (mean $\\pm$ sem). The MS model still outperformed those by $\\Delta R^{2} = 0.10$. \n\nIn practice, we used 1K epochs because we observed that they were enough for the training objective of all model to converge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. The reviewers pointed out that our unified multi-session, multi-subject modeling approach "is a substantial step forward compared to traditional single-subject approaches" (pdN2) that "provides ideas for future research" (VKBz). 

Some highlights from the reviewers:
* Method: "the paper highlights a novel approach to sEEG decoding which enables training of their model on multiple subjects" (VKBz), " The detailed methodology [...] ensures the robustness of the proposed approach" (pdN2), "The ability to pretrain the model on a larger multi-subject dataset and fine-tune it for new subjects with minimal training is a valuable feature, enhancing the model's practicality and applicability" (o6UF)
    
* Impact: "Authors present a method which has a value in itself and provides ideas for future research in this direction" (VKBz), "The paper addresses an important issue of data heterogeneity in sEEG recordings among participants which makes constructing large dataset challenging" (j3GS), "The proposed framework effectively addresses the heterogeneity across subjects, a significant challenge in sEEG data processing" (o6UF), "Generalization across subjects is overall a very important and difficult task to achieve" (VKBz)

* Experiments \& Evaluation: "The model's performance is validated using a substantial dataset (21 subjects), showing consistent improvement in performance for most subjects" (o6UF), "The experimental details are clearly provided" (j3GS), "The ablation study is insightful" (j3GS)

* Writing \& Presentation: "The paper is well-organized and clearly written, making it accessible to readers from both neuroscience and machine learning backgrounds" (pdN2), "well-written and easy to follow" (j3GS),"paper clearly presents the ideas and performed experiments" (VKBz)

Based upon reviewer comments, we ran a number of new experiments, and are currently working on the following revisions to our manuscript:

* _Comparisons with single-subject baselines_. As suggested by reviewers j3GS & VKBz, we compared the performance of our model against other traditional and state of-the-art models (see Table 1). The results indicate that our model outperforms other approaches across the board. When trained on single-subjects, our model outperforms the baselines by $\Delta R^{2} >= 0.03$, while when trained on mutli-subject data, our model beats the baselines by $\Delta R^{2} >= 0.12$. Importantly, the single-subject models obtained by finetuning multi-subject models to new subjects also beat the baselines by $\Delta R^{2} >= 0.10$. Those results demonstrate the power of multi-subject approaches compared to single-subject ones. 

_Table 1. Comparison of our model's performance against baselines. Results report mean $\pm$ sem._
Model | $R^{2}$
--- | ---
PCA + Wiener Filter | 0.13 $\pm$ 0.18
PCA + L$_{2}$-Regression | 0.17 $\pm$ 0.14
PCA + XGBoost | 0.17 $\pm$ 0.06
MLP | 0.23 $\pm$ 0.06
CNN + MLP | 0.27 $\pm$ 0.07
PCA + L$_{1}$-Regression | 0.27 $\pm$ 0.04
Ours (Single-Subject) | 0.30 $\pm$ 0.05
Ours (Multi-Subject) | 0.39 $\pm$ 0.05
Ours (Multi-Subject + Finetune) | 0.41 $\pm$ 0.05
Ours (Single-Subject + Finetune) | 0.37 $\pm$ 0.06


* _Comparison of other positional encodings (PEs) against ours_: All reviewers pointed out that model performance only benefits slightly by our spatial PE. To address this, we trained variants of our architecture using different PEs and compared them against ours (see Table 2). The results suggest that our proposed PE performs as well as, or better than other approaches. However, we echo the reviewers' concern that the gains are modest. Therefore, we plan to include these results along with a discussion of other possible PEs, based on whole brain MRIs (j3GS) or based on atlases other than MNI (o6UF), in our revised manuscript. 

_Table 2. Comparison of different PE schemes. Results report the mean $\pm$ sem._
PE Type | $R^{2}$
--- | ---
Vaswani (2017) | 0.16 $\pm$ 0.04
MNI-Fourier | 0.39 $\pm$ 0.05
No PE | 0.37 $\pm$ 0.05
MNI-RBF (ours)  | 0.39 $\pm$ 0.05


* _Real-Time applicability_. Reviewer pdN2 suggested we check whether our model can be used in real time. To test this, we measured our model's inference time on two machines (see Table 3). The results show that our model can be used in real time, since its inference time is on the order of msec, while real time systems run on the order of 100 msec. 

_Table 3. Inference time on different hardware._
Machine | CPU | GPU | Units
-- | -- | -- | --
AMD EPYC 7502P + Nvidia A40 | 9.1 | 5.1 | msec 
Inter Core i9 + Nvidia A2000 | 4.0 | 7.9 | msec 

* _Separate attention in space & time vs 2-D attention over space & time_. Reviewer j3GS suggested we check whether our model's decoding performance would benefit by employing one 2-D self-attention mechanism over the time & space dimensions of our data, instead of two separate self-attention mechanisms over the aforementioned dimensions. We identified that our method outperforms the 2-D attention variant by $\Delta R^{2} = 0.06$ (see Table 4). Our proposed architecture trains faster, too.

_Table 4. Comparison of our approach vs the 2-D attention variant. Results report the mean $\pm$ sem._
Model Variant | $R^{2}$ score | Training Time
--- | --- | ---
Combined attn in time & space | 0.33 $\pm$ 0.05 | 141.6 $\pm$ 3.23 mins 
Separate attn in time & space (Ours) | 0.39 $\pm$ 0.05 | 25.7 $\pm$ 0.03 mins 

**Impact of this work**: This work is the first to train a unified, multi-session, multi-subject models for neural decoding based on sEEG. We demonstrate our approach on a very diverse dataset, with data from 21 subjects whose electrodes are heterogenously placed in brain locations that are unique to each subject. We show that pretrained multi-subject model can be transferred to new subjects, demonstrating their practicality and applicability. This work highlights the power of multi-subject approaches for neural decoding based on sEEG.

Meta-review: This manuscript proposes a novel architecture for decoding stereotactic electroencephalography (sEEG) signal and to predict response time in a cognitive task. The main innovation is to leverage a much larger multi-subject data, in contrast to existing works based on single-subject models, by dealing with the inter-individual variability (e.g. location of electrodes, different SNR, biological differences). The learning strategy can be summarized in three main components: (i) tokenization of the neural signal using convolutions to extract temporal dependencies with self-attention, (ii) the extraction of effective spatio-temporal neural representations by a positional encoding scheme based on MNI coordinates and a mechanism of spatial attention to select the most informative electrodes, (iii) a subject-specific prediction layer based on multi-head regression layer to deal with inherent patient-to-patient variability. An empirical evaluation on a dataset collected from 21 subjects performing a visual color-change task, shows that the proposed architecture significantly boosts the decoding performance when using multi-subject data as compared to single-subject data.

The Authors were really responsive to the criticism of the Reviewers and devoted a meaningful effort to take the opportunity to improve their work. The rebuttal was very effective in improving the submitted manuscript with several additional analyses.

Two major concerns didn’t find an answer in the discussion: (i) the limited size of the dataset, and (ii) the negligible effect of spatial encoding scheme. While the first limitation is inherent to the domain (scaling up with sEEG is hard and this is the reason of shortage of larger open dataset), the failure of dealing with spatial inter-individual differences reveals that a major challenge in across individual decoding is still open. 
For this reason this work better deserve a poster presentation.
Decision: Accept (poster)