BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity
Authors: Lucine L Oganesian
Saba Hashemi
Maryam M. Shanechi
Venue: NeurIPS 2025 poster ()
Keywords: self supervised learning
masked modeling
tokenization
transformers
neural decoding
Primary area: neuroscience_and_cognitive_science

Abstract: Intracranial recordings have opened a unique opportunity to simultaneously measure activity across multiregional networks in the human brain. Recent works have focused on developing transformer-based neurofoundation models of such recordings that can generalize across subjects and datasets. However, these recordings exhibit highly complex spatiotemporal interactions across diverse spatial scales, from the single-channel scale to the scale of brain regions. As such, there remain critical open questions regarding how best to encode spatial information and how to design self-supervision tasks that enable the learning of brain network patterns and enhance downstream decoding performance using such high-dimensional, multiregional recordings. To allow for exploring these questions, we propose a new spatiotemporal transformer model of multiregional neural activity and a corresponding self-supervised masked latent reconstruction task, designed to enable flexibility in the spatial scale used for token encoding and masking. Applying this model on publicly available multiregional intracranial electrophysiology (iEEG) data, we demonstrate that adjusting the spatial scale for both token encoding and masked reconstruction significantly impacts downstream decoding. Further, we find that spatial encoding at larger scales than channel-level encoding, which is commonly used in existing iEEG transformer models, improves downstream decoding performance. Finally, we demonstrate that our method allows for region-level token encoding while also maintaining accurate channel-level neural reconstruction. Taken together, our modeling framework enables exploration of the spatial scales used for token encoding and masking, reveals their importance towards self-supervised pretraining of neurofoundation models of multiregional human brain activity, and enhances downstream decoding performance.

BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity
Lucine L. Oganesian" Saba Hashemit2 Maryam M. Shanechi 1-4 University of Southern California, Los Angeles, CA (loganesi, saba. hashemi, shanechi)@usc..ooo
Abstract
Intracranial recordings have opened a unique opportunity to simultaneously mea- sure activity across multiregional networks in the human brain. Recent works have focused on developing mansformer-basee neurofoundation models of such record- ings that can generalize across subjects and datasets. However, these recordings exhibit highly complex spatiotemporal interactions across diverse spatial scales, from the single- channel scale to the scale of brain regions. As such, there remain critical open questions regarding how best to encode spatial information and how design self-supervision tasks that enable the learning of brain network patterns and enhance downstream decoding performance using such high-dimensional, multiregional recordings. To allow for exploring these questions, propose new spatiotemporal transformer model of multiregional neural activity and a cor- responding self-supervised masked latent reconstruction task, designed enable flexibility in the spatial scale used for token encoding and masking. Applying this model publicly available multiregional intracranial electrophysiology (iEEG data, we demonstrate that adjusting the spatial scale for both token encoding and masked reconstruction significantly impacts downstream decoding. Further, we find that spatial encoding at larger scales than thannel-leve encoding, which is commonly used in existing iEEG transformer models, improves downstream decod- ing performance. Finally, we demonstrate that our method allows for region-level token encoding while also maintaining accurate channel-level neural reconstruc- tion. Taken together, our modeling framework enables exploration of the spatial scales used for token encoding and masking, reveals their importance towards self-supervised pretraining of neurofoundation models of multiregional human brain activity, and enhances downstream decoding performance.
Introduction
Intracranial electroencephalograayy (iEEG) provides direct window into the human brain by abling the simultaneous recording of high-dimensional neural activity across multiple brain regions, thus measuring diverse spatial scales from single channels large-scale brain networks. Enabling the modeling of such recordings can provide unique opportunity to study functional brain networks associated with complex behavioral and cognitive processes [+ and develop translational technolo- gies such as brain-computer interfaces [ 6). Compared with non-invasive approaches such as fMRI
Corresponding author: shanechi @ ussc.edu.
Equal contribution.
Miing Hsich Department of Electrical and Computer Engineering, University Southern California
"Thomas Lord Department of Computer Science, University of Southern California
"Alfred F. Mann Department Biomedical Engineering. University of Southern California
"Neuroscience Graduate Program, University of Southern Californa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

or scalp EEG, iEEG yields a more direct measurement of brain activity with rich temporal dynamics. Furthermore, while intracortical recordings of spiking activity typically focus on measuring neuronal populations within local brain circuit or region (e.g., motor cortex), iEEG data is typically collected from aparsely-placed electrodes across much larger spatial scales of several brain regions at once. As such, modeling of iEEG presents distinct challenges due to its complex spatiotemporal structure. Towards the goal of learning rich spatiotemporal representations for iEEG activity, there has been keen interest in developing iEEG neurofoundation models that can generalize across different subjects and datasets, paralleling recent efforts for spiking data [7-10], non-invasive EEG [Л], and fMRI [12 13.. To do so, recent works have leveraged large transformer- based models, often pretrained with suservuuervis,,, to learn rich representations of human iEEG data with demonstrated efficacy in downstream tasks and cross-subject generalization [9) Despite the progress that has been made, there still remain critical open questions on how best to incorporate spatial information when designing and training such models.
First, while prior works have largely used standard positional encoding methods for providing temporal information to the transformer model (e.g., sine-cosine, rotary, learnable), there still exists no unified approach for encoding space during neural tokenization here defined as the process of transforming continuous neural recordings into finite-dimensional input tokens for the transformer encoder. Previous approaches have either not encoded space [14), collapsed it across prespecified channels chosen based neuroscientific knowledge [17), encoded space but at the scale of single channels [16] 18 19]. As such, developing models that enable a larger than chamnel-level spatial scale for token encoding and studying the effect of such larger-scale encoding remains unexplored. Second, it is not clear if and how spatial information should be incorporated into self-supervised model pretraining. Prior works have pretrained spatiotemporal models of iEEG activity, using either supervised [8] or self-supervised methods [1 19]. and demonstrated transferability across tasks, subjects. or sessions. Among these, one approach has used a discrimination task identify if channel had been randomly replaced in an ensemble of channels [9]. However, a critical remaining question is how different spatial scales would impact self-supervised pretraining. Indeed, within the context of masked pretraining, none of the existing approaches have explicitly incorporated the notion of space within their masking strategy and have instead typically selected random channels to mask and reconstruct. Thus, it remains unclear if channel- bbsed masking is preferred over larger scales of masking, such as brain region-based, when modeling multiregional neural activity.
Contributions Here we address the above challenges by developing neural tokenization and spatial encoding scheme that maintains individual channel temporal statistics while also enabling spatial encoding larger spatial scales. Further, to study the impact of spatial scales model pretraining with a self-supervised masked reconstruction task, we also develop an end-to-end training procedure that trains a model to reconstruct targets that are masked based on spatial meta-information. supporting masking both at the single channel scale as well as larger brain region scales. We call our modeling framework BaRISTA. In summary, our contributions are the following:
1. We develop a spatiotemporal transformer model of intracranial neural activity and an associ- ated masked latent reconstruction pretraining task. Within our framework, we dissociate the selection of spatial encoding from spatial masking to isolate the effects of one from the other on learned representations and overall pretrained model performance.
2. Using our framework, we investigate the impact of spatial resolution on model pretraining and downstream task performance, observing that spatial encoding at larger spatial scales improves downstream decoding performance over channel-level encoding.
3. We demonstrate with a downstream masked channel reconstruction task that our modeling and pretraining approach is able to incorporate larger-scale spatial information without sacrificing knowledge of individual channel temporal statisties.
2 Related Work
Spatiotemporal models of intracranial neural activity Several prior works have proposed spa- tiotemporal models of iEEG activity using different approaches for encoding spatial information. Brant and its subsequent iterations did not explicitly encode the spatial axis and only utilized standard
positional encoding [20] of the temporal axis in their models [15] [6]. Zheng et al. [17] proposed an

---

approach to model iEEG activity within preselected brain regions by pooling all channels within region, collapsing out the spatial axis, and thereby precluding the need for explicit spatial encoding. Finally, both Mentzelopoulos et al. [8] and Chau et al. [19] encoded space at the single-channel scale by incorporating neuroanatomical information and utilizing each channel's volumetric 3D coordinate to construct the corresponding token's 's spatial encoding vector. However, to our knowledge, no prior work on modeling iEEG data has looked at maintaining channel-level tokens while encoding spatial information at larger spatial scales (as we do here), such the brain regions in which the channels are located.
Self-supervised masked modeling of neural data Paralleling demonstrations in population spiking 99. fMRI 12 [ and EEG [11] neurofoundation models, there have been recent efforts showing the utility of masked self-supervised pretraining for spatiotemporal models of iEEG data [15]77] Most of these prior works have typically used random masking strategy at the level of individual channels, following standard procedure in other domains such as vision [21 22 and language [23]
However, random channel-level masking strategy for target selection may necessarily be the most effective for iEEG data due to the unique statistical properties and functional roles associated with spatially distributed channel recordings. As such, here we develop masked model pretraining task that allows for flexible specification of masking targets based on user-specified meta-informmtioo. for example domain knowledge of neuroanatomy or that of functional brain network activity. This allows us to explore masking both at the single-channe scale and at larger scales. Finally, we further differentiate from prior approaches by training our model, which consists of neural tokenizer and a combined spatiotemporal encoder, end-to-end to perform reconstruction in the latent rather than observation space.
3 Methods
To assess the impact that spatial encoding and masking each have representation learning and downstream model performance, we developed a new spatiotemporal transformer model and a corresponding pretraining framework that allowed us to independently adjust the spatial scales used for token encoding and target selection the masked reconstruction task. We first describe how we chose the spatial scales tested and how we flexibly incorporated that spatial information into framework. We then present our transformer model architecture and our self-supervised pretraining procedure. Finally, discuss evaluation schemes.
3.1 Spatial scales investigated
Intracranial neural recordings are multivariate time-series collected from electrode channels that span multiple brain regions. As such, there is an inherent notion of space, both at small (e.g., 3D) channel coordinates) and large (e.g., brain regions) spatial scales. Here, we explore the choice of spatial scale within the context of masked self-supervised pretraining. For our investigation, we choose three spatial scales based on neuroanatomical meta-information to test (see Figure | and Appendix for details):
(1) Channel Similar 18 and 119. channel (æ, y, (left,, posterior, and inferior, LPI (24)
coordinates in MRI volumetric space are used for spatial token encoding. In this regime, channels are randomly selected and masked.
(2) Atlas parcellations Spatial encoding and masking is based on electrode localization and channel assignments to cortical parcellations using standard brain atlases (e.g.. Destrieux or - atlases in Freesurfer). Here we choose to use parcel assignments based on the Destrieux (25] atlas as it contains more parcels and therefore permits finer mnainne analyses. We also additionally include subcortical structures (e.g.. hippocampus) that were annotated in the provided dataset (Section 4.1
(3) Lobes Spatial encoding and masking is performed at scales corresponding to brain lobes. We also include regions that are not considered lobes (e.g., cingulate). but are often regions of interest across various neuroscience domains. Lobe identities for each channel are designated based the Desikan-Killian atlas as per the appendix of (26).
3

---

Spatial Encoding Scales
Speech
Nonsaeoch
Time
Time!
Tirr
Linwer
Channal LPI Coordinates
Atlas Farcels
Lobes
Downstream
Subject 1
Decoder
Convolutional
Transformer Encoder
Tokenizer
2
Time
Neural
Latent Embeddings
Patched Data
Tokens
Figure 1: BaRISTA model architecture. Subject data is first channel-wise patched along the time axis and encoded using a tokenizer (dilated convolutional temporal encoder and linear projection layer). Then, spatial information is encoded based on a prespecified spatial scale; here we explore channel-level volumetric LPI coordinates, atlas parcels, and lobes. Neural tokens are passed as inputs to the encoder transformer, which provides the embeddings used for downstream tasks.
3.2 Model architecture
Our model architecture and tokenization scheme are shown in Figure Given a multivariate time- series of neural activity X E RCCT, where C denotes the number of recording channels and T denotes time, we first tokenize channels as univariate signals (i.e., agnostic to space), following common practice [14 1
28). We create temporal patches of each channel that are of length L (e.g., 250 milliseconds), such that Pij E R&- indicates the i-th patch of length for the j-th channel. Our tokenizer, denoted by F. consists of temporal encoder and a linear projection layer. In the first step tokenization, each temporal patch is passed through a shared temporal encoder. In practice this encoder can take any form; here choose a dilated convolutional neural network (CNN) (29) both to account for the input signal's continuous nature and because of prior domain knowledge about the importance of oscillatory features in neural activity [I 33). Next, we apply linear layer the output of the temporal encoder to create tokens of dimension d, such that B.j = F(P.j) E Rd denotes the token corresponding to patch Pij-
18
27
To encode space we add a learnable embedding vector, denoted by E, == eP) Rd that corre- sponds to the j-th channel's 's spatial category, sp(j). Note, this category depends on the selected scale among the three spatial scales explored here (Section 3.1) and refers to the channel's spatial designation within the selected scale. At larger scales, two channels may have the same spatial encoding if they belong to the same category (e.g., the same parcel assignment). The number of unique categories within a given spatial scale determines the size |C] of the learnable spatial em- bedding dictionary for that scale (more details about spatial categories are provided in Appendix D). Using the spatially-encoded tokens, denoted as S.y- = B,j + Ej. we create the transformer input token sequences of length nCC, where indicates the number of temporal patches for one channel. Specifically, we order all channels' tokens within an input sequence as
S [S S12, Sic, S21, S.cc (nC)xd
such that temporal and spatial information interleaved. This allows have single encoder transformer that can attend to space and time concurrently unlike some prior work that cascaded the temporal and spatial transformers [15 [8 [9]. We also note that because transformer input sequences can be of variable lengths C' and n here are also not fixed, meaning our method can support modeling sessions with differing channel and patch counts. To encode temporal information the token level, we use rotary positional embeddings (RoPE) in our transformer's ' attention layers [34]. Finally, the outputs of our spatiotemporal encoder transformer model, Z = G()) E (RC))xd are used as the neural embeddings for all downstream tasks (Section 3.4). In Appendix (Tables] [4] [5). present ablation results on our choice of temporal encoder (CNN) and the combined attention module. Comprehensive details on model architecture are provided in Appendix B
3.3 Spatially masked latent reconstruction
Our training procedure is shown in Figure We train BaRISTA using a self-supervised masked token reconstruction task, which differs from prior work in two ways. First, use the selected spatial

---

scale to guide masking, rather than only masking mendomll-selecteer channels [1 16 or tokens [17] Second, unlike some prior iEEG models [T7 19]. we simultaneously train both the tokenizer and encoder transformer to perform masked reconstruction in the latent token space.
During training, we randomly select subset of spatial categories within the input data mask, denoted by SP target. We use all the tokens that correspond to the selected spatial categories as our target tokens, Bearger, such that
Btarget = {Bij (G)) targes
We note that the selection of target spatial categories is constrained such that the total number of masked tokens |Barget| corresponds model. All remaining tokens are used as observation tokens, Bobs- While observation tokens are obtained using our original tokenizer (top row, Figure 2), we use a separate target tokenizer F for the target tokens (bottom row, Figure (A). The target tokenizer is updated with exponential moving average (EMA) of the original tokenizer weights. In our online network (top row Figure A the target tokens replaced with shared learnable mask token, M. The spatial encoding for each token is added to the masked input sequence as described Section 3.2
desired masking percentage hyperparameter of our
So, for example, if SP target
contains channels 1 and 2. then the input sequence
S [S11. S12: S13, ... Sic, S21 S22, ... "
would become the masked input sequence
Smasked = MM + E,, M + E2, S13, Sic, M M E,, Swc] E ((CC)xd
where E, denotes the spatial encoding for the corresponding j-th masked token. Temporal position for masked tokens was encoded using RoPE, as in Section 3.2
After obtaining the latent embeddings Z from the transformer, we pass the embeddings for the masked tokens to a predictor network, H. to perform target token reconstruction (Figure(A) Here, we use a multi-layer fully-connected network (MLP) as our predictor, H. Our training loss is the average mean-squared error between predicted tokens, B:j H(G(M + Ej (Smaskaa)) and target tokens, Bi F(P.)):
C
Σ
BB, B
B Brarget iE JESP surges
For all downstream tasks, we use the EMA-updated target tokenizer with the transformer backbone our pretrained model. For our channel reconstruction downstream task described in Section 3.4 (Figure 3), we retain the trained predictor network H. Additional details on model training are provided in Appendix C
3.4 Downstream evaluation
We evaluate the validity of our training procedure and the effectiveness of our learned model using several downstream tasks. We also evaluate the impact of spatial scale, both for token encoding and masking, on the same tasks. To do so, we first validate our pretrained model's performance on two language-related downstream tasks used in [14] 19]: classification of speech vs. non-speech audio and identification of words that correspond sentence onsets. Classification performance reported an average across all hold-out test sessions for 5 finetuning seeds each (see Section [. and Appendix A.. As baselines, we compare our pretrained model's finetuned performance against finetuned, handomly-niitaliizz version of itself and two state-of-the-art (SOTA) spatiotemporal iEEG models: Population Transformer (PopT)) [99 and Brant 15)
Second, we use the flexibility afforded by our framework to pretrain BaRISTA with different spatial token encoding and masking scales, and we compare the different configurations based on their performance on the language-related downstream classification tasks. Third, we also evaluate our pretrained model's finetuned performance on masked neural reconstruction the observation space as another downstream task. We finetune models pretrained with different spatial configurations using mean- -quuared error reconstruction loss computed on an individual channel basis. Distinct from the language- reguaaee-ellted related classification tasks above, here we also finetune the prediction head 7. from our pretraining task to perform masked channel reconstruction (Figure Details on the classification tasks and the reconstruction task setup, including the training procedure we had to develop to teach the model to reconstruct channel activity from masked tokens, are provided in Appendices and respectively.
5

---

Pretraining
Downstream
Reconstruction
Spare
Time
Takenizer
Encoainng
Spatial
Transformer
MUP Predictor
Time
Predicted Tokens
Patched
Masked Data
EMA
Masked
Latent Embeddings
Neural Tokens
Rec Reconnstuuted Patches
Time
Target Tokenizer
Loss
SG
Target Patches
MSE
Target
Patched Target
Neural Tokens
Figure 2: BaRISTA is pretrained with a masked latent token reconstruction task. A. We randomly select observed and target spatial categories, which are encoded with an online (top) and target (bottom) tokenizer, respectively. Target tokens replaced with a learnable mask token before being embedded by the transformer (top). The embeddings for the masked tokens are used to predict the target tokens as per mean-squared error loss. The trained encoder transformer and target tokenizer are used for downstream tasks. SG=stop gradient. B. We use a linear layer to reconstruct raw channel time-series activity from masked target tokens, using the predictions provided by the pretrained predictor network. A mean-squared error loss between true and reconstructed neural activity is used for finetuning.
4
Experimental results
4.1 Dataset and evaluation methods
For our experiments used the publicly available Brain Treebank dataset [35], which consists of intracranial recordings from 10 epilepsy patients collected over total of 26 sessions as they watched Hollywood films. Film transcripts that are aligned to neural activity are also provided. The iEEG recordings cover multiple brain regions across both hemispheres, including the temporal and frontal lobes, which are known to support auditory and language processing. Neural data is provided at sampling rate of 2048 Hz. We followed similar preprocessing procedures on raw data (e.g., filtering) as outlined in 14 ways to enable two sets of evaluations (details are in Appendices and K For our main evaluation, we generated overlapppn 3-second-long neural data segments and randomly assigned them to 80/10/10 train/valid/test splits; we present the results of this analysis in Sections 4.2 and 4.3 However, since enforcing no overlap requires dropping some of the labeled segments, we also performed an alternative evaluation that let use more of the annotations provided by the Brain Treebank dataset 35| for downstream training. In this evaluation, we allowed for overlapping neural segments and generated the 80/10/10 train/valid/test splits chronologically in time avoid any overlap between these splits. This procedure increased the amount of labeled data and additionally enabled evaluation on 2 more downstream tasks. We provide the results of the second evaluation in Appendix K Our findings across both evaluation schemes were consistent, thus providing a rigorous validation of our conclusions. Finally, to further validate framework and our baseline comparisons, that we were able to reproduce the PopT downstream classification results reported in [19 when using their original downstream segments (see Appendix [..). For all downstream classification tasks we report the average performance (+/- standard error of measure, s.e.m.) over the 7 test hold-out sessions, with 5 finetuning seeds for each task.
but generated our downstream data segments differently two
confirmed
Lastly, for pretraining, we generated 3-sccond-lowe long non-overlapping neural segments which we separated into 80/10/10 train/valid/test data splits. We pretrain 17 of the sessions and hold-out and sessions for validation and test, respectively [4] 19
4.2 BaRISTA's flexible spatial encoding enables decoding improvements over baselines
In Table we report the average classification ROC-AUC over all test sessions and finetuning seeds (n 35 points total). Our results show that our model outperforms all alternative models by enabling flexibility over spatial encoding. First, our pretraining improves downstream performance

---

Table 1: Classification results (mean AUC + s.e.m.). Within each task, asterisk indicates the best pest-performin (bolded) model is significantly better than second-bes (underlined) model with p-value <le-5 (Wilcoxon signed-rank test).
Model
Sentence Onset
Speech/oon-Speech
Brant 15
0.767 + 0.017
0.691 + 0.017
PopneBrainbert 19
0.795 1 0.014
0.775 1 0.016
BaRISTA (channels/channels)
0.778 1 0.019
0.764 1 0.020
BaRISTA (pancels/channels)
0.862 = 0.016
0.869 1 0.016
BaRISTA (random initialization, channels)
0.688 + 0.017
0,616 = 0.019
BaRISTA (random initialization, parecls)
0.683 = 0.017
0.627 = 0.018
Table 2: Downstream classification results of different spatial encoding/masking configurations (mean
AUC +/- s.c.m.). Best results in bold.
Mask
Channels
Parcels
Lobes
Random Init.
Encode
Channels
0.778 + 0.019
0.710 0.017
0.654 0.019
0.688 0.017
Sentence
Parcels
0.862 + 0.016
0.861 + 0.014
0.841 = 0.015
0.683 + 0.017
Onset
Lobes
0.842 + 0.016
0.816 + 0.017
0.840 = 0.014
0.681 = 0.017
Channels
0.764 + 0.020
0.668 = 0.015
0.652 0.017
0.616 = 0.019
Speech
Parcels
0.869 + 0.016
0.866 = 0.015
0.845 = 0.017
0.627 = 0.018
Lobes
0.841 + 0.019
0.823 = 0.015
0.840 0.015
0.628 + 0.017
compared randomly initialized versions of our model. Moreover, pretraining using channel-level encoding and masking yields performance roughly on with recent iEEG models, both of which use channel-level encoding (none of the differences between our channel-level model and baselines were significant, except for our model being significantly better than Brant for the speech task, Wilcoxon signed-rank p-value 3. 3.869e-05). Interestingly, however, when using larger-scale parcel-level encoding and channel-level masking, our model achieves higher overall downstream performance compared to these SOTA iEEG models (difference with PopT significant with Wilcoxon signed-rank p-values 5e014e-06 and 2.328e-10 10 on sentence onset and speech tasks, respectively). Overall, the results Table demonstrate that by affording flexibility over the spatial encoding seale during masked reconstruction pretraining, subject performance we refer readers to Appendix Table [[] Similar results held our second evaluation with chronological splits (see Appendix Table 16).
model can improve downstream task performance. For individual
4.3 Larger scale spatial encoding enhances downstream performance
Next, we investigated the impact of spatial scale in both token encoding and masking and used our framework's 's flexibility to dissociate these two effects. To do so, pretrained our model using distinct spatial encoding/masking combinations with the 3 different spatial scales described in Section 3.1 and evaluated each pretrained model's performance on the same language-related tasks in Table We present both finetuned and random initialization results Table 2 we note that secoding/masking combinations presented in Table [ are subcomponents of the complete results presented in Table 2
First, we find that the choice of spatial scale has a significant impact on the performance of the pretrained model (Table[aan and Figure Second, we see that the choice of spatial encoding, rather than spatial masking, has larger impact on final downstream performance for both tasks. Third, interestingly, we find that channel-level encoding underperforms larger spatial scale encodings regardless of the spatial masking scale. To further isolate and quantify the sources of variability, we performed a two-way ANOVA (36) with spatial encoding and spatial masking as the independent variables and the ROC-AUC values the dependent variable; we Bonferroni correct p-values to account for tested conditions (e.g., two downstream tasks, etc.). The two-way ANOVA revealed that both independent variables had statistically significant effect downstream task performance with no significant interaction (sentence onset: encoding < le - 3, masking p = 0.010; speech: encoding p le 3, masking p 0.037). As another observation, by using BaRISTA's flexibility in designating encoding and masking spatial seales, we found that when using channel- encoding,
7

---

Classification Performance for Different Encoding/Masking Combinations
1.0
Sentence Onset ROC- AUC
06-
64
0.-
Subject
Subject
Subject 3
Subject 4
6ubject 6
Subject 7
Subject 10
Aggrogated
Speech vs Nonspeech ROC-AUC
1.0-
000
Subject
Subject
Subject 3
Subject 4
Subject 6
Subject 7
Subject 10
Aggregated
Channels/Ch.
ChuFarcels
Chulobes
Parcels/Ch
Parcels/Parcess
Parcelsluoees
Lobes/Ch
Lobes/Parcels
Lobindiobes
Figure 3: Channel-level spatial encoding underperforms parcel- and lobe-level encoding across all subjects for both downstream tasks, suggesting the importance of larger spatial scales in masked reconstruction pretraining, Scatter points correspond to individual trials (3 pretraining and finetuning seeds), error bars correspond to s.e.m. Aggregated results pool trials across all subject sessions for each condition. Two-sided Wilcoxon signed-rank tests were conducted between spatial encoding pairs, with * and indicating p-values E [le 5, le - 10) and S le 15, respectively. Ch.=channels.
channel-level masking works better than masking at larger scales, which may be an important consideration if a given application requires channel-level encoding. Furthermore, we note that the choice of spatial encoding has no impact in the aandomly-nittialzzed setting. Per-subject results presented in Figure 3 Here and in Section 4.2 we present results for a single pretraining seed per spatial encoding/masking category that was selected based on validation hold-out performance in the two downstream language tasks; we do this to be consistent with prior works that presented results for single pretraining seed (e.g.. 19). We also present downstream classification results averaged across 3 different pretraining seeds, in addition to the 5 finetuning seeds, in Appendix [F] (Table 10). We find similar results in our second evaluation with chronological splits (see Appendix Table 17).
In summary, our results show that larger than channel-level spatial scales, particularly for neural token encoding. can critically improve downstream classification performance, demonstrating that the choice of spatial scale can be important in self-supervised masked reconstruction pretraining. Additional model interpretability results are presented in Appendix (Figures 7 and 8
4.4 BaRISTA can maintain channel-level reconstruction with larger-scale spatial encoding
Beyond looking at higher-order language-related tasks, we also considered pretrained model perfor- mance masked channel reconstruction task in the observation space. We first used the same setup as our pretraining task to predict the target tokens from the masked tokens, using our pretrained model and the pretrained predictor network H (Figure(A)) To reconstruct the target channel's 's raw time-series activity from the predicted neural tokens, we added a linear head after the predictor net- work, H. that maps the predicted tokens, By H( + E, (maakeed)) to the corresponding raw time-series patch, Pes (Figure 2B) During evaluation, mask out one channel at time and report the average reconstruction mean-squared error (MSE) and coefficient of determination (R²) across all masked channels for the 7 held-out test sessions (Section 1 in Table As a baseline, we include models that use the same spatial encoding. Interestingly, we
the performance of see that finetuned models using parcel-level spatial encoding are able to achieve reconstruction perfor- mance comparable to finetuned channel-level encoded models. This suggests that the framework is capable of modeling larger than channel-leve spatial interactions without loss of individual channel
8

---

information. For further illustration, we show example reconstruction traces for 2 of our pretrained models (channel/channel and parcel/parcel) in Figure We can observe qualitatively that our method more accurately reconstructs low-frequency vs. high-frequency content. We quantitatively confirm this observation by performing a spectral analysis the reconstruction results in Appendix | Finally, full experimental details are provided in Appendix E.3 and subject-specific performance is provided in Appendix H (Table (2).
Table 3: Masked channel reconstruction performance (mean + s.e.m.). Best results in bold, second- best results underlined. Ini==initializaton.
Mask
Channels
Parcels
Lobes
Random Init.
Encode
Channels
0.397 + 0.040
0.354 1 0.032
0.478 1 0.036
0.566 1 0.028
MSE +
Parcels
0.391 = 0.019
0.413 = 0.023
0.417
0.027
0.846 + 0.028
Lobes
0.753 0.039
0.951 = 0.014
0.029 0.853
0.965 + 0.022
Channels
0.603 + 0.040
0.646 + 0.032
0.522
0.036
0.434 + 0.028
t
Parcels
0.609 + 0.019
0.587 1 0.023
0.583
0.027
0.155 + 0.028
Lobes
0.247 + 0.039
0.049 1 0.014
0.147 + 0.029
0.035 + 0.022
Channels/Channels Parcels/Parcels
Ground Truth
seconds
3 seconds
Figure 4: Example reconstruction traces from masked tokens using models pretrained with different spatial aucoding/masking pairs for two different 3-second segments. Parcel-level spatial encoding performs comparably to channel-level encoding in channel reconstruction performance, suggesting that channel-specific information is not lost when modeling with larger spatial scales. For visualization purposes, raw reconstruction outputs have been smoothed using SciPy's 's (37 Savitzky- Golay filter with window size of 5 and polynomial order 2.
4.5
Pretrained BaRISTA generalizes to unseen subjects and scales with pretraining data
To assess the ability of BaRISTA to generalize to completely unseen subjects, we conducted an analysis using our downstream language tasks in which we held-out all sessions for a test subject during pretraining and evaluated the resulting model's classification performance for the unseen subject. We performed this analysis for each the test subjects specified Appendix Table used the parcel/channels model configuration reported in Table [ Average results are presented in Table F and individual subject results are presented in Appendix Table While minor performance degradation is seen, as expected, the performance unseen subjects still higher than the two SOTA baselines and our randomly initialized models (Table TT. We also compared the downstream classification performance of the same parcels/channels BaRiSTA model when pretrained using 5%, 10%, 25%, 50%, and 75% of the total available pretraining data. Doing so, we observed that our the same downstream language tasks successfully scaled with
model's downstream performance more pretraining data (Figure 55.. To get the desired percentage, we added sessions randomly one by one, such that their total number of segments matches the desired data percentage. To ensure the results were not biased by specific sampling order, we repeated this process with 5 different random seeds. We also adjusted the number of epochs for pretraining when using lower percentage of data, such that the total number of parameter updates for each of the data size percentages was roughly comparable. We find similar patterns of generalizability and scaling using our second evaluation with chronological splits, provided in Appendix (Table 19 and Figure D).
5 Discussion and future directions
There are several interesting directions for future work that may further improve our modeling framework. First, although here we defined our spatial scales based on anatomical designations, our

---

Table 4: Generalizability to new subjects: downstream results of our parcels/channels model for both standard pretraining and pretraining with the target subject completely held-out (mean +/- s.e.m.). Results are averaged across finetuning seeds.
Model
Sentence Onset
Speech/Non-Spechh
BaRISTA (pareels/channels, Held-out)
0.841 0.016 0.862 + 0.015
0.852 0.013 0.869 + 0.016
BaRISTA (parcels/channels, Included)
model is ficxible in terms of what "spatial" definitions to use. As such, alternative definitions, for example based the functional roles of brain regions regardless of their anatomical designation [3] can also be utilized within our framework. Indeed, in future work it may be interesting to use the flexibility of spatial encoding enabled by our framework for hypothesis-driven testing of encoding scales on a variety of downstream tasks that exhibit different degrees of complexity, including simpler sensory tasks. Doing may yield further improvements in model performance and/or insights about the encoding of various behavioral and cognitive states.
Second, in all of our experiments we used spatial-only masking in order to study the impact of spatial scales on model pretraining. Future work can explore integrating more diverse masking procedures 13, such masking across space and time, to further improve overall model performance and to potentially help facilitate learning of richer representations of iEEG recordings. Finally, we used a dilated CNN for temporal encoding and saw that modeling framework, even when using larger than channel-level scale, was able to maintain channel-level temporal statistics to perform reconstruction (Sections 3.3 and 4.4). Nevertheless, exploring alternative temporal encoding schemes, such as temporal pyramid pooling 38 or a combination of short-term and long-term encoders [3). may further improve channel reconstruction and will be interesting explore in the future.
4.44..4)
Sentence Onset RUC-AUC
0.95
Speech/Non-Speed ROC-AUC
0.95
0.90
0.50
0.85
0.60
0.80
0.75
0.75
0.65
0.65
0.60
0.60
5%
10%
25%
50%
75%
100%
5%
10%
25%
50%
75%
100%
Data Percentage
Data Percentage
Figure 5: BaRISTA's downstream classification performance scales as function of pretraining data size. Downstream classification results of our best model using different amounts pretraining data, denoted as percentage of the full training data (Appendix A Lighter seatter points represent the average performance different subsets of training sessions over finetuning seeds; we used different random subsets per percentage. The darker point is the average across these subsets.
6 Conclusion
Here we present BaRISTA, a modeling framework that enables flexible use of spatial scales to- wards spatiotemporal modeling of multiregional intracranial neural activity. First, we introduce transformer-basse model that allows for encoding at larger than channel-level spatial scales. Next, we develop and validate latent masked reconstruction pretraining task that uses spatial meta-infee meta-information for masking target tokens, thus also enabling larger spatial scales for masking. We show that utilizing spatial scale larger than channel-level during pretraining allows model to improve downstream task performance compared to SOTA iEEG models. Further, the scale of spatial encoding has greater impact on performance than that of spatial masking. Taken together, our results suggest that the choice of spatial scales during masked pretraining, encoding more so than masking, are important for enhanced model performance, especially towards building neurofoundation models of multiregional human intracranial neural activity. Furthermore, by affording flexibility in spatial encoding, our model may serve as a tool to explore hypotheses about the role of brain networks in behavior and cognition.
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by the National Institutes of Health (NIH) Awards R01MH123770. R61MH135407 DP2-MH1263 and RFIDA056402. We thank Dr. Danil Tyulmankov and Eray Erturk for helpful discussions.
References
[1] Joshua Jacobs and Michael J. Kahana Direct brain recordings fuel advances in cognitive electrophysiology Trends in Cognitive Sciences, (241):162-171.Applll 2010. ISSN 1364-6613. doi: 10. 1016/) 2010..
URL
[2] Jean-Philippe Lachaux, Nikolai Axmacher, Florian Mormann, Eric Halgren, and Nathan E, Crone. High- frequency neural activity and human cognition: Past, present and possible future of intracranial EFG
research. Progress in Neurobiology. 98(3):279-301 September 2012. ISSN 0301-0082. doi: 10.
1016/i.pneurobio22122 503010082120010622
URL fttppss/wwww.siieneediree wttp://wweeeeeeeeeeeeeeeeeaaaaaa
[3] Sean A. Guillory Krzysztof Bujarski. Exploring emotions using invasive methods: review of 60 years of human intracranial electrophysiology. Social Cognitive and Affective Neuroscience, 9(1)): December 2014. ISSN 1749-5016, doi: 10.10933/cam/nss000, URL http:t:st////oooor
1093/scan/nsu002
[4] Josef Parvizi and Sabine Kastner. Promises and limitations of human intracranial clectroenecchhalggrppy Nature Neuroscience, 21(4):474 483, April 2018. ISSN 1546-1726. doi: (0938//41193-018--000
URL httpss//www. nawure.ww....u.eaaaaaaaaaa http:/artiicccee////1199999 018 0108- Group.
Publisher: Nature Publishing
Maryam M Shanechi. Brain-machinc interfaces from motor mood. Nature neuroscience, 22(10): 1554-1564, 2019. htLp:/t::/www http://wwwnnaur..eeeem .ntuure.eee.... http://////a./.ccc.o/o////////// 0488-
[6] Lucine L Oganesian and Maryam Shanechi.
interfaces for neuropsychiatric disor- ders. Nature Reviews Bioengineering. 2:653-670, 2024. URL. URI.
[7] Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural Data Transformer 2: Multi-context
Pretraining for Neural Spiking Activity. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 80352- 80374. Curran Associates, Inc., 2023. URL httpp::/pproceeddngg/ http://wwwoe.ne..nn..rrereeeeeee wttp:pppp.cpppppp..eeeepe files/ fopp://222ffiieeeeeeeeeeeee16112
Papel----------aeeee Conference pdf
Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. Unified, Scalable Framework for Neural Population Decoding. Advances in Neural Information Processing Systems, 36: 44937 44956, December 2023. URL htttps::/papooe nips cc/ ccpppppr_.. faappr_ffleee fwwes/ill/eleeelle wwwes//pppplleeee wwwpr/apr//eaee Seaper/2023.. 22/2222222hahaaaaa (ww.:22664412227771111121 58887- Abstract- Conf erence. html
[9] Yizi Zhang, Yanchen Wang. Donato M. Jiménez-Benetó, Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung. Olivier Winter, The International B. Laboratory, Eva Dyer, Liam Panin- ski, and Cole Hurwitz. Towards "Universal Translator" for Neural Dynamics at Single-Cell, Single-Spike Resolution. Advances in Neural Information Processing Systems, 37:80495-80521, wwwppper.ccpppp..ppppmm httpc/pppr_ffeeee__eeeeeeeeeeeee faleer_ffllepppppee 2204r//////2///hh/ December 2024. URL https:: htttp:://proceedingg: Conference html
[10] Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva L. Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. October 2024. U htt.::ts///peeeeeeieeee wttp://wwww.eeeoom/ddddduoooooea http://wg...ppooooeerreeereeeeea
[11] Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large Brain Model for Learning Generic Represen- tations with Tremendous EEG Data in BCI, May 2024. URL httt:://ariii org/ aba/ 220./ 18765 arXiv:2405. 18765 [cs].
[12] Josue Ortega Caro, Antonio Henrique de Oliveira Fonseca, Syed A. Rizvi, Matteo Rosati, Christopher Averill, James Cross, Prateek Mittal, Emanuele Zappala, Rahul Madhay Dhodapkar, Chadi Abdallah, and David van Dijk. BrainLM: A foundation model for brain activity recordings. October 2023. URL https://openrest net/forumm hodt/foormmidd-Rl//22E22

---

[13] Zijian Dong, Ruilin Li, Yilei Wu, Thuan T. Nguyen, Joanna S. Chong, Fang Ji, Nathanael R. Tong. Christopher L. Chen, and Juan H. Zhou. Brain-JEPA: Brain Dynamies Foundation Model with Gradient Positioning and Spatiotemporal Masking. Advances in Neural Information Processing Systems, 37: 86048-86073, 86073, December 2024, URL httpttp::/pproceeaaae wewpiinnnn..uuuuppee wttppssccpppperetteeeepppeee fomer_ffie www.sseeppppeeee 2024/hash/ 91399992828810000 000aa00000000000006666606000---3 Abstract- Conference html
[14] Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu, BrainBERT: Self-supervised representation learning for intracranial recordings, February 2023. URL http: http://awwi..rraa//////a//////// 14367 ar arXiv:2302. 14367 [cs, eess, q-bio].
[15] Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen. Jingjing Wang, and Yafeng Li. Brant: Founda- tion Model for Intracranial Neural Signal. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 26304- 26321. Curran Associates, Inc., 2023. URL httpp:://proceeddngg/ http://oooeeniinnneeerrirprppppp neuripa. www.wee.....pppeeeeeeee wwtpeepeppeeaiaeeeeeeee files/
Paper- Conference. pdd
[16] Zhizhang Yuan, Daoze Zhang, Junru Chen, Gefei Gu, and Yang Yang. Brant-2: Foundation Model for
Brain Signals, March 2024. URL htt.ktt:t///izi..rrrtttr/ wtt.:/rg//aaa///////////// 10251 arXiv:2402.1025 [q-bio]
version: 4.
[17] Hui Zheng, Hai-Teng Wang, Wei-Bang Jiang, Zhong- Tao Chen, Li He, Pei- Yang Lin, Peng-Hu Wei, Guo-Guat Guang Zhao, and Zun-Zhe Liu. Du-IN: Discrete units- guided mask modeling for decoding speech from Intracranial Neural signals. Advances in Newral Information Processing Systems, 37: 79996-80033, December 2024. URL aatpaaaaeeeeeeeaaaaaaaaaaaooette Confereeee htmll
[18] Georgios Mentzelopoulos, Evangelos Chatzipantazis, Ashwin G. Ramayya, Michelle J. Hedlund, Vivek P. Buch, Kostas Daniliidis, Konrad P. Kording, and Flavia Vitale. Neural decoding from stereotactic EEG: Advances accounting for electrode variability across Addvacess in Neural Information Processing Systems, 37: 106600-108624, December 2024. URL htttpppps://ccocepanageoo/// neur ips. httpscccppaaeeeeeeeeeeeeeeeeeeee cc/paper_fiieeeccppppe sapoer_rileee www.seeespppprrrr...
2024/hash/ http://hwccc777000000000000223/0 -wt.://777....0---babaaaacccaa Abstract- Conference oot.:eebooo.bbatataccoooooreeeee html
[19] Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Socdarmadji, Yisong Yue, Boris Katz, and Andrei Barbu, Population Transformer Learning Population-level Represen-
tations of Neural Activity, March 2025. URL http:///://aaxx www.aaawwww...oaaaaa.aea wwwp/t///////////////////eeee/// abtp///bb///.....//// 2406 03044 2XXv:2406.03044 [cs].
[20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL btttps/:://// 1ataer/210///hh///353 httaa///ffeee2244/44eee11100....
wwwp.cccpaappppeeeee ccppaper.p.pp....pppppeeeee.. files/ 2tt.a//dee001tb000000144444aa c4a00555ac1.a44aaaaa. c4a844aaabbstracct........aaaaaa www.ract..aaat.e... html
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoen- coders Are Scalable Vision Learners, December 2021. URL. arXiv:2111.061 [es].
wwwpa/wwwwwrg/////1/.aa1a 06377
[22] Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT Pre- Pre-Tnainin of Image Transformers, September 2022. URL htttp///arx wwtpeiiiioorr/aaee org/abg://////oo aas////b//220 08254 arXiv:2106.0225 [cs].
[23] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers). pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/N1199144
[24] Graham Wideman. Orientation and Voxel-Order Terminoloyy: RAS. LAS, LPI. RPI, XYZ and All That. or sentterms...... htt
UtL/http::// htttt:://ww... /www www.ggrraaawddmaaaaaa. com.aaann.cco.aaam wwt. wttp://///////aa/rreeeeeeeeee/ee
[25] Christophe Destrieux, Bruce Fischl, Anders Dale, and Eric Halgren. Automatic parcellation of human
cortical gyri and sulci using standard anatomical nomenclature. Neurolmage, 53(1):1-15, October 2010. com/
ISSN 1053-8119. doi: 10. 10.10oiij wwwroomage..210...ooomm 2010.06.010. URL, hhttpe://sauu.cieeceee /iakace//rraaccee////////1111111 /tt////0001111111000011
[26] Aro Klein and Jason Tourville. 101 Labeled Brain Images and a Consistent Human Cortical Label- ing Protocol. Frontiers in Neuroscience, 6, December 2012. ISSN 1662-453X. doi: 10.3389/fnins. 2012.00171. URL https: /wwww frontiersin orghttps: :/tp::::://w/w.//// wwwntiersin.ooooee wrreeeinin/.eeeeeeeeeeeeeeoeeeee org/jouraass//////ooooooo heorosscienne//rtrccleess/// 3389 fnins. 2012. 00171/full Publisher: Frontiers.
12

---

[27] Yuqi Nie. Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A Time Series is Worth 64 Words: Long-tern Forecasting with Transformers. September 2022. URL /ttpitttt///ppeeeiiiee Forum?5d==JodeOrToeol
[28] Yong Liu, Tengge Hu, Haoran Zhang. Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. Transformer: Inverted Transformers Are Effective for Time Series Forecasting, March 2024. URL.
http://arxiv. wwg. 06625 arXiv:2310.06625 [es].
[29] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio, September 2016. URL peattp://arxiv. http://aaxii...gabbb/a////sa wwwp/r//////////oeee 03499 PeXiv:169903440 [cs].
[30] Shaojie Bai, Zico Kolter, and Vladien Koltun. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, March 2018. URL URL httpt::///xaxii./ www.axii....g/abooooo abe/ 1803
01271v2
[31] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8))8980-898)))) June 2022. ISSN 2374-3468, 2159-5399 5399. doi: 10. 6009/aai..v36i
20881. URL httpp://////s aaai www//rrgg.....eere/eeeeee php/ AAAI aww.aaeetiileeeee view/ 20881
[32] Xiaochen Zheng, Xingyu Chen, Manuel Schürch, Amina Mollaysa, Ahmed Allam, and Michael Krautham- Simts: Rethinking contrastive representation learning for time series forecasting. arXiv preprint arXiv:2303. 18205, 2023.
[33] György Buzsáki and Andreas Draguhn. Neuronal Oscillations in Cortical Networks. Science, 304 15679):1926-192, June 2004. ISSN 0036-8075, 1095-9203, doi: 10. 1112//ccinnce1009975. URL
attps://wuw.sciett
/trg//oo//0..// 1126/science. 1099745
[34] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023. URL http://arxiv httpaaawwii.....aaae...a org/abs/ 2104 09864 arXiv:2104.09866 [cs].
[35] Christopher Wang, Adam Yaari, Aaditya K. Singh, Vighnesh Subramaniam, Dana Rosenfarb, Jan DeWitt, Pranav Misra, Joseph R. Madsen, Scellig Stone, Gabriel Kreiman, Boris Katz, Igna- cio Cases, and Andrei Barbu. Brain Treebank: Large-scale intracranial recordings from natu- ralistic language stimuli. Advances in Neural Information Processing Syxtems, 37:96505-96540. December 2024. URL https: //pp://edinscccc....ooooaaoa neuripa www..pa.ccpppppp......eeep wttp:/aaee.eellle//pppeeeeeeeeee www..e/appppp///////.. /204//ash/ aofa2385b3f 15266442222088dd Abstract Datasets Benchmanhsarrks.. Track html
[36] Skipper Scabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
[37] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas. Denis Laxalde, Josef Perktold. Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python, Nature Methods, 17:261-272, 2020. doi: 10. 10.1033/s41 000-59922-9---------1
[38] Mehdi Azabou, Michael Mendelson, Maks Sorokin, Shantanu Thakoor, Nauman Ahad, Carolina Urzay, and Eva I.. Dyer. Learning Behavior Representations Through Malli--imescale Bootstrapping, June 2022.
URL htRp:tt:://// http:////aaivv.//aa org/ abaaabas///// 07041 arXiv:2206.0704 [cs].
[39] Mehdi Azabou, Michael Mendelson, Nauman Ahad, Maks Sorokin, Shantanu Thakoor, Carolina Urzay, and Eva L. Dyer. Relax, it doesn't 't matter how you get there: a new self supervvid approach for multi- timescale behavior analysis. In Proceedings of the 37th International Conference on Newral Information Processing Systems, NIPS '23, pages 28491 1- -85509, Red Hook, NY, USA, December 2023. Curran Associates Inc.
[40] Ariel Goldstein, Haocheng Wang, Leonard Niekerken, Mariano Schain, Zaid Zada, Bobbi Aubrey, Tom Sheffer, Samuel A. Nastase, Harshvardhan Gazula, Aditi Singh, Aditi Rao, Gina Choe, Catherine Kim, Werner Doyle, Daniel Friedman. Sasha Devore, Patricia Dugan, Avinatan Hassidim, Michael Brenner. Yossi Matias, Orrin Devinsky, Adeen Flinker, and Uri Hasson. A unitied acoustic- coousticcto-sppee to-speech-to wwwecch-t-languuaee to-language embedding space captures the neural basis of natural language processing in everyday conversations. Nature Human Behaviour, pages 1-15, March 2025. ISSN 2397-3374, doi: 122102//41162-025-02111
13

---

URL httpp:://wet httpa//rc......r.cce//////////-/ Group.
Publisher: Nature Publishing
[41] Biao Zhang and Rico Sennrich. Ront Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019, ptpptppppp..tttppp/pppeersss .html
nipp.ccppppprp. cc/paper
[42]
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs), June 2023. URL http:
1106e.//66..... 084155 arXiv:1606.08415 [es].
[43] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in Hour, April 2018. htRp: pttttp://arxivv /ttp//xxxiv....///a///// 1et.//706.... 02677 arXiv:1706.0266) [cs].
[44] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. September 2018. URL https://openreevee
[45] Ting Chen, Simon Korbblith. Mohammad Norouzi, and Geoffrey Hinton, A Simple Framework for Contrastive Learning of Visual Representations. 2020. htRp::tts:///tttt com googgee------eech
simclr. arxiv: 2002.05709v3.
[46] Felix Pei, Joel Ye, David Zoltowski, Angi Wu, Raced H. Chowdhury, Hansem Sohn, Joseph F. O'Doherty, Krishna V. Shenoy, Matthew T. Kaufman, Mark Churchland, Mehrdad Jazayeri, Lee E. Miller, Jonathan Pillow, Il Memming Park, Eva L. Dyer, and Chethan Pandarinath. Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity, January 2022. URL URL http://arxivv org/
abs/2109. 00466
Siiv::209.004663 [cs].
[47] Nilearn contributors. nilearn. URL httph::tt/t:thhiieeee
[48] Jianbin Wen, Tao Yu, Li Liu, Zhenhong Hu, Jiaqing Yan, Yongjie Li, and Xiaoli Li and. Evaluating the roles of left middle frontal gyrus in word production using electrocorticooraapyy Neurocase, 23(5-6); 263-269, 2017. doi: 10. 10.1080013150001 1080/13554794.200 2017.138777. URL http:: //doi org/ 1080/13554794
2017. 1387275
Publisher: Routledge _eprint: https://doi..rg To.
[49] Sally Rosario Hazem, Mariam Awan, Jose Pedro Lavrador, Sabina Patel, Hilary Margaret Wren, Oeslle Lucena, Carla Semedo, Hassna Irzan, Andrew Melbourne, Sebastien Ourselin, Jonathan Shapey, Ahilan Kailaya-Vasan, Richard Gullan, Keyoumars Ashkan, Ranjeev Bhangoo, and Francesco Vergani. Middle Frontal Gyrus and Area 55b: Perioperative Mapping and Language Outcomes. Frontiers in Neurology, 12, March 2021. ISSN 1664-2295. doi: 10. 00.3389/fnnur:22211 646075. URL httpsst///// : //www. frontiersin org/ htt////ounasneeeeeea neuro wogllllggyyooogyyl https//rtiiilsseeee / 10. 3389/fneur 2021. 646075/ full Publisher: Frontiers.
[50]
Lars Buesing, Jakob Macke, and Maneesh Sahani. Spectral learning of linear dynamics from generalised- lincar observations with application to neural population data. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran As- sociates, Inc., 2012. URL https: http:///c//p/ce//pe///
wwtiggsnnerriiee weourpp..cc/ppeeeeee http:/ppppprttttsspppprprr/////p /201////1///1///llle (tt.:68822200222218888888 c0518e80be- Satee0000e-PPPee....eee
(580722be282066622ca2222
[S1] Scott Linderman, Mathew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, pages 914-922 PMLR, April 2017, URL mlr wrwppressssee.... srress/54// v54/ win/ermannddnnneeee 17 17 html
ISSN: 2640-3498.
[52] Omid G. Sani, Yuxiao Yang, Morgan B. Lee, Heather E. Dawes, Edward F. Chang. and Maryam M. Shanechi. Mood variations decoded from multi-site intracranial human brain activity. Nature Biotechnology, 66:954-961, September 2018. doi: 10. 1038//bb..4200. URL https:ttss/////o //doi. org/1og 10. 1038/ 4200
[53] Omid G. Sani, Hamidreza Abbaspourazad. Yan T. Wong, Bijan Pesaran, and Maryam M. Shanechi. Mod- eling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuro- science, 24(1):140-149, January 2021. ISSN 1097-6256, 1546- 1726. doi: 10. 1011038/41153-000-0000 httreicccee///199933 s41593111993991 020 020000030--000033-000000
URL http::/www nature. com/ wwwaicteteeeeeeeeeeeeeeeee
[54] Parsa Vahidi. Omid G. Sani, and Maryam M. Shanechi. Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior. Proceedings of the National Academy of Sciences, (7) 12887121, 2024. doi: 10. 10.13/3/pmas.221 2887121. URI.https. /wtp///www//////// /twp:/pwppaaaaa////.. org/
doi/ 10. 1ew1007/ppaa 2w133///ass... 2212887121
[55] Lucine L Oganesian, Omid G. Sani, and Maryam Shanechi. Spectral learning of shared dynamics between generalized-Iinee linear processes. In The Thirty-eighth Annual Conference on Neural Information Processing
Systems, 2024.
14

---

[56] Yuanjun Gao, Evan W Archer. Liam Paninski, and John P CCnnnnnhamm Linear dynamical neural population models through nonlinear embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associales, Inc., 2016. URL /tttps://papers. nips
/
1bwdbbccbbbb-ccc Abstract.
[57] Chethan Pandarinath. Daniel J. O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey Stavisky, Jonathan Kao. Eric M. Trautmann, Matthew T. Kaufman, Stephen Ryu, Leigh R. Hochberg, Jaimie M. Henderson, Krishna V. Shenoy, L.. F. Abbott, and David Sussillo. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods, 15(10):805 815, October 2018. ISSN 1 7091, 1548-7105. doi: 10.
URL http::/www nature com/ http://attiilleeeeee -4192-
Number: 10.
[58] Qi She and Anqi Wu. Neural Dynamies Discovery via Gaussian Process Recurrent Neural Networks. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 454-464, PMIR, August 2020. URI htttps://proceeeaiiii
mlr. wpwess/v15///eeee/e http///111///ee.e.eeee she./a.eee0000 html ISSN: 2640-3498.
[59] Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with Neural Data Transformers. Neurons, Behavior, Data analysis, and Theory, 5(3), August 2021. ISSN 2690-2664. doi: 10.51628/00c..77355
http://atp::iiii org/ absa 2108. 01210 (.Xix:2108.0122 [es, q-bio].
[60] Colc Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted Neural Dynamical Modeling. In Advances in Neurul Information Processing Systems, volume 34. pages 29379-29392 Curran Associates, Inc., 2021. URL htttps:/paapers: nips cc/ pste//ape/20000 /021
haat///cccee7777hh000cccccaaeeee Aww.taat.bstttt...........o..... htmml
[61] Trung Le and Eli Shlizerman. STNDT: Modeling Neural Population Activity with Spatiotem- poral Transformers. Advances in Neural Information Processing Systems, 35:17926- December 2022. URL
neurips http:/tcccc1221111121171026e0ee2 - Abstract Conf erence..e... html
[62] Hamidreza Abbaspourazad. Eray Erturk, Bijan Pesaran, and Maryam M. Shanechi. Dynamical flexible inference of nonlinear latent factors and structures in neural population activity. Nature Biomedical Engineering, 8(1):55-108 108. Jamuary 2024. ISSN 2157-846X doi: 10.1038/441551-033 1030001106-111111111111111111111 URL Publisher: Nature Publishing Group
https://www. nature. com/ wwt.aaaaiilaaaeeeeee s4155e1e///1111111 10111151---0----0---11--
[63] Steffen Schneider, Jin Hwa Lee, and Mackenzic Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360- 368. May 2023. ISSN 1476-4687 doi:
10. 11B28844158--023-00001---6 URL htt.::ttttt///teee http://www.nttreeeeeeeee naturantuee... http//artiiceeee//115------- 0200 Publisher: Nature Publishing Group.
[64] Omid G. Sani, Bijan Pesaran, and Maryam M. Shanechi. Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks. Nature Neuroscience, 27(10): URL https://www Publisher: Nature Publishing Group.
2033-2045, October 2024. ISSN 1726. doi: 10. 10.100141113-002--000
nature comure..cmmeee
014-1111-22
[65] Parsa Vahidi, Omid G. Sani, and Maryam Shanechi. BRAID: Input-driven nonlinear dynamical modeling of neural-behavioral data. In The Thirteenth International Conference Learning Representations, 2025.
URL http:ttt////opeeeeiieee net. wwwp/eort/////eeee 3ddddmm
[66] Sayed Mohammad Hosseini and Maryam Shanechi. Dynamical modeling of behaviorally relevant spa- tiotemporal patterns in neural imaging data. In Forty-second International Conference on Machine Learning, 2025. URL httphttts////oenreee wwt. http//orruunddd////0t00a
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize our methodological innovations and experimental results in the abstract, and provide summary of main conclusions. All these method innovations, results, and conclusions are supported in the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a summary of our limitations in the Discussions section and propose future directions to address these limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification: Our paper is not theoretical but experimental in nature. We provide equations for all of our loss terms but these require stated assumptions proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We used publicly available iEEG datasets and provide citation and reference to access this data for reproducibility. Furthermore, we provide comprehensive details the main manuscript and supplementary how the data was preprocessed and segmented, how the train/valid/tess splits were generated, how the downstream labels were generated, and how all models were trained (both pretraining and downstream). We provide citations to prior work as necessary and will release our model code upon manuscript acceptance. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility.
17

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will provide the pretrained weights and corresponding model code with tutorials upon acceptance of the manuscript. The dataset used in our analysis is already publicly available and we provide the reference to it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
Seapii/ggiides/Cooetttiiiiss oollicicynnnooino for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co... ctaipss.c/pubicc//uuic http:/gguudeeeoeeeSuuubbiiiii wwt.submiisooooooiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide comprehensive training and test details in both the main manuscript and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We provide error bars for all reported values tables and figures presented. We have performed statistical analyses, the results of which we also report in the manuscript.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide these details in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::/neuripss http://rrr.ppp.pppppcccccccaaaaa
Answer: [Yes]
Justification:
We have followed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
19

---

Justification: We have included discussion about broader impacts in the Conclusions section. Our model may help study multiregional neural representations of behavior and cognition. thus advancing our understanding of the brain as a positive societal impact. Our work does not entail any negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This is not relevant to the main contributions or results of this manuscript. All analyses were performed dataset that is already publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the manuscripts corresponding to the baseline SOTA models we have compared against. We also cite scientific Python libraries that we used in our analyses. We also cite the publicly available dataset that was used in our manuscript.
20

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will make the pretrained weights and corresponding model code publicly available upon manuscript acceptance and include sufficient instructions in the form of READMES and tutorials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We did not use crowdsourced and/or other research with human subjects in this study. We used an iEEG dataset that is already publicly available and was published in NeurlPS 2024.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
21

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We used data that was already released publicly and did not collect any data ourselves. The corresponding technical paper reported that the data was collected with Harvard/Boston Children's 's Hospital IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs
important, original, non-standard component of the methods presented in our manuscript.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standtad components.
Please refer to LLM httppp////errpp.ccc/////oeeerees for what should or should not be described.
22

---

A Dataset details
a publicly available dataset of 10 epilepsy patients collected while they were watching movies from set of 21 animated/action Hollywood movies. Each subject watched one or more movies while iEEG was being recorded. There total of 26 sessions across all subjects, each being 2.07 hours long on average. Electrodes are mapped to common brain atlases that can be used analyze activity in each brain region. For each session, we first remove corrupted/noisy channels before Laplacian re-referencing the rest, excluding channels with insufficient neighbors for re-referencing - as [9]. We also additionally removed channels that had been localized to ventricles of the brain. The final number of channels, parcels, and lobes used per subject is reported in Appendix Table From the 26 available sessions, 17 were used for pretraining, 2 were held out as downstream validation, and the remaining were held out for downstream testing, as specified in Appendix Table
Brain Treebank 35
Table 5: List of available sessions in the Brain Treebank dataset, indicating those used for pretraining. downstream validation, and downstream testing. We also report the duration of each session and the number of segments used in pretraining (where relevant).
Subject
Session
Duration (hrs)
Split
Pretraining Segment #
Subject
Session
1.91
Pretrain
1828
Session
2.9
Test
Session
2.07
Pretrain
1989
Subject
Session
2.6
Pretrain
2498
Session
2.42
Pretrain
2342
Session
2.66
Pretrain
2515
Session
Pretrain
2903
Session 5
3.73
Pretrain
3567
Session
1.85
Validation
Session
3.52
Test
Subject
Session
1.9
Test
Session Session
2.94
Pretrain
2796
4.06
Pretrain
3924
Subject
Session
1.87
Test Pretrain
Session 2
1.75
1672
Session
1.31
Validation
Subject
Session
1.54
Pretrain
1482
Subject 6
Session
0.81
Pretrain
780
Session
1.32
Pretrain
1267
Session
1.6
Test
Subject
Session
1.67
Test
Session
1.77
Pretrain
1696
Subject 8
Session
1.41
Pretrain
1350
Subject
Session
Pretrain
960
Subject 10
Session
1.57
Test
Session
2.33
Pretrain
2240
For pretraining, we segment data into second non-overlapping intervals (6144 samples at 2048 Hz), resulting in a total of 35,089 pretraining segments corresponding to 29.2 hours. The number of pretraining segments per session are reported in Appendix Table We use the same pretraining segments for the channel reconstruction task (Section 4.4). As noted in Section 4.1 for our main evaluation and analyses we generate non-overlapping 3-second segments for the language-related downstream tasks and assign labels using the following protocol: lasitlee-labeled segments center word-aligned and correspond to sentence onsets or speech whereas negative-labe labeled samples (for both tasks) are 3-second-long intervals that correspond to speech content in their entirety; note
23

---

that this definition of negative samples is distinct from the definition used by [9], which only considered the speech content of the center I-second interval for the label assignment. In Appendix 6 we report the number of training, validation, and test segments for each downstream task and hold-out session. Training, validation, and test segments are randomly selected from the generated segments to hit the desired 80/10/10 ratio, similar to [14 19]. Positive and negative labels were balanced for the classification task before split generation.
In all of our analyses, we z-score standardize the 3-second segments. Further, for both pretraining and finetuning, we generate n = 12 temporal patches of length 512 (corresponding to 250 ms) for each 3-seoond segment. The subsegment length was chosen based on prior work looking at the timescale of language processing [14 19 35 40] but can be treated as a tunable hyperparameter.
Table For each hold-out session, the number of training, validation, and test segments used in the downstream tasks. Note, these counts correspond to the test sessions in Appendix Table
Sentence Onset
Speech/Non-Spechh
Channel Reconstruction
Subject
Train
Valid
Test
Train
Valid
Test
Train
Valid
Test
Subject 1488
186
186
2156
269
269
2787
348
348
Subject
1036
129
129
1470
183
183
3385
422
422
Subject
1066
133
133
1072
133
133
1825
228
228
Subject
1022
127
127
1468
183
183
1795
224
224
Subject
480
60
60
638
79
79
1532
191
191
Subject
650
81
81
624
78
78
1604
200
200
Subject 10
944
117
117
916
114
114
1506
188
188
B Model architecture
The temporal encoder in tokenizer F was dilated convolutional neural network (CNN) 29 composed of convolutional block layers, with the inner hidden layers having a hidden dimension of 5. Each convolutional block had a kernel width of 3, stride length of and exponentially increasing dilations as a function of layer depth (i.e., dilation 2' where corresponds to the depth, starting from i 0). The CNN operated on univariate channel recordings, and thus the input and final output dimensions were of size 1. Layer normalization was applied on the CNN outputs within each block. A linear layer then used to transform the CNN output, which is of length into the 64.
final neural tokens with dimensionality
For our model backbone G, we used encoder transformer with 12 hidden layers, self-attention heads, and hidden dimension of d - the same dimensionality as the neural tokens. In each layer, we first apply Root Mean Square (RMS) normalization [41], then perform self-attention followed by a 10% dropout layer, another RMS normalization, and finally a feed-forward MLP. Our predictor network H. used in both the pretraining and downstream channel reconstruction task, is 5-layer fully-connected network, with hidden layers each followed by a GeLU activation function 42 and 10% dropout layer. We also GeLU activation function after the final layer.
We use a target masking percentage of 30% during pretraining. EMA updates to the target tokenizer F happened according to linear warm-up schedule of 10 epochs starting from and increasing to target momentum of 0.996. In Appendix Table 7 we present the model parameter count for BaRISTA and our baselines (PopT and Brant). Note that despite being significantly smaller than other two SOTA models (20x smaller than Pop'T and 500x smaller than Brant), BaRISTA was able to achieve significantly better downstream performance when using larger than channel- spatial scales. Model code is publicly available at: wtw.gghhww.oootSSSaiiiaaaaaaaeaa
C Training details
Here we present training details, including computational cost, for both our model and our baseline models. BaRISTA models were pretrained using effective batch size of 128, with local batch size of 32 parallelized over NVIDIA RTX 6000 Ada NVIDIA RTX A6000 GPUs. We used linear warm-up of 5 epochs to the target learning rate 43 followed by exponential decay rate of
24

---

Table Comparison of the parameter count and total training time between different iEEG models.
Model
Size
Device
Training Time
BaRISTA
1M
NVVIDI RTX 6000 Ada
4 hours
PopTT (as reported in 191)
20M
NVIDIA TITAN RTX
days
PopT (our pretraining, see E.
20M
4 NVIDIA RTX A6000
22 hours
Brant (as reported in [15
500M
4 NVIDIA Tesla A100
2.8 days
y 0.99. We used the AdamW 44 optimizer with target learning rate of 1e-3 and decay rate of le-2 for pretraining. All BaRISTA models were pretrained for 70 epochs, which amounts to 19, 19,500 update steps. Pop'T' pretraining involved 500,000 update steps and Brant reported 750,000 update steps 13].
For the downstream tasks, we again used effective batch size of 128 for BaRISTA. We finetuned our model for 30 epochs, with a 15-epoch early stopping schedule based on validation performance. As with pretraining, we had a 5-epoch linear warm-up exponential decay with decay factor = 0.99. Here, we again used the A optimizer with a decay of 1e-2. Our learning rate was le-4 for the pretrained model and 1e-3 for the downstream linear layers. We note that during finetuning we only update the learned spatial encodings, e (see Appendix D) and the transformer encoder backbone, while keeping the temporal encoder (dilated CNN) frozen. We empirically found that the difference downstream classification performance was small when finetuning the CNN as well, and therefore opted to keep the model frozen for the sake of computational efficiency. Randomly initialized versions of our models follow the same downstream learning rate schedules as the pretrained ones.
target learning rate followed by an
For finetuning our baselines, PopT and Brant, we matched as closely as possible the training configurations reported in [15] and [19]. For both models, we finetuned for 75 epochs for each hold-out session and used Adam W with decay rate le-2. Moreover, for both models we used a linear warmup of 50 update steps to target learning rate, followed by step decay scheduler with a step size of 20 updates and decay factor y = 0.95. For PopT, the learning rate for the pretrained model was 5e-4 and 5e-5 for the linear classification layer. For Brant, the learning rate was le-3 for downstream layers and 1e-7 for the pretrained model. Training batch size for PopTT' was 128, whereas batch size was 64 for Brant.
Because we ran training for a fixed number of epochs, the total number of finetuning update steps was also dependent on the downstream task and subject (i.e., the number of training segments available), in addition to the batch size. For finetuning on the speech vs. non-speech and sentence onset downstream tasks, the average number of updates for BaRISTA across 7 test sessions was 252, for PopT it was 629 update steps, and for Brant it was 1258 update steps. We chose the larger number of update steps for the baseline models to ensure they converged as we wanted to validate performance against their best performance. We trained Brant the longest as its finetuning learning rate was 1e3 times smaller than BaRISTA's and 1e2 times smaller than PopT's; s; we note that the learning rates used reflect the rates used by the authors in the original works. Finally, we trained all models using mixed floating-pons point precision for both pretraining and finetuning.
model's
D Spatial scale definitions
Appendix Table & defines the within-scale categories discussed in Section 3.2 The number of distinct categories used in our model for each subject can be viewed in Appendix Table For spatial scales consisting of multidimensional spatial information (e.g., the three LPI coordinates), we maintain a distinct embedding table of size |X] for each dimension (e.g., 3 tables, one for each of the LPI coordinates). The final spatial encoding for a channel is equal to the sum of the embedding vectors Ispi (P),,.()), where |sp] denotes the number of dimensions w= corresponding each dimension: E, >bmm the spatial scale being used and aP denotes the w-th dimension's embedding vector for channel j. As an example, our final spatial encoding for the j-th channel's LPI coordinates can be expanded as E, = (P, + (i) + ()), where each embedding vector (P e Rd.
25

---

Table 8: Description and examples of the spatial scales defined in Section 3.1 categories include hemisphere designations. L/R-Left/Rihht. Sup.=Sup
Atlas parcels and lobes
Spatial Scale
Description
Dimension Example
(x, y/ z) where each element is integer between 0 to 200
Channel
LPI coordinate
24
3
Discrete neuroanatomical
L/R Sup. Temporal Sulcus
Atlas Parcels
subdivisions of the cortex;
L/R Postcentral Gyrus
subcortical regions also included
L/R Hippocampus
Brain lobe or equivalently large anatomical region; subcortical regions also included
L/R Frontal Lobe
Lobes
L/R Hippocampus
L/R Temporal Lobe
Table 9: Number spatial categories per subject.
Subject
Channels (LPI Coordinates)
Atlas Parcels
Lobes
Subject 1
91
25
6
Subject 2
100
25
8
Subject 3
91
20
4
Subject 4
151
36
9
Subject 5
109
25
6
Subject 6
134
30
6
Subject 7
205
47
9
Subject 8
121
29
7
Subject 9
72
19
5
Subject 10
173
42
8
E Experimental details
E.1
Baselines
We note few key points with respect to our baseline comparisons.
First, we used the pretrained Brant model as provided by the a For PopT 19 we used the publicly available codebase to pretrain Pop'T. To ensure performance reproducibility, we used the scripts made available by the authors to perform pretraining, black-box, with the only difference being the hardware used (see Appendix Table 7}. We verified our pretrained PopT's validity by reproducing the downstream classification results reported in [19]. To do so, we used the publicly available codebase2 generate the same train/valid/test splits used in [99] and evaluated our pretrained PopTT on the sentence onset and speech discrimination tasks, achieving 0.883 + 0.008 AUC and 0.925 + 0.010 AUC (averaged on 5 finetuning seeds), respectively: the original work reported 0.90 + 0.01 AUC and 0.93 + 0.02 AUC for sentence onset and speech/non-speeee speechh,,,,, respectively [19]
Second, Brant's model architecture expects temporal patches of length 1500 samples (the original work had pretrained the model using 6-second-long patches at a 250Hz sampling rate). However, the data segments used here were of length 6144 (3-second-long at 2048Hz sampling rate). In order to use the same train/valid/test data segments across all three baselines, we chose to downsample each of our 6 6144le segments 1500 samples (per segment) before providing them Brant; empirically found that this approach worked better than subsegmenting the original segment (i.e., breaking the original 6144 6144-s segment into subsegments of length 1500 each).
Brant Codebase: http://gtthbbccmm/yyy5577//aa
PopT Codebase: http://sthi...oooooaaaaa/ppaaaap
26

---

E.2 Classification tasks
For all downstream classification tasks, we use a lightweight linear decoder to evaluate the quality of each model's learned embeddings, as is common practice 46]. To do so, we train a logistic regression with the latent embeddings using a binary cross-entropy loss. For both our model and Brant, we apply linear projection on all latent embeddings in a sequence to get a single "average" embedding before classification. For Pop'T, we use the [CLS] token in the original paper [9]
E.3 Reconstruction task
For the reconstruction task, we perform linear regression from predicted masked tokens Bij E Rd to patched neural time-series data, Pij R2, where d 64 is neural token dimension and == 512 is our temporal patch length. We use our pretrained predictor network H to generate the predicted masked tokens, such that Bij = HM + Ej (masked))) We finetune our model using a mean-squared error loss between true and reconstructed neural temporal patches, such that
Learget
Σ
|.s - Payl8
(I)
Brarget
where Bearget defined as in Section 3.3 and n denotes the total number of reconstructed patches.
However, using only the predicted masked tokens to learn the mapping from neural tokens to the temporal patches is challenging, this would require the network model the true relationship between tokens and patches using a "noisy" (i.e., masked) token prediction. Thus, to help facilitate learning of the mapping from neural tokens their corresponding temporal patches, we also perform reconstruction of the temporal patches that correspond to the observed ("unmasked") tokens, denoted by B Bobs- We compute the mean- meeansquared error for the observed token reconstruction
Lobs
Σ
PPj - Pip
Bobe
(.............................. Pearget
and augment the training loss to be a weighted combination of Equations and 2
such that
L Ltorget + aLober
where is adjustable parameter that regulates the influence of observed (i.e., unmasked) tokens during training. We start with a constant value of a = for the first 10 epochs and then linearly decrease it to 0 afterwards. Note that the observed tokens evaluation, we mask out temporal patches one channel at a time, and use the linear head to reconstruct
only used during finetuning. For the patches directly from just the predicted masked tokens. B:y.
For this reconstruction task. used learning rate of 1c-3 for the pretrained model (predictor network H. included) and a learning rate of le-2 for the linear layer. Optimizer scheduling was the same as the classification tasks above (Appendix C). We evaluated seed per hold-out session and finetuned the models for 20 epochs.
F Encoding and masking spatial scale analysis
In Appendix Table 10 we present classification performance for the same encoding/masking con- figurations reported in T but here we also average across pretraining seeds. As before, we can see that classification performance increases when using larger spatial scales, with parcel-level encoding doing the best on average. Also as before, we see spatial encoding having greater impact than spatial masking.
To better dissociate the impact of each factor on downstream classification performance, we visualize the interaction plots between spatial encoding and spatial masking in Appendix Figure In panels 6A and 6C,, we can see that larger than channel-level spatial encoding scales boost downstream classification, across all masking strategies. panels 63 and 6D, the difference between masking strategies becomes more evident, with the choice of strategy having the greatest impact in the configuration with channel-level spatial encoding.
27

---

Table 10: Downstream classification performance various spatial encoding/masking configurations averaged across all 3 pretraining seeds and 5 finetuning seeds (mean AUC +/- s.c.m.).
Mask
Channels
Parcels
Lobes
Random Init.
Encode
Channels
0.735 + 0.013
0.681 + 0.012
0.665
0.012
0.688 + 0.017
Sentence
Parcels
0.836 + 0.010
0.843 + 0.009
0.844
0.009
0.683 + 0.017
Onset
Lobes
0.835 + 0.010
0.829 + 0.010
0.011 0.811
0.681 = 0.017
Channels
0.705 + 0.012
0.651 + 0.009
0.664
0.010
0.616 + 0.019
Speech
Parcels
0.847 + 0.010
0.843 + 0.010
0.848
0.010
0.627 + 0.018
Lobes
0.837 1 0.010
0.829 1 0.010
0.806
0.011
0.628 1 0.017
Sentence Croset AUC
0.85
Sentenze Omet AUC
0.85
0.80
0,80
Encoding:
0.75
Masking:
0.75
Chaannel Parcels
Channels Parcels
0.70
0.70
Lobes
Lobre
0.65
0.65
Channels
Parcels
Lobes
Channels
Parcels Spatial Masking
Lobes
Spotial Encoding
speechvs Nonggeech AUC C
Sprech vs Naneperch AlIC
0.85
0.80
0.80
Encoding:
0.75
Masking: Channels Farcels
0.75
Parcels
0.70
0.70
Lobes
0.65
Lobes
0.55
Channels
Parcels
Lobes
Channels
Parcels
Lobes
Spatial Encoding
Spatial Masking
Figure 6: For all masking strategies, downstream performance the language tasks improves with greater than channel-level spatial encoding scales, whereas the choice of spatial masking scale has the greatest impact in the configuration with channel-level encoding. For all panels, solid traces show the average AUC across pretraining seeds and 5 finetuning seeds (shaded areas denote s.e.m.). A. Sentence onset classification performance as function of spatial encoding. Each colored trace corresponds to a different spatial masking strategy, as indicated by the legends. B. Sentence onset classification performance as a function of spatial masking strategy; each colored trace corresponds to different spatial encoding, as indicated by the legends. C-D. Same as A-B. but for speech vs. non-speech task.
G Interpretability analysis
We also performed an interpretability analysis which we used the weights the linear projection that computes "average" embedding from all latent embeddings during the sentence onset classifi- cation task (described in Appendix E..2 By doing so, we aimed to identify the brain regions that model found to be most critical for decoding sentence onsets. As we detail below, found that the regions with higher weight loadings indeed corresponded to well-known regions implicated in language tasks, thus suggesting the biological consistency of our learned representations (Appendix Figures and R)
To perform the interpretability analysis, we first compute the absolute value of the weights, which are of size nC', where C₂ denotes the number of channels in the q-th test session and n is the number of temporal patches. Here we had = 12 patches at 250ms for a total duration of 3 seconds (Appendix A). Next, we group channel weights within each of the Destrieux parcels 25 (used here for the sake of visualization) and use the 75th- Stth-eercentiee weight represent each parcel. Finally, use session-wii wise min-max normalization to scale all values to be between 0 and 1. We denote these normalized linear weights by 1 RaR,, with Rg being the number of Destrieux parcels in the test session q. We present two different visualizations of these normalized weights (prepared using
28

---

Left Hemisphere
Right Hemisphere
Normalized weights
0.75
0.50
0.25
Middle Frontal Gyrus
Wernicke's
Broca's
Auditory Cortex
Figure Normalized linear projection weights have higher loadings on language-related regions across all test sessions. Weights from our sentence onset classification task are averaged across test sessions and visualized within Destrieux parcels. Bottom visualization depicts the locations of various cortical regions associated with language-related processes.
Nilearn 47 (1) aggregated across all test sessions and (2) as a function of time (i.e., across the n temporal patches) for single test session.
For the first visualization, we first aggregate weights across test sessions for each temporal patch, by scaling each session's weights by the associated downstream classification AUC and then forming the weighted average. The aggregated weights, denoted by Vagg. allow us to visualize the task-relevant information across the union of all parcels in all test sessions. Lastly, then average the aggregated weight for each parcel across all temporal patches to compute an average weight per Destrieux parcel, corresponding to each of our 3-second segments. Our results, presented in Figure 7 show larger weight loadings in temporal cortical areas, both in lower-level perceptual regions, such as auditory cortex, well in higher-level language processing regions, such Wernicke's s area. Interestingly, we also saw high loadings in the left middle frontal gyrus, which may have language- related implications suggested prior work [8 49]. These results suggest that our model has learned biologically interpretable embeddings.
In the second visualization, aimed to better understand the neural dynamics during sentence onset. To do so, in Appendix Figure we visualized the weights for an example test session over time, ie., over 12 consecutive 250ms-long temporal patches. We observed an increase in normalized weight loadings for temporal cortical areas shortly after the onset, which corresponds to Oms this figure. These results indicate that our embeddings also capture temporal information in the neural data during language tasks.
H Subject-specific downstream performance
Per-subject performance for all models on the downstream classification tasks is presented in Ap- pendix Table For BaRISTA present both the standard pretraining results ("Included" columns) as well as the within-subject generalization results ("Held-out" columns). Subject-specific channel reconstruction results for three of the models reported in Table are provided in Appendix Table T2
I Spectral analysis of channel reconstruction results
After observing the channel reconstruction results presented in Section 4.4 we explored our method's reconstruction in low high frequency ranges. We found that the majority of the spectral power in the reconstructed signal was in the low-frequency range (approximately $25Hz on average). For analysis, first reconstructed 1 1162 3-second segments across the 7 test sessions and filtered both the true and reconstructed signals for the low-frequency (<40Hzz) and high-frequency (40-
29

---

Normalized weights
0.75
250ms
Oms
250ms
500ms
750ms
1000ms
0.50
Sentence Onset
0,25
-1500
-1250
-1000
-750
-500
-250
250
500
750
1000
1250
1500
Time (ms)
Figure 8: Normalized weight loadings capture the dynamics of language-prooessinng processiooeennnooeeeeeeeeeennnge during sentence onset detection. We visualize the normalized linear projection weights for single test session during the course of seconds, where Oms indicates sentence onset. Weights achieve their maximal values shortly after sentence onset. The inset is a zoomed version of the dynamics from -250ms to 1000ms relative to sentence onset.
Table 11:
Downstream classification results of our model for both standard pretraining and pretraining with the target subject completely omitted (mean +/- s.c.m.).
Sentence Onset
Subject
Included
Held-out
Brant
PopT
Random Init.
Subject
0.922 + 0.002
0,921 + 0.002
0.871 + 0.001
0.844 + 0.007
0.818 + 0.008
Subject
0.859 = 0.006
0,841 = 0.011
0.771 + 0.008
0.787 = 0.010
0.692 : 0.006
Subject
0.956 + 0.002
0.949 = 0.005
0,887 + 0.009
0,898 + 0.007
0.799 + 0.021
Subject
0.890 = 0.006
0.841 = 0.006
0.822 = 0.008
0.829 = 0.007
0.619 # 0.027
Suhject
0.850 0.031
0.825 0.024
0.635 + 0.039
0.823 # 0.014
0.601 # 0.036
Subject
0.686 + 0.036
0.645 +
0.642 + 0.009
0.622
0.556 + 0.021
Subject 10
0.879 = 0.008
0.868 + 0.008
0.739 + 0.011
0 0.764
0.695 + 0.013
Average
0.862 + 0.015
0.841 + 0.016
0,767 + 0.017
0,795 + 0.014
0.683 + 0.017
Speech/Non-Speech
Subject
Included
Held-out
Brant
PopT
Random Init.
Subject
0.950 + 0.003
0.942 + 0.004
0.844 + 0.002
0.853 + 0.006
0.808 + 0.024
Subject
0.874 # 0.005
0.863 # 0.004
0.745 # 0.008
0.799 # 0.006
0.691 = 0.013
Subject
0.977 = 0.003
0,952 = 0.001
0.749 = 0.012
0.897 = 0.008
0.675 # 0.034
Subject
0.910 0.004
0.863 0.012
0.702 0.014
0.763
0.505 # 0.020
Subject
0.827 = 0.015
0.798
0.624 + 0.022
0.766
0.602 = 0.027
Subject
0.708 = 0.047
0,701 0.012
0,526 + 0.029
0.0100
0,568 + 0.018
Subject 10
0.839 0.01
0.843 0.007
0.644 + 0.026
0.757 0.020
0.540 + 0.016
Average
0.869 + 0.016
0.852 + 0.013
0.691 + 0.017
0.775 + 0.016
0.627 + 0.018
150Hz) ranges. We then computed the reconstruction error the filtered signals and compared the performance between the low- and figh-frequency ranges. Below we present the results of our analysis for encoding/masking pairs (channels/channels, parcels/parcels, lobes/lobes). We computed the normalized mean-squared error, NMSE (i.e., MSE normalized by the variance of the target signal). better metric for comparing the two regimes since the high-frequency filtered
We found that this signal had lower amplitude than the low-frequency filtered signal (due to the 1/f nature of neural activity). The results are presented in Table and, expected, the reconstruction error for the high-frequency range is higher than for the low-frequency range.
30

---

Table 12: Per-subject channel reconstruction performance for three encoding/masking pairs.
Chans=channels.
Chans./Chans
Parcels/Parcels
Lobes/Lobes
Subject
MSE
R²1
MSEJ
R²t
MSEJ
R²1
Subject
0.44
0.56
0.35
0.65
0.82
0.18
Subject
0.40
0.60
0.40
0.91
0.10
Subject
0.54
0,46
0.40
0.60
0.71
0.29
Subject 4
0,47
0,53
0,48
0.52
0.89
0.11
Subject 6
0.37
0,63
0.51
0.49
0,94
0.06
Subject 7
0.21
0.79
0.40
0.60
0.85
0.14
Subject 10
0.35
0.65
0.36
0.64
0.86
0.14
mean s.e.m.
0.40 0.04 0.60 1: 0.04
0.41 1: 0.02
0.59 0.02
0.85 0.03
0.15 1:
Table 13: Channel reconstruction results within low- and high-frequenny ranges averaged across 5 finetuning seeds (mean NMSE +/- s.e.m.). Low-frequency NMSEJ
Model (Encode/Mask)
High- NMSE
Channels/Channels Parcels/Parcels
0.380 + 0.005
1.370 0.029
0,405 + 0.007
1.246 = 0.015
Lobes/Lobes
0.879 0.012
1.163 0.018
J Architectural ablations
We performed architectural ablations evaluate our choice of temporal encoder and interleaved space-time since-time attention. Results are presented below.
J.1 Choice of temporal encoder
Here we chose to dilated CNN
temporal encoder based prior works modeling
uni/multivariate time-series activity (29 To evaluate this choice, we compared the downstream classification performance of our model when using one of three possible temporal encoders: (I) a dilated CNN (default), (2) a linear projection (i.e., a linear layer the size of our patch length L, similar [5],, and (3) single layer univariate CNN with kernel size (to match the dilated CNN kernel size). In Appendix Table [4] we present the downstream classification performance on the language tasks (average AUC over pretraining and finetuning seeds) for the parcel/channels encoding/masken
configuration presented in Table [ Our results show that the dilated CNN encoder achieves higher performance than the other two temporal encoders.
J.2 Choice of combined vs. separate space-time attention modules
We made the decision to use interleaved tokens (i.e., the vector) and a single space-time attention module enable our model to better learn spatiotemporal relationships between channels. Here we empirically show that this interleaved approach outperforms having separated attention modules through an ablation study: we performed ablation on the S vector by first passing our sequences through a temporal attention module (i.c., self-attention on the patches within each channel indepen- dently) and then passing the output into a spatial attention module (i.e., self-attention on the channels within each patch) resulting in separated attention modules similar to prior works [15] the fairness of comparison, we split our 12-layer transformer into 6-layer transformers, each with 4 attention heads and the same hidden dimension of 64. In Appendix Table we present the results for the parcels/channels encoding/masking pairing (used Table T); AUC scores are averaged
I8 19]. For
across 3 pretraining and 5 finetuning seeds for each of the 7 test sessions. Our results show that the combined attention module achieves higher downstream performance.
31

---

Table 14: Downstream classification performance of pretrained parcels/channels BaRISTA models using different temporal encoders (mean +/- s.e.m.).
Temporal Encoder
Sentence Onset
Speech/Non-Speech
Linear projection
0.776 + 0.01
0.763 0.01
Single layer univariate CNN (kemel=3)
0.749 = 0.013
0.752 = 0.012 0.847 = 0.010
Dilated CNN (default)
0.836 + 0.010
Table 15:
Downstream classification performance of pretrained parcels/channels BaRISTA models using either combined (interlcaved) or separate attention modules (mean +/- s.c.m.).
Attention Module
Sentence Onset
Speech/Non-Speech
Separate attention
0.828 + 0.010
0.825 + 0.011
Interleaved attention (default)
0.836 + 0.010
0.847 + 0.010
K Extended downstream evaluations on chronological splits and additional tasks
As mentioned in Section 4. to extend the evaluation of our model also performed an alternative (second) evaluation of our main results by generating our downstream segments and creating the train/valid/test splits differently from what was described in Appendices A and B.2 with the goal of increasing the amount of labeled data available for downstream training.
Our main evaluation used non-overlapping segments that were randomly assigned to train/valid/tes. splits. Since enforcing no overlap requires dropping some of the annotated segments, in our second evaluation we relaxed the constraint on generating positive-labeled non-overlapping segments for the downstream language tasks, while also generating the train/valid/test splits chronologically in time to avoid any overlap between these splits. Specifically, we again generated 3-second center word-aligned neural segments, but allowed for these segments to overlap. As a reminder, positive here denotes segments that correspond to sentence onset speech-containing audio; negative-labeled samples were generated as before (Appendix (A). By allowing for overlaps, we were able to better utilize the richly-annotated information provided by the Brain Treebank dataset [35) and not restrict ourselves to only a subset of the language-related features. However, to prevent any possible overlap between training and test data due to random split assignments, instead generated 5 different 80/10/10 train/valid/test splits by partitioning the data chronologically (e.g.. the beginning of the recording session for training vs. the end for testing).
In addition to providing a second evaluation for the language-related downstream tasks, this alter- native evaluation method provided enough labels for us to also add 2 more downstream tasks: (i) classification of word loudness or softness, and (ii) discrimination of high vs. low magnitude global optical flow in the video stimuli [35]. For these tasks, we again generated center word-aligned segments, each with an associated volume and optical flow measure, and use the oop/bottom-quarile approach described in [19 to generate positive and negative labels.
We evaluated the same models from Table on all downstream tasks using the new chronological splits. In Appendix Table 16 we report the average AUC over all test sessions, finetuning seeds, and chronological splits (n = 175 points total). The conclusions are the same as before: our model's flexibility in using larger spatial encoding scales during pretraining improved downstream classification performance compared to the bascline models across all tasks. To further verify that our results were consistent with those in Section 4.2 used a Wilcoxon signed-rank test to assess significance. First, we observed that our channel-leve model and PopT were not statistically different in these 4 tasks, but our channel-level model was significantly better than Brant in 3 tasks (p-value<< 1c 3). i.e., all but the sentence onset task in which they were not statistically different. Second, importantly, our parcels/channels pretrained model
significantly better than both the SOTA baseline models across all 4 tasks (p- (p-value<< le 5) for both Brant and PopT.
We then investigated if the same trends observed in Table 2 regarding the choice of spatial encod- ing/masking pairs held with the new chronological splits across the 4 downstream tasks. To do so, evaluated the same 9 models pretrained using distinct spatial encoding/masking combinations
32

---

Table 16: Classification results (mean AUC + s.e.m.) across chronological split and finetuning seeds. Best-pe rest-performing model is bolded and second benttdeettt is underlined model. chans=channels,
RI=random initialization.
Model
Sentence Onset
Speech/Non-Speech
Volume
Optical Flow
Brant 15]
0.772 0.009
0.650 + 0.009
0,571 = 0.006
0.531 + 0.005
PopT+Brainbert 19
0.776 + 0.009
0.724 + 0.011
0.584 = 0.006
0.551 + 0,006
BaRISTA (chans/chans)
0.778 + 0.009
0.733 + 0.011
0.609 0.007
0.562 + 0.006
BaRISTA (parcels/chans)
0.853 0.007
0.834 + 0.010
0.698 0.009
0.585 + 0.006
BaRISTA (RI, chans)
0.693 0.009
0.594 + 0.008
0.566 = 0.005
0.529 + 0.004
BaRISTA (RI, parcels)
0.697 + 0.009
0.608 + 0.009
0.564 + 0.005
0.527 + 0.003
with the 3 different spatial scales described in Section 3.1 initialization results in Appendix Table [7
We present both finetuned and random
As in our main evaluation, we find that the choice of spatial scale has a significant impact on the performance of the pretrained model, with spatial encoding scale having greater impact than spatial masking scale. To verify this observation, we again performed a two-way ANOVA [36] with spatial encoding and spatial masking as the independent variables and the AUC values as the dependent variable - Bonferroni correcting p-values to account for the 4 downstream tasks. The results of the ANOVA were consistent with those in the first evaluation, revealing that both independent variables had statistically significant effects on the downstream tasks with only I of 4 tasks (optical flow) demonstrating significant interaction between encoding and masking (sentence onset: encoding
p 1e 3, masking p < 1e 3; speech: encoding p le 3, masking p < 1e 2; volume: encoding < 1 - 3, masking < le 2; optical flow: encoding < le 3, masking < le 2, interaction p le
Table 17: Downstream classification results of different spatial encoding/masking configurations
(mean AUC +/- s.e.m.) across chronological splits and 5 finetuning seeds. Best results in bold.
Mask
Channels
Parcels
Lobes
Random Init.
Encode
Channels
0.778 + 0.009
0.710 + 0.008
0.680
0.010
0.693 + 0.009
Sentence
Parcels
0.853 + 0.007
0.838 + 0.009
0.842 0.008
0.697 + 0.009
Onset
Lobes
0.832 + 0.008
0.839 + 0.008
0.829 0.008
0.69 0.009
Channels
0.733 + 0.011
0.643 + 0.010
0.669
0.010
0.594 + 0.008
Speech
Parcels
0.834 + 0.010
0.829 + 0.010
0,828 + 0.011
0.608 + 0.009
Lobes
0.820 1 0.010
0.824 1 0.010
0.812 + 0.011
0.606 1 0.009
Channels
0.609 + 0.007
0.572 + 0.005
0.555 + 0.005
0.566 + 0.005
Volume
Parcels
0.698 + 0.009
0.698 + 0.01
0,676 0.009
0.564 + 0.005
Lobes
0.693 +:2 0.011
0.683 + 0.010
0.677 0.008
0.565 1 0.005
Channels
0.562 + 0.006
0.527 = 0.003
0.519
0.003
0.529 + 0.004
Optical
Parcels
0.585 + 0.006
0.582 + 0.007
0.582 0.006
0.527 + 0.003
Flow
Lobes
0.581 + 0.006
0.578 + 0.007
0.571 0.006
0.529 + 0.003
In Appendix Table 18 we report the average number of training, valid, and test samples for each of the 4 tasks when using chronological splits (compare with Appendix T As before, positive and negative labels were balanced prior to generating the splits.
K.1 Data scaling and generalizibality of chronological splits
Similar to Section 4.5 we assessed BaRISTA's ability to generalize completely unseen subjects for our second evaluation method on the sentence onset and speech vs non-speech tasks. Results are provided in Appendix Table 19 Consistent with the first evaluation method (Table 4. we observe minor performance degradation as expected, while still achieving higher performance compared to baselines. Additionally, we also examined the scalability of downstream performance when pretraining using 5%, 10%, 25% 50%, and 75% of the total available pretraining data, and observed
33

---

Table 18: For each hold-out session, the number of training, validation, and test segments used in the downstream tasks averaged across the 5 chronological splits. Note, these counts correspond to the test sessions in Appendix 5
Sentence Onset & Speech/Non-Speech
Volume &
Optical Flow
Subject
Train
Valid
Test
Train
Valid
Test
Subject
2469
308
308
4086
510
510
Subject 2
1626
202
203
2560
318
319
Subject 3
1066
132
133
4048
506
506
Subject 4
1276
158
159
2540
316
317
Subject 6
823
102
102
1000
124
125
Subject
650
80
81
4049
506
506
Subject 10
944
116
117
3336
416
417
performance improvement with more pretraining data (Appendix Figure similar our results for the first evaluation method (Figure
Table 19: Generalizability
subjects holds for chronological folds: downstream results of our parcels/channels model on chronological folds evaluation for both standard pretraining and pretraining with the target subject completely held-out (mean +/- s.c.m.). Results are averaged across 5 finetuning seeds and 5 chronological folds.
Model
Sentence Onset
Speech/Non-Speech
BaRISTA (parcels/channels, Held-out) BaRISTA (parcels/channels, Included)
0.841 0.007
0.819 0.010
0.853 + 0.007
0.834 + 0.010
Speech/Non Speech ROC-AUC
Sentence Onset ROC-AUC AUC
0.90
0.85
0.80
0.80
0.75
0.75
0.70
0.65
0.60
0.60
5%
10%
25%
50%
75%
100%
10%
25%
50%
75%
100%
Data Percentage
Data Percentage
Figure 9: BaRISTA'S downstream classification performance chronological folds also scales as a function of pretraining data size. Downstream classification results of our best model using different amounts of pretraining data, denoted percentage of the full training data. Lighter scatter points represent the average performance of different subsets of training sessions over chronological splits and 5 finetuning seeds; we used 5 different random subsets per percentage. The darker point is the average across these subsets.
L Single-session vs. multi-session models
There has been significant progress developing models of invasive neural recordings for various modalities such as spikes, local field potentials, and iEEG, for example using state-space models 55 or deep learning approaches 66). Many of these approaches have primarily focused on training models for each individual recording session separately. Recently, developing transformer- based neurofoundation models for multi-session training has gotten significant attention for such neural modalities 17-10 19 due to their potential to enable accurate and generalizable modeling of neural datasets by aggregating data across sessions and subjects. Here we show that the scale of spatial encoding and masking are important toward developing neurofoundation models of multiregional human intracranial neural activity and enhancing their downstream decoding performance.
34


Reviews:
[{"review_id": "MZReAwuVV8", "reviewer": "Reviewer_xW3g", "summary": "This paper introduces BaRISTA, an architecture that learns to interpret brain activity from iEEG recordings, using both spatial and temporal characteristics of the data. The authors introduce a masked latent reconstruction task to pre-train their model. They then investigate the performance impact of different spatial encodings, evaluating on various downstream tasks. The paper considers three distinct spatial scales, which are (from finest to coarsest): channel, atlas parcellation, and lobe. For the two coarser scales, the spatial encodings of channels within the same parcel or lobe are clustered together. The proposed architecture contains a dilated-convolution tokenizer, followed by a transformer-based component, whose outputs are fed into a downstream decoder. In the masked reconstruction task, the model receives a partially hidden input that it attempts to complete. The evaluation is performed on classification tasks, as well as a per-channel reconstruction task, with publicly available data.", "questions": "The authors are encouraged to consider the following items:\n- Defending the claim that different scales are beneficial: The first action item is to attempt retraining or fine-tuning baselines (wherever possible) under different scales. The second is to complete Tab. 1 and Tab. 3 of the main paper with the missing scale combinations for BaRISTA. If any of the above are not applicable, please explain why that is so.\n- Motivating and ablating the design choices of Sect. 3.2: Particular emphasis should be given to the S-vector and the attention encoder module. This is a core design component in the pipeline that requires further understanding. The first action item is to replace the single attention module with separate attention modules for space and time, and then compare the performance. The second action item would be to explain any additional benefits that motivate this design (e.g., computational savings, if any). Motivating or ablating the tokenizer and readout head designs would also be helpful.\n- Although there is some information presented in App. B of the supplement, a more explicit elaboration of how fairness in the baseline comparisons is ensured would be appropriate. The total number of gradient updates for pre-trained and fine-tuned BaRISTA models should ideally be the same as that of the baselines. The action item is to present accurate counts of training steps taken for each compared system, including a breakdown of how many steps were taken during pre-training versus fine-tuning.", "limitations": "none identified", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper's core contributions, as presented, are technically sound.\n- The proposed tasks are well-defined, and the architecture follows established methods in ML.\n- The theoretical analysis is sufficient for understanding the proposed ideas.\n- The supplemental materials provide many clarifying details and results (especially Tab. 3, Tab. 6, and Fig. 6).\n\nWeaknesses:\n- In processing the signal, the proposed pipeline essentially performs geometric clustering of channels based on a pre-defined atlas, and by using different atlases of varying granularity, we obtain different scales for the system. However, these clusterings rely on handcrafted models that, in turn, depend solely on anatomical structure. No concrete reasons are provided why such clusterings could not be learned directly, e.g., in the token space. This is partially addressed in Sect. 5, where [11] is cited as an approach for functional clustering. Given a downstream task, the question of where on the spectrum of possible clusterings lies the optimal (and how to find it), remains open.\n- The use of a neurobiological prior (atlases) suggests that channel clustering may be universally beneficial for the examined tasks. If that is the case, the baselines' performance might show a similar improving trend when re-optimized under the same training protocol. However, since they are only trained on channel data and are not retrained at different scales, it is difficult to make a comparison.\n- Several ideas introduced in Sect. 3.2 are not clearly motivated. For the encoder, it is not immediately obvious why a dilated CNN module is the best approach to capture continuous and/or oscillating features; several other options could be considered here. Similarly, the decoder being a linear layer is one possible choice, but could be ablated against differently sized MLPs, or even recurrent networks. Finally, and perhaps most crucially, there is the interleaved S vector and its associated attention mechanism. It is unclear what the benefit is of having a single attention module for space and time, as opposed to separate modules, and how it compares to other attention paradigms used in previous literature.", "was_revised": true, "final_justification": "The authors addressed all points raised.", "rebuttal": "We thank the reviewer for their thorough comments, recommendations, and excellent questions. We reply inline below.\n\n> retrain or fine-tune baselines under different scales\n\nThis is an interesting point. First, we clarify that there was no notion of spatial encoding based on metadata (e.g., channel coordinates) in the iteration of Brant against which we compare [1]; vanilla index-based positional encoding [2] was used. Introducing spatial encoding to Brant would be adding new functionality and would require validation against the original pretrained model. The fairest way to perform such a comparison and validate the new functionality would be to pretrain on the same dataset the existing model was pretrained on. However we did not have access to that dataset nor did we have access to their pretraining code. Thus retraining/finetuning/validating Brant with different spatial scales would be outside of the scope of this work. The request to pretrain PopT with different spatial scales is possible, as we have the dataset the model was originally pretrained on and their code. We tried pretraining PopT with region-level encoding by replacing the $(x,y,z)$ encoding used in the original work with parcel-level spatial encoding. Using the default pretraining settings available on PopT's GitHub, we were unable to successfully pretrain the model: performance on downstream tasks was chance (~0.5 AUC on average). We suspect that this result may be due to the difference in pretraining tasks (we use a masked reconstruction task while PopT was trained on discriminative tasks). In order to train PopT to support different spatial scales, the pretraining task and/or optimization configurations would most likely need to be modified. Considering this result, we will revise the manuscript to clarify that the impact of spatial scales was seen for masked reconstruction pretraining, and that more work needs to be done to investigate the impact of spatial encoding in other pretraining tasks (e.g., discriminative tasks).\n\n> complete Tab 1 & Tab 3 of the main paper\n\nWe clarify that Tab 2 is the complete version of Tab 1. We chose to present the comparisons against baseline models separately from the full 3x3 scale combinations in Tab 2 for the sake of readability; we will revise the text to make the relationship between Tab 1 and 2 clearer. Below we present the results of completed Tab 3 which we will add to the manuscript. We will revise section 4.4 to reflect these more comprehensive results, specifically noting that the granularity of spatial encoding may impact the model’s ability to learn individual channel properties, as evidenced by the drop in channel reconstruction performance with the lobe-level spatial encoding model.\n\n|Model|MSE $\\downarrow$|$R^2 \\uparrow$|\n|-|-|-|\n|Channels/Chans|0.397 &plusmn; 0.040|0.603 &plusmn; 0.040|\n|Chans/Parcels|0.354 &plusmn; 0.032|0.646 &plusmn; 0.032|\n|Chans/Lobes|0.478 &plusmn; 0.036|0.522 &plusmn; 0.036|\n|Parcels/Chans|0.391 &plusmn; 0.019|0.609 &plusmn; 0.019|\n|Parcels/Parcels|0.413 &plusmn; 0.023| 0.587 &plusmn; 0.023|\n|Parcels/Parcels|0.417 &plusmn; 0.027|0.583 &plusmn; 0.027|\n|Lobes/Chans|0.753 &plusmn; 0.039|0.247 &plusmn; 0.039|\n|Lobes/Parcels|0.951 &plusmn; 0.014|0.049 &plusmn; 0.014|\n|Lobes/Lobes|0.853 &plusmn; 0.029|0.147 &plusmn; 0.029|\n|Random init (chans encoding)|0.566 &plusmn; 0.028|0.434 &plusmn; 0.028|\n|Rand init (parcel enc)|0.846  &plusmn; 0.028|0.155  &plusmn; 0.028|\n|Rand init (lobe enc)|0.965 &plusmn; 0.022|0.035 &plusmn; 0.022|\n\n> Motivate & ablating the design choices of Sect. 3.2; explain additional benefits\n\nThe choice to use a dilated CNN was based on prior works modeling uni/multivariate time-series activity [3-5]. We have now also performed an ablation on the temporal encoder choice and present results comparing against a linear projection (linear layer the size of our patch length $L$, similar to [1]) and a single layer univariate CNN with kernel size 5. For all models we used the best-performing encoding/masking pair presented in Tab 1 (parcels/channels) and report average AUC over 3 pretraining & 5 finetuning seeds (as in Tab 7). We will complete a more comprehensive ablation (use all spatial encoding/masking scales) and add it to the supplementary materials. Our results show that dilated CNN encoder achieves higher performance. With respect to computational savings: a dilated CNN is slower to train than a fully connected linear model but requires fewer parameters (302K for the linear projection vs 71.8K for the dilated CNN in our implementation). An RNN or SSM would have also been suitable alternatives but the training time for RNNs/SSMs is longer than that of a CNN. Since we wanted to pretrain several encoding/masking configurations in our analysis, we chose to optimize running time for the sake of experimentation and therefore opted for a CNN over a RNN/SSM. We will include this discussion on to computation savings/design choices in the manuscript.\n\n|Encoder (Parcels/Chans)|Sentence Onset|Speech|\n|-|-|-|\n|Linear projection|0.776  &plusmn; 0.01|0.763 &plusmn; 0.01|\n|Single layer univariate CNN (kernel=5)|0.749 &plusmn; 0.013|0.752 &plusmn; 0.012|\n|Dilated CNN (default)|**0.836 &plusmn; 0.010**|**0.847 &plusmn; 0.01**|\n\nWe decided to use interleaved tokens (i.e., $\\mathbf{S}$ vector) to give our model the capacity to learn spatiotemporal relationships between the channels within the same attention module (rather than each axis individually). We now show that this interleaved approach outperforms having separated attention modules through an ablation study in which we first pass our sequences through a temporal attention module (i.e., self-attention on the patches within each channel independently) and then through a spatial attention module (i.e., self-attention on the channels within each patch), similar to [1,6,7]. For fairness of comparison, we split our 12-layer transformer into 2 6-layer transformers each with 4 attention heads and the same hidden dimension 64. We used the best performing encoding/masking pair in Tab 1, but will add a more comprehensive analysis with all pairs in the supp materials. AUC scores are averaged across 3 pretraining & 5 finetuning seeds for each of the 7 test sessions. Our results show that the combined attention module achieves higher downstream performance.\n\n|Model|Sentence Onset|Speech|\n|-|-|-|\n|Interleaved $\\mathbf{S}$|**0.862 &plusmn; 0.016**|**0.869 &plusmn; 0.016**|\n|Separate attention modules|0.828 &plusmn; 010|0.825 &plusmn; 0.011|\n\nWe use a lightweight linear decoder to evaluate the quality of the learned embeddings/representations, as is common practice [8,9]. We were concerned that a more complex downstream decoder would have boosted the downstream performance regardless of the embedding/representation quality. We will clarify our choice of decoder in the text.\n\n> counts of training steps taken\n\nWe thank the reviewer for the recommendation. For pretraining our model had 19,500 update steps, PopT pretraining involved 500,000 update steps, and Brant had 750,000 update steps. Because we ran training for a fixed number of epochs (as reported in Supp D.2), the total number of finetuning update steps was dependent on the downstream task and session (i.e., the number of training segments available). The average number of updates for BaRISTA across 7 test sessions and 2 downstream tasks was 252 (range 393 steps), for PopT it was 629 (range 982), and for Brant it was 1258 (range 1964). We chose the larger number of update steps for the baseline models to ensure they converged because we wanted to validate our model against their best performance. We trained Brant the longest as its finetuning learning rate was 1e3 times smaller than BaRISTA’s and 1e2 times smaller than that of PopT’s; we note that the LRs used reflect the rates used by the authors in the original works. We will include these numbers in the revised manuscript.\n\n> No concrete reasons are provided why such clusterings could not be learned directly…question of where on the spectrum of possible clusterings lies the optimal and how to find it remains open\n\nWe agree with the reviewer that future work can explore using clusterings that are learned in a data-driven way, e.g., based on functional relationships or signal correlations. However, the use of handcrafted features in this work does not necessarily reflect a limitation in our model. We designed our model (1) to support both hand-crafted and data-driven (with minor code modifications) channel clusterings, and (2) to enable hypothesis testing by allowing users the flexibility to specify specific clustering labels based on side meta-information, which may not come from the data at hand but instead reflect domain knowledge (e.g., brain anatomy) and/or information from other modalities (e.g., structural images). For example, by utilizing clusters based on brain regions or functional relationships, users of our model can assess the significance and/or involvement of different regions or functional brain networks on various downstream tasks of interest. We will further clarify these points in the Discussion.\n\n**References**\n\n[1] D. Zhang, et al. Brant: Foundation Model for Intracranial Neural Signal\n\n[2] A. Vaswani et al. Attention Is All You Need\n\n[3] A. Oord, et al.. WaveNet: A Generative Model for Raw Audio\n\n[4] Z. Yue, et al. TS2Vec: Towards Universal Representation of Time Series\n\n[5] S. Bai, et al. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling\n\n[6] G. Mentzelopoulos et al. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects\n\n[7] G. Chau et al. Population Transformer: Learning Population-level Representations of Neural Activity\n\n[8] F. Pei et al. Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity\n\n[9] T. Chen et al. A Simple Framework for Contrastive Learning of Visual Representations"}, {"review_id": "95Jut9SHcf", "reviewer": "Reviewer_54tZ", "summary": "The authors develop a model, BaRISTA, in which varying spatial scales are used to decode iEEG activity in a complex task. They achieve noticeable improvements in performance via flexibility in larger spatial encoding scales, a novel addition to previous models using only channel-level spatial encoding. Instead of randomly selecting channels during model training, the authors chose spatial scales for the masked reconstruction in latent space, thus resulting in a high fidelty model.", "questions": "Do spatial scales vary given the classification task/experimental stimuli? E.g. smaller spatial scale needed to encode visual features in environments vs larger spatial scale for more cognitively demanding tasks such as the language tasks here?\n\nAlthough the current model performs quite well, it would be interesting to test if the same performance gains could be achieved using a second dataset in which people engage in a far less cognitively demanding task (such as viewing images of scenes). I.e. Are the benefits of varying spatial encoding less advantageous in situations where shorter, less flexible temporal and spatial scales are needed to detect changes in one's environment?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Pros:\n\t\n* One of the major achievements of this model is that the authors consider varying spatial scales of human brain (via iEEG) in modeling neural activity, which is very important when capturing high-level cognitive processes.\n\n* The authors use a dataset well-suited to capture spatially complex brain patterns. Movie watching, even at shorter timescales, give rise to high variability within and across individuals' spatial and temporal patterns of activity.\n\n* Varying spatial encoding substantially outperforms other models and non-variable encoding methods.\n\n### Cons:\n\t\n* Please clarify earlier on in your intro the working definition of neural tokenization. Although the context throughout the paper resolves some questions about the term, an earlier definition would improve readability. \n\n* The current model was tested on a single dataset, which is understandable given the nature of iEEG datasets. As it stands, I wonder if  the flexibility of spatial encoding is only marginally advantageous in a task that is less cognitively demanding in which rich information processing is not needed. A simulation testing the current model's parameters and the necessity of spatial encoding flexibility could be very useful here.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their recommendations, questions, and comments. We address these below inline.\n\n> Do spatial scales vary given the classification task/experimental stimuli? E.g. smaller spatial scale needed to encode visual features in environments vs larger spatial scale for more cognitively demanding tasks such as the language tasks here? …it would be interesting to test if the same performance gains could be achieved using a second dataset in which people engage in a far less cognitively demanding task (such as viewing images of scenes). I.e. Are the benefits of varying spatial encoding less advantageous in situations where shorter, less flexible temporal and spatial scales are needed to detect changes in one's environment?\n\nThis is a great question and will be an interesting hypothesis to test in follow-on work. Based on the analyses performed here (i.e., the language-related decoding task and the channel reconstruction task), we hypothesize that the reviewer’s expectation may indeed be the case and that different applications will benefit from different spatial encodings – which can be assessed by incorporating and testing each scale using our model and its flexible spatial encoding capability. For example, we observed that models pretrained using smaller encoding spatial scales were able to perform channel-level neural reconstruction more accurately than models pretrained with larger encoding spatial scales. On the other hand, larger spatial scales helped more with language-related classification tasks.\n\nTo address the reviewer’s question, we decided to try decoding one of the visual features (global optical flow magnitude) provided by the Brain Treebank dataset [1]. We used the two encoding/masking configurations highlighted in Table 1 (i.e., parcel/channels and channels/channels), and found that the parcel/channels model outperformed the channel/channel model. Our results are presented in the table below (average AUC across the 7 test sessions and 5 finetuning seeds each); as baselines we compared against PopT and randomly initialized BaRISTA models with channel-level and parcel-level encoding.\n\n|Model|AUC &plusmn; standard error of measure|\n|-|-|\n|Random initialization (channels)|0.525 &plusmn; 0.008|\n|Random initialization (parcels)|0.527 &plusmn; 0.006|\n|Channels/Channels|0.561 &plusmn; 0.010|\n|Parcels/Channels|**0.604 &plusmn; 0.013**|\n|PopT|0.549 &plusmn; 0.009|\n\nThese results indicate that the flexibility of spatial encoding offered by BaRISTA is beneficial for decoding visual stimuli as well. However, it will be very interesting to use the flexibility of spatial encoding enabled by our model for hypothesis-driven testing of spatial encoding scales on a variety of downstream tasks that exhibit different degrees of complexity in the future – including simpler sensory tasks. We will include discussion of this point as a possible future research direction in our manuscript.\n\n> Please clarify earlier on in your intro the working definition of neural tokenization.\n\nWe thank the reviewer for their recommendation. We will revise the manuscript accordingly.\n\n**References:**\n\n[1] Christopher Wang et al. Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli."}, {"review_id": "AEbQKvogQC", "reviewer": "Reviewer_uiXd", "summary": "In this work, the authors present a self-supervised pretraining framework for stereotactic EEG (sEEG) data. Unlike prior methods, their approach introduces a flexible mechanism for incorporating spatial information about electrode locations, allowing the model to be informed at varying levels of granularity, ranging from individual electrodes to anatomical parcels or entire lobes. Another innovation of the framework is a novel reconstruction-based task in which the reconstruction targets are selected based on spatial meta-information, rather than through random sampling. The authors evaluate their method on the publicly available Brain Treebank dataset. They demonstrate that their pretraining strategy enhances decoding performance when finetuned on two downstream tasks, surpassing the performance of leading pretraining baselines, including PopT and Brant. Further analysis reveals that the spatial scale at which location information is encoded significantly affects downstream performance: encoding at the parcel level yields better results than encoding at the electrode level. Importantly, they also show that parcel-level representations retain nearly all channel-level information, as evidenced by performance on a channel-level reconstruction task. In supplementary experiments, the authors show that their framework scales with more training data and that the learned representations transfer effectively to subjects not seen during pretraining.", "questions": "Major:\n1. Brain Treebank is a dataset that has labels to enable testing your framework on more downstream tasks than the ones used in this work. For example, PopT also evaluates their model on volume, and pitch discrimination tasks as well. Please justify why you did not evaluate your model on those tasks or evaluate the utility of your pretrained model on those tasks as well. See details here: https://arxiv.org/abs/2406.03044.\n2. Based on Table 3 of the Supplement, it seems like your model is substantially smaller than the baselines it is compared against (Brant & PopT). Please justify why you did not parameter match your model with your baselines or even better, report the results shown in table 1 of the main text after training versions of Brant and PopT that have approximately the same number of parameters as your model. \n3. Based on Figure 4 of the main text, it looks like your pretraining strategy does a good job at capturing the low-frequency information in the sEEG signal. However, it looks like the model is doing a bad job at capturing the high-frequency details of the signal. This is a common problem encountered when pretraining on reconstruction tasks with MSE Losss (since there is a ~1/f relationship between frequency and power in biological signals). Note: This observation might be exacerbated due to the Savitzky-Golay filter used for plotting purposes. \n    1. Do you have a sense of whether your modelling approach is better at capturing high-frequency information (i.e. frequency in the range 70-150 Hz) compared to Brandt and PopT? A relatively easy way to check this would be: using the pretrained model, (1) reconstruct a large amount of sEEG traces (similar to what you describe in section 4.4 of your main text), (2) filter the reconstructed and ground truth signal in the frequency range 70-150 Hz (and notch filter line noise, if necessary), (3) calculate the MSE between the reconstructed and ground truth signals. Report the results of this procedure for your model as well as Brant and PopT.   \n    2. I believe this work would benefit by mentioning this limitation in the discussion as a directions for future work.\n\nMinor\n1. Throughout the text, the authors use \\mathcal{R} to denote the set of real numbers. I believe a more standard notation would be to use \\mathbb{R}. Consider replacing.\n2. It would be helpful for the reader if you included in the captions of tables (or at least of table 1) what does bold, underline, and * indicate.\n3. Typo in lines 10-11 of Supplement. I believe the main text mentions that 17 sessions were used for pretraining but supplement mentions 16. \n4. Please explain why you chose to not show Table 7 of the supplement in place of Table 2 in main. Consider replacing Table 2 in main with Table 7 of supplement, as the results shown in Table 7 of supplement are a more robust estimate of your model's performance.\n5. The results shown in Supplement E and F are really strong. I believe that moving them to the main text on your manuscript (possibly after revisions where you get an additional page for the main text) would help the reader appreciate the utility of your approach.", "limitations": "Yes. For a suggestion please see major question 3.2.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Flexibility to incorporate spatial information at different spatial scales. To my knowledge, parcel or lobe level encoding has not been used to inform models with spatial information during pretraining before.\n2. Insight into the spatial scale of encoding. The majority of previous works have encoded spatial information at the electrode level. However, the results shown in this work support that encoding information at larger scales (parcel level) result in better models. This could steer the field towards adopting a better positional encoding scheme (tailored to sEEG) in future works.\n3. The introduced framework + architecture scales to large amounts of data and is transferable to new subjects. \n4. The manuscript is well written and easy to follow.\n\nWeaknesses:\n1. Limited evaluation. While the authors show their pretraining strategy on a large dataset, they only evaluate the utility of the learned representation on 2 downstream decoding tasks.\n2. Unclear whether this method is good at capturing \"high-frequency\" information of sEEG singals (see major question 3).", "was_revised": true, "final_justification": "After careful consideration of the authors’ rebuttal, I have decided to increase my score and recommend that this paper be accepted for publication.  \n\nFrom the initial submission, I believed that this work addressed an important and unresolved problem in neural decoding: how to best incorporate spatial information into models. The authors provide compelling evidence that parcel-level information may be more important than channel-level information.  \n\nDuring the rebuttal, the authors addressed two key concerns:  \n1. They demonstrated that their modeling approach captures low-frequency information better than high-frequency information, a common limitation for neural decoding models, and committed to including this point in their limitations and discussion section.  \n2. They conducted additional experiments showing that their models perform on par with PopT and Brant.  \n\nOverall, I believe this work offers valuable insights into an important, previously unaddressed problem in neural decoding: determining the optimal spatial scale for incorporating information into models. The results presented here can be highly useful for the field, and therefore I recommend this paper be accepted for publication.", "rebuttal": "We thank the reviewer for their feedback, questions, and comments. We reply inline below.\n\n> Limited evaluation\n\nThis is an important point. There are two reasons for not evaluating on those tasks. First, we clarify that the primary focus of our work was to assess the impact of different spatial scales during pretraining on downstream model performance. As such, we prioritized performing comparisons of our pretraining framework under different spatial encoding/masking configurations, rather than demonstrating our model’s utility/generalizability on a variety of downstream tasks. Second, we did not have sufficient labeled data for a per-subject pitch and volume discrimination task to enable robust model training. This is because we generated our downstream data segments differently than how the authors of PopT generated their data segments: as noted in our manuscript, we controlled our data splits to use *non-overlapping* segments to avoid any potential information leakage between training and test splits (especially since neural data is highly correlated over time), therefore enabling a rigorous evaluation of spatial scales. In contrast, the original segments in PopT did contain overlap. Because of our stringent no overlap requirement, combined with the quartile label generation (as used in [1,2] for the pitch/volume tasks), we did not have sufficient training samples left over to reliably train the model. Indeed, we observed chance performance across all models (PopT, Brant, our variants) for pitch/volume discrimination due to the small training set size. That is why we decided against evaluating on pitch/volume and instead focused on the speech/non-speech and sentence onset tasks.\n\nHowever, to address the reviewer’s concern about limited evaluation we decided to modify our segmentation scheme to generate more labels, while still addressing the information leakage concern. To do so, we modified our scheme to allow overlapping segments but selected train/valid/test splits across time (i.e., the beginning of the recording session for training vs. the end for testing), instead of randomly shuffling the segments and then selecting splits – as is done in PopT’s segment generation. Based on Fig. 9 of the Brain Treebank technical paper [3], we decided to decode volume which showed high SNR across multiple brain regions. We present our results (average AUC across 7 test sessions, 5 seeds each) using PopT and our pretrained models from Tab. 1 (channels/channels and parcels/channels). As we can see, similar trends on the effect of spatial scales hold for this downstream task as well, thus expanding our model’s evaluation.\n\n|Model|Volume|\n|-|-|\n|Channels/Channels|0.656 &plusmn; 0.015|\n|Parcels/Channels|**0.714 &plusmn; 0.016**|\n|PopT|0.639 &plusmn; 0.009|\n\n> \"high-frequency\" information of sEEG signals; is your model better at capturing high-frequency information compared to Brant and PopT; mention limitation in the discussion\n\nWe thank the reviewer for their excellent suggestion of an interesting analysis. We looked into our method’s ability to capture high-frequency information and indeed observed that the majority of the spectral power in the reconstructed signal was in the low-frequency range (approximately <=25Hz on average). As per the reviewer’s recommendation we performed the following analysis: we reconstructed 1162 3-second segments across the 7 test sessions which we low-pass filtered for the low-frequency (<40Hz) range and band-pass filtered for the higher frequency (40-150Hz) range. We then computed the reconstruction error on the filtered signals and compared the performance between the low and high-frequency ranges. Below we present the results of our analysis for 3 encoding/masking pairs (channels/channels, parcels/parcels, lobes/lobes). We computed the normalized MSE (i.e., MSE normalized by the variance of the target signal), which we found was a better metric for comparing the two regimes since the high-frequency filtered signal had lower amplitude than the low-frequency filtered signal (due to the 1/f relationship noted by the reviewer). As the reviewer hypothesized, the reconstruction error for the high-frequency range is higher than for the low-frequency range. \n\n|Model|Low-frequency NMSE $\\downarrow$|High-frequency NMSE $\\downarrow$|\n|-|-|-|\n|Channels/Channels|0.380 &plusmn; 0.005|1.370 &plusmn; 0.029|\n|Parcels/Parcels|0.405 &plusmn; 0.007|1.246 &plusmn; 0.015|\n|Lobes/Lobes|0.879 &plusmn; 0.012|1.163 &plusmn; 0.018|\n\nWe also compared against Brant by finetuning Brant for the channel reconstruction task, but model convergence was very slow and the performance was poor. We did not compare against PopT because we do not expect PopT to have the ability to faithfully reconstruct the channel activity. This is because PopT’s default temporal encoder (pretrained Brainbert) summarizes the input segments by computing an average temporal representation using the middle ~500ms (center 10 samples) – regardless of the input segment length. As such, we don’t believe the model has sufficient temporal information to reconstruct the channel activity for the full 3-second interval. We will revise the discussion section to explicitly make note of the limitation of high-frequency reconstruction and list it as a direction for future work. We will also include a more comprehensive supplementary section with the analysis discussed above.\n\n> justify why you did not parameter match your model with your baselines\n\nThis is also a good point. First we clarify that our primary goal in this work, as stated in the first response, was to investigate the impact of spatial scales on model performance. As such, we optimized our model to be lightweight enough to permit multiple pretrainings in a reasonable amount of time (to test the various encoding/masking combinations), while still maintaining sufficient modeling capacity to perform well on downstream tasks. This is why we did not parameter match the larger models. Indeed, the primary reason for including the comparisons against the baseline models in Tab. 1 was to demonstrate our proposed model’s & pretraining framework’s validity (by showing that we are able to achieve results comparable to prior published works when we used channel-level encoding). \n\nThe reviewer’s suggestion to train tiny versions of Brant and PopT is interesting. We do not have access to the original dataset the larger Brant model was pretrained on and thus don’t think it would be suitable to train smaller versions of Brant without this dataset. Moreover, Brant's pretraining code is not publicly released for us to pretrain a tiny version on a different dataset. However, having access to PopT's codebase (on GitHub) and the Brain Treebank dataset (used to pretrain PopT), we did try pretraining a smaller version of PopT. We identified a model configuration that allowed for ~1M (1790594) parameters by changing the hidden dimension from the default 512 to 64. However, pretraining using the default pretraining configurations provided by the authors was unsuccessful; we evaluated downstream performance on one task for a single test session and found the performance was close to chance. We also tried a ~3M (3772930) parameter model (hidden dimension 128), but pretraining was again unsuccessful with their default settings. Pretraining a tiny PopT may require more substantial changes to the pretraining configuration and/or the model configuration itself (i.e., number of layers, etc.) than is within the scope of this work. Finally, the larger Brant and PopT models are likely better than their tiny versions on the downstream tasks, as is generally the case with foundation models. Thus, we used the best possible performance from our baselines as a comparison point for our model.\n\n> explain why you chose to not show Table 7 of the supplement in place of Table 2 in main; consider replacing Table 2 with Table 7\n\nWe thank the reviewer for their question/comment. The primary reason for showing the best pretraining seed in Tables 1 and 2 was to maintain consistency with common practice and for fairer comparison with our baselines. In Table 1, we chose a single model to fairly compare against our baseline models which reflected just one pretraining seed (i.e., a single model); we used validation set decoding to select the model. Then, for the sake of consistency, we used the same criteria to select the best performing model within each category for Table 2. Finally, for completeness we presented the most robust estimate of our model’s performance for each encoding/masking combination in Table 7. Thus, we ended up choosing the simplified Table 2 to present in the main manuscript largely for ease of exposition/explanation and to maintain consistency with the criteria used for Table 1. Our results and conclusions are consistent in both Tables 2 and 7. We will additionally provide Table 7 in the main text of our manuscript following the reviewer’s suggestion.\n\n> Replace $\\mathcal{R}$ with $\\mathbb{R}$; include captions of tables (or at least of table 1); typo in lines 10-11 of supplement\n\nWe thank the reviewer for their recommendations and for catching the typo. We will make the suggested changes to improve readability of the text and to be more aligned with standard notation.\n\n> Supplement E and F are really strong; move them to the main text\n\nWe thank the reviewer for describing our results to be strong and for the suggestion. Upon manuscript acceptance we will move supplements E and F into the main manuscript. \n\n**References**\n\n[1] C. Wang et al. BrainBERT: Self-supervised representation learning for intracranial recordings.\n\n[2] G. Chau et al. Population Transformer: Learning Population-level Representations of Neural Activity.\n\n[3] C. Wang et al. Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli."}, {"review_id": "4sPqy1hl2I", "reviewer": "Reviewer_eXGi", "summary": "The authors introduce an iEEG-based JEPA-style foundation model, BaRISTA, which explores the impact of different spatial encoding/masking configurations on decoding performance. BaRISTA achieves SOTA performance on Brain Treebank dataset [1].\n\n**Dataset**: The authors evaluate their model on Brain Treebank dataset [1], which includes `10` subjects and `~55` hours iEEG recordings in total. The authors evalutes two sub-tasks, i.e., sentence onset detection and speech/non-speech classification.\n\n**Model**: The authors introduce an iEEG-based JEPA-style foundation model, BaRISTA, whose architecture is similar to LaBraM [2], composing a CNN-based tokenizer and a 12-layer Transformer to model temporal-spatial relationships among iEEG patches.\n\n**Experiment**: BaRISTA explores the impact of different spatial encoding/masking configurations on decoding performance.\n\n**References**:\n\n[1] Chau G, Wang C, Talukder S, et al. Population Transformer: Learning population-level representations of neural activity[J]. ArXiv, 2025: arXiv: 2406.03044 v4.\n\n[2] Jiang W B, Zhao L M, Lu B L. Large brain model for learning generic representations with tremendous EEG data in BCI[J]. arXiv preprint arXiv:2405.18765, 2024.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n1. BaRISTA explores the impact of different spatial encoding/masking configurations on decoding performance, and demonstrates that spatial encoding at larger scales than channel-level encoding improves downstream decoding performance.\n\n2. The text has a good structure and is well-written.\n\n**Weaknesses**:\n\n**Major**:\n\n1. The reproduced results for PopT+BrainBERT underperform the original paper [1] -- specifically, `0.80` versus `0.90` on sentence onset detection, and `0.78` versus `0.93` on speech/non-speech classification.\n\n2. I note that the supplementary [code](https://openreview.net/notes/edits/attachment?id=nbRHQko3FE&name=supplementary_material) for PopT [1] on OpenReview implements a lobe-based position embedding approach, which is not described in the PopT manuscript. Given the concerns raised in Major Weakness 1, the contribution of this embedding technique to model performance remains unclear.\n\n**Minor**:\n\n1. A minor implementation difference exists in trial length -- `3` seconds in this work versus the original paper's `5`-second segments.\n\n**References**:\n\n[1] Chau G, Wang C, Talukder S, et al. Population Transformer: Learning population-level representations of neural activity[J]. ArXiv, 2025: arXiv: 2406.03044 v4.", "was_revised": true, "final_justification": "I acknowledge the interesting finding of this work, “explor[ing] the impact of different spatial encoding/masking”. As my concerns related to the PopT baseline have not been resolved, I will maintain my score as `3`.", "rebuttal": "We thank the reviewer for their feedback and we reply to the comments inline below.\n\n> The reproduced results for PopT+BrainBERT underperform the original paper [1] -- specifically, 0.80 versus 0.90 on sentence onset detection, and 0.78 versus 0.93 on speech/non-speech classification. \n\nYes, this is a correct observation. First, we clarify that the reason for this is that we did not use the same data segments used by the authors of Brainbert/PopT [1,2] for our analysis. As such, the reproduction results would not necessarily exactly match. Specifically, those authors generated 5-second overlapping segments which they then partitioned into training/valid/test splits. Here we generated non-overlapping segments to minimize any possible information leakage between segments for a rigorous evaluation of spatial scales. \n\nHowever, to address the reviewer’s concerns we now demonstrate that our pretrained PopT model can achieve the same performance reported in the original work using our evaluation framework. To do so, we used the publicly available code on GitHub to generate the downstream data splits used in [2] and evaluated our pretrained PopT model on these data splits. In this setup, we achieved **0.883 &plusmn; 0.008 on sentence onset** and **0.925 &plusmn; 0.01 on speech**, which are almost the same as the numbers reported in the original manuscript. We note that the small discrepancies can be due to differences in the evaluation framework, such as seeding and the channels used. In [2] the authors only used 1 finetuning seed (to the best of our understanding) whereas we use 5 to achieve a more robust estimate of model performance. Further, here we use all channels for each session rather than selecting a subset, which is done in [2] (the authors used 90 of the channels). \n\nThe above analysis shows that our pretrained PopT model is valid and we are able to use it to reproduce the original reported results.\n\n> A minor implementation difference exists in trial length -- 3 seconds in this work versus the original paper's 5-second segments.\n\nAlthough this is a valid observation, we clarify that the 3-second vs 5-second distinction is not a limitation of our approach. We chose to use the 3-second interval for two reasons. First, as we noted in the previous response, we chose to generate non-overlapping data segments (distinct from the original work [1,2]). As a result, we would have had less (pre)training data had we used the same 5-second intervals as the original work. Thus we opted to use 3-second intervals. Second, we based our choice of 3-seconds on intervals used in prior works considering language processing [3-5]. For example, the technical paper associated with the Brain Treebank dataset [3] used intervals of 4 seconds for their analysis.\n\n> I note that the supplementary code for PopT [1] on OpenReview implements a lobe-based position embedding approach, which is not described in the PopT manuscript. Given the concerns raised in Major Weakness 1, the contribution of this embedding technique to model performance remains unclear.\n\nWe thank the reviewer for directing us to the supplementary code on OpenReview. First, we clarify that we have addressed weakness 1, as we have now shown reproduction of the original numbers (described above). Second, we have integrated the OpenReview implementation of region-based positional embedding into a local copy of the publicly available PopT codebase (from GitHub), and have pretrained a new PopT with region-based encoding. Using the default pretraining settings, the resulting pretrained model underperformed the channel-level encoding version of PopT that was reported in the published paper. We hypothesize that there can be several reasons for this:\n\n1. The pretraining configurations provided by the default codebase may not be tuned for using parcel-based spatial encoding and thus may require more extensive modification and experimentation, which are beyond the scope of this work. Indeed, the original PopT paper does not report any results with parcel-based encoding, suggesting that they may have seen no improvement with the larger spatial scale.\n\n2. It could be that the specific discriminative pretraining task used to train PopT benefits from channel-level information more than parcel-level information. For example, the model may have a harder time performing the channel-wise discrimination task if it has the parcel-level encoding rather than a coordinate-level encoding for the channel – as the individual channel’s identity may be critical for the task.\n\nWe will revise our manuscript to clarify that our results show the benefit of flexible spatial scales in masked reconstruction pretraining. The impact of spatial encoding in other pretraining approaches (e.g., discriminative tasks such as PopT's) remains an open question and is an interesting direction for future research. Overall, our model enables flexibility in choosing spatial scales for neurofoundation modeling with masked pretraining, shows the importance of considering such scales for strong model performance, and provides the capability for future work to study spatial scales in various downstream motor, cognitive, and sensory tasks.\n\n** References:**\n\n[1] Christopher Wang et al. BrainBERT: Self-supervised representation learning for intracranial recordings\n\n[2] Geeling Chau et al. Population Transformer: Learning Population-level Representations of Neural Activity.\n\n[3] Christopher Wang et al. Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli.\n\n[4] Ariel Goldstein et al. A unified acoustic-to-speech-to-language embedding space captures the neural basis of natural language processing in everyday conversations.\n\n[5] Ariel Goldstein et al. Shared computational principles for language processing in humans and deep language models."}]

Author rebuttal:

Meta-review: This paper introduces BaRISTA, a JEPA-style pretraining framework for iEEG that flexibly incorporates spatial encoding at multiple scales (channel, parcel, lobe). Using a masked latent reconstruction objective, the authors systematically assess how spatial encoding/masking scales influence downstream decoding performance. Experiments on the Brain Treebank dataset show that encoding at parcel-level yields stronger results than channel-level encoding, while still preserving fine-grained information, and that the framework scales effectively to additional tasks such as speech/non-speech discrimination and sentence onset detection.

The strengths of this submission are clear. Reviewers appreciated the technical soundness and novelty of enabling flexible spatial scales in neurofoundation modeling (uiXd, 54tZ), with one noting this could “steer the field towards better positional encoding schemes for sEEG.” The paper is well written, supported by comprehensive ablations, and situates its contributions clearly. Reviewers also valued the empirical insight that larger spatial scales can improve high-level cognitive decoding while smaller scales preserve low-level reconstruction fidelity.

Some weaknesses remain. Evaluation was limited to a small set of tasks, and the PopT baseline raised concerns (eXGi). However, the rebuttal included additional experiments (e.g., K-fold evaluations, new volume and optical flow decoding tasks) and clarified baseline differences, which alleviated most concerns.

Overall, this is a technically solid and insightful contribution that advances understanding of spatial encoding in neural signal modeling. I recommend acceptance.
Decision: Accept (poster)