Population Transformer: Learning Population-level Representations of Neural Activity
Authors: Geeling Chau
Christopher Wang
Sabera J Talukder
Vighnesh Subramaniam
Saraswati Soedarmadji
Yisong Yue
Boris Katz
Andrei Barbu
Venue: ICLR 2025 Oral (2025)
Keywords: representation learning
neuroscience
self supervised learning
Primary area: applications to neuroscience & cognitive science
TL;DR: Representation learning of neural data

Abstract: We present a self-supervised framework that learns population-level codes for arbitrary ensembles of neural recordings at scale. We address key challenges in scaling models with neural time-series data, namely, sparse and variable electrode distribution across subjects and datasets. The Population Transformer (PopT) stacks on top of pretrained temporal embeddings and enhances downstream decoding by enabling learned aggregation of multiple spatially-sparse data channels. The pretrained PopT lowers the amount of data required for downstream decoding experiments, while increasing accuracy, even on held-out subjects and tasks. Compared to end-to-end methods, this approach is computationally lightweight, while achieving similar or better decoding performance. We further show how our framework is generalizable to multiple time-series embeddings and neural data modalities. Beyond decoding, we interpret the pretrained and fine-tuned PopT models to show how they can be used to extract neuroscience insights from large amounts of data. We release our code as well as a pretrained PopT to enable off-the-shelf improvements in multi-channel intracranial data decoding and interpretability. Code is available at https://github.com/czlwang/PopulationTransformer.

Published as a conference paper at ICLR 2025
POPULATION TRANSFORMER: LEARNING POPULATION- LEVEL REPRESENTATIONS OF NEURAL ACTIVITY
Geeling Chaul"
Christopher Wang"
Sabera Talukder'
Vighnesh Subramaniam
Saraswati Soedarmadji
Yisong Yue!
Boris Katz²
Andrei Barbu²
California Institute of Technology
(gchau, sabera, ssoedarm, yyue) (yyee/cclteeeee @caltech. edu
MIT CSAIL, CBMM {ezw, vsub851, boris, abarbu] @mit edu
ABSTRACT
We present self-supervised framework that learns population-level codes for arbi- trary ensembles of neural recordings at scale. We address key challenges in scaling models with neural time-series data, namely, sparse and variable electrode distribu- tion across subjects and datasets. The Population Transformer (PopT) stacks on top of pretrained temporal embeddings and enhances downstream decoding by enabling learned aggregation of multiple spatially-sparse data channels. 'he pretrained Pop'T' lowers the amount of data required for downstream decoding experiments, while increasing accuracy, even on held-out subjects and tasks, Compared to end-to-end methods, this approach computationally lightweight, while achieving similar better decoding performance. We further show how our framework is generalizable multiple time-series embeddings and neural data modalities. Beyond decoding, we interpret the pretrained and fine-tuned Pop'T models to show how they can used to extract neuroscience insights from large amounts of data. We release our code as well pretrained PopT T to enable off-the-shelff improvements in multi-channel intracranial data decoding and interpretabilitty Code is available at https://githan com/ cz wwt.:///.aan///oouuutiiiotaaaaaa
INTRODUCTION
Building effective representations of neural data is an important tool in enabling neuroscience research. Recordings from the brain such as intracranial (iEEG) and scalp (EEG) electroencepholloraapyy consist of time series recorded simultancously from multiple channels. The relationships between these time series complex and governed by the underlying functional connectivity that exists between brain regions. Our goal is to build an effective model of maalti-channel activity. Recently, improvements have been made modeling time-series (Wang et al. 2022 Ansari et al. 2024). This suggests an approach for learning multi-channel representations
2022 Talukder et al. [2024] Yue
aggregating temporal embeddings. However, this is not a trivial task. For brain recordings, particularly iEEG, one must contend with sparse and variable electrode layouts, which change the semantics of input channels from subject subject. This forces many Brain Machine Interface (BMI) and neuroscience studies to rely on expensive schemes, in which models are retrained for each new participant, requiring large amounts of data for calibration (Faezi et al.) [22T]] Herff et al.) 2020; Martin et al., 2018, Metzger et al.) 2023) Willett et al.) (023). To this end, we propose self-supervised learning framework, Population Transformer (PopT), which specifically designed to aggregate single-chin channel encodings across variable electrode layouts.
Self-supervised pretraining on unannotated data has been shown to be effective for creating generic representations that are useful for many downstream tasks (Bommasani et al. 2022). Prior work has shown how to pretrain subject-specific (Le & Shlizerman (022) or channel-specific (Wang et al.
2022) models of iEEG, but such techniques ignore inter-channel relationships commonalities that
Equal contribution. Code: httphh/ppp///ghhtooooocooa/aaaaa

---

Published as a conference paper at ICLR 2025
might exist across subjects. Recent end-to-end self-supervised learning approaches downsample signals heavily to make training across hundreds of channels feasible (Zhang et al. 2024 Yang et al. 2024: Jiang et al., 2024). This is particularly problematic for high-fidelity iEEG signals, which capture sub-millisecond changes in neural activity. Our approach leverages existing rich temporal embeddings to represent signal, freeing the model to foeus on learning effective aggregation.
We propose Population Transformer (PopTT), self-supervised pretraining approach that learns subject- generic representations of arbitrary electrode ensembles. Transformers offer the flexibility to learn aggregate information across channels, but large amounts of data are needed to train the attention weights (Devlin al. 2019). During pretraining, we train on large amounts of unannotated data and simultaneously optimize both channel-level and ensemble-level objective. These tasks encourage the model to develop subject-geners representations by modeling (1) individual channels in the context of surrounding channels and (2) channel ensembles and their relations across time.
Our PopT approach is modular, and builds on top of powerful single-channel temporal embeddings. which provides two key advantages. First, by separating single-channel embedding and multi-channel- aggregation into different modules, we make our approach agnostic to the specific type of temporal embedding used, leaving room for future independent improvements along either the temporal or spatial dimension (an approach that has been validated in video modeling (Arna et al. 2021 Second, by taking advantage of learned channel embeddings. PopT training is computationally lightweight compared to their end-to-end counterparts (Appendix B and baseline aggregation approaches (Figure A) allowing for adoption in lower compute resource environments.
Empirically, we find that our pretrained PopT outperforms commonly used aggregation approaches 2021), and is competitive with end-to-end trained methods (Zhang al.) 2024] Yang et al. 2024 You et al. 2019). Moreover, we find that these benefits hold even for subjects not seen during pretraining, indicating its usefulness for new subject decoding. We also show that the pretrained PopT weights themselves reveal interpretable patterns for neuroscientific study. Finally, we demonstrate that our proposed framework is agnostic the underlying temporal encoder, further allowing it to adapt to other neural recording modalities.
(Ghosal & Abbasi-Asi
Our main contributions are:
1. a generic self-supervised learning framework, Population Transformer (PopT), that learns joint representations of arbitrary channel ensembles across neural datasets,
2. a demonstration that pretraining systematically improves ensemble representations for downstream decoding even for held-out subjects,
3. new method for brain region connectivity analysis and functional brain region identification based on the pretrained and fine-tuned PopT weights,
4. trained and usable off off-tf model that computes population-level representations of high temporal resolution intracranial neural recordings.
2 RELATED WORK
Self-supervised learning on neural data Channel independent pretrained models are a popular
approach for neural spiking data (Liu et al. (002)), intracranial brain data (Wang et al. [002 Talukder & Gkioxari 2023), and general time-series (Talukder al. 20244.. Additionally,
neural datasets, approaches exist for EEG (Chien et al. 2022 Kostas et al. 2021 Yi et al. 2023), fMRI
2023). and calcium imaging (Antoniades et 20.2)) datasets. However, these approaches do not learn population- levell interactions across datasets with different recording layouts. either due a single-channel focus or the assumption that the channel layout is fixed. Several works pretrain spatial and temporal dimensions across datasets with variable inputs (Zhang et al. 2024] Yang et al. 2024 Jiang et al., 2024) Ye al. 2024, Cail et al., 2023), but most simultancously learn the temporal embeddings with the spatial modeling, which make them challenging to interpret and computationally expensive to train, especially for high temporal resolution signals. To knowledge,
(Thomas et al. 2022, Kan et al. 2022 Ortega Caro et al.
the first study the problem of building pretrained channel aggregation models on top of pre-existing temporal embeddings trained across neural datasets with variable channel layouts, allowing for modeling of high quality neural data.
Modeling across variable input channels Modeling spatial representations on top of temporal embeddings has been found to be beneficial for decoding (Faezi et al. 2021 Le & Shlizerman
2

---

Published as a conference paper at ICLR 2025
2022
Azabou et al. 2024), but prior works supervised labels, do not leverage large amounts of unannotated data. The brain-computer-inteen interface field has studied how to align latent spaces
(Pandarinath al. 2018 Karpowicz et al. 2022 Degenhart et al., 2020; Jude al. Ma M al. 2023) which either still requires creating an a alignment matrix to learn across datasets or only provides post-training alignment mechanisms rather than learning across datasets. Other approaches impute
missing channels or learn latent spaces robust to missing channels (Tluudeer et al. 2022 Zhang et al.
202T Chau al., 2024), but these are more suited for the occasional missing channel rather than largely varying sensor layouts. We directly learn spatial-leve representations using self- self-supervised
learning across datasets to leverage large amounts of unannotated intracranial data.
POPULATION TRANSFORMER APPROACH
PopulationTTaanttorme
Ensemble-wise discrimination Is close to
TransformerEncoder (x6)
PopulationTransformme
CLS
C.S
Temporal Emb.
+
DD pusition
0+0
Mean
Temporal encoder
00000000000 ... 00000000000
Brain state time
Brain state at time
BrainBERT
*
BrainBERT
Channel- wise discrimination
Does t=t?
00000000000000
000
PopulationTransfommer
Seg
S mome
ooo
Brain state at time
Time
Time
Figure 1: Schematic of our approach. The inputs to our model (a) are the neural activities from collection of electrodes in given time interval (bottom). These are passed to a frozen temporal embedding model (dotted red outline: BrainBERT (Wang et al., (202) shown), which produces set of time embedding vectors (yellow). The 3D positions of each electrode (red) are summed with these vectors to produce the model inputs (orange, lower). PopT produces space-contextual embeddings (orange, top) for each electrode and [CLS] token (blue, top), which can be fine-tuned for downstream tasks. In pretraining, PopT learns two objectives simultancously. In the first, (b) PopT determines whether two different sets of electrodes (orange brown) represent consecutive non-consecutive times. the second objective, (c) Pop'T must determine whether an input channel has been replaced with activity random other time that inconsistent with the majority of inputs.
Figure I overviews our Population Transformer (PopT) approach. The key ideas are: (1) to learn generic representation of neural recordings that can handle arbitrary electrode ensembles: and (2) to employ modular system design that uses transformer architecture to aggregate information from existing per-channel temporal embeddings. To do so, we employ self-supervised pretraining approach to learn ensemble and channel level representations. Afterwards, downstream decoding tasks. In addition to offering strong decoding results, including generalization to subjects with different electrode configurations than training subjects (see Section 5) the modular system design is computationally lightweight (see Appendix [B), can benefit from improved temporal representations, and is readily interpretable (see Section 6).
fine-tune PopT on
Architecture A schematic our Population Transformer (PopT) approach is shown in Figure [ We adopt a transformer backbone due to its ability accommodate variable channel configurations.

---

Published as a conference paper at ICLR 2025
Consider given subject with N channels indexed by C {1, [1....,Ne] and an arbitrary subset of channels 5 " C. Let 4 E R" denote a time window of activity from channel i that begins at time t, where T is the number of time samples in the interval. The PopTI takes input a collection of such channels activities, X = { E 5). as well as a special [CLS] token. Per channel, each interval of brain activity is passed through a temporal embedding model B, in the figure's case BrainBERT (Wang et al. 2022). to obtain a representation of each channel's s temporal context, B(x{) E RR where d's is the embedding dimension. For BrainBERT, the first step of pre-processing involves obtaining the STFT for the signal, but preprocessing will differ depending on the embedding model used.
To allow the model to learn a common brain state representation across layouts. each channel's embedding is summed with its 3D position. so that the final processed input to the PopT is X3 = ( (2²) + pos(i)laf E X*). The PopT receives this as an x d matrix. Spatial location is given by the electrode's 's Left, Posterior, and Inferior coordinates for iEEG electrodes (Wideman 2024). and XYZ. positions for EEG electrodes. Membership in a particular ensemble (see below: ensemble-wis loss) is also encoded. The four encodings are concatenated together to form the position embedding (is(i) = [Cieti Epost: Cint' femseblle], where e is given using a sinusoidal position encoding that represents scalar coordinate
unique combination of sines (Vaswani et al. 2017
The core of PopT consists of a transformer encoder stack (see Appendix A Architectures). The output of the PopT are spatial--conextual embeddings of the channels Y {Br and an embedding of the CLS token Yets- During pretraining, the Pop'T additionally is equipped with a linear head for the [CLS] ] token output and separate linear heads for all other individual token outputs. These produce the scalars Dets and Y; respectively, which are used in the pretraining objective (Figure [p and c).
Self-supervised loss Our loss has two discriminative components: (1) ensemble-wise the model determines if activity from two ensembles occurred consecutively, requiring an effective brain state representation the ensemble-level, (2) channel-wise - the model identifies outlier channels swapped with a different timepoint's 's activity, requiring sensitivity to surrounding channel context.
key aspect of our method the fact that our objective is discriminative, rather than reconstructive, as is often the case in self-supervision (Liu et al. 202T Wang et al., 2022). In practice, the temporal embeddings often have low effective dimension (see Wang et al. (2022)). and reconstruction rewards the model for overfitting to "filler" dimensions in the feature vector (Section 5
Pretraining In ensemble-wise discrimination (fig. [D.. two different subsets of channels SA.SB C C are chosen with the condition that they be disjoint SA n SB 0. During pretraining, the model receives the activities from these channels at separate times X {-f i E SA} and X = {=f i E Sp). The objective of the task is then determine whether these states X and X have occurred consecutively in time (Jt - e'| = 500ms) or are separated by some further, randomly selected interval. Given the output of the classification head, the loss function LN is the binary cross-entropy (BCE). We select disjoint subsets for ensemble-wise discrimination to prevent the model from solving tasks through trivial copying. We also randomly vary |S| during sampling to ensure the model handles ensembles of different sizes.
In channel-wiss wise discrimination (fig. [ the model must determine whether channel's 's activity has been swapped with activity from a random time. Precisely, activity from each channel i is drawn from a time 14 = for all channels. Then, 10% of the channels are randomly selected to have their activity replaced with activity from randomly selected channel at a random point in time t, * t. For each of the token outputs of PopT, the channel-wise loss function Lo is the BCE, Our complete objective function is L LN + Lc. A detailed formulation of the objective is given in Appendix A
Fine-tuning In fine-tuning, given the [CLS] token, which is a d-dimensional vector, the PopT produces the intermediate representation, Vota E Rd, which is passed through single layer linear to produce a scalar prediction Wets E R. BCE loss is used for our binary decoding tasks (Section (4)
EXPERIMENT SETUP
Data We use two types of neural time-series data: intracranial and scalp electroencephologrpppy (iEEG and EEG). iEEG probes are surgically implanted within the 3D brain volume and record local electric signals from the brain at very high temporal resolution and spatial precision. EEG electrodes lie the scalp, and record electric signals that are smeared by the skull, which results
4

---

Published as a conference paper at ICLR 2025
in low temporal and spatial resolution. EEG montages typically tile the whole scalp, while iEEG electrodes are often only inserted a comparatively smaller number locations. These cover two resolution extremes of neural time-series data modalities.
iEEG: We use the publicly available subject data from Wang et al. (2024). Data was collected from 10 subjects (total 1,688 electrodes, with mean of 167 electrodes per subject) who watched 26 movies (19 for pretraining, 7 for downstream decoding) while intracranial probes recorded their brain activity. To test decoding with arbitrary ensemble sizes, we select subsets of electrodes based on their individual linear task decodability, with the smallest subsets containing the electrodes with highest decodability. We follow the trialization and data preprocessing practices used in Wang al. 2022
EEG: We use the Temple University Hospital EEG and Abnormal datasets, TUEG and TUAB Obeid
Picone, 2016), for pretraining and task data respectively. We remove all task subjects from the pretraining set and follow the data preprocessing practices in Yang et al. [024] Jiang et al. (2024)
Decoding Tasks We evaluate on 5 different classification tasks: auditory-linguistcc tasks used in the evaluation of Wang et al. (2022) and widely evaluated abnormal EEG detection task from Obeid Picone (0016). Of the muditory-linguiste whether word is spoken with a high or low pitch and determining whether word is spoken loudly or softly. And two of the tasks have more linguistic focus: determining whether the beginning of sentence is occurring determining whether any speech at all is occurring. The TUAB abnormal EEG detection task is a binary classification of pathological or normal EEG recording.
tasks, two the tasks are audio focused: determining
Baselines For controlled baselines, concatenate the single-channel temporal embeddings and train a linear (Linear) or non-linear (Deep NN) aggregator on the decoding task. These enable us to directly assess how much PopT improves upon existing aggregation approaches (Ghosal & Abbasi- Asl 2021). These approaches cannot be pretrained across subjects due to the changing meaning and quantity of inputs. To test the effectiveness of pretraining, we also compare against a non-pretrained Pop'T.
Methods compared For the iEEG experiments, we also compare against Brant (Zhang et al. 2024). which is an end-to-end iEEG encoder. We take the fully pretrained Brant model, and fine-tune on our iEEG tasks combining channels with linear aggregation. For the EEG experiments, we compare against reported BIOT (Yang al. 2024) and LaBraM (Jiang et al.. 2024) results. They also train both temporal and spatial encoders together in contrast to our modular approach.
Temporal encoders To test the generalizability of our approach, we train with a variety of temporal encoders: BrainBERT (Wang et al. (022)). which is designed for iEEG data, TOTEM (Talukder et al. 2024) which learns tokenization the imput, Chronos (Ansari al. 2024) which is large general time-series encoder, and Vec (uue et al. (022) which has hierarchical convolutional architecture.
Hidden dimensions of these encoders vary from 64 to 768. More details are in Appendix
Model
Pitch
Volume
Sent. Onset
Speech/Non-speech
BrainBERT:
Linear Agg.
0.59 + 0.03
0.66 0.03
0.70 + 0.04
0.71 + 0.04
Deep NN Agg.
0.56 + 0.03
0.64
0.71 0.03
0.70 0.04
non-pretrained PopT
0.53 1 0.02
0.61 1 0.05
0.74 0.04
0.70 1 0.03
Pretrained PopT
0.74 1 0.03*
0.87 + 0.03*
0.90 0.01'
0.93 1 0.02*
TOTEM:
Linear Agg.
0
0.67
0.80 0.03
0.77 0.77 + 0.05 0.77 + 0.05
Deep NN Agg.
0.58 +: 0.02
0.70 0.02
0.80 +: 0.03
Non-pretrained PopT
0.53 + 0.01
0.62 0.02
0.80 + 0.03
0.77 + 0.05
Pretrained PopT
0.64 + 0.03
0.79 + 0.02
0.90 + 0.02*
0.88 + 0.05*
End-to-end:
Brant (Zhang al.
2024
0.61 + 0.03 0.74 0.03
0.03 0.80 + 0.04
0.80 + 0.03
Table 1: Pretraining PopT is critical to downstream decoding performance (iEEG data). We test variety of audio-linguistic decoding tasks (see Section 4 with 90 channels as input. The temporal encoder used for aggregation in sections and 2 are denoted in the section header. We also evaluate against an end-to-end pretrained iEEG model in section 3. Shown are the ROC-AUC mean and standard eтror across subjects. Best per section are bolded. Asterisks indicate that the bolded model significantly better than the pecan model (p < 0.05, Wilcoxon rank-sum).
5

---

Published as a conference paper at ICLR 2025
Model
Balanced Accuracy
ROC AUC
Chronos:
Linear Agg.
0.7754 + 0.0008
0.8563 + 0.0003
Deep NN Agg.
0.7881 + 0.0057
0.8678 + 0.0049
Non-preteen pretrained PopT
0.7763 + 0.0047
0.8631 1 0.0016
Pretrained PopTT
0.7963 1 0.0021
0.8822 1 0.0013"
TS2Vec:
Linear Agg.
0.7554 = 0.0005
0.8549 + 0.0011
Deep NN Agg.
0.7835 = 0. 0.0039
0.8723 + 0.0029
Non-pretrained PopT
0.7896 + 0.0037
0.8782 1 0.0018
Pretrained PopT
0.8060 + 0.0025
0.8883 0.0008
End-to-end:
BIOT Yang et al 2024
0.7959 = 0.0057
0.8815 + 0.0043 0.9162 + 0.0016
LaBraM (Jang etal. 2024
0.8258 + 0.0011
Table 2: Pretraining Pop'I' is critical to downstream decoding performance (EEG data). We test on an abnormal EEG detection task (see Section 4 with 21 channels as input. The temporal encoder used for aggregation in sections I and 2 are denoted in the section header. We also evaluate against end-to-ent pretrained EEG models in section 3 (values from the original works). Shown are mean and standard deviation across 5 random seeds. Best section are bolded. Asterisks indicate that the bolded model is significantly better than the second- place model (p 0.05, Wilcoxon rank-sum).
5 RESULTS
Decoding performance We find that using a pretrained PopT significantly benefits downstream decoding compared to baseline channel aggregation techniques across tasks, data modalities, and temporal encoding models (Tables and and Figure 2}. To test our method's ability to handle multiple types of channel encodings, we applied our framework to 4 different channel encoders: (1) an iEEG-specific temporal encoder: BrainBERT (Wang et al. 2022), (2) a general tokenization-
time-series encoder: TOTEM (Talukder et al. 2024), (3) pretrained general time-series encoder: Chronos (Ansari et al. 2024). and general convolution- based time-series encoder: TS2Vec Yuue) et al. 2022). We significant improvements in performance with the pretrained PopT in all cases when comparing with baseline aggregation approaches (Figure 2 Additionally, the pretrained PopT scales well with increasing ensemble sizes (Figure 3 challenging task for the baseline aggregation approaches due to limited downstream task data and increasing input size.
We also find that PopTT can achieve competitive performance against pretrained end-to-end models, such as Brant (Zhang et al. 2024 for iEEG, and BIOT (Yang et al. 2024) and LaBraM (Jiang al., 2024) for EEG (Tables) and 2). For instance, PopT outperforms Brant (Zhang et al. 2024) in decoding iEEG data with our pretrained PopT + BrainBERT combination, likely due to PopT's ability leverage spatial relationships. Whereas Brant leaves the channel aggregation problem open.
BrainBERI
TOTEM
Chroncs
TS2Vec
Fretraine PopT
Pretrained PopT
Pretrained PopT
0.89
APPapT netrained Popt
Pretra ind Dopt
Prett vetrainn ained POD
ROC AUC
0.6
0.7
MM
Pretrinnd PopT
Linarr
Deep Wi
Pretr Pretreimed Papl
Linea
Deep VN
Uinaa
Deep NM
InPrupTs
Pretreined Popt
Linear
Deep NN
NFRaaT T
Linear
drep UN
witepp
Linear
Deep NiV
NPRRpT
Linesr
Deep WP
NiPboT
ROC AUC
0.85
Inner
Deep WN
UPPupp
pettrince Popt
Untar
Deep NN
Beep
APPopT
Ipropt
0.4
0.83
(a)
(b)
Volume
Sentence Onset
Speech Non-speech
EEG TUAB
Figure 2: Compared to common aggregation approaches, pretrained PopT consistently yields better downstream decoding across tasks, data modalities, and temporal embedding types. NPopT = Non- pretrrerddd PoPT. (a) performance on four audio-linguistic iEEG tasks with 90 elec- trodes. Grey bars denote standard error across subjects. (b) performance on an abnormal detection EEG task with electrodes. Grey bars denote standard deviation across random seeds.

---

Published as a conference paper at ICLR 2025
Pitch
Volume
Sentence onset
Speech vs, Non-speech
0.9
0.9
0.9
0.9
ROC AUC
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.6
0.5
0.5
1
10
20
30
40
50
1
10
20
30
40
50
1
10
20
30
40
50
1
10
20
30
40
50
Ensemble size
Pretralned PopT
Non- poon-prerralned PopT
Uiner
Deep NN
Figure Pretrained PopTT downstream performance scales better with ensemble size. Increasing channel ensemble size from 1 to 50 (x-axis), we see pretrained Pop'T (green) decoding performance (y-axis) not only beat non-preen con-pretrained approaches (orange, purple, grey), but also continually improve more with increasing channel count. Shaded bands show the standard error across subjects.
PopT is competitive with recent end-to-end trained BEG models (Yang al., 2024 Jiang al. 2024 on EEG TUAB abnormal detection task. This impressive, since models such as LaBraM were specifically developed for this application, whereas PopTT was trained on top of generic time-series embeddings. We find that PopT can offer an efficient and competitive alternative to large end-to-end models for these decoding tasks, due to the effectivenesss of our pretraining task for learning spatial and functional relationships between channel input embeddings.
To verify that the weights of the pretrained PopT capture neural processing well even without fine- tuning, we also train a linear-encoder on top of the frozen PopT [CLS] token and find the same trends (Figure [7p. This point in particular is important in building confidence in the results of our interpretability studies (Section 6). in which we use the frozen pretrained weights to analyze connectivity. For the remaining analyses described below, we use a PopT with BrainBERT inputs.
Sample and compute efficiency Our PopT learns spatial relationships between channels, in way that makes downstream supervised learning more data and compute efficient (Figure and Figure 5). Compared to the preetrrrd baseline models, fine-tuning the pretrained PopT can achieve the same decoding performance as other aggregation techniques with an order of magnitude fewer samples. The pretrained РoрТ surpasses the performance achieved by all other aggregation techniques by 500 samples out of the full dataset (roughly 5-10k examples depending on subject and task) (Figure The pretrained PopTT also converges a low number of steps. This greatly contrasts with the non-pretin non-pretrained PopT. The Linear and Deep NN baselines can similarly compute efficient, but occasionally may require 2k or more steps (Figure
in the case of Speech Non-speech.
Pitch
Volume
Sentence onset
Speech vs. Non-speech
0.9
0.9
0.9
ROC AUC
0.8
0.8
0.8
0.8
0.7
0.7
0.6
0.6
0.6
0.6
0.5
0.5
0.5
0.5
200
350
500
50
200
350
500
50
200
350
500
50
200
350
500
Number of fine tuning samples
Pretrained PopT
Non-pretrained PopT
Linear
Deep
Figure 4: Pretrained PopT is more sample efficient when fine-tuning. Varying the number of samples available to each model train time (x-axis), we see that the pretrained PopT is highly sample efficient, requiring only fraction of samples (fewer than 500 samples out of 5-10k of the full dataset) reach the full performance level of baseline aggregation approaches (dashed lines). Bands show standard error across test subjects. Stars indicate performance with full fine-tuning dataset.
7

---

Published as a conference paper at ICLR 2025
Pitch
Volume
Sentence onset
Speech Vs, Non-speech
0.9
0.9
0.9
0.9
ROC AUC
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
750
1500
750
1500
750
1500
750
1500
Number of steps
Pretrained PopT
Non Non-pretralned PopT
DeepNN
Linear
Figure 5: Pretrained PopT is consistently compute efficient when fine-tuning, Number of steps required for each model to reach final performance during fine-tuning (dashed lines). The pretrained PopTT consistently requires fewer than 750 steps (each step an update on a batch size of 256) to converge. Bands show standard error across subjects. Stars indicate fully trained performance.
Generalizability To test if our pretrained weights will be useful for subjects not seen during training, we conduct hold-one-out analysis. We pretrain model using all subjects except for one, and then fine-tune and evaluate on the held-out subject. We find that missing subject from pretraining does not significantly affect the downstream results (Figure (). This raises our confidence that the pretrained weights will be useful for unseen subjects and for researchers using new data.
ROC AUC
0.9
0.8
0.4
Pitch
Volume
Sentence onset
Speech Non-speech BrainBERT
All
Held-out
Non-pretrained PopT
Figure 6: Gains in decoding performance available to subjects. A minimal decrease in downstream decoding performance is found if the subject held-out from pretraining (Held-out vs All). Results cross-validated across all test subjects. For BraTn we report performance the channel with the best finear-decodability Markers show mean and standard error across subjects.
Scaling with amount of pretraining data To investigate the effect of scaling pretraining data on our model, we pretrain versions of PopT using only 2%, 5%, 10%, 25% and 50% of our data. Evaluation is performed on all test-subjects. We find a general improvement in downstream decoding when we increase the amount of pretraining data available across all our downstream decoding tasks (Figure 7). suggesting decoding improvements as we double our pretraining dataset. We expect that the framework could benefit from more diverse data, as slight plataeuing effect is seen with the current pretraining dataset.
1.00
ROC AUC
0.75
0.50
Pitch
Volume
Sentence onset
Speech vs. Non-speech
0%
2%
5%
10%
25%
50%
100%
Figure 7: Pretraining with more data leads better downstream performance. We pretrain PopT with different percentages of our full pretraining dataset (colors) and test on our decoding tasks (x-axis). Markers show mean and standard error across test subjects.
8

---

Published as a conference paper at ICLR 2025
Ablation of loss components and position information An ablation study confirms that both ensemble- wise and whannel-wise losses contribute to downstream performance (Table Further- more, including the 3D position information for each channel and using discriminative losses critical. Removing our positional encoding during pretraining and fine-tuning drops the performance significantly. Attempting to only an I.1 reconstruction term for our pretraining objective results in poorer performance. Our discriminative loss requires the model to understand the embeddings in terms of how they can be distinguished from one another, which leads the model to extract representations that are more beneficial for decoding.
Pitch
Volume
Sent.
Onset
Speech/Non-specch
PopTT'
0.74 0.03
0.87 + 0.03
0.90 = 0.01
0.93 + 0.02
Pop'T' w/o channel-wise loss
0.71 + 0.03
0.86 + 0.03
0.88 - 0.02
0.92 + 0.02
PopTT w/o ensemble-wise loss
0.70 + 0.03
0.87 0.02
0.87 # 0.01
0.92 + 0.02
PopT w/o position encoding
0.62 + 0.07%
0.76 + 0.07
0.81 = 0.09
0.83 + 0.10"
PopTT reconstruction loss only
0.56 + 0.04 0.65 + 0.08
0.73 = 0.10"
0.74 + 0.10
Table 3: Pop'T' ablation study. We individually ablate losses and positional encodings during pretraining then decode on the resulting models. Shown are ROC-AUC mean and standard error across subjects evaluated 90 electrodes. The best performing model across all decoding tasks uses all of our proposed components. Here, V denotes ablations which are significantly worse than the full model (p- < 0.05, Dunnett's test).
INTERPRETING LEARNED WEIGHTS
Connectivity Traditional neuroscience analyses typically use cross-correlation measure of region connectivity (Wang et al. 2021). Our PopTT allows for an alternative method of determining connectivity, based on the degree to which channels sensitive to each other's context. In this method, each channel is masked in turn, and then model performance on the pretraining channel-wise objective for the remaining unmasked channels measured. We use the degradation in performance as a measure of connectivity. We can construct plots (Figure 8 that recapitulate the strongest connectivity of the cross- correcoorelation maps. Note that while some approaches for modelling brain activity explicitly build this into their architecture (Cai et al. (223),, we recover these connections purely as result of our sup-ruuee learning. Additional method details available Appendix G
Connectivity from coherence analysis Connectivity from pretrained PopT
Banks: banks of the superior temporal sulcus inferiar parictal
Latorb: lateral orbitafrontal
ParsOp: pars operculants
ParsOrb: pars orbitails
PostCient
ParsTri: pars triangularis
PostCent: postcentral
PeeCen
Stace Tompo
IreeCam
PreCen: precentral
RosMitFront: rostral
midd frontal
Sup Temp: superier temporel
SupMarg: supramarqinal
TransTemp: transverse temporal
Figure 8: Probing the pretrained model for inter-channel connectivity. Traditionally, connectivity analysis between regions is done by computing the coherence between electrode activity (left). We propose an alternative analysis purely based on the contextual sensitivity learned during pretraining. Briefly, we select an electrode, mask out its activity, and then measure the degradation in the channel- wise objective function for the remaining electrodes. Plotting the values of this delta (right) recovers the main points of connectivity. Plots for all test subjects can be seen in Appendix
Candidate functional brain regions from attention weights After fine-tuning our weights on a decoding task, we can examine the attention weights of the [CLS] output for candidate functional brain regions. We obtain a normalized Scaled Attention Weight metric across all subjects to analyze candidate functional brain regions across sparsely sampled subject datasets (Figure 9) The Scaled Attention Weight is computed from raw attention weights at the [CLS] token passed through the attention rollout algorithm (Abnar & Zuidemal 2020). The resulting weights from each channel are then grouped by brain region according to the Desikan- Desikan-Killian- Killiany-Tourvllle Tourville (DKT) region atlas (Klein & Tourville, 2012). A full description of the method is available in Appendix G

---

Published as a conference paper at ICLR 2025
Volume
Speech vs. Non-speech 0.5
St Scaled Atterttin Weights
Broca
Wernicke
Pitch
Sentence onset
Auditory
Cortex
Figure 9: Attention weights from fine-tuned Pop'T identify candidate functional brain regions Candidate functional maps be read from attention weights of PopTT fine-tuned on our decoding tasks. For the Speech vs Non-speech and Sentence onset tasks, note the weight placed on regions near Wernicke's ' area (black arrows). all tasks, the auditory cortex is attended to. Center brain figure highlight regions related to auditory-linguiss processing; figure credit: (aph) 2017
The resulting weights reveal expected functional brain regions related the tasks decoded (Figure 9 with low-level auditory tasks highlighting primary auditory cortex and higher-level language distinc- tion tasks highlighting language- -eeegge--pcifcccee areas. Given the training PopT undergoes, these weights provide technique for discovering candidate regions (see Appendix H for quantitative comparison).
7 DISCUSSION
We presented a self-supervised scheme for learning effective joint representations of neural activity from temporal embeddings. Our approach improves decoding and reduces the samples required to learn downstream tasks, which is especially critical for neural data modalities given subject constraints. key aspect of our approach the fact that we focus on spatial aggregation of cxisting channel embeddings, rather than training large end-to-end model. By decoupling temporal and spatial feature extraction, we are able to leverage existing temporal embeddings to learn spatiotemporal representations efficiently and with smaller number of parameters. This makes model available for use in low compute-resouree settings. Furthermore, this separation of considerations opens up the possibility for future independent improvement in temporal modeling, whether that be from domain specific model or a more general time-series encoder. The generality of this approach allowed us to train on two very different neural modalities: scalp EEG and invasive iEEG. Our success these domains suggest that this approach could even be extended to settings outside of neuroscience that also contend with sparsely and variably distributed time-series data channels, as is often the case with geophysical or climate data.
Limitations and Future Work We proposed a strategy for aggregating signals, provided that meaningful spatial coordinates are available, but it remains to be seen how to extend this approach to settings without such coordinates. Electrode layouts are highly variable, so it is important that some notion of positional encoding be given. Future work could experiment with automatic functional identification for each channel, such as that explored in neural spiking data (Azabou et al., 2024)) but it is currently unclear how to do so with neural recordings that have lower SNR.
CONCLUSION
We introduced a pretraining method for learning representations of arbitrary ensembles of intracranial electrodes. We showed that our pretraining produced considerable improvements in downstream decoding and efficiency, that would not have been possible without the knowledge of spatial rela- tionships learned during the self-supervised pretraining stage. These benefits were found across data modalities, decoding tasks, and temporal encoders used, speaking the generality of approach. We further showed that this scheme produces interpretable weights from which connectivity maps and candidate functional brain regions can be read. Finally, we release the pretrained weights for our PopT with BER inputs as well as our code for pretraining with any temporal embedding.
10

---

Published as a conference paper at ICLR 2025
9 ACKNOWLEDGEMENTS
We would like to thank Xuefei (Julie) Wang and Kejun (Amy) Li for helpful comments and discussion.
We would like to thank our funding sources: Caltech Chen Institute, Caltech Carver Mead New Adventures Fund, Center for Brains, Minds, and Machines, NSF STC award CCF-1231216 the NSF award 2124052, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the CBMM-Siemens Graduate Fellowship. the DARPA Mathematics for the Discovery of ALgorithms and Architectures (DIAL) program. the DARPA Knowledge Management at Scale and Speed (KMASS) program, the DARPA Machine Common Sense (MCS) program, the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-0388
and the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence The views and conclusions
Accelerator under Cooperative Agreement Number 1A8750-19-2-1000 contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Govermment. The U.S. Government is authorized reproduce and distribute reprints for Govemmennt purposes notwithstanding any copyright notation herein.
REFERENCES
What is aphasia? - types, causes and treatment, Mar 2017. URL httpp::/www.n..ce nih
gov/health/aphass
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint
arXiv:2005.00928, 2020.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint
2024.
Antonis Antoniades, Yiyi Yu, Joseph Canzano, William Wang, and Spencer LaVere Smith. Neu- roformer: Multimodal and multitask generative pretraining for brain data. arXiv preprint
arXiv:2311.00136, 2023.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6836-6846, 2021.
Mchdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoic, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36, 2024.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lec, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li. Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang,
11

---

Published as a conference paper at ICLR 2025
Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities petttps://arxitt
and risks of foundation models, 2022.
abs/alas...../// 07258
Donghong Cai, Junru Chen, Yang Yang. Teng Liu, and Yafeng Li. Mbrain: A multi-channel self- supervised learning framework for brain signals. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 130-141, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305. 3599426. URL http:::tttooiii http://i..ogggg////// 1tt../........1...55.......0 315111555553355.99222
Geeling Chau, Yujin An, Ahamed Raffey Iqbal, Soon-Jo Chung, Yisong Yue, and Sabera Talukder. Generalizability under sensor failure: Tokenization+ transformers enable more robust latent spaces. arXiv preprint arXiv:2402. 18546, 2024.
Hsiang- Yun Sherry Chien, Hanlin Goh, Christopher M Sandino, and Joseph Y Cheng. Maeeg: Masked auto-encoder for eeg representation learning. arXiv preprint arXiv:2211.02625, 2022.
Nilearn contributors. nilearn. URL StL.htts:////tthbbbe com http///wwnaannnnneeeeaaeeeee
Alan D Degenhart, William E Bishop, Emily R Oby, Elizabeth Tyller-aabrr,, Steven M Chase, Aaron P Batista, and Byron M Yu. Stabilization of a brain-computer interface via the alignment of low-dimensional spaces neural activity. Nature biomedical engineering. 4(7): 6855 2020.
Rahul S Desikan, Florent Ségonne, Bruce Fischl, Brian T Quinn, Bradford C Dickerson, Deborah Blacker, Randy L Buckner, Anders M Dale, R Paul Maguire, Bradley T Hyman, et al. An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage, 316)):96--980, 2006.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Sina Faezi, Rozhin Yasaei, and Mohammad Abdullah Al Faruque. Htnet: Transfer learning for golden chip-free hardware trojan detection. In 202/ Design, Automation & Test Europe Conference & Exhibition (DATE), pp. 1484-1489. IEEE, 2021.
Gaurav R Ghosal and Reza Abbasi-Asl. Multi-modal prototype learning for interpretable multivariable time series classification. arXiv preprint arXiv:2 09636, 2021.
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, et al. Meg and ceg data analysis with mne-python. Frontiers in neuroscience, 7:70133, 2013.
Christian Herff, Dean J Krusienski, and Pieter Kubben. The potential of stereotactic-eeg for brain- computer interfaces: current progress and future directions. Frontiers in neuroscience, 14:483258, 2020.
ildoonet. 2024. http:///....taoooooooooooo/o/tta
lidbone/ppoocch-mmmaall///uuuprr wwwp:p/pww--GGuuuallaaaaauaaaaaa learning rate scheduler scheeulle for pytorch, com
Weibang Jiang. Liming Zhao, and Bao liang Lu. Large brain model for learning generic representa- tions with tremendous EEG data in BCI. In The Twelfth International Conference on Learning Representations, 2024. URL http:::s///pppeevieeeee /
net. (tte://fooommmG==0001r0r//
Justin Jude, Matthew G Perich, Lee E Miller, and Matthias H Hennig. Robust alignment of cross- session recordings of neural population activity by behaviour via unsupervised domain adaptation. feb 2022. doi: 10.45 40.585 arXiv preprint 2xXiv.200..06.59.
Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems, 15:25586-25525 2022.
Brianna M Karpowiez, Yahia H Ali, Lahiru N Wimalasena, Andrew R Sedler, Mohammad Reza Keshtkaran, Kevin Bodkin, Xuan Ma. Lee E Miller, and Chethan Pandarinath. Stabilizing brain- computer interfaces through alignment of latent dynamics. BioRxiv, pp. 2022-04, 2022.
12

---

Published as a conference paper at ICLR 2025
Arno Klein and Jason Tourville. 101 labeled brain images and consistent human cortical labeling protocol. Frontiers in neuroscience, 6:171, 2012.
Demetres Kostas, Stephane Aroca-Quellette, and Frank Rudzicz. Bendr:
using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15:653659, 2021.
Trung Le and Eli Shlizerman. Stndt: Modeling neural population activity with spatiotemporal transformers. Advances in Neural Information Processing Systems, 35:17926-17939, 2022.
Andy T. Liu, Shang- Wen Li, and Hung-yi Lee. TERA: self-supervised learning of transformer encoder representation for speech. IEEE ACM Trans. Audio Speech Lang. Process., 29:2351 2021. doi: 10. 1000/TASL..220233305566 2021 1 369
URL https:://bi..org////eee/
Ran Liu, Mehdi Azabou, Max Dabagia, Jingyun Xiao, and Eva Dyer. Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers. Advances in neural information processing systems, 35:2377-23991 2022.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05001, 2017.
Xuan Ma, Fabio Rizzoglio, Kevin L Bodkin, Eric Perreault, Lee Miller, and Ann Kennedy. Using adversarial networks to extend brain computer interface decoding accuracy over time. elife, 12: e84296, 2023.
Stephanie Martin, Iñaki Iturrate, José del R Millán, Robert T Knight, and Brian N Pasley. Decoding inner speech using electrocorticographyy: Progress and challenges toward speech prosthesis. Frontiers in neuroscience, 12:367292, 2018.
Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian Dougherty, Jessie R Liu, Peter Wu, Michael A Berger, et al. nigh-performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976): 1037-1046, 2023.
Iyad Obeid and Joseph Picone. The temple university hospital eeg data corpus. Frontiers in neuroscience, 10:196, 2016.
Josue Ortega Caro, Antonio Henrique Oliveira Fonseca, Christopher Averill, Syed A Rizvi, Matteo Rosati, James Cross, Prateck Mittal, Emanuele Zappala, Daniel Levine, Rahul M Dhodapkar, et al. Brainlm: A foundation model for brain activity recordings. bioRxiv, pp. 2023-09, 2023.
Chethan Pandarinath, Daniel J O'Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):0058115 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-leamd Machine learning in python. Journal of machine learning research, ((10ct::825-2833) 2011.
Sabera Talukder, Jennifer J Sun, Matthew Leonard, Bingni W Brunton, and Yisong Yue. Deep neural imputation: A framework for recovering incomplete brain recordings. arXiv preprint arXiv:2206.08094, 2022.
Sabera Talukder, Yisong Yue, and Georgia Gkioxari. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint arXiv:2402.164122 16412, 2024.
Sabera Talukder and Georgia Gkioxari. Time series modeling at scale: universal representation across tasks and domains. 2023.
Armin Thomas, Christopher Ré, and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances in Neural Information Processing Systems, 35:21255- 2 21269, 2022.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems. 30, 2017.
Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Brainbert: Self-supervised representation learning for intracranial recordings. The Eleventh International Conference on Learning Representations. 2022.
Christopher Wang, Adam Uri Yaari, Aaditya K Singh, Vighnesh Subramaniam, Dana Rosenfarb, Jan DeWitt, Pranav Misra, Joseph R Madsen, Scellig Stone, Gabriel Kreiman, et al. Brain treebank: Large-scale intracranial recordings from naturalistic language stimuli. Advances in Neural Information Processing Systems, 2024.
Jiarui Wang, Annabelle Tao, William S Anderson, Joseph R Madsen, and Gabriel Kreiman. Meso- scopic physiological interactions in the human brain reveal small-world properties. Cell reports, 36(8), 2021.
Graham Wideman. Orientation and voxel-order terminology: Ras, las, Ipi, rpi, xyz and all that, 2024.
ori prientterms.cm htm
Francis Willett, Erin M Kunz, Chaofei Fan, Donald Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew Glasser, Leigh R Hochberg, Shaul Druckmann, al. A high- performance speech neuroprosthesis. Nature, 620(79766: 2023.
Chaoqi Yang, M Westover, and Jimeng Sun. Biot: Biosignal transformer for cross-data learning in the wild. Advances Neural Information Processing Systems, 36, 2024.
Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer 2: multi-context pretraining for neural spiking activity. Advances in Neural Information Processing Systems, 36, 2024.
Ke Yi, Yansen Wang, Kan Ren, and Dongsheng Li. Learning with geometry-aware modeling. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
eeg representations
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes. arXiv preprint arXiv: 1904.999042.2 2019.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8980-89 0980,8987,000 2022.
Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foundation model for intracranial neural signal. Advances in Neural Information Processing Systems, 36, 2024.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zimik, Graph-guided network for irregularly sampled multivariate time series. arXiv preprint 17X:v:2110.05357. 2021.
14

---

Published as a conference paper at ICLR 2025
A ARCHITECTURES AND TRAINING
Pretrained PopTT The core Population Transformer consists of a transformer encoder stack with 6 layers. 8 heads. All layers in the encoder stack are set with the following parameters: dB 512.
H = 8, and Pdropout 0.1. We pretrain the Pop'I' model with the LAMB optimizer Youu et al. 2019) (Ir = 5e 4), with batch size of fbbcch = 256, and train/val/test split of 0.89, 0.01, 0.10 of the data. We pretrain for 500,000 steps, and record the validation performance every 1,000 steps. Downstream evaluation takes place on the weights with the best validation performance. We use the intermediate representation at the [CLS] token d 512 and put linear layer that outputs to dout 1 for fine-tuning on downstream tasks. These parameters for pretraining were the same for any PopT that needed to be pretrained (across temporal embeddings, hold-one-out subject, ablation studies).
Pretraining task: Ensemble-wise pretraining Two different subsets of channels SB C C are chosen with the condition that they be disjoint SA n SB = 0. During pretraining, the model receives the activities from these channels separate times XA { i E SA} and XB = (z i SB}- The sets XA and XB can be written as an SA x d matrix and SB x d matrix respectively. The Pop'T' receives these matrices input, along with the token [CLS]. The objective of the task is then to determine whether these states XA and XB have occurred consecutively in time or are separated by some further, randomly selected interval. The PopT produces outputs for all inputs, including the classification head, Nets E IRd Then, Mets passes through linear layer to produce a scalar Vels E R. The objective function is the BCE between this prediction and the label y
CN 1 Sn E; Vala log(p((ets)) + (1 yo.)) log(p((ets)), where you. 1( - - < 500ms)
Pretraining task: Channel-wise pretraining The token level objective is to determine whether channels activity has been swapped with activity from a random time. Precisely, activity from each channel is drawn from a time 4- All channels are drawn from the same time 4 = 4, and then 10% of the channels are randomly selected to have their activity replaced with activity from a randomly selected channel, taken from a random point in time t; * t. Then, the channel-wise outputs, Us IRd, of the Population Transformer are passed through a linear layer obtain scalar predictions Di- The objective function is the BCE between these predictions and the labels y
Lo = A B Σ, of log(p(g)) + (1 - xt log(p((r)) where y = (t, # t). Then, the complete pretraining objective is C = Lo + Ln
Non-pretrained PopTT The architecture for the non-pretrained PopTT is the same as the pretrained PopTT (above). However, no pretraining is done, and the weights are randomly initialized with the default initializations.
Linear The linear baseline consists of a single linear layer that outputs to dout = 1. The inputs are flattened and concatenated BrainBERT embeddings demb = 756, TOTEM embeddings demb 64, Chronos embeddings demb = 512, or TS2 embeddings demb 320 from a subset of channels SCC C. Thus, the full input dimension is diaputt demb
|S).
Deep NN The inputs are the same as above, but the decoding network now consists of 5 stacked linear layers, each with d₁, 512 and a GeLU activation.
Downstream Training For Pop'T models, we train with these parameters: Adam W optimizer Loshchilov Hutter 2017 Ir 5e-1 where transformer weights are scaled down by a factor of 10
(Ir; = 5c ") Abotch = 128 a Ramp Up scheduler (ildoonet (202)) with warmup 0.025 and Step LR gamma 0.95, reducing 100 times within the 2000 total steps that we train for. For Linear and DeepNN
models, we train with these parameters: AdamW optimizer (Loshchilov & Hutter 2017). lr le Tbatch = 256,2 a Ramp Up scheduler (ildoonet, 2024) with warmup 0.025 and Step LR gamma 0.95, reducing 100 times within the 17, 7,000 total steps we train for. For all downstream decoding, we use a fixed train/val/test split of 0.8, O. 1, 0. of the data. For BranBrainB models, we provide configuration files for downstream decoding parameters in our codebase.
Compute Resources To run all our experiments (data processing, pretraining, evaluations, inter- pretability). one only needs I NVIDIA Titan RTXs (24GB GPU RAM). Pretraining PopT takes days on GPU. Our downstream evaluations take a few minutes to run each. For the purposes of data processing and gathering all the results in the paper, we parallelized the experiments on 8 GPUs.
15

---

Published as a conference paper at ICLR 2025
B MODEL AND COMPUTE REQUIREMENTS
e5
e50
e90
PopT
20M
Deep NN
3M
20M
36M
Linear
3.8k
38k
69k
Brant Zhhagg al. 2024
500M
LaBraM (Jiang et al. 2024)
350M
BIOT Yang et al. 2024
3M
Table 4: Parameter counts. Since Pop'T' takes existing temporal embeddings as input, the number of trainable parameters an order of magnitude less than some recent end-to-end approaches.
GPUs
GPU type
Time to train
TELOPS
PopT
NVIDIA TITAN RTX (24GB)
2 days
2.1M
Brant Zhang et al.
2024
4
NVIDIA Tesla A100 (80G)
2.8 days
18.8M
LaBraM
Drang et
2024
8
NVIDIA Tesla A800 (40G)
BIOT
Yang
2024
NVIDIA RTX A6000 (48G)
Table 5: Pretraining compute requirements Based on published train times (none were given for LaBraM and BIOT) evident that Pop'T has smaller hardware and shorter training time requirements.
C DECODING TASKS
We follow the same task specification in Wang al. 2022),, with the modification that the pitch and volume examples are determined by percentile (see below) rather than standard deviation order to obtain balanced classes.
Pitch The PopT receives an interval activity and must determine it corresponds with high or low pitch word being spoken. For the duration of given word, pitch was extracted using Librosa's piptrack function over a Mel-spectrogram (sampling rate 48,000 Hz, FFT window length of 2048, hop length of 512, and 128 mel filters). For this task, for given session, positive examples consist of words in the top-quartile of mean pitch and negative examples are the words in the bottom quartiles.
Volume The volume of given word computed the average intensity of root-mean-square (RMS) (rms function, frame and hop lengths 2048 and 512 respectively). As before, positive examples are the words in the top-quartile of volume and negative examples are those in the bottom quartiles.
Sentence onset Negative examples intervals of activity from periods during which speech is occurring in the movie. Positive examples are intervals of brain activity that correspond with hearing the first word of sentence.
Speech vs. Non-speech Negative examples are as before. Positive examples are intervals of brain activity that correspond with dialogue being spoken in the stimuli movie.
16

---

Published as a conference paper at ICLR 2025
D DATASET DETAILS
Subj.
Age (yrs.)
Elec- Movie
Recording
Held-
trodes
time (hrs)
19
91
Thor: Ragnarok
2.07
Fantastic Mr. Fox The Martian
1.91
2.90
x
12
100
Venom
2.60
Spider-Man: Homecoming
2.42
Guardians of the Galaxy
2.66
Guardians of the Galaxy 2
3,00
Avengers: Infinity War Black Panther
3.73
1.85
Aquaman
3.52
18
91
Cars 2
1.90
X
Lord of the Rings
2.94
Lord of the Rings 2 (extended edition)
4.06
12
152
Incredibles Shrek 3
1.31
1.87
X
Megamind
1.75
109
Fantastic Mr. Fox
1.54
9
135
Megamind
0.81
Toy Story
1.32
Coraline
1.6
X
11
205
Cars 2
1.67
Megamind
1.77
4.5
121
Sesame Street Episode
1.41
16
72
Ant Man
1.0
10
12
173
Cars 2
1.57
Spider-Man: Far from Home
2.33
Table 6: Subject statisties Subjects used in BrainBERT training, and held-out downstream evaluation. The number of uncorrupted, electrodes that can be Laplacian re-referenced shown in the second column. The average amount of recording data per subject 5.55 (hrs).
HOLD OUT SUBJECT PRETRAINING GENERALIZABILITY
1.0
ROC AUC
0.8
0.6
0.4
Pitch
Volume
Sentence onset
Speech vs. Non-speech
All
Held-out
Non-pretrained PopT
Figure 10: Gains in decoding performance are available to new subjects even TOTEM pretrained POP'.. Same experiment as Figure 6 but with TOTEM embedding.
17

---

Published as a conference paper at ICLR 2025
F
RANDOM ELECTRODE ENSEMBLE PERFORMANCE
Pitch
Volume
Sentence onset
Speech vs. Non-speech
ROC AUG
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.6
0.5
0.5
0.5
10
20
30
50
10
20
50
10
20
50
10
20
40
50
Ensemble size
top Hlec set
Random blec set
Kandom Elec set
Random Elec set 3
Figure II: Downstream decoding performance on random electrode subsets. To check if our original channel ensemble ordering inflated performance, perform downstream decoding on randomly generated electrode ensembles. The random electrode ensembles perform roughly similar to our reported values, with the exception of few low-electrode count ensembles for Sentence Onset. These exceptions may be due to strong decodability of Sentence Onset at specific electrodes. Shaded bands show the standard error across subjects.
G INTERPRETATION METHODS
Connectivity analysis We start with pretrained PoP'.. To test particular channel's contribution to connectivity, we omit it from the imput (more details in Appendix Then, we consider the remaining unmasked channels and ask: how does this change the pretraining channel-wise loss? Recall that this objective is to determine if a channel has had inputs swapped with random activity. If the change in loss is large, we infer that the masked channel provided important context. Using the magnitude of this delta as a measure for connectivity, we then average across the Desikan Desikan-Killiany regions (Desikan et al. 2006) and produce a plot using mne-mceee canee-connectvitte (Gramfort et al. 2013).
Scaled Attention Weight First, we obtain an attention weight matrix across all trials which includes weights between all tokens. Then, perform attention rollout (Abnar & Zuidemal 2020) across layers to obtain the contributions of each input channel by the last layer. We take the resulting last layer of rollout weights for all channels, where the target is the [CLS] token, normalize within subject, and scale by ROC AUC to obtain the Scaled Attention Weight per channel. Finally, we take the 0.75 percentile score per DKT region (Klein & Tourville 2012), average across fine-tuning runs, and plot using Nilearn (contributors).
H FUNCTIONAL BRAIN REGION COMPARISON
Socked Allunton Weight Praztion Signl Dhsttodd
41
Gastem
movetor fortal
msrene femerral
Weads
conrrrtee propitde
indero rereovel
arthrnor Litouiale
mittle Sanial
Hain Easepparrt pieere periical
baskd tag
Figure 12: Comparison of functional maps identified by method vs traditional measures.
Scaled Attention Weight vs Fraction of Significant Electrodes per DKT (Klein & Tourville] 2012) region for the Speech vs. Non-speech task. Fraction of significant electrodes from Wang al. (2024). The Pearson's s r correlation coefficient is 0.4 between the two metrics. Error bars are standard deviation across 5 fine-tuning runs.
18

---

Published as a conference paper at ICLR 2025
I CONNECTIVITY
1. Measure baseline
2. Measure perturbation Losses (given by channel-wise objective)
Losses (given by channel-wise objective)
0000000000
000
PopulationTransformer
PopulationTransformer
000 000000
Figure 13:
Schematic of connectivity analysis determine the influence of some channel, on another channel j. first measure the baseline performance of the pretrained PopT on the replace-only objective. Then, we omit from the input, and measure how the performance on channel-wise objective is perturbed j. See also Algorithm
Algorithm 1 Connectivity measurement between channels and j
Require: j < i, x E ER Nexd P(x:)
Nc is the number channels, is the embedding dimension. P is pretrained Pop'T, Ütuseline E Ne
Youseline
while n I N Naaples do
Tomitted &- Concat(zr|: :), x 1:])
b Remove the jth channel from the input
Uperturbed P( (mooieted)
Influence = Voaseline pperturbed
>> How much did the prediction change?
AvgConnectivity + AvgConnectivity + Influencelji/n
end while
Coherence
PopT
0.010
12
16
20
21
0.008
46
48
48
52
52
56
56
0.004
60
60
0.002
84
84
88
88
Figure 14: Electrode level connectivity. Connectivity between all channels for the same subject shown in F Outliers at the 2-percentile are snapped to color map floor and ceiling.
19

---

Published as a conference paper at ICLR 2025
Connectivity from coherence analysis
Connectivity from pretrained PopT
Connectivity from coherence analysis
Connectivity from pretrained PopT
Figure 15: Region connectivity for test subjects. Continued from Figure g this figures shows the rest the test subjects. We compare between traditional connectivity analysis performed coherence (top in each section) and the analysis based our PopT pretrained weights (bottom row each section). We note that our analysis usually recovers the strongest points connectivity from the traditional analysis. Coherence computed using seikit-leam's 's (Pedregosa al. 2011 a i aigalll ccherence.
20

---

Published as a conference paper at ICLR 2025
Connectivity from coherence analysis
Connectivity from pretrained PopT
Connectivity from coherence analysis
50
Connectivity from pretrained PopT
Figure 16: Electrode connectivity for test subjects. Continued from Appendix this figures shows the of the test subjects. Order given in Figure 15
Subject
Correlation
Subject 0.42
Subject 0.66
Subject 0.54
Subject 0.55
Subject 0.44
Subject 0.44
Subject 10 0.50
Table 7: Pearson's correlation coefficients between connectivity matrices for test subjects shown in Table and Figure 16
21

---

Published as a conference paper at ICLR 2025
J ADDITIONAL ABLATIONS
Pitch
Volume
Sent.
Onset
Speech/Non-specch
PopTT
0.74 0.03
0.87 0.03
0.90 + 0.01
0.93 0.02
Pop'T' w/ gaussian blur
0.73 = 0.03
0.86 = 0.03
0.88 = 0.02
0.91 + 0.08
Pop'T w/o channel randomize
0.71 = 0.02
0.85 + 0.03 0.89 = 0.01 0.91 + 0.03
Table 8: PopT additional ablation study. We pretrain additional variations of PopT to see their effect downstream decoding. In 'PopT w/ gaussian blur' we fuzz the input coordinate values with a Gaussian, (N = 0, a = 5). before position encoding. We hypothesized that augmenting the coordinates during training would help the model generalize better, but improvements were shown. 'PopT w/o channel randomize' replaces a channel with a channel's s own activity at another time part of the channel-wise pretraining task. We hypothesized that this would help the model identify the channel's specific variability across time, but no improvements were shown. Shown are ROC-AUC mean and standard error across subjects evaluated 90 electrodes.
K
FROZEN ENSEMBLE SCALING
Pitch
Volume
Sentence onset
Speech vs, Non-speech
0.8
0.8
0.8
0.8
ROC AUC
0.7
0.7
0.7
0.7
0.5
0.5
0.5
10
20
30
Ensemble size
Frozen Pretrained PopT
Frozen Non-pretrained PopT
Figure 17: Pretraining is critical to frozen PopT performance that scales with the number of channels. Transformer weights are frozen; only the linear classification head is updated during fine-tuning. As in Figure 3 we see that pretraining results in better downstream decoding and scales with the number of added channels. Bands show standard error across subjects.
22


Reviews:
[{"review_id": "Ayq8tcDvS1", "reviewer": "Reviewer_jUF4", "summary": "The manuscript introduces the Population Transformer (PopT). This self-supervised framework tackles two key challenges in neural time-series analysis: sparse electrode distribution and variable electrode placement across subjects and datasets. PopT uses a transformer architecture that combines temporal embeddings with spatial information and trains through self-supervised learning to aggregate signals across variable electrode layouts. The model outperforms baselines on neural decoding tasks, requires less training data, generalizes to new subjects, and reveals interpretable brain connectivity patterns.", "questions": "- Figure 1 has a short-time Fourier transform, but I did not find it to be discussed anywhere in the text.\n- It would be great to see how much Gaussian fuzzing contributes to the final performance. EEG is spatially very smoothed.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Validation across two types of neural recordings: EEG and intracranial EEG (iEEG)\n- Thorough comparison against common aggregation methods and state-of-the-art models\n- Strong ablations to test the importance of ensemble-wise loss, channel-wise loss, position encoding, and reconstruction loss", "weaknesses": "- In Tables 1, 2, and 3, consider adding statistical testing (e.g., Wilcoxon) between the best model and others. Then, correct p-values for multiple comparisons (e.g., Holm). Otherwise \"We see significant improvements in performance\" is not justified.\n- I am confused about construction for \" (1) ensemble-wise —the model determines if activities from two channel ensembles occurred consecutively, requiring an effective brain state representation at the ensemble-level\". How do we know if activities from two channels occurred consecutively? In other words, could you explain how you construct pairs? How do you get the states?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6f6qFiZAX", "reviewer": "Reviewer_VeNz", "summary": "The authors introduce a self-supervised learning framework called PopT to learn embeddings for EEG and iEEG recordings for the improving downstream decoding analysis. In general, the problem is hard because of the spatially varying channel placements in different experiments. Their method leverages unannotated data to optimize channel-level and ensemble-level objectives, which helps them build generic representations and also allows to capture some dynamical relationships. Their tests show improved decoding on held-out subjects. Based on the interpretation of the weights, they propose a new method for brain region connectivity analysis and for identifying candidate brain regions.", "questions": "In Fig. 8, how does the connectivity shown compare with cross-correlation or some other metrics that are used in the field?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- They tackle an important and difficult problem for the field\n\n- Interesting approach backed with strong empirical results.\n\n- I appreciate the work authors put in doing new analyses and interpreting the weights. \n\n-Well written and interesting.", "weaknesses": "I have some questions about their connectivity analyses below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dxw3b0qGez", "reviewer": "Reviewer_LNxL", "summary": "The authors introduce Population Transformer (PopT), a transformer-based and contrastive learning approach for decoding neural time series data. The PopT aggregates and decodes channel-wise temporal encodings of neural time series data from, e.g., a pretrained BrainBert but is agnostic to the encoder. The authors introduce two contrastive loss terms for training the PopT. One contrastive loss term requires the PopT to learn temporal representations for each channel, and the other contrastive loss term requires the PopT to learn spatial representations across channels. The PopT receives electrode placement as input, allowing it to decode neural activity across varying electrode placements (e.g., from several subjects). In an application with EEG and iEEG, the paper presents that the PopT converges with fewer samples than other shown methods, achieves superior decoding performance compared to other shown methods, and provides evidence for the PopT's ability to generalize across subjects. The authors also suggest measuring and interpreting functional connectivity based on the ablation of individual channel input to their PopT and performance degradation for all other channels. They also suggest interpreting the weights of the model to map the task-associated function to specific electrodes.", "questions": "# **Major Questions**\n\n1. You show that the decoding performance increases with more subjects and decreases with holding out a subject. However, these experiments are limited because they consider a validation on specific channels of one specific subject, or holding out one specific subject. Is this also valid across arbitrary channels and arbitrary subjects?  \n2. Figure 4: Do the baseline aggregation approaches have fewer free parameters and nonlinearities than the PopT? Could you report the number of free parameters and nonlinearities of all networks and factor this into the discussion of your results?  \n3. Results 366-370: The model's ability to generalize is a central claim. However, the hold-one-out analysis seems somewhat limited. What if the held-out subject was a lucky draw? Can this comparison be done in a crossvalidation? Can the performance decrease be computed for hold-k-out and presented as a function of k? Separately, is there a different model class that can generalize to provide a comparison group in this test?  \n4. 371-377 and Figure 6 and 7: I would have expected that \"All subjects\" and \"0 subjects\" correspond to \"All\" and \"Non-pre-trained PopT\" but the numbers are different. Can I rely on this data? Why is there a mismatch?  \n5. Figure 7: What if \"across channel ensembles 5-30 on a held out test subject\" is a lucky draw? Could this be more robustly supported with an extended analysis?  \n6. 414-416: To say this, I would expect an analysis of representations resulting from different loss functions. Can one show that the contrastive learning objective results in the clearest association/separation of inputs?  \n7. 468-470: One obtains N-squared performances, where N is the number of channels. Can one present the raw functional connectivity matrices inferred this way in addition to the aggregate? This could help to gauge how effective the approach is.  \n8. 468-470: Can one compare the results to the functional connectivity inferred with traditional cross-correlation methods on the measurements? This would help to illustrate the true utility of the PopT-based analysis.  \n9. 475-485: Same here. Can one compare to neuroscience ways to characterize the activated brain area, e.g., z-score based?  \n10. Results 296-297: What are differences in, e.g., architecture or training that makes LaBraM outperform the PopT? Could one provide possible explanations for why LaBraM outperforms the PopT on EEG decoding (whereas end-to-end approaches specifically designed for iEEG do not)?  \n11. Figure 4: there is a much more significant improvement for the sentence onset and speech vs. non-speech tasks over the improvement in the pitch and volume tasks. Could one provide an explanation for this in the results section?  \n12. In contrast to one of the central claims in the abstract, 23-24, the codebase is messy, and lacks documentation, clear examples, instructions, and tutorials. The readme title is BrainBert 2.0. Could the codebase please be significantly cleaned and documented?\n\n# **Clarifications and Minor Questions**\n\n13. Abstract 16-18: The sentence \"lowers the amount of data for downstream decoding experiments\" is compatible with the presented results (Fig. 4 and 5). However, the verb 'lowers' requires a comparison group, and 'downstream decoding experiments' is vague. The meaning of 'downstream' is also ambiguous. Could you please rephrase this?  \n14. Abstract 18-20: Calling the PopT computationally lightweight in comparison to end-to-end approaches is misleading because the PopT only works with pretrained weights from BrainBert. Could you please rephrase this?  \n15. Abstract 18-20: Calling the PopT \"more interpretable\" when not comparing the interpretations to existing traditional neuroscientific methods for interpretability is misleading. Could you please rephrase this?  \n16. Abstract 18-20: What does 'retaining competitive performance' mean? In all iEEG data, it outperforms the end-to-end method used, in all EEG data it is outperformed by LaBraM. Could you please provide this information without ambiguity?  \n17. Abstract 21-23: What are \"massive\" amounts of data according to the authors?  \n18. Discussion 524-525: \"\\[ … \\] could provide an even higher resolution latent space \\[ … \\]\". Higher than what? Could you please rephrase this?  \n19. Conclusion 538-539: I wouldn't know how to use it \"for plug-and-play pretraining\". The codebase is messy and lacks clear examples, instructions, or tutorials. Is this a work in progress or the cleaned-up codebase? \n20. 206-208: The explanation is not reader-friendly and not comprehensible for someone who does not know the types of neurorecordings. Could you please explain to the reader why intracranial and scalp electroencepholography lead to two different types of time-series data, and why and which resolution extremes these represent?  \n21. 3 Population Transformer Approach 178-179: these are rather \"contrastive\" components and not \"discriminative\" components. Could one please adopt \"contrastive\" instead of discriminative throughout the text?  \n22. Figure 2, 6, 7: Could one use violin (or box) plots instead of bar plots to convey the variability and scatter the individual data points on top to convey the sample size?  \n23. 375: \"potentially due to adaption to the temporal embeddings used\" meaning unclear. Could you please rephrase this?  \n24. 363-365: These sentences seem contradictory, which might be easily resolved with rephrasing them. Could you please rephrase this? Part of the Figure 5 caption could go here into the main text.  \n25. Results Figure 4: What is 'highly' sample efficient?\n\n# **Minor Formatting**\n\n26. 214-215: Could you please remove the parenthesis inside the parenthesis?  \n27. Table 1: Could you please emphasize your result better by bolding the label Pretrained PopT or star it for \"best overall\"?  \n28. Results 264-269: hard to read. Could you please remove the parenthesis inside the parenthesis?  \n29. Figure 3 is referred to before Figure 2 in the text. Could you please renumber the figures to maintain clarity?  \n30. Table 1 and Figure 1: Could you please make latex keep them inside the section where they are referred to?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The claimed contributions of the PopT would present an exciting leap forward in decoding neural time series data. This is because unreliable and slow decoding of neural activity limits applications and scientific insights from models. A method that requires comparably few data, no labeled data, and learns representations that generalize across subjects regardless of electrode layout makes decoding more fast and reliable for new subjects and unlocks new applications. The ability of the PopT to generalize across subjects is a major strength of the approach. Another strength is that the PopT relies on pre-trained encoding models that it is agnostic to. The formulations of the contrastive learning objectives seem original. The suggested ways to interpret the model to understand the data are interesting.", "weaknesses": "The paper presents multiple weaknesses that could be addressed to improve its clarity and reliability. \n\nFirst, the abstract describes PopT's benefits, such as \"computationally lightweight\" and \"more interpretable,\" which lacks precision without clear comparison groups or quantitative metrics. For example, calling PopT computationally lightweight in comparison to end-to-end trained models is misleading since it relies on pre-trained, e.g., BrainBert weights, which already bear significant computational demands. As another example, the interpretability analysis does not include any comparison method.\n\nIn addition, the generalizability claims lack thorough validation: the hold-one-out subject analysis does not sufficiently establish robustness. This experiment could benefit from a broader validation, e.g., cross-validation approaches. \n\nAdditionally, when comparing the performances to other models the analysis does not account for the differences in free parameters and nonlinearities across models. \n\nLast, the codebase requires significant improvements, as it is messy, poorly documented, and lacks proper instructions or tutorials. For example, the readme is incorrectly titled BrainBert 2.0. This contrasts the authors claim of it being \"plug-and-play.\"\n\nI am not up-to-date and fully familiar with related work. This is why my expertise might be insufficient to fully weigh the significance of the central contribution against the current limitations of the paper in the clarity and reliability of the results. \n\nI am generally excited about the direction of the work and happy to give a higher evaluation once the presentation of the current version of the paper is significantly improved and the detailed questions below are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3uZp7NCKB", "reviewer": "Reviewer_b3HQ", "summary": "This paper introduces a self-supervised framework, the Population Transformer (PopT), designed to learn population-level representations for large neural recording datasets.  PopT addresses challenges related to sparse and variable electrode distribution across subjects and datasets by using pretrained embeddings. By using a modular approach the model is more flexible and lightweight. The authors claim that the approach is computationally efficient, interpretable, and performs competitively against end-to-end methods.", "questions": "1. Table 3 refers to “PoPT w/o group-wise loss”, is the group-wise loss the same of ensemble-wise loss. To improve clarity, could the authors consider using consistent terminology throughout?\n2. It would be insightful to see how one of the baseline models, specifically BrainBERT from Table 1, performs on the hold-out dataset, similar to the performance reported for PoPT in Figure 6. This comparison would provide additional context for the robustness of PoPT relative to other methods. Could the authors report the performance of BrainBERT on the hold-out dataset in a format similar to Figure 6?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "This paper demonstrates strong rigor by evaluating the proposed method across two types of neural time-series data (iEEG and EEG), enhancing the generalizability and robustness of its findings. The authors plan to share both the data and code upon acceptance, promoting transparency and reproducibility within the community. Additionally, they test their method using hold-out subject tests to validate the model’s performance on unseen data, which is essential for assessing real-world applicability. The method’s effectiveness is benchmarked against a diverse set of models, providing a comprehensive view of its performance relative to other approaches. Furthermore, ablation studies are conducted to examine the contribution of different components in the proposed framework, offering valuable insights into how each part enhances overall model performance.", "weaknesses": "The study offers a comprehensive evaluation of PopT, and assess it using various experiments. However, the paper would benefit from clearer organization and writing, particularly in emphasizing the significance of each experiment.\n\n- **Introduction and Field Context**: Despite the extensive experimentation, the paper lacks a clear introduction to the field and a focused statement of its goals. Important background information and foundational definitions are often buried in the appendix rather than integrated into the main text.\n    - For instance, a key innovation of the study is the use of channel-level and ensemble objectives. Providing a more detailed literature review on current approaches to these objectives would help the reader understand the limitations of existing methods and the advantages of the proposed improvements. This contextualization would make the study’s contributions clearer and more impactful.\n    - The connection between experiments is also unclear, with critical information about the pretrained PopT model only appearing in the appendix. Providing a clear description of the data used for pretraining within the main text would help readers understand the study’s foundation and goals more effectively.\n    - Similarly, Table 3 introduces PopT with an L1 reconstruction loss as an additional experiment. However, this experiment is not discussed in the text, and it appears tangential to the core contributions of the study. Omitting such details could allow space to expand on more relevant analyses.\n- **Benchmarking Choices**: While Chronos/TS2Vec and BrainBERT/TOTEM are used as benchmarks for PopT, it remains unclear why these specific models were selected. The authors could strengthen the study by discussing the criteria for choosing these benchmarks.\n- **Minor Presentation Issue**: In Figure 8, the plot slightly overlaps with the title on the left image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ev1h5HDGaX", "reviewer": "Reviewer_vWCS", "summary": "The Population Transformer (PopT) is a self-supervised framework designed to learn brain-wide neural activity representations. The approach aims to address common challenges faced by invasive iEEG and non-invasive EEG neural recordings, namely 1) sparsity of recording channels within the brain and 2) variability in channel positioning and reliability. The authors propose a smart data embedding and combining two loss functions that allow pre-training of the network using data from multiple datasets and individuals. Pre-training is shown to be fundamental to lowering the amount of data necessary for fine-tuning neuroscientific tasks, such as decoding experiments. Finally, the authors show experiments and analyses to support the claim that this framework is interpretable and generalizable.", "questions": "**Clarity**:  the paper is not really neat. Moreover, there is some sloppiness in the mathematical notation and technical explanation of the method. Finally, the figures are uneven and quite messy. Follows a list of detailed complaints:\n- general lack of dimensionality: in many points, it is not clear what the dimensions are of the variables. for example, lines 160-161, what is the dimension of $X_B^t$? Is the “+” used for concatenation or for an actual sum? What is the output dimensionality of the temporal embedding model $B$? Some of these are in Appendix A, but it would be useful to streamline and clarify.\n- $[CLS]$ token: it is stated nowhere that this is the token used for downstream decoding/classification tasks, and what space it belongs to. Is it only binary? Please clarify in the text or figure legend.\n- Self-supervised Loss (line 173-197): the losses are described in words, but it would be useful and much clearer to write them in formulas. Right now, this paragraph is a bit messy and somewhat colloquial.\n- Fine-tuning (line 199-201): colloquial and relies on details only available in the appendix. It would also be useful to formally define what is meant by a decoding task.\n- Figures: Fig 1 is unclear and messy. Why does it go from bottom to top, rather than the opposite? Currently, the “Population Transformer inputs” title of Fig 1a is just above the output, making it very confusing. Moreover, the figure has at least four different font sizes. The “+” used in temporal embedding + 3D position is confusing; does it refer to concatenation or addition. The title of the colorbar near the STTF is nearly illegible. Panels b and c could provide clearer information on the loss. Fig 3-4: font sizes widely different across figures; Fig 3 text is hard to read on print. Same for Figs 6-7-8. Fig 8 left title is partially covered by the plot.\n\n**Interpretability**: the claim of interpretability requires some work to be convincing/useful. For example, the connectivity analyses are far from clear (also after reading the Appendix!). It is not immediately clear how one infers (pairwise) connectivity by masking out the activity of one channel at the time and measuring the degradation in channel-wise loss. Can you add a few details on that? And do we learn anything new from this analysis as compared to previous coherence or cross-correlations analyses? Can we have a few more qualitative and/or quantitative comparisons? Same questions go for the functional brain regions from attention weights. Moreover, what are the possible caveats of inferring connectivity or functional regions from PopT?\n\n**Minor comments**:\n- line 153: *adapt* -> adopt?\n- lines 132, 158, …: BrainBERT was used but never cited.\n- line 163: *sinusoidal position encoding*: is there a simple formula or a couple of words to explain this method without necessarily having to read the cited paper?\n- line 171: and $\\tilde y_i$ *and(?)* respectively, …\n- line 188: …is *again(?)* the binary cross-entropy…\n- line 374, 478, 483: Figure X -> (Figure X)\n\n**Additional questions**:\n*[These questions popped up while first reading the paper; some got clearer on a second read, some not. I don’t think it’s necessary for the authors to answer all these questions, but hopefully, they’ll be useful for improving discussion and intro, or future experiments, or mybe just food for thoughts.]*\n\nThe paper claims to learn representations of neural activity; do the authors mean the final output of the PopT or only the CLS output? Can one also learn a neural representation without using a CLS token? Can this representation be thought of and used as a dimensionality reduction method? Who are the intended users of this method? Can this method be used for non-human invasive research, such as calcium imaging, neuropixels, etc? Is it possible that the discriminative nature of the pre-training objective leads to potentially misleading representations in case of noisy/faulty channels?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "This paper has many merits. The idea and approach, from a computational neuroscience perspective, are novel and very interesting. The proposed framework could benefit the neuroscience community at large: combining data from multiple datasets, and learning brain-wide representations that are not idiosyncratic to a specific individual are key to advancing neuroscience research. Many points made throughout the paper and several experiments are convincing and valuable, and I don’t think much work is required in terms of experiments/compute.", "weaknesses": "On the flip side, the paper is, at some points, lacking in precision and clarity, at some points a bit colloquial, and at other points, unnecessary jargon. I think that the paper would be stronger, and more accessible, if the authors put some effort into clarifying and streamlining it. Moreover, the claims of interpretability are currently unclear and potentially overstated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1r648s8HrN", "reviewer": "Reviewer_P8gU", "summary": "In this paper, the authors propose a self-supervised learning framework for neural time series data. The model is trained using ensemble and channel-wise discrimination strategies to address the problem of sparse and variable electrode distribution across subjects and datasets. The authors presented the validity of the proposed method on two datasets with diverse analyses.", "questions": "Page 3, line 161: According to the paper's description, the input to the temporal encoding is a single value of a channel at a time t. Is this correct?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "In this paper, the author tried to solve the problem of sparse and variable electrode distribution across subjects and neural time series data datasets by modeling with a single-channel embedding method.\n\nThe effectiveness of the proposed method was supported through various experiments and analyses.\n\nIt achieved high performance with fewer computational resources compared to existing pre-trained models.", "weaknesses": "It is unclear how exactly the position of each channel is defined in spatial position encoding. The reference provides coarse positions only, which is important information in spatial relation learning. \nPage 4, line 166: The notations used in position embedding are not defined. \n\nSupporting materials must be presented for the Gaussian fuzzing in channel embedding to ensure its effectiveness in avoiding overfitting.\n\nThe experiments are somehow limited. More rigorous and thorough experiments should be conducted over diverse datasets. Other more challenging tasks, e.g., TUEV in the TUH EEG corpus, should be conducted and compared with the comparative methods.\n\nThe results in the ablation study (Table 3) suggest that position encoding played a significant role in performance improvements, while other proposed loss functions had marginal effects. In this regard, the related contexts should be provided in detail. \n\nRegarding the connectivity construction in Section 6, it is not straightforward to understand how the connectivity is measured. More detailed explanations and rationales should be provided regarding why the proposed method can estimate channel connectivity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)