A foundation model with multi-variate parallel attention to generate neuronal activity
Authors: Francesco S. Carzaniga
Michael Hersche
Abu Sebastian
Kaspar Schindler
Abbas Rahimi
Venue: ICLR 2026 Poster (2026)
Keywords: time-series
ieeg
neurology
foundation model
attention
transformer
Primary area: applications to neuroscience & cognitive science
TL;DR: We introduce MVPA, a new attention mechanism for multi-variate time-series, and use it to build MVPFormer, a foundation model that achieves state-of-the-art performance on multiple tasks and is trained on the largest open iEEG dataset to date.

Abstract: Learning from multi-variate time-series with heterogeneous channel configurations remains a fundamental challenge for deep neural networks, particularly in clinical domains such as intracranial electroencephalography (iEEG), where channel setups vary widely across subjects. In this work, we introduce multi-variate parallel attention (MVPA), a novel self-attention mechanism that disentangles content, temporal, and spatial attention, enabling flexible, generalizable, and efficient modeling of time-series data with varying channel counts and configurations. We use MVPA to build MVPFormer, a generative foundation model for human electrophysiology, trained to predict the evolution of iEEG signals across diverse subjects. To support this and future efforts by the community, we release the SWEC iEEG dataset, the largest publicly available iEEG dataset to date, comprising nearly 10,000 hours of recordings from heterogeneous clinical sources. MVPFormer leverages MVPA to achieve strong generalization across subjects, demonstrating expert-level performance in several iEEG tasks. MVPFormer surpasses state-of-the-art (SOTA) Transformer baselines in seizure detection across the SWEC, the MAYO, and the FNUSA datasets, while also achieving SOTA performance on four Brain TreeBank iEEG decoding tasks (volume, pitch, onset, and speech). We further validate MVPA on standard time-series forecasting and classification tasks, where it matches or exceeds the performance of existing attention-based models. Together, our contributions establish MVPA as a general-purpose attention mechanism for heterogeneous time-series and MVPFormer as the first open-source, open-weights, and open-data iEEG foundation model with SOTA clinical performance. The code and weights are available at https://github.com/IBM/multi-variate-parallel-transformer. The SWEC iEEG dataset is available at https://huggingface.co/datasets/NeuroTec/SWEC_iEEG_Dataset.

Published as a conference paper at ICLR 2026
A FOUNDATION MODEL WITH MULTI- VARIATE PARAL- LEL ATTENTION TO GENERATE NEURONAL ACTIVITY
Francesco S. Carzaniga
Michael Hersche
Abu Sebastian
IBM Research- - Zurich
IBM Research- - -urich
IBM Research - Zurich
Bern University Hospital
Kaspar Schindler Bemn University Hospital
Abbas Rahimi IBM Research Zurich
ABSTRACT
Learning from multi-variate time-series with heterogeneous channel configura- tions remains fundamental challenge for deep neural networks, particularly in clinical domains such as intracranial clectroencehhhalgrrppy (iEEG), where chan- nel setups vary widely across subjects. In this work, introduce multi-variate parallel attention (MVPA), a novel self-attention mechanism that disentangles content, temporal, and spatial attention, enabling flexible, generalizable, and ef- ficient modeling of time-series data with varying channel counts and configura- tions. We use MVPA build MVPFormer, generative foundation model for human electrophysiology, trained to predict the evolution of iEEG signals across diverse subjects. To support this and future efforts by the community, we release the SWEC iEEG dataset, the largest publicly available iEEG dataset to date, com- prising nearly 10,000 hours of recordings from heterogeneous clinical sources. MVPFormer leverages MVPA to achieve strong generalization across subjects, demonstrating expert-level performance in several iEEG tasks. MVPFormer sur- passes state-of-the-art (SOTA) Transformer baselines in seizure detection across the SWEC, the MAYO, and the FNUSA datasets, while also achieving SOTA per- formance four Brain TreeBank iEEG decoding tasks (volume, pitch, onset, and speech). We further validate MVPA on standard time-series forecasting and clas- sification tasks, where it matches or exceeds the performance of existing attention- based models. Together, our contributions establish MVPA as a general-purpose attention mechanism for heterogeneous time-series and MVPFormer as the first open-source, open-weights, and open-data iEEG foundation model with SOTA clinical performance. The code is available at /tattts::/githubb com/ IBM/ The SWEC IEEG dataset is available http:///////ggggggeeeeeeaaa//aaa
INTRODUCTION
The increasing availability of multi-variate time-series data across domains, from financial data to sensor networks to clinical recordings, has driven demand for seneral-purpose neural architectures capable of learning from such data (Nic et al.) [203]] Jin al. 20244) Wang et al.) (222b)] Guetschel et al. 2024). A fundamental challenge in this setting is channel heterogeneity: different sensors (or channels) often earry information that both structurally and semantically non-uniform, while the number and the location of channels may vary across instances. This is especially pronounced in intracranial electroencephalography (iEEG; NNuezz & Srinivasan 2006), where each subject's elec- trode layout is unique and tailored to clinical needs. iEEG models (Kuhlmann al. 2018 Cho &
Jang 2020 Thuwajit et al.) 2022 Wang et al. 2023 Saab et al. (024) often require subject-specine adaptation account for setups, yet they still struggle generalize. Consequently, effec-
"Correspondence Frrezurrica ch. ib. com

---

Published as a conference paper at ICLR 2026
Duwry
6
MVPA
A
oh
Content
Channel
Solee Queryy
MVFA
Figure 1: Multi-variate parallel attention (MVPA). (a) The input signal is divided into temporal and spatial segments. Each query-key interaction is computed for all keys within local window. (b) MVPA decomposes attention into three components: content-based, computed per segment without positional encoding; time-based, shared across channels and dependent only on temporal distance; and channel-basede shared across time steps and dependent only on spatial distance. (c) The final attention is the sum of these three components, each capturing a distinct aspect of the data.
tive learning from multi- variate time-series demands models that are flexible and Channel-agnosse without sacrificing locality or the ability to generalize.
In this work, we introduce multi-variate parallel attention (MVPA. Figure D.. a novel self-attention mechanism addressing the structural challenges of channel heterogeneity. MVPA decomposes at- tention into three components: content-, time-, and channel-based components. Thus, it allows the model to separately learn the semantics of the signal, its temporal dynamics, and spatial (inter- channel) structure. MVPA enables flexible and efficient processing of time-series data, without relying on fixed channel positions or global positional encodings.
To highlight MVPA's ability handle heterogeneous and clinically relevant time-series, we apply it to the particularly challenging domain of iEEG. Indeed, mentioned above iEEG recordings present an idcal testbed for models designed to handle variable multi-channel structure. We use MVPA build MVPFormer, foundation model for human electrophysiology trained via generative pre-training to predict the evolution of brain signals.
MVPFormer is trained the SWEC iEEG dataset, the largest available iEEG corpus to date with nearly 10,000 hours of multi-channel recordings (or 540,000 channel-hours), collected over a decade in clinical settings and made publicly available part of this work. Using this long-term ictal iEEG dataset, we show that MVPFormer not only models neuronal activity during both normal and ictal states, but also generalizes across patients zero-shot within the same task, and outperforms pre- vious approaches on clinically relevant benchmarks. At the same time, MVPFormer also enables diverse downstream applications through finetuning, including seizure detection on multiple other institutional datasets (Nejedly et al., (020) and four iEEG decoding tasks from the Brain TreeBank dataset (Wang et al., (2024)). Remarkably, MVPFormer surpasses an equivalent purely discrimina- tive version which has not undergone generative pre-training, strengthening the validity of founda- tion models in iEEG.
We further evaluate MVPA on classical time-series benchmarks, including ETTh and Weather (Zhou cet al.) 2021 Wu et al. 2021) for forecasting and EthanolConcentration, FaceDetection, and oth- (Liu & Wang 2024) for classification. Here, MVPA matches outperforms tfate-of-the-an (SOTA) models. These results establish MVPA as a competitive attention mechanism for general- purpose time-series beyond iEEG.
Our contributions are: (1) Multi-variate parallel attention (MVPA), a novel self-attention mecha- nism that separately attends content, temporal, and spatial structure, enabling strong generaliza-
2

---

Published as a conference paper at ICLR 2026
tion across multi-variate time-series with heterogeneous channels; (2) MVPFormer, foundation model for human electrophysiology, powered by MVPA and trained on the largest iEEG corpus available, showing superior generalization across subjects and clinical tasks compared to models which use vanilla attention like Brant-2 (Yuan et al. 2024a); (3) The release of the SWEC iEEG dataset, the largest iEEG dataset publicly available date, with almost 10,000 hours of highly curated and labeled iEEG recordings.
Moreover, make all our contributions open source, realizing the first open-data, open-code, and open-weights iEEG foundation model.
2 MULTI-VARIATE PARALLEL ATTENTION (MVPA)
This section introduces vulti-variate parallel attention (MVPA), the first contribution of this work, shown in Figure [ We start with vanilla attention for 1D sequences, then present dual-coded attention for 2D sequences, which has higher computational costs. Building on this, we derive MVPA, which efficiently attends both temporal and spatial aspects of multi-variate time-series data.
2.1 MVPA OVERVIEW
Vanilla attention (Vaswani et al. 2017) operates on sequences of embeddings (æ1, 22, TT) of dimension d (æy E ).. It computes the attention between two tokens at positions (i, as follows
a_ angilll (æ; + S.) W We (æ, + Ss),
where az the query token and 2j the key token. S is the positional encoding, a vector with the same dimensionality (d) that helps to distinguish between different positions in the sequence. W and W3 are the learnable query and key matrices.
While vanilla attention has been successfully applied to ID sequences, its extension to multi-variate time series (i.e., 2D sequences) is not obvious. Specifically, we aim to process sequences of the form (æ1,1, Z1,2 .... Fe.t; fec........ .... #c,r). where indicates the space and t the time dimension, while maintaining the embedding dimensionality (i.c., &e,t I.. One approach is flatten the 2D data to a ID sequence (c.g., as done by the Vision Transformer; Dosovitskiy et al. 202T]; however, this will yield loss in spatial structure. Instead, introduce two separate learnable positional codebooks, representing space (C) and time (T). By equipping self-attention with this dual encoding can treat the two dimensions individually, which is fundamental in recovering their interplay and would not be possible with vanilla attention:
a o,t,e'' = (xe,+ + T, C.)"" Wa + T Cor).
Dual attention allows us to exploit the relationship between time and space at the attention level, the most basic computational unit of a Transformer. We believe this allows the architecture to model the time-series at a lower level, and hence more effectively. However, the dual attention mechanism is computationally expensive, as it computes second- -oddr correlations between time and space.
For this reason, want to squash these correltoos.....ss..............c Specifically, want to push much of the spatio-temporal computation as possible to the lower levels of processing without overwhelming it. In contrast, all Transformer models equipped with vanilla attention require ancillary structures to process any relation between time and space (Nie et al. 2023 Zhang & Yan 2023] Wen et al.) 2022). Inspired by Transformer-Xl. (Dai al.] (2019), we encode the relative distance in the two dimensions between the segments separately and introduce new learnable bias terms (u, v, w). In contrast Transformer-XL, however, MVPA operates 2D signals by treating the two dimensions differently and providing sub-quadratic solution to learn these interactions (see Appendix A.2 for a detailed comparison).
To do so, we operate the following modifications to disentangle space and time:
(Te CT. Wea -> u WK. Tete
For better readahility, we describe the attention computation for single head without activation. In prac- tice, we generalize multi-head attention and apply a consecutive softmax non-linearity.
3

---

Published as a conference paper at ICLR 2026
(Te "C,+"
Wania [Te-e/Co-ee)
which characterize the relative enror with respect the full quadratic dual-encoding attention.
Finally, after removing the second-order effects we rearrange the expanded equation into three re-
lated groups:
act,ce . = Fe,, WTTTVVV Ws. Fe'r + u Wx. vew
MVPA
Content-based attention
(1)
1h W Wk. Te- + v² Wh, Te-
Time-based attention
(2)
+ x W Wx. Co- + wD W.
Channel-bused attention
(3)
and compute the final softmax attention as:
A softmax (a M
(4)
vd
The three terms above are the attentional components of MVPA. Content-based attention only at- tends to the content of query and key, without any positional encoding. In this component we compute the relationship between two raw segment embeddings, so modulate the final attention output without considering any structure of the signal. Time-based attention only attends to the query and the distance in time with the key. In this component, only the relative distance in time is considered, allowing for arbitrary signal lengths without loss of generality. Finally, channel-based attention only attends to the query and the distance in space with the key. Similarly the time-based component, also the distance in the channel-base -based component is relative.
This feature is particularly interesting for the channel-based component, given the heterogeneity of possible channel setups. Specifically, the channel component uncovers the hidden connection map between the spatial locations from its initial random initialization, as shown in Appendix [G.IT] In fact, while the use of the absolute position of the electrodes has produced notable work (Tiraa et al.) 2023), much of the literature has shown that such information might not be necessary (Schindler et al. 2006; 2008). Therefore, our relative encoding scheme affords maximum flexibility while not sacrificing performance, as MVPFormer outperforms the SOTA both on the seizure detection task (see Section [5.2.]] and on the four tasks of the Brain TreeBank dataset (see Section 5.2.2 which explicitly provides the absolute channel positions.
2.2 EFFICIENT IMPEEMENTATION OF MVPA
As MVPA's computational cost is still quadratic in space and time, employ several techniques to further reduce the complexity and enable the efficient processing of very large signals. We present here the main techniques, while the details can be found in Appendix A
Efficiently computing the time- and channel-based terms requires two main techniques. First, we recognize that it is unnecessary to compute the full attention matrix, which is quadratic in the text length (i.e., both time and space). By design, all elements of the time-based attention are the same for each channel (see Figure [], the green components are all equal), and all elements of the channel-based attention are the same for each time point (see Figure [e. the blue components are all equal). Hence, complexity is quadratic in one dimension and constant in the other. We then simply repeat the elements along the right dimension at no additional cost. Second, we employ the shifting operation (Dai et al. 2019) to compute all relative embeddings in one pass.
Content attention remains the most expensive component. To further reduce the cost, we make use of a local attention window (Child et al. 2019) that focuses on the most recent (in our case 10 segments, or 50 seconds) time points. Since time-based attention is not limited, the lookup window still spans the entire context. Thus, for T. & T, the total complexity of MVPA is O(T2 C CT+ T C²). quadratic in each dimension but subquadratic in the context length. Combining all techniques, MVPA pushes the effective total context length on an NVIDIA A100-80GB GPU to over 10,000 (e.g., 100 channels and 100 time segments).
Additionally, we use grouped query attention (Ainslie et al. 2023) to reduce the number of heads without loss of performance. Moreover, develop FlashMVPA based on FlashAttention (Dao ctt al. 2022 Dao 2024), implemented in the OpenAl Triton language, providing us with lover-level access to CUDA primitives and superior performance (see Appendix A
4

---

Published as a conference paper at ICLR 2026
013
MVPFormer
Siillaity
Attention
Limwer
Targer
Confounders
Linear
+ Simllatty
stantion
Linear
xN
Targh
Donfoundees
oTwo-step alaasto
Consinouus emboddinge
Avsrage
#3.
013
Non-ithll
422
6. 03 05 07 0S
simility
Wavelet
Time
Time 2
Time
Time
Ch.3 Ch
Figure 2: MVPFormer architecture and forward pass. iEEG signals are segmented in time and space, encoded via wavelet-based encoder, and arranged into 2D embedding grid. These con- tinuous embeddings are processed by MVPA to model temporal, spatial, and content-bast based depen- dencies. MVPFormer predicts the next-in-time embedding while reducing similarity confounders from same or other subjects. Notched the bottom right the resulting cosine similarity with the true target and the confounders after training. The two-step target is the signal twice removed in the future.
MVPFORMER
MVPFormer is novel Transformer-basee predictive unimodal foundational model equipped with MVPA, that processes heterogeneous multi-variate iEEG data (see Figure While is customary for language-based Transformer models to employ finite vocabulary words, such definition is non-trivial for iEEG. At the same time, recent works have challenged this discrete paradigm in
favor of continuous latent representations (LeCun) 2022 Tack et 2025 Tschannen et al. 2025) and continuous chain-of-tie thouhht mechanisms (Hao et al. 2024| Geiping et 2005). In similar vein, MVPFormer predicts the development of neuronal activity in a continuous embedding space governed by wavelet encoder. We build MVPFormer following the foundational paradigm, with a pre-training dedicated to predicting the future iEEG embedding using a contrastive loss function. Moreover, we show that successive fine-tuning using LoRA (Hu et al. (022) and simple clas- sification head allows MVPFormer to perform downstream classification tasks. particular, our results indicate that model trained in this fashion surpasses equivalent purely discriminative model (i.e., without generative pre-training), strengthening the validity of foundation models in the iEEG domain well. Appendix provides the full details the architecture.
3.1 TRAINING
Generative pre-training MVPFormer is pre-trained to generate neuronal activity predicting successive input segments in time. During pre-training, random input segments from batched win- dows are used confounding targets (Z (21, )n)), which plausible but different from the

---

Published as a conference paper at ICLR 2026
true target. We compute the contrastive loss as follows
Ce, = - log exp(sim(Oc.t: Co,++1//7)
(5)
ΣxEez exp(sim(Oc.s =x)/T)²
where oc, is the model's output embedding and c,,441 the ground -ruth ntxt-state embedding. Sum- ming over every c, gives us the optimization target for the generative task. The temperature is 0.1 and the number of confounders n. is 30. The contrastive setting (Chen et al. (020) provides clearer distinction between segments that are effectively similar, allowing MVPFormer to better model the dynamics of the signal without incurring into the typical pitfalls of L2 distance (such as
neural collapse: |Han al. (022). Given this generative setup, MVPFormer predicts the development in the latent space, rather than the raw signals themselves. See Appendix C.1 for more details.
Validation of pre-training Given the architecture of MVPFormer, we need to ensure that the true target and the confounders are sufficiently well-separated in cosine similarity. We evaluate MVPFormer's ability to predict embeddings of future iEEG signals by comparing the predicted embedding at time t to the ground truth embedding at t + 1. To do so, we introduce two references: (1) the embedding at t + 2, which is highly correlated with + 1, and (2) a random future segment sampled within the next two minutes. Given the high auto-correlation of iEEG signals, naive prediction model could simply predict again time I, and be moderately successful due to its similarity with + 1. Our results (see Figure 2 and Appendix [F) show that the wavelet- based encoder ensures signal features are well preserved, mapping even mildly similar signals to distinct embeddings.
LoRA fine-tuning for downstream tasks For downstream tasks, we use small classification head (i.e., a linear layer). This layer has input size equal to the decoder's block output size, and output size equal to the dimensionality of the classification task (i.e., 2 for seizure classification). The input to this classification head is either the channel-averaged (for seizure detection) or the channel- concatenated output of the last signal segment in time (for the pitch, volume, onset, and speech tasks). The output of the classification head is then passed through a softmax to compute the binary cross-entropy loss. We further adopt LoRA (Hu et al., 2022) to perform parameter-efficien fine- tuning. We only fine-tune the and v layers of the self-atenntion in the base MVPFormer model, with a LoRA rank of and and alpha of 16. This leads to a number of trainable parameters during fine-tuning of approximately 0. %% of the base model.
4 SWEC IEEG DATASET
The lack of publicly available large-scale iEEG datasets has been significant obstacle to the devel- opment of iEEG foundation models. In fact. while EEG datasets are abundant (Tangermann et al.) 2012 Shoeb 2010) and large (Obeid & Picone 2016), with many thousands of hours, such resources lacking in the 1EEG domain. Due to barriers tied to data collection and privacy, available iEEG datasets cover few hours (35 subjects and 290 hours; Nejedly et al. (020)) and subjects (10 subjects and 43 hours; Wang et al. 2024a), while larger datasets are kept private Yuan et al. (2024b).
In an effort towards addressing this issue, together with this work we open open-source the SWEC iEEG dataset, a large-scale iEEG dataset consisting of total of 68 subjects, 9328 hours of recording. and 704 ictal events. To our knowledge, the SWEC iEEG dataset is the largest publicly available iEEG dataset, fully curated and labelled by experienced clinicians. Due institutional data privacy concerns, the dataset does not contain information about the location of the channels in the brain. Appendix D reports more details and illustrates two example recordings.
5 EXPERIMENTS
5.1 SETUP
Pre-training We pre-train MVPFormer on 18 subjects, leaving the remaining 50 subjects for test- ing. MVPFormerM is pre-trained for 1.2M steps a single node with 8 NVIDIA A A100-80GB GPUs for two weeks. The chosen optimizer is FusedAdam with 0.1 weight decay, from the Deep- speed library compiled on the specific machine. The training used bfl6-mixed DeepSpeed stage 2 without activation checkpointing. The learning rate is fixed to 10 44 The training environment includes PyTorch 2.0, Py Torch Lightning 2.0, and Triton 1.0

---

Published as a conference paper at ICLR 2026
Fine-tuning After pre-training, we further fine-tune the MVPFormer for each task. For the seizure detection task, we fine-tune on the same 18 subjects of the SWEC iEEG dataset, and then test in a zero-shot manner nearly 7,000 hours of iEEG data from 50 unseen subjects, all suffering from epilepsy. To keep computational cost moderate, we use subset of the channels of each subject for testing: we select them based combination of variance and kurtosis, excluding the noisier ones (see Appendix E.I). In a real-worid clinical scenario selection would comprise minimal additional burden for the expert. The number of channels chosen fixed to 32 simplify comparisons with the other baselines, but we also provide ablations using manual channel selection and no channel selection at all (see Appendix G.8). For the four tasks of the Brain TreeBank dataset, we follow the same procedure as BrainBERT (Wang et al., (023) and PopT (Chau et al., 2025) by first fine-tuning on the specific subject on subset of the data and then testing on the remaining data. As before, we also evaluate the robustness of MVPFormer with respect to the channel selection in Appendix G. 16
5.2
IEEG TASKS
5.2.1 SEIZURE DETECTION TASK
We begin by evaluating MVPFormer on the seizure detection task on iEEG data. First, we con- sider clinically realistic setup that compares model predictions to a board-certified neurologist annotations using Cohen's Kappa. To do so, the predictions are post-processed to yield episodic results (see Appendix D Cohen's Kappa (Danker-Hople et al.) 2004 Schlögl et al. 2005 McHugh 2012) is widely used to quantify inter-rater reliability in seizure classification. The Landis and Koch criteria (Landis & Koch 1977) (see Appendix EE] are often used in practice to evaluate hu- man performance. Expert-level performance in the seizure classification task varies considerably, from 0.58 (Halford et al.] (015) to 0.53 (Grant et al.] 2014) in EEG, to 0.57 (Quigg al., 2015) in iEEG. We consider Kappa values above 0.53 to be expert-level. We must also consider that when comparing the decisions of human raters, only few (in the order of tens) curated episodes are evalu- ated. In contrast, our setup involves many more subjects and ictal events (in the order of thousands), making this task more challenging for MVPFormer. MVPFormer achieves an average Kappa of 0.61 across 50 unseen subjects from the SWEC iEEG dataset, matching human expert performance (see Figure B3). Importantly, agreement varies by subject, reflecting the clinical reality that seizure presentation complexity strongly affects classification (see Appendix G.
Overall, MVPFormer demonstrates expert-level seizure classification across a large, heterogeneous cohort. This performance, combined with its low false positive rate (0.15 fp/h), positions MVP- Former as a promising clinical assistant for real-world iEEG analysis.
Second, we consider a conventional evaluation (see Appendix G.3 based on Fl-score, sensitiv- ity, specificity, and number of false positives per hour (fp/h). We compare MVPFormer against three strong baselines: (1) Brant-2 (Yuan et al. 2024a), SOTA Transformer for iEEG, tine-tuned here starting from the published weights. Brant-2 requires all subjects to have the same number of channels for classification; hence, were not able to test all subjects with it. (2) Brain- BERT (Wang et al., 2023), another SOTA iEEG model with public weights. (3) MV-Llama, an ablation of MVPFormer-S that replaces MVPA with vanilla attention, is trained identically to MVP- Former. After undergoing the same task-specific finetuning of MVPFormer, we test the considered models zero-shot across the unseen subjection of our SWEC iEEG dataset, and also apply them on the MAYO and FNUSA datasets (Nejedly et al., 2020 (see Appendix G.12).
As shown in Table D all basclines fail to generalize on our SWEC iEEG dataset, achieving Kappa scores of just 0.11. 0.05, and 0.00, while MVPFormer achieves 0.61 and 0.57 in medium and small configurations, respectively. Specifically, BrainBERT always fails to detect a seizure, while Brant- 2's behaviour is more nuanced. We provide in Appendix G the complete per-subject statistics, including 95% confidence intervals. Moreover, MVPFormer outperforms the baselines on MAYO (highest fl-score of 0.36) and is par on FNUSA. As further baseline to validate our choice of pre-training - we also compare MVPFormer-S with an equivalent model built without the genera- tive base task (i.e., without the initial contrastive loss-based training, see Appendix G.13). With this setup. the purely discriminative model only reaches a Kappa score of 0.52, inferior to the equivalent MVPFormer-S which reached 0.54. The full set of results can be found in Appendix G.1
7

---

Published as a conference paper at ICLR 2026
Table 1: Results on the iEEG seizure detection tasks. We compare MVPFormer with multiple baselines across 3 iEEG datasets. The best results are bolded.
SWEC
MAYO
FNUSA
Model
Attention
Kappa
fl
fl
MVPFormer
MVPA
0.61
0.59
0.36
0.46
MVPFormer-S
MVPA
0.57
0.53
0.35
0.46
MV-Llama
Vanilla
0.11
0.01
Brant-2
Vanilla
0.06
0.01
0.19
0.46
BrainBERT
Vanilla
0.00
0.00
5.2.2 BRAIN TREEBANK DECODING TASKS
We validate the generalization of MVPFormer by testing on the four tasks of the Brain TreeBank dataset (Wang et al. 2024a), described in [Wang et al. (2023); Chau et al. (2025). The four tasks are: 1) discrimination of volume level (volume), 2) discrimination of pitch (pitch). 3) classifica- tion of sentence onset (onset), and 4) classification of speech (speech). All four tasks involve the discrimination high-level cognitive behaviors from iEEG recordings. As such, they represent a significant testbed for MVPFormer outside of its design environment of seizure detection. Table shows the results of MVPFormer against the SOTA baselines represented by Pop'T (Chau et al.,
2025 BrainBERT (Wang al. 2023), and Brant (Zhang al. 2023), as reported by PopTT (Chau
et al. 2025).
The full results can be found in Appendix G.2
Table 2: Results on the Brain TreeBank tasks. We compare MVPFormer with multiple baselines the 4 tasks of the Brain TreeBank dataset. The models requiring the electrodes' position are indi- cated by t. The best results without the electrodes' position are bolded, while the results where the electrodes' position is beneficial are underlined.
Pitch
Volume
Onset
Speech
Model
Attention
acc
acc
acc
acc
MVPFormer-S
MVPA
0.83
0.88
0.87
0.90
MV-Llama
Vanilla
0.63
0.77
0.80
0.81
Brant
Vanilla
0.61
0.74
0.80
0.80
BrainBERT
Vanilla
0.59
0.66
0.70
0.71
PopT t
Vanilla
0.74
0.87
0.90
0.93
PopT
Vanilla
0.62
0.76
0.81
0.83
The Brain TreeBank dataset contains information about the 3D location of the electrodes. the one hand, this information is often unavailable in datasets, it, by autonomously building an implicit channel map (see Appendix [..II). On the other hand, Pop'T was specifically developed for the Brain TreeBank dataset, taking into account the electrodes' physical location. Nonetheless, MVPFormer surpasses all baselines on the pitch and volume tasks, providing further evidence that MVPA well-suited iEEG tasks and generalizing the model's ability beyond its original task. On the remaining two tasks, MVPFormer is second behind PopT, but still superior to PopT without the electrodes' location. These results indicate that, while the electrodes' position might be beneficial in some instances, MVPA's implicit channel map produces superior results overall by being more flexible and adaptable a wider variety of existing datasets.
explicitly design MVPA not to require
EVALUATION ON GENERAL TIME-SERIE
We have shown that MVPFormer achieves SOTA results in its native modality of iEEG. However, the design of MVPA should allow it to make use of the tiem and channel information intrinsic to any multi-channel time-series. To this end, also provide baseline evaluation of MVPA against
8

---

Published as a conference paper at ICLR 2026
established alternatives in the time-series domain. We compare MVPFormer with existing SOTA architectures on classical long-term forecasting and classification tasks.
6.1 TIME-SERIES FORECASTING
Table B reports the results of MVPFormer, the vanilla Transformer (Vaswani al. 2017),
PatchTST (Nie et al. )023), TimesFM (Das et al., 2024). TimeMixer (Wang et al.) 2024b). and
WPMixer (Murad al. 2025) the ETThi, ETTh2, and Weather datasets (Zhou et al. 202 Wu
et al. 2021). MVPFormer always equals or surpasses the baselines (see Appendix G.17 for the full
results).
Table Results on the time-series forecasting task. We report the mean- seaannd error (MSE) and mean-absolute error (MAE) averaged over all forecasting lengths.
Model
MVPFormer Transformer
PatchTST
TimesFM
TimeMixer
WPMixer
Dataset
Length
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAF
MSE
MAE
MSE
MAE
ETThl
Avg.
0.45
0.45
1.00
0.80
0.45
0.45
0.45
0.45
0.45
0.44
0.45
0.44
ETTh2
Avg.
0.38
0.41
3.37
1.48
0.39
0.41
0.38
0.41
0.39
0.41
0.38
0.41
Weather
Avg.
0.25
0.28
0.59
0.53
0.26
0.28
0.26
0.28
0.25
0.28
0.25
0.28
6.2 TIME-SERIES CLASSIFICATION
Moreover, we evaluate MVPA on common classification tasks, against the vanilla Transformer and PatchTST on the EthanolConcentration (EıCo), FaceDetection (FaDe), Hand Writing (HaWr), Heart- beat (HaBe), Japanese Vowels Vowels (Ja PEMS-SF (PEMS), SCP1, SCP2, Spoken (SpAr), and Uwave datasets (Liu & Wang 2024).
Table 4: Accuracy on time-series classification tasks. We report the accuracy per task.
EtCo
FaDe
HaWr
HaBe
JaVo
PEMS
SCP1
SCP2
SpAr
Uwave
MVPFormer
0.33
0.66
0.21
0.70
0.95
0.86
0.86
0.54
0.97
0.80
Transformer
0.29
0.64
0.20
0.70
0.91
0.84
0.83
0.54
0.95
0.80
PatchTST
0.29
0.67
0.23
0.72
0.95
0.85
0.83
0.51
0.97
0.82
TimesFM
0.29
0.68
0.23
0.71
0.93
0.84
0.83
0.52
0.99
0.82
TimeMixer and WPMixer are forecasting-only architectures, so we could test them. Table shows that MVPFormer achieves SOTA results on general classification tasks as well. At the same time, these results highlight the generalization capability of MVPFormer that, in contrast other models, is effective in both forecasting and classification.
6.3 ABLATION THE THREE COMPONENTS
Finally, we make use of the general time-series setting validate the contribution of the three differ- ent components of MVPA: content, time, and channel. In Sections 5.2.1 and 5.2.2 we confirmed that MVPA provides a notable increase in performance to MVPFormer in native iEEG environment.
In Table 5 we report the results of MVPA, content-only MVPA, time-only MVPA, channel-only MVPA, and no-component MVPA on the ETTh1, ETTh2, and Weather datasets. MVPA obtains the best performance over all datasets, providing further evidence that the three components jointly learn the different aspects of the signal. Interestingly, the performance gap on the Weather dataset larger, is its number of channels: 21 instead of 7 of ETTh1 and ETTh2. This result supports our design of MVPA, which is able to take advantage of the information content in the strongly multi-variate time-series better than its simpler counterparts. The full results be found in Appendix G.18

---

Published as a conference paper at ICLR 2026
Table Ablation of the components of MVPA on the time-series forecasting task. We report the mean-squared error (MSE) and mean-absolute error (MAE) averaged over all forecasting lengths.
Model
MVPA
Content-onl only
Time-only
Channel
None
Dataset
Length
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
Avg.
0,45
0.45
0.46
0.45
0.46
0.45
0.46
0.45
0.47
0.46
ETTh2
Avg.
0.38
0.41
0.38
0.40
0.38
0,41
0.38
0.41
0.40
0.41
Weather
Avg.
0.25
0.28
0.27
0.29
0.27
0.29
0.26
0.28
0.27
0.29
7 RELATED WORKS
Single-channel data has been treated as 1D sequences for tasks like speech recognition, where the signal is divided into patches that serve as tokens (Schneider et al., 2019 Gulati et al.] 2020). Ex- tending vanilla attention to nulti-dimensional data, such as images, is more complex. The Vision Transformer (Dosovitskiy al., 2021) processes images by flattening 2D patches into 1D quence, losing spatial structure in the process. However, this approach inflexible and unsuitable for generalizing images with different heights and widths. Other mechanisms (Ho et al.) 2019 Huang et al.] 2019 Child et al. 2019 Bulat et al., 2021; Arnab et al. (021) alternative to vanilla attention have been developed speed up computation in the ID case or to extend it to 2D. We
compare MVPA against such alternatives in more detail in Appendix A.2 AA.2
For multi-variate time-series, such as EEG, Transformers face challenges due to the need to pre- serve both time and channel information (Wen et al., 2022] Cui & Lv (0224). Channel-independen approaches (Nie et al. 2023) reuse vanilla self-attention and discard all information content in the time dimension, while channel-mixing promises to preserve it by either fusing the channels (Zhou jet al., 2022) or processing them sequentially (Zhang Yan (023). For neural spike data, a fusion of channel and time aspects has been proposed (Le & Shlizerman 2022). albeit without complete integration at the attention level. Specifically for iEEG and EEG, there exist few Transformer-baas based solutions (Zhang al., 2023: Yuan et al. 2024a). Since electrode placements vary widely across subjects, these models struggle e with the heterogeneous nature of the data.
Some models (Chau et al. 2025) tackle this issues by requiring the absolute position of the elec- trodes, limiting their applicability to datasets that do have such information. Due to practical and ethical concerns, the publicly available datasets without the absolute position of the electrodes (Bur-) rello et al.) 2018 2019; Nejedly et al. 2020 Li et al. 2021), including ours, notably outsize the ones that do (Wang et al. [2024a] Keles al. 2024 Zada al.) 2025). Overall, the complex interplay between time and space, where distant brain regions may be more strongly connected than nearby ones, makes it difficult for conventional attention mechanisms effectively process iEEG signals.
8 CONCLUSION
We introduce MVPA, novel attention mechanism designed to effectively process multi-variate time-series data, exemplified by its application to iEEG signal analysis. MVPA enables MVP- Former, foundation model trained on our novel SWEC iEEG dataset, capture complex inter- actions between time and spatial dimensions in multi-variate time-series. We also contribute the SWEC iEEG dataset itself, as the largest iEEG dataset currently publicly available. MVPFormer is trained following the foundational paradigm to predict the next brain states, and then further fine- tuned on multiple tasks. MVPA ensures robust performance across several iEEG tasks and dataset. It reaches high inter-rater agreement (0.61 Kappa score) on our large scale and challenging SWEC iEEG dataset, notably surpassing the SOTA Brant-2 (0.08). It also achieves SOTA results on the four tasks of the Brain TreeBank dataset, even surpassing models specifically designed for them. More- over, MVPA equals or surpasses the SOTA also in classical time-series forecasting and classification tasks. Overall, our results show that MVPA affords MVPFormer superior generalization capabilities while maintaining computational efficiency and scalability, marking a significant advancement in the analysis of time-series data and iEEG in particular.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work supported by the Swiss National Science foundation (SNF), grant no. 200800.
ETHICS STATEMENT
During the collection of the SWEC iEEG dataset, all the subjects gave written informed consent that their iEEG data might be used for research and teaching purposes. The decision on the necessity for iEEG recordings, the electrode implantation scheme, and the decision about surgical therapy were made entirely on clinical grounds. These decisions were taken prior to and completely independently from the compilation of this dataset.
REPRODUCIBILITY STATEMENT
This paper describes the MVPA algorithm in Section and Appendix A and the architecture of MVPFormer in Section 3 and Appendix B
All the hyperparameters are in Appendix B
The setup used for training and evaluating our model are in Section 5.1
The SWEC iEEG dataset is publicly available at htttss//huggiinffccccccoatstes NeuroTec/SWE3 iEEG Dataser
The code and the weights of MVPFormer are publicly available at https://github.crm IBM/multi-varatte www.araaaaaaaaaaleeeeeeeaeeaeeee transsfortrassooereee
REFERENCES
Hojjat Adeli, Ziqin Zhou, and Nahid Dadmehr. Analysis of EEG records in an epileptic patient using wavelet transform. Journal of neuroscience methods, 123()):69-87,200)) 2003.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, 2023.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučic, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Mark Beenhakker and John R. Huguenard. Neurons that fire together also conspire together: Is normal sleep circuitry hijacked to generate epilepsy? Neuron, 2009,
Richard F. Betzel, John D. Medaglia, Ari E. Kahn, Jonathan Soffer. Daniel R. Schonhaut, and Danielle S. Bassett. Structural, geometric and genetic factors predict interregional brain con- nectivity patterns probed by electrocorticography Nature Biomedical Engineering, 2019.
Elisa Bruno, Pedro F. Viana, Michael R. Sperling, and Mark P. Richardson. Seizure detection at home: Do devices on the market match the needs of people living with epilepsy and their care- givers? Epilepsia, 2020.
Adrian Bulat, Juan Manuel Perez Rua, Swathikiran Sudhakaran, Brais Martinez, and Georgios Tz- imiropoulos. Space-time mixing attention for video transformer. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2021.
Alessio Burrello, Kaspar Schindler, Luca Benini, and Abbas Rahimi. One-shot learning for iEEG seizure detection using end-to-end binary operations: Local binary patterns with hyperdimen- sional computing. In 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS), 2018.
Alessio Burrello, Lukas Cavigelli, Kaspar Schindler, Luca Benini, and Abbas Rahimi. Laelaps: An energy-efficient seizure detection algorithm from long-term human iEEG recordings without false alarms. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2019.
11

---

Published as a conference paper at ICLR 2026
Geeling Chau, Christopher Wang, Sabera J Talukder, Vighnesh Subramaniam, Saraswati Soedar- madji, Yisong Yue, Boris Katz, and Andrei Barbu. Population Transformer: Learning population- level representations of neural activity. In International Conference on Learning Representations (ICIR), 2025.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML). 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Kyung-Ok Cho and Hyun-Jong Jang. Comparison of different input modalities and network struc- tures for deep learning- based seizure detection. Scientific Reports, 2020.
Wenhui Cui, Woojae Jeong, Philipp Thölke, Takfarinas Medani, Karim Jerbi, Anand A. Joshi, and Richard M. Leahy. Neuro-GPT: Developing a foundation model for EBG. arXiv preprint arXiv:2311.03764, 2023.
Xiandai Cui and Hui Lv. Steft: Spatio-temporal embedding fusion transformer for traffic prediction. Electronics, 13(19), 2024.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-Xe XL: Attentive language models beyond fixed- context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Heidi Danker-Hopfe, D. Kunz, G. Gruber, G. Klösch, J. L. Lorenzo, S. L.. Himanen, B. Kemp, T. Penzel, J. Röschke, H. Dorn, A. Schlögl, E. Trenker, and G. Dorffner. Interrater reliability between scorers from eight european sleep laboratories in subjects with different sleep disorders. Journal of Sleep Research, 13(1):63-66 69, 2004.
Tri Dao. FlashAttention-22: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with 10-awareness. In Advances Neural Information Process- ing Systems (NeurIPS), 2022.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In International Conference on Machine Learning (ICML), 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.055 011.552..551,1122 2025.
Jean Gotman. few thoughts on "what a seizure?". Epilepsy & Behavior, 22:S2-S3, 2011.
Arthur C. Grant, Samah G. Abdel-Baki, Jeremy Weedon, Vanessa Arnedo, Geetha Chari, Ewa Koziorynska, Catherine Lushbough, Douglas Maus, Tresa McSween, Katherine A. Mortati, Alexandra Reznikov, and Ahmet Omurtag. EEG interpretation reliability and interpreter con- fidence: A large single-center study. Epilepsy & Behavior, 32, 2014.
Pierre Guetschel, Thomas Moreau, and Michael Tangermann. S-JEPA: towards seamless cross- dataset transfer through dynamic spatial attention. In Graz Brain- Computrrr Interface Conference, 2024.
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentte transformer for speech recognition. arXiv preprint 2005:20005008100 2020.
12

---

Published as a conference paper at ICLR 2026
J.J. Halford, D. Shiau, J.A. Desrochers, B.J. Kolls, B.C. Dean, C.G. Waters, N.J. Azar, K.F. Haas, E. Kutluay, G.U. Martz, S.R. Sinha, R.T. Kern, K.M. Kelly. J.C. Sackellares, and S.M. LaRoche. Inter-rater agreement identification of clectrographic seizures and periodic discharges in icu eeg recordings. Clinical Neurophysiology, 126(9), 2015.
X.Y. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamies the central path. In International Conference on Learning Representations (ICLR), 2022.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- bert with disentangled attention. In International Conference on Learning Representations (ICLR), 2021.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint 11X.v 2019.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. Advances Neural Information Processing Systems (NeurIPS), 36, 2022.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representations (ICLR), 2022.
Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CC- Net: Criss-cross attention for semantic segmentation. Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision (ICCV), 2019.
Herbert H. Jasper. Report of the committee on methods of clinical examination in electroencephalog. raphy: 1957. Electreencehhalggappy and Clinical Neurophysiology, 10(2):370-375, 1958.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Zhang, Xiaoming Shi, Pin- Pin-Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In Iternational Conference on Learning Representations (ICLR), 2024.
Viktor Jirsa, Huifang Wang. Paul Triebkorn, Mcysam Hashemi. Jayant Jha, Jorge Gonzalez- Martinez, Maxime Guye, Julia Makhalova, and Fabrice Bartolomei. Personalised virtual brain models in epilepsy. The Lancet Neurology, 22(5))443-455 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.00 20036666,010,,.. 2020.
Umit Keles, Julien Dubois, Kevin J. M. Le, J. Michael Tyszka, David A. Kahn, Chrystal M. Reed, Jeffrey M. Chung, Adam N. Mamelak, Ralph Adolphs, and Ueli Rutishauser. Multimodal single- neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients. Scientific Data, 11(1), 2024.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations (ICLR), 2020.
Levin Kuhlmann, Klaus Lehnertz, Mark P. Richardson, Björn Schelter, and Hitten P. Zaveri. Seizure prediction - ready for a new era. Nature Reviews Neurology, 14(10), 2018.
J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159, 1977.
13

---

Published as a conference paper at ICLR 2026
Trung Le and Eli Shlizerman. STNDT: Modeling neural population activity with spatiotemporal
transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open
Review, 62(1):1-65 62,1-66,222 2022.
Adam Li, Chester Huynh, Zachary Fitzgerald, Iahn Cajigas, Damian Brusko. Jonathan Jagid, An- gel O. Claudio, Andres M. Kanner, Jennifer Hopp. Stephamie Chen, Jennifer Haagensen, Emily Johnson, William Anderson, Nathan Crone, Sara Inati, Kareem A. Zaghloul, Juan Bulacio, Jorge Gorralez-Martinez, and Sridevi V. Sarma. Neural fragility as an EEG marker of the seizure onset 7one. Nature Neuroscience, 24(10), 2021.
Xinhe Liu and Wenmin Wang. Deep time series forecasting models: A comprehensive survey. Mathematics, 12(10), 2024.
Michael I.. Martini, Aly A. Valliani, Claire Sun, Anthony B. Costa, Shan Zhao, Fedor Panov, Saadi Ghatan, Kanaka Rajan, and Eric Karl Oermann. Deep anomaly detection of seizures with paired steroodectroencchallognee
and video recordings. Scientific Reports, 11(1), 2021.
Marry McHugh. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.
Md Mahmuddun Nabi Murad, Mehmet Aktukmak, and Yasin Yilmaz. Wpmixer: Efficient multi- resolution mixing for long-term time series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025.
Petr Nejedly, Vaclav Kremen, Vladimir Sladky, Jan Cimbalnik, Petr Klimes, Filip Plesinger, Filip Mivalt, Vojtech Travnicek, Ivo Viscor, Martin Pail, Josef Halamek, Benjamin H. Brinkmann, Milan Brazdil, Pavel Jurak, and Gregory Worrell. Multicenter intracranial EEG dataset for clas- sification of graphoelements and artifactual signals. Scientific Data, 7(1), 2020.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations (ICLR), 2023.
Paul L. Nunez and Ramesh Srinivasan. Electric Fields of the Brain. Oxford University Press, 2006. ISBN 9780195050387.
Iyad Obeid and Joseph Picone. The Temple University Hospital EEG data corpus. Frontiers in Neuroscience, 10, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03775. 2018
James C. Pang. Kevin M. Aquino, Marianne Oldehinkel, Peter A. Robinson, Ben D. Fulcher, Michael Breakspear, and Alex Fornioo. Geometric constraints on human brain function. Nature, 618(7965), 2023.
Mark Quigg, Felice Sun, Nathan B. Fountain, Barbara C. Jobst, Victoria S. S. Wong, Emily Mirro, Sarah Brown, and David C. Spencer. Interrater reliability in interpretation of electrocorticooraphic seizure detections of the responsive neurostimulator. Epilepsia, 56(6), 2015.
Khaled Saab, Siyi Tang, Mohamed Taha, Christopher Lee-Messer, Christopher Ré, and Daniel L. Rubin. Towards trustworthy seizure onset detection using workflow notes. npj Digital Medicine, 7(1), 2024.
Kaspar Schindler, Howan Leung. Christian E. Elger, and Klaus Lehnertz. Assessing seizure dynam- ics by analysing the correlation structure of multichannel intracranial EEG. Brain, (:30(1):65-77, 2006.
Kaspar A. Schindler, Stephan Bialonski, Therese Horstmann, Christian E. Elger, and Klaus Lehnertz. Evolving functional network properties and synchronizability during human epileptic seizures. Chaos: An Interdisciplinary Journal of Nonlinear Science, 18(3), 2008
14

---

Published as a conference paper at ICLR 2026
Alois Schlögl, Felix Lee, Horst Bischof, and Gert Pfurtscheller. Characterization of four-class motor imagery EEG data for the BCI-competition 2005. Journal of Neural Engineering, 2(4):L14-L22, 2005.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In Interspeech 2019, 2019.
V. Shah, I. Obeid, J. Picone, G. Ekladious, R. Iskander, and Y. Roy. Validation temporal scoring metrics for automatic seizure detection. In 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2020.
Mingkan Shen, Peng Wen, Bo Song, and Yan Li. An EEG based real-time epilepsy seizure detection approach using discrete wavelet transform and machine learning methods. Biomedical Signal Processing and Control, 77, 2022.
Ali Shoeb. CHB-MIT scalp EEG database. 2010.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billionn parameter language models using model par- allelism. arXiv preprint (e.Xiv:9999000000 2019.
Leila Abrishami Shokooh, Dènahin Hinnoutondji Toffa, Philippe Pouliot, Frédéric Lesage, and Dang Khoa Nguyen. Intracranial eeg seizure onset and termination patterns and their associa- tion. Epilepsy Research, 176, 2021.
Xiaopeng Si, Zhuobin Yang, Xingjian Zhang, Yulin Sun, Weipeng Jin, Le Wang. Shaoya Yin, and Dong Ming. Patient-independent seizure detection based on long-term iEEG and novel lightweight CNN. Journal of Neural Engineering, 20(1), 2023.
Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuan- dong Tian, Jason Weston, and Xian Li. LLM pretraining with continuous concepts. arXiv preprint arXiv:2502.08524, 2025.
Michael Tangermann, Klaus-Robert Müller, Ad Aertsen, Niels Birbaumer, Christoph Braun, Clemens Brunner, Robert Leeb, Carsten Mehring, Kai J. Miller, Gernot R. Müller-Putz, Guido Nolte, Gert Pfurtscheller. Hubert Preissl, Gerwin Schalk, Alois Schlögl, Carmen Vidaurre, Stephan Waldert, and Benjamin Blankertz. Review of the BCI competition IV. Frontiers in Neuroxcience, 2012.
Punnawish Thuwajit, Phurin Rangpong, Phattarapong Sawangjai, Phairot Autthasan, Rattanaphon Chaisaen, Nannapas Banluesombatkul, Puttaranun Boonchit, Nattasate Tatsaringkansakul. Tha- panun Sudhawiyangkul, and Theerawit Wilaiprasitporn. EEGWaveNet: Multiscale CNN-based spatiotemporal feature extraction for EEG seizure detection. IEEE Transactions on Industrial Informatics, 18(8), 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2337.09228. 2023.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. GIVT: Generative infinite-vocabulary transformers. In Computer Vision - ECCV 2024, pp. 292-309, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72998-00
Anouk Van de Vel, Kristien Verhaert, and Berten Ceulemans. Critical evaluation of four differ- ent scizure detection systems tested on one patient with focal and generalized tonic and clonic seizures. Epilepsy & Behavior, 37, 2014.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017.
Ben Wang. Mesh-Transfor Transformer-AA Model-Paralles Implementation of Transformer Lan-
guage Model with JAX, 2021. URL https:: /ttp::://gttub.buu com htcmmiinggflooe///
mesh-transforme
15

---

Published as a conference paper at ICLR 2026
Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. BrainBERT: Self-supervvsed representation learning for intracranial recordings. In International Conference Learning Representations (ICLR), 2023.
Christopher Wang, Adam Uri Yaari, Aaditya K Singh, Vighnesh Subramaniam, Dana Rosenfarb, Jan DeWitt, Pranav Misra, Joseph R. Madsen, Scellig Stone, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024a.
Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang. and JUN ZHOU, Timemixer: Decomposable multiscale mixing for time series forecasting. In International Conference on Learning Representations (ICLR), 2024b.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with Auto- Corr-Corrlltionn for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and
Vikas Singh. Nyströmformer: A nyström- based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Zhizhang Yuan, Fanqi Shen, Meng Li, Yuguo Yu, Chenhao Tan, and Yang Yang. Brant-2: Founda- tion model for brain signals. arXiv preprint arXiv:2402.1025v4. 2024a.
Zhizhang Yuan, Fanqi Shen, Meng Li, Yuguo Yu, Chenhao Tan, and Yang Yang. Brainwave: A brain signal foundation model for clinical applications. arXiv preprint arXiv:2402. 1025v7, 2024b.
Zaid Zada, Samuel A. Nastase, Bobbi Aubrey, Itamar Jalon, Sebastian Michelmann, Haocheng Wang, Liat Hasenfratz, Werner Doyle, Daniel Friedman. Patricia Dugan, Lucia Melloni. Sasha Devore, Adeen Flinker, Orrin Devinsky, Ariel Goldstein, and Uri Hasson. The "Podcast" ECoG dataset for modeling neural activity during natural language comprehension. Scientific Data 12 (1), 2025.
Hitten P. Zaveri, Björ Schelter, Catherine A. Schevon, Premysl Jiruska, John G.R. Gregory Worrell, Andreas Schulze-Bonhar Rasesh B. Joshi, Viktor Jirsa, Mare Goodfellow, Christian Meisel, and Klaus Lehnertz. Controversies on the network theory of epilepsy: Debates held during the ictals 2019 conference. Seizure, 78, 2020.
Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foundation model for intracranial neural signal. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross- noss-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations (ICLR), 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting. In P Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning (ICML), 2022.
Saeedeh Ziyabari, Vinit Shah, Meysam Golmohammadi, lyad Obeid, and Joseph Picone. Objective evaluation metrics for automatic classification of EEG events. arXiv preprint arXiv: 2017.
16

---

Published as a conference paper at ICLR 2026
A
DETAILS ON MULTI-VARIATE PARALLEL ATTENTION (MVPA)
Algorithm Iillustrates the varti-v parallel attention algorithm.
Algorithm 1: Computation of MVPA
Input: ce,t "aaied output token of Encoder; Nembed 768
Output: Oct E Rhheeto output attention
Data: time encoding; e. channel encoding; u, y, un biases; h E [1,..., [h.xst,
hR E ;
def MVPAttent (æe,z)
Compute query separately from key and value due to GQA t. LINEARNOBIAS(()))
ket"", .uat he. + LINEARNOBIAS(E
Compute the three components of MVPA KCDsAe)) & (d +uu"")"'"
Time and channel components are independent of the key content, they do not need to be recomputed o (g t c + (q + (
Shift the time and channel components to avoid recomputation, from Transformer-Xl.
8/ (MFFTTIMB:(
1 < SHIFTCHANNEL,
# Apply window and causal mask
m ((c.))ee CAUSAI (ett)) + the + )
10
((c.))(e + WINDDWWMASKI (ee.))))))
# Apply structured dropout
di (ct)).e.)) < STRUCTURESDDROPUUT.
Compute final attention value
a (cA))())) + SIGMOID( dd ((n)))))))
12
di Ec. a (c. (a..)))))
13
return %% 14
A.1 FURTHER MOTIVATION OF MVPA
Single-channel data can be treated equivalently to sentences, by dividing the signal into ID patches, which form the tokens. This modality has attracted considerable interest (Schneider et al. 2019 Gulati et af. 2020), frequently for speech recognition tasks that are once again related to the natural language domain.
There is no straightforward extension of vanilla attention to the 2D case. The Vision Trans- former (Dosovitskiy al. 202I) processes images by extending the notion of the patches to the 2D case. carves images into a collection of patches, which then flattens into a ID sequence. Each patch has 2D coordinates (i.j) which get flattened by an arbitrary function into ID index (I). This is simple way to recover the ID case, but it has several drawbacks. First, by flattening the patches we lose any notion of spatial structure, as nearby patches in space are no longer necessarily close in the sequence. Any information about the structure of the patches lost. However, if the size of the images, the number patches, and the flattening direction are kept constant, then the Transformer might autonomously learn it. If it learns the structure, then it cannot be exposed to different images as would completely misinterpret them; if does not learn the structure, then it is missing critical information. This leads to inflexible model which cannot easily generalize to different images. One possible solution is to choose bijective f, such as the Cantor pairing function, to have one-to-one correspondence between the position of the patch in
N x N N
17

---

Published as a conference paper at ICLR 2026
the image and in the sequence. This solution is, however, quite unintuitive. Second, the Vision Transformer does not distinguish between the two dimensions of height and width, i.c., it does not distinguish between up, down, left, and right. For images this limitation is not too impactful, as most of the information is conveyed in the closeness of two patches and not their relative position in any dimension.
The patching schema of the Vision Transformer unsuitable to multi-variate time-series, as the two dimensions of time and channels require delicate handling. Transformers for time-series are well- known problem in the field (Wen et al. 2022). reuse vanilla self-attention and discard all information content the time dimension, while channel- mixing promises to preserve it by either fusing the channels (Zhou et al. (022) or processing them sequentially (Zhang & Yan. 2023). The second family of solutions is more promising addressing the issue but is still limited either with respect to computational expense or expressiveness.
approaches (Nie et al. 2023)
recordings of the brain. Transformer- based approaches to EEG are sparse (Si et al., [023] Cui et al., (023), due to the often unmanageable complexity of the data. In iEEG recordings, the subjects are implanted with electrodes directly in multiple areas of the brain for the purpose of clinical diagnosis. There is no standardized location, even number of electrodes, for intracranial implants. This makes iEEG an extremely heterogeneous data modality, intractable for conventional attention approaches. The channels present a fundamental source of information, as electric fields spread in different areas of the brain on different time-scales and with different intensities depending the strength of the connection between the areas. Moreover, the relationship between brain regions is not always proportional to their spatial closeness, as distant areas might be more strongly connected than close ones. There is tremendously intricate interplay between space and time, which the Transformer must exploit.
EEG signals are
A.2 COMPARISON WITH ALTERNATIVE ATTENTION MECIIANISMS
We further compare MVPA with other existing alternatives better characterize the features of MVPA. particular, we draw our main inspiration for the disentanglement and relative positional encoding from Transformer-Xi (Dai al. 2019) and DeBERTa (He et al. 202T). which were the first to introduce this concept. We now compare MVPA against a selection of relevant alternative attention mechanisms.
Table 6: Summary of the differences between MVPA and existing attention mechanisms.
Complexity
Disentangled
Relative position
Simultaneous space
Receptive
Vanilla
ID
Quadratic
No
No
No
Global
Transformer XI.
ID
Quadratic
Yes
Yes
No
Global
DeBERTa
ID
Quadratic
Yes
No
No
Global
Axial
2D
Subquadratic
No
No
No
Local
Criss-cross
2D
Subquadratic
No
No
Local
Localized
2D
Quadratic
No
No
Yes
Local
Space-time
Subquadratic
VIVIT
2D
Subquadratic
Yes
No
No
Global
Ours
2D
Subquadratic
Yes
Yes
Global
Axial attention (Ho et al. 2019) consists of separate attention mechanisms, RowAttention and ColumnAtenttinn, each of which attends to one row (one channel) one column (one timepoint) only. The layers are then stacked sequentially to recover the full receptive field. MVPA, in contrast, attends to both time and space simultaneously, and has global receptive field built-in at every layer.
Criss-cross attention (Huang et al. 2019) computes the attention between each point and all the other points in its row column via the affinity operation. Once again, the layers are applied recursively to obtain the full receptive field. One of the most significant differences between MVPA and criss- cross attention is in the encoding of the position. In fact, MVPA treats rows and columns differently through two independent positional codebooks, while criss-cross attention distance in rows and heights is equivalent. This is natural consequence of the design choices, criss-cross attention is designed for images, where the two dimensions are indeed equivalent. Moreover, MVPA again has global receptive field.
18

---

Published as a conference paper at ICLR 2026
Localized sparse transformers (Child et al. 2019) use separate heads with separate connectivity patterns to improve on the computational requirements of the full attention. As before, the full receptive field is only recovered with multiple applications. MVPA, on the other hand, computes the full 2D attention over the entire input in every head. Moreover, the separate positional codebooks allow MVPA treat the dimensions differently, which localized sparse transformers cannot do.
Space-time mixing (Bulat et al.) 2021 reduces the computational complexity of the full quadratic self-attention by only computing attention across the channels, and then performing a simple av- eraging in time. On the other hand, MVPA fully integrates both dimensions of the signal in the computation the attention level.
Finally, ViVIT (Arnab et al. 2021) disentangies space and time through the use of sequential en- coders, similar the channels mixing approaches mentioned in Section
Table 6
summarizes the main features of MVPA with respect to the considered alternatives.
A.3 EFFICIENT COMPUTATION OF MVPA
Vanilla attention is quadratic in the number of input elements, and this often represents significant computational roadblock (Kitaev et al., 2020). The input becomes intractable as the number of channels increases, especially for multi-variate time-series. At the same time, more channels imply more sources of information, and we cannot simply discard them.
MVPA is also quadratic, but we employ a number of techniques to significantly reduce the compu- tational complexity and make the processing of very large signals feasible. Letting T be the number of time segments and C' be the number of channels, the context length of the Transformer becomes T C' and number of terms necessary to compute for vanilla attention O(T x C²). Given reason- able estimation of 100 segments and 50 channels the context length would be 5,000, until recently intractable even for language models.
By dividing MVPA into three components we gain considerable advantages (see Table for the complexity of each term). Efficiently computing the time- and channel-based terms requires two main techniques. First, recognize that is not necessary to compute the full square matrix, which would be quadratic in the context length (i.e., both time and space). By design, all elements of the based attention the same for each channel, and all elements of the channel-based attention are the same for each time point. Hence, complexity is quadratic in one dimension and constant in the other. We then simply repeat the elements along the right dimension at no additional cost. Second, we employ the shifting operation described in Supplementary Section B of Transformer- XL (Dai et al. 2019 compute all relative embeddings in one pass.
Time-based Channel-based Content-bascee (w/ window) Content-based (vanilla) O x C) O(T x C²) O x C2)
Complexity
O x C2)
Table 7: Complexity of each component of MVPA. T is the number of time segments in the signal, C' is the number of channels, and L is the size of the local window. Content-band based attention without window has the same complexity vanilla attention.
Content attention, though stripped of positional encoding, remains the most expensive component. To further reduce computational cost, with little impact performance, we make use of local attention window (Child et al., 2019) which focuses on the most recent L time points discarding ones that have little information content. Since time-based attention is not limited, the lookup window still spans the entire context (though it is affected, see Figure (9p). Thus, for L, < T, the total complexity of MVPA is O C C C²), quadratic in each dimension but subquadratic the context length. Combining all techniques, MVPA pushes the effective total context length over 10,000.
Given the three components are independent of each other, it is possible to exclude any one and reduce computations even more. As an additional cost-saving measure, we use grouped query atten- tion (Ainslie et al. 2023) to reduce the number of heads without loss of performance. In summary, MVPA correctly treats time and space as unrelated dimensions, forcing the model to consider them separately, all with little computational overhead.
19

---

Published as a conference paper at ICLR 2026
A.4 TRITON IMPLEMENTATION OF MVPA
MVPFormer's training effectiveness is heavily affected by batch size, as its training routine draws the negative samples from the batch. The bigger the batch size, the more variety the negative samples and the better the model generalizes. Given the large context size of MVPFormer, up to 10k, pure Python implementation of scaled dot product attention would consume too much VRAM be useful. FlaehAttention (Dao et al. 2022) and Flashnttention-22 (Dao 2024) provide the blueprint to solve this problem, though they only apply to vanilla attention. Using tiling. FlashAttention makes VRAM consumption linear instead of quadratic in the context length, enabling training much longer context.
We develop FlashMVPA using the same technique in the OpenAl Triton language, which gives lower-level access to CUDA primitives. While CUDA implementation could likely deliver better performance. the choice of Triton is dictated by the much lower coding time, though Triton is less robust and more prone to unexpected behaviors at this point. The time-based and channel-based components of MVPA (Transformer-XLL trick) and added using Triton, while the content-based component is fully imple- mented in Triton. This is due to limitations in Triton. FlashMVPA reaches 20 TFlops on an A100.
computed using PyTorch's own matrix-multiply. but then shifted
Algorithm 2: Computation of FlashMVPA
Input: Te,t E IR Fraubel output token of Encoder; Tlembed 768
Output: Oct Rllerbal output attention
Data: t time encoding; с channel encoding; u, y. w biases; h E [1,..., Abeads::
hg.v [1,.... R2gual
def "lashMVPAttent.
(
Compute query separately from key and value due to GQA
dh LINEARNOBIAS((
ha Vaa he. + LINEARNOBIA((E...)
Need to compute time and channel components outside Triton
8 -(g,, + vhe, y
ta + (qh, + (ette)Tchhe
Triton MVPA combines all computations into one kernel o t TRITONMVPA (g"., st, 1. u, u, w)
return oi.
A.5 RELATIVE SHIFTING
By design, MVPA requires the computation of relative time and channel encodings, which can no- tably slow down the overall operation. While this does not affect vanilla attention, other relative attentions provide us with an elegant solution to this problem. In particular, the shifting opera- tion from Transformer-Xl provides us with an efficient alternative to recomputing the time- and channel-based attention components. To keep notation simple, let = x.. WT. Pe x WT T, = WA, Tr- and C; Cho The shift time can be performed in the original implementation
To To
Ti ...
TT 91 TT
9o Tr
0 qiTr-1
T
9:
SHIFTTIME
To
-r-1T
Tr-1
9r-1T
tn T
(6)
The right triangular matrix is zeroed out requisite of autoregressive training, i.e., cannot attend to keys in the future. The entire time shifting operation can be performed efficiently and quickly using tensor manipulation in PyTorch.
Thus, the time attention component does not require recomputation for each time position, i.c. each row in the matrix of the time component.
20

---

Published as a conference paper at ICLR 2026
The shift in channels is more involved
PoCo
PoC
PoCo-1
PoCC-1 PoCo-2 PcCc-2 P:Co-1 Pi
PoCp P.Ci
PıCo
P.Ci
PcCo-1
SHIFTCHANNEL,
Pc-1Ch PC-1Ci
PC-1CC-11
PC-1Co
PCC
Here, no element is zeroed out, as all channels can attend to all other channels. The channel shifting operation does not (to our knowledge) enjoy an implementation which as efficient as the time shifting one in Py Torch, but requires relatively complex index manipulation which cannot be streamlined.
As before, thanks to this shifting operation the channel attention component does not require recom- putation for each channel position.
We provide a Triton implementation for both operations which is much more efficient and must be preferred when training model.
A.6 STRUCTURED ATTENTION DROPOUT
Dropout is a common technique to improve the generalization performance of neural networks. In Transformers, it is often applied inside the attention block to randomly zero-out some query-key attentions, to avoid over-reliance of the model on specifie connections.
dropout
Conventional dropout
Time
Time 2
Time 3
Time
Time
Time S
Man
6
Mappy MM
v0.5 0.3
0.5
Figure 3: Structured dropout. (a) Our structured dropout blanks entire channels and time steps, to reduce the number of correlated segments. The dropout rate is computed maintain the same num- ber of dropped out segments conventional dropout. (b) Conventional dropout blanks segments randomly. This is less effective with time-series because adjacent segments in time or space contain much of the same information.
Dropout usually applies to all elements with equal probability and creates uniform holes in the attention matrix. This is not efficient in the case of multi-variate time-series, as for each hole the neighboring segments are likely to carry very similar information, reducing dropout's effectiveness. We introduce a structured dropout technique which blanks entire channels and time points instead of individual segments. This technique in principle much more effective by removing all segments which more likely be strongly correlated. We keep the same parameters as in conventional Dropout and compute the channel-specific and time-specific dropout rates as
tdrop
Cdrop 1
Tdrop
This ensures that approximately the same overall number of elements zeroed (see Figure 3).
For the specific dropout rates and the location of the structured dropout layers refer to the description of the architecture in Appendix B
A.7 PERFORMANCE COMPARISON
To provide a clearer evaluation of the computational performance benefits of MVPA, and in particu- lar FlashMVPA, we compare the inference speed and VRAM usage of multiple attention implemen- tations. Specifically, we test:
Naïve self-attention
21

---

Published as a conference paper at ICLR 2026
FlashAttention 2
Linear attention (Nyströmformer, Xiong et al. 2021)
MVPA
FlashMVPA
We test all implementations with a batch size of 64 and size of 768, with 6 numeric type. to maintain realistic scenario. We use 12 heads and no GQA, as it is not natively implemented for Nyströmformer. We vary both the number of time windows (T) and the number of channels (C) from I to 50, and report the runtime and memory consumption for forward pass. We test the attention module in isolation to avoid introducing confounding variables, and we follow the best practices in GPU benchmarking.
Tables X and show the full quadratic scaling of MVPA and the full self-attention. However, the tricks we employ for efficient computation still results in lesser memory usage for MVPA, albeit at a higher computational cost.
On the other hand, Tables T0 and report the much more favorable scaling of the two dedicated implementations, with FlashMVPA having the same memory consumption as the more optimized FlashAttention 2.
Finally, Table 12 indicates that while Nyströmformer attention has theoretically favorable linear complexity, its implementation is much less performance than either FlashAttention 2 or Flash- MVPA.
Table 8: Computational performance of MVPA. Runtime and VRAM consumption of the naïve implementation of MVPA.
T(C
1
10
20
30
40
50
/0.02
/0.05
0.06
us 0.07
0.00
2.12 /..10
10
1.72 us/ 0.05 GB
1.81 us 0.18 05 GB
2.59 us/ 0.43 GB
5.11 us/ GB
7.53 1.33 GB
11.83 us/ 1.92 GB
20
1.69 us 0.06 GB
2.64 us 0.43 GB
7.59 us 1.30 GB
15.29 us /.64 GB
25.30 us / /4,47 GB
38.15 us 6.75 GB
30
1.77 0.07 GB
5.20 0081 GB
15.27 us 2.64 GB
34.52 us/ 5.54 GB
67.64 us 9.50 GB
111.33 14.52 GB
/0088
1.31
25.15 us /4447
67.48 us 9950
16.43
161.07 25.25 GB
50
1.74 us 00.10 GB
11.86 us 1.92 GB
37.77 us 6.75 GB
110.74 us 14.52 GB
160.72 us 25.25 GB
278.60 us 38.90 GB
Table 9: Computational performance of self-attention. Runtime and VRAM consumption of the naive implementation of vanilla self-attention. T(C
20
30
40
1.08 u 0.03 GB
1.21 0.05 GB
1.15 us 0.06 GB
1.18 00,06 GB
1.18 / 0.07 GB
1.19 us 0.08 GB
10
1.17 us 0000 GB
1.36 us 0.16 GB
2.17 us 0.43 GB
3.72 0 GB
4.93 us 1.41 GB
7.63 / 2.11
20
1.16 us 1.06 GB
2.17 2.17 us
/0.43 GB
4.93 us/ 1.41 GB
8.99 8.99 us.97 GB
14.37 us 5.12 GB
20.97 us 7.85 GB
30
1.21 us / 0.06 GB
3.72 us/ 0.84 GB
8.99 2.97 GB
20.79 us 6.41 GB
35.93 11.17 GB
59.22 17.25 GB
400
1.24 us / 0.07 GB
4.95 u 1.41 GB
14.35 us 5. GB
35.93 us 11.17 GB
56.17 us 19.56 GB
81.37 us 30.31 GB
50
1.21 us/ 0.08 GB
7.63 us/ 2.11 GB
20.98 us 7.85 GB
59.21 us 17.25 GB
81.38 us/ 30.31 GB
143. us 47.03 GB
Table 10: Computational performance of FlashMVPA. Runtime and VRAM consumption of the Triton implementation of MVPA. T(C
10
20
30
40
50
1.34 us /0.02 GB
1.40 us 0.00 GB
1.39 us 0.06 GR
1.42 us 0.07 GR
1.41 us /0.08 GR
1.49 us /0.9 GB
10
1.41 us / 0.05 GB
1.45 us / 0.14 GB
1.78 us 0.24 GB
2.83 us 1034 GB
4.10 us 0045 GB
4.53 0.56 GB
20
1.42 us 0.06 GB
1.69 us 0.24 GB
4.09 us 0.44 GB
5.66 us / 0.65 GB
10.04 us 0.87 GB
12.10 us/ 1.09 GB
30
1.42 us / 0.07 GB
2.92 us / 0.34 GB
6.00 us 0.65 GB
12.09 us 0.96 GB
18.93 us 1.29 GB
23.98 us/ 1.63 GB
40
1.47 us 0.08 GB
3.98 us 0.45 GB
12.43 us 0.87 GB
16.17 us/ 1.29 GB
27.29 us 1.73
37.68 us 2.19 GB
50
1.44 us / 0.09 GB
4.79 us / 0.55 GB
11.69 us/ 1.08 GB
23.45 us/ 1.62 GB
36.28 us 2.19 GB
60.36 us 2.75 GB
22

---

Published as a conference paper at ICLR 2026
Table 11: Computational performance of FlashAttention 2. Runtime and VRAM consumption of FlashAttention 2. a custom CUDA implementation of self-attention.
Tic
10
20
30
40
50
1.09 us /0.04 GB
1.08 us /0.05
1.08 us 0.06 GB
1.11 us 0.06
1.11 us / GB
1.12 us /.008
10
1.10 us 0.05 GB
1.15 0..1 GB
1.29 us 0.21 GB
1.68 us / 0.30 GB
2.09 us / 0039 GB
2.40 0.44 GB
20
1.10 us / 0.06 GB
1.30 us 0.21 GB
2.07 as 0.39 GB
4.56 us 0.55 GB
4.70 us 0.73 GB
5.16 us 0.90
30
1.12 10lus/ 0.06 GB
1.66 0.30 GB
5.27 us 0.55 GB
4,73 0.81 GB
6.17 1.07 GB
8.11 us 1.33 GB
40
1.17 us 0.07 GB
2.14 us 0.39
4.71 us 0.73 GB
6.19 us 1.07 GB
8,46 us 1.41 GB
10.69 us 1.76 GB
50
1.15 / 0.08 GB
2.41 0.44
5.07 0.90 GB
8.16 1.33
10.64 1.76 GB
13.99 2.19 GB
Table 12: Computational performance of Nyströmformer. Runtime and VRAM consumption of the linear attention from Nyströmformer Xiong et al. T/c
(02T).
20
0.62 us / 0.03 GB
0.72 us / 0.05 GB
0.70 us /0.06 GB
0.72 us 0007 GB
u. GB
0.73 us /0.09 GB
10
0.73 / 0.05 GB
1.56 us 0.15 GB
2.62 /.32 GB
4.44 / 0.55 GB
5.63 us/ 0.85 GB
8.00 us/ 1.19 GB
20
0.72 us / 0.06 GB
2.62 us 0.32
5.63 us 0.85 GB
us 1.60 GB
12.52 us / 2,60 GB
16.61 us / 3.81
30
0.71 us / /0.07 GB
4.44 us / 0.55 GB
8.95 us 1.60 GB
17.73 3.17 GB
31.13 us 5.27 GB
57.89 us/ / 7.90 GB
40
0.76 7.08 GB
5.63 0.85
12:51 us/ 2.60 GB
31.16 5.27 GB
43,59 8889 GB
57.37 us 13.45 GB
50
0.74 us 0.09 GB
7.99 us/ 1.19 GB
16.63 3.88 GB
$7.85 us 7.90 GB
57.35 us 13:45 GB
122.60 us 20.46 GB
23

---

Published as a conference paper at ICLR 2026
B MVPFORMER ARCHITECTURE
MVPFormer is part of family of predictive deep learning models with 74 million (MVPFormer-S) 1.2 billion (MVPFormer-M, or simply MVPFormer for brevity) parameters based on the Trans- former Vaswann et al. 2017) architecture, capable of generating iEEG signals.
Wavelet encoder The first processing step maps the raw iEEG signal to continuous embeddings. We begin by partitioning the raw data into segments of five seconds. Each segment passes inde- pendently through a db4 wavelet decomposition, which has been shown to be highly effective for biosignals (Adeli et al., 2003 Shen et al., (022). Depending on the model's 's overall size, it is then linearly projected onto smaller latent space. This projection,
feature vector, is the embedding. Our method is inspired by wav2vee (Schneider et al. (2019), though we use learnable embeddings. We apply the encoding channel-wise, meaning each segment remains one-dimensional.
Decoder MVPFormer is based on the Llama2 architecture (Tovrron et al. 2023 with parallel at- tention and MI.P blocks inspired by Megatron-LMM (Shoeybi et al. 2019). This choice was informed by the selection of generative model powerful enough to process brain iEEG signals and compu- tationally light enough to enable extensive testing. We provide two models to evaluate the scaling of our foundational model: MVPFormer-S with 75M parameters and MVPFormer-M (or simply MVPForme) with 1.2B parameters.
See Tables and 14 for a breakdown of the models sizes and hyperparameters.
Table 13: Breakdown of the parameters of MVPFormer-S. The dimensions are indicated for each of the components of MVPFormer-S.
Transformer
Encoder
Signal
Mayyrr 12
Tinput 2560
Wlength 500
Thhads f- 12
Rembed +- 768
Reggments <- 100
ggg <- 4
eegmentt 5s
eembed <- 768
7 meegatives < 30
Tiineer 1728
Tijacal +
Tdrop 0.1
Table 14: Breakdown of the parameters of MVPFormer-M. The dimensions are indicated for each the components of MVPFormer-M.
Transformer
Encoder
Signal
Taaaers f- 24
Tiipuut f- 2560
Whongth + 500
Theads 16
Teembed f- 1024
Usegments 100
Tlyya 8
Wsegment 5s
Tlembed 2048
nongattives 30
Thjnner &- 5362
7ooocal +- 10
Tdrop f- 0.1
B. INFERENCE
The full end-to-end inference procedure is reported in Algorithm
B.2 ENCODER
The Encoder block is detailed above. The algorithmic overview is presented in Algorithm
24

---

Published as a conference paper at ICLR 2026
Algorithm: 3: Full inference with MVPFormer
Input: a E RCXT raw input; C,T I'' number of channels and length resp.; Maayrrs 12
Data: c E [...... C]; t [I, [I...., / +
Output: (t(tt1)) E R generated embedding; Rembed 768
def Inference ( Ee,t k SEGMENT(Z)) Ce,t ENCODER (Fe,t) for < 1 to Mayyers do
L Cc,t DECODER(EC.)
(c,(t-1) - Cc,t return (c,(t-))
Algorithm 4: Encoder block of MVPFormer
Input: Zo,t E Fliqu raw input segment; Timput 2560; c [l,..., Cj: [l....,
Output: 2 RReebtt output token; Rembed 768
Data: maximum decomposition level given Pinput
def Encoder (#
de.t <- WISSRETTEWW..AATEEECOOSSTTTTTTT (Ze,t;
Ze.t RMSNORM(de.)
Oct
f LINEAR (Z),1
return Oc,t
B.3 DECODER
The collection of vectors resulting from the Encoder block is flattened into a ID sequence to provide unified input interface the Transformer decoder blocks, consistent with conventional Transform- ers. All the encoded segments corresponding to window form the input to the Transformer module, which computes the MVPA among all the segments. The segments are sequentially processed by multiple Transformer layers, composed of attention and MLP blocks in deep network config- uration. The attention blocks are masked to guarantee that MVPFormer only has access to past segments to generate the target. The model produces one output embedding for each input segment. The algorithmic overview presented in Algorithm 5 while the MLP block in Algorithm
Algorithm 5: Decoder block of MVPFormer
Input: Oc,t E Remted input tokens; Nemmed = 768
Output: Oc,t Reasbs
def Decoder (Oc,t):
Ze,t + RMSNORM(O.::)
Compute attention
ac,t +- MVPATTENTION(Z...
de DROPOUT[LIEARNNOBIS((....
Compute feedforward residtals in parallel with iitention Wang 2021
Sc,t + MLP(ze,)
# Sum residuals and attention
Oc, f Oc,t + de,t + 8,,
return Oc,
25

---

Published as a conference paper at ICLR 2026
Algorithm 6: MLP block of MVPFormer
Input: Ze,t noetel normalised Decoder output; Tembeed 768
Data: Ue Ic,t E Tioeer Plinner 1728
Output: Se,t Reaeee
def MLP (Zc,t)
We.t <- LINEARNOBIAS(Z..)
Jo.t SILU(LIINARNOBBIAS(E...
Sc,t f LINEARNOBIAS(Ue, + 9c,t)
return S,,1
26

---

Published as a conference paper at ICLR 2026
C DETAILS ON TRAINING
C.I GENERATIVE PRE-TRAINING
MVPFormer' is used to generate neuronal activity while in the base prediction task. During training, the target for each output is the successive imput segment in time, not in space. First, we divide each recording into windows of 500 seconds each, with a stride of five seconds. Then, each window is divided into 100 segments (each five seconds long), yielding total of 39B total training segments.
For each target, we sample random input segments from the rest of the batched windows to create the confounding targets Z ==11..., ). These segments still represent actual iEEG signals, so they are plausible, but they are expected to be very different from the true target.
This scheme strikes the correct balance between too much similarity and too little. The objective of MVPFormer to generate future iEEG signals, so we choose a contrastive loss to increase the cosine similarity of its output with the true target, while decreasing it with the confounding targets. As training progresses, MVPFormer starts to produce outputs that look like encoded segments, i.e., its inputs. MVPFormer becomes more and more capable of choosing the right target and thus is able to predict the future signal.
Extraction of positive and negative examples Out of the entire dataset, B windows are chosen at random to form a batch. Each window W; [)] has arbitrary sample rate and C, channels. First, the sampling rate is normalized to 512 Hz, then the windows are divided into T non-overlapping segments per window. Each segment is passed in
segments per-channel, leaving parallel through h the encoder. For the sake of simplicity, suppose one window W (with O* channels) selected at random as the positive window, and all the others as the confounding windows. The embeddings of W form the input context E with length C* x T.
with C2
For each segment, embeddings are selected random from the confounding windows form the negative samples Z. Each Ze, has n elements, thus Z has size C* x T x n. Z is excluded from backpropagation.
MVPFormer processes the entire E at once and produces an output O also of size C* x T. We then compute the losses and iteratively optimize to train the model.
Contrastive loss
We train MVPFormer using contrastive loss (Oord et al. 2018) and auxiliary loss. To compute the contrastive loss, we rely on having other windows in the batch, so a larger batch size leads in general e", i E .1....,1) B] be the outputs of the signal Encoder and o', the Decoder stack, for B the batch size. For each i", we select at random Nnegatives elements from e', # act as our negative samples n'*, Clearly, the bigger the batch the greater the entropy. We compute the contrastive loss for each i" follows:
more stable training and better generalization performance. Let
[L,..., B} the outputs of
C -oo Exez exp(sim(O... zx)/T)
(9)
Finally summing over every i, c, gives us the optimization target for the generative task.
The loss is invariant to the channel which encourages all the outputs to be the same regardless of channel. The temperature 7 is 0.1.
C.2 GENERATION OF NEURONAL ACTIVITY
The generation of brain signals during inference proceeds analogously as during training. However, we do not have access to the same source of entropy as in training since the batches are limited to one subject at a time. This limitation implies that the evaluation scores of MVPFormer must be more punishing than the training objective, since cannot reliably estimate the accuracy with which MVPFormer chooses the right target. For this reason, we measure the cosine similarity directly in way reference scheme. First, consider the cosine similarity of the output with the true target. Second, we consider the similarity with the maximally correlated target. Third, we measure the cosine similarity with the highest form of entropy available, random segments in the batch that
27

---

Published as a conference paper at ICLR 2026
are still close by in time. This measurement ensures that the difference in similarity between the true and confounding targets remains significant.
28

---

Published as a conference paper at ICLR 2026
D SWEC IEEG DATASET
The SWEC iEEG dataset is presented in Section 4 The iEEG signals were recorded intracranially with a sampling rate of either Hz or 1024 Hz. which was then normalized 512 Hz before training MVPFormer. The signals were median-referenced and digitally band-pass filtered between 0.5 and 120 Hz using a fourth-order Butterworth filter, both in a forward and backward pass to minimize phase distortions. All the recordings were inspected by an expert for identification of seizure onsets and offsets, and remove channels corrupted by artifacts.
This dataset may only be used for research. For other applications any liability is denied. In partic- ular, the dataset must be used for diagnostic purposes.
Here, Table 15 shows the full details of the dataset in srbect-b--subject breakdown. Finally, Figure shows two annotated seizures in the dataset.
Table 15: Per-subject details of our SWEC iEEG dataset. Ch. the number electrodes, fs
the sampling frequency in Hz, Rec. [h] is the length of the recording in hours, and Ev. is the number
of seizures. The entire dataset contains 68 subjects, 9328 hours of recording and 704 ictal events.
Subject
f. [Hz]
Rec [h]
Ev.
Subject
Ch.
f. [Hz)
Rec [h]
Subject
J. [Hz)
Rec [h]
IDOI
88
512
293.4
2
ID24
32
1024
40.7
14
ID47
32
1024
330.4
ID02
66
512
235.2
2
ID25
128
512
109.4
ID48
57
1024
28.4
1D03
64
512
158.4
4
ID26
87.6
ID49
60
512
140.4
ID04
32
1024
40.7
ID27
32
1024
146
ID50
64
1024
177.2
2
IDOS
128
512
109.4
4
ID28
75
512
69
IDS1
89
512
161.5
1024
69
ID07
75
512
69
ID30
48
1024
40.9
27
IDS3
22
1024
134.9
ID08
61
1024
143.8
70
ID31
32
1024
42.4
17
ID54
54
1024
202
ID09
48
1024
ID32
212.2
24
1024
152.1
IDI0
32
1024
42.4
17
ID33
104
512
53.6
ID56
62
1024
130.5
IDII
1024
212.2
ID34
191.4
ID57
40
1024
90.7
ID12
56
1024
1914
ID35
104
92
512
138.2
IDI3
64
1024
104
ID36
24
1024
161.4
60
ID39
54
1024
107.3
15
24
1024
161.4
ID37
195.9
512
IDI5
98
512
195.9
2
ID38
1024
177.1
ID61
76
512
89.6
ID16
34
1024
177.1
ID39
1024
129.6
ID62
60
1024
235.
ID40
64
179.8
IDI8
42
1024
205.1
ID41
33
1024
82.7
ID64
56
1024
36.3
20
ID19
29
1024
21.7
ID42
87.8
ID65
49
1024
139.7
ID20
88
512
293.4
2
ID43
126
512
63.2
39
1024
212.3
ID21
66
$12
235.2
2
ID44
60
1024
150.3
ID67
63
512
111.7
ID22
64
512
158.4
4
ID45
157.3
ID68
32
1024
167.8
ID23
32
1024
42.4
ID46
512
140.5
21
29

---

Published as a conference paper at ICLR 2026
Patient
The seizure can be essily di datiingished from the surrounding signal
Patient
The seiiure in not immediately recognisable
Figure 4: iEEG activity of two patients with different ictal patterns. (a) Patient 17 of the SWEC iEEG dataset presents typical ictal events. The seizure can be clearly distinguished even by a non- expert, and MVPFormer performs very well on this patient. The number of channels is reduced from the original recording to facilitate comparison with the more difficult presentation. (b) Patient 14 of the SWEC iEEG dataset does have typical events. The neuronal activity during seizures for this patient cannot be clearly distinguished, and assessment by experts would diverge considerably. As expected, MVPFormer has high level of disagreement the original recording are presented to exclude the chance of some channels carrying additional information.
this patient. All the channels of
30

---

Published as a conference paper at ICLR 2026
E GENERATION OF RESULTS
During evaluation, the target and data prepared according each task and model's specification, analogously to the pre-training task. For MVPFormer, we first divide each recording into windows of 500 seconds each, with stride of five seconds. Then, each window is divided into 100 segments (each five seconds long). In cases where datasets are too small for MVPFormer's context window, we shorten it accordingly. The classification decision taken on the last window.
E.1 CHANNEL SELECTION
In order to ensure a consistent setup across all baselines and to speed up evaluation perform the seizure detection task on the SWEC iEEG dataset using a subset of the channels (see Appendix G.8 for ablation of the selection mechanism). Specifically, with the fixed-channel models such as Brant-2.
always choose 32 channels comply
First, compute the variance and kurtosis for all channels within the first 30 minutes of each subject's recording. We exclude channels based on the following:
Variance above the 99th percentile and below the 1st percentile
Kurtosis above the 95th percentile
This results in a first quality filtering. Next, we rank the channels based on a simple combination of variance and kurtosis with the following:
var (C)
TC = 1+ kurt(C)
(10)
and choose the first 32 channels. In the cases where there not enough channels, we also include some the channels excluded above to reach 32.
E.2 EPISODIC SEIZURE POST-PROCESSING
For episodic evaluation we apply three post-processing steps the model output:
Merge events happening within 5 minutes of each other
Remove events shorter than 20 seconds in length
Remove events with less than positive responses
Moreover, when the subject has multiple seizures in one minute we merge them into one.
E.3 ONLINE SEIZURE THRESHOLDING
In the clinical evaluation setup apply simple thresholding decide whether report seizure or not. We set 3 positive seconds out 10 to be the lower limit for detecting a seizure, to deter false positives; events shorter than 3 seconds are thus reported, and additional latency of seconds is to be considered. We find this trade-off has limited drawbacks in practice, as there is often large disagreement even among neurologists about very short events.
E.4 KAPPA SCORE ESTIMATION
To estimate the Kappa score, we choose 300 random segments per subject to compare their classi- fication from MVPFormer and the labels. We perform multiple iterations to ensure no bias this computation. Figure indicates that our choice of 250 iterations is sufficient for stable results.
31

---

Published as a conference paper at ICLR 2026
E.5 LANDIS AND KOCH CRITERIA
Table 16 reports the commonly used Landis and Koch criteria for qualitative inter-rater agreement estimation from Kappa scores.
Kappa
Agreement
0 - 0.20
Slight
0,21 - 0.40
Fair
0,41 0.60
Moderate
0.61 0.80
Substantial
0.81 1.00
Almost perfect
Table 16: Landis and Koch criteria. Landis and Koch criteria (Landis & Koch 1977) for eval- uating Cohen's kappa in the context of inter-rater agreement between human experts seizure classification.
32

---

Published as a conference paper at ICLR 2026
0.3
Kappa ((ogeaaa))
0.2
1.50
200
200
Iteration
100
150
200
250
leration
Figure 5: Mean absolute error in Cohen's kappa estimation. Our estimation scheme for Cohen's kappa converges after very few iterations. The error is computed per-subject as the absolute dif- ference between the running averages at each consecutive iterations; the running average the average of all preceding steps. The average and standard deviation across all subjects reported here. We compute up to 250 random iterations ensure precise reporting.
33

---

Published as a conference paper at ICLR 2026
F PREDICTION OF IEEG SIGNALS
We evaluate the effects of the size of the model and the attention mechanism on the iEEG prediction task. Figure 6 shows that both MVPA and the vanilla attention are effective at predicting the next brain states. Scaling up the model size from MVPFormer-S to MVPFormer-M has the effect of shortening the tail of the true distribution, effectively increasing the concentration of the cosine similarity towards the maximum.
Two-stop
Random
True
FAHEEG-S
MV-Liama
0.2
0.3
0.4
0.5
0.7
0.9
Coeine similarty
Figure 6: Performance of MVPFormer-M. MVPFormer-S, and MV-Llama on the iEEG pre- diction task. We report the average two-step, random, and true target cosine similarities for the three different models. All three are effective at predicting iEEG activity, while the larger model takes advantage of the increased embedding size by increasing the concentration of cosine similari- ties towards 1.
Complete details are available for the iEEG prediction performance of MVPFormer-M (see Ap- pendix F.1 MVPFormer-S (see Appendix F2 and MV-Llama (see Appendix F3).
34

---

Published as a conference paper at ICLR 2026
F.1 PREDICTION OF IEEG AND ICTAL ACTIVITY
MVPFormer is primarily a neuronal prediction model, trained to generate neuronal activity re- gardless of whether such activity is pathological physiological. To understand the behavior of MVPFormer with anomalous brain states, we evaluate its performance in generating ictal neuronal activity. The precise relationship between ictal and interictal states is point of ongoing discus- sion (Beenhakker & Huguenard 2009; Zaveri et al., 2020), but many consider an approach to seizures as anomalies (Martini et al. 2021) the most appropriate. The SWEC iEEG dataset con- tains many ictal events, so we are able to evaluate the performance of MVPFormer in generating anomalous activity. particular, this dataset the ratio between non-ictal and ictal states is ap- proximately 500:1.
Figure 7 shows that ictal states are not anomalous for MVPFormer. particular, the prediction similarity of MVPFormer does not degrade when generating ictal activity. Moreover, the prediction similarity in the ictal state is neither significantly different from the average similarity nor from the non-ictal similarity. This indicates that MVPFormer's understanding of the mechanisms of genera- tion of neuronal activity encompasses the pathological ictal state as well. Therefore, MVPFormer must model patterns found both in physiological and pathological brain states. Finally, MVPFormer incorporates a model of seizure generation as a by-product of its predictive task, which is particularly noteworthy.
35

---

Published as a conference paper at ICLR 2026
Similarity prediction with larget and controls
Two-step
Handom
True
Subject
Subject
Subject 13
Average
0.1
0.2
0.3
0.4
0.5
0.8
0.7
0.8
0.8
Cceine simiarily
Similaiy of prediction for nonvictal, lotal, and boundary activtyy
Two-step
Handom
True
Non-ictal
Owvlfisats
Cosine simiartly
Figure 7: Performance of MVPFormer on the prediction task. (a) A three-reference evaluation scheme is used assess MVPFormer's performance. The true target is the immediate future, i.c. the next five seconds of iEEG signal. The two-step target is the signal twice removed in the future, iee the five seconds of iEEG signal coming after the true target. Finally, the random target chosen from iEEG signals which close by in time with the true target. The distribution of the average similarity across the entire recording is shown together with the similarity within three representative subjects (with maximum, median, and minimum average similarity). (b) The prediction similarity is computed again for all three targets, distinguishing between targets which lie within an ictal event, without, at the boundary. There is no significant difference in the performance of MVPFormer in predicting ictal or non-ictal activity, indicating that MVPFormer can encompass anomalous brain states as well, together with the transitions between physiological and anomalous.
36

---

Published as a conference paper at ICLR 2026
F.2 EFFECTS OF THE SCALE OF THE MODEL
Figure shows the full details on the performance of MVPFormer-S on the SWEC iEEG dataset in the iEEG prediction task.
Simliarity of prodiction with target
stop
Non-ictal
Ictal
Onlofietta
Average
0.1
0.2
0.3
0.4
0.5
0.8
0.7
0.8
0.9
Cosine similarity
Figure 8: Performance of MVPFormer-S the prediction task. Performance of MVPFormer-S in generating neuronal activity of unseen test subjects. The prediction similarity is computed for all three targets, distinguishing between targets which lie within ictal event without. There is no significant difference in the performance of MVPFormer-S in predicting ictal or non-ictal activity.
37

---

Published as a conference paper at ICLR 2026
F.3 EFFECTS OF THE ATTENTION MECHANISM
Figure 9
shows that vanilla attention is also effective in predicting the development of iEEG signal.
Simitarity of prediction with true larget and controls
Two-step
Random
True
Nor-ictal
Ictal
Boundary
Average
0.1
0.2
0.3
0.5
0.8
0.7
0.8
0.9
Cosine similanty
Figure 9: Performance of MV-Llama on the prediction task. The prediction similarity com- puted for all three targets, distinguishing between targets which lie within an ictal event without. There is no significant difference in the performance of MV-Llama in predicting ictal or non-ictal activity. This indicates that vanilla attention with a proper positional encoding scheme effec- tively generate neuronal activity. However, this does not translate to improved performance in the seizure classification task (see Table 21
Table 20
38

---

Published as a conference paper at ICLR 2026
F.4 PER-SUBJECT COSINE SIMILARITY
We provide a detailed per-subject breakdown of the maximum cosine similarity measure for MVP- Former. Figure shows the per-patient global similarity. F the per-patient similarity within an anomaly. Figure] 12 shows the per-patient similarity at the boundary of an anomaly.
True
Random
TwO-step
Cosine similarity
0.4
0.2
1002
1001
1O0T
6001
2101
9001
8001
0201
200
1201
920
820)
060
260
1801
9801
880
0PO
1042
1901
0901
Patient
Figure 10: Breakdown of total cosine similarity per-patient. Maximum cosine similarity of MVP-
Former's 's output with the true, random, and two-step targets the entire SWEC iEEG dataset. The data is shown patient-by-patiens
39

---

Published as a conference paper at ICLR 2026
True
Random
Two-step
1
0.8
Cosine similarity
0.6
0.4
0.2
1001
100
100T
6001
1112
+704
9T01
8TO0
020
2200
1201
0220
8201
1030
2800
9801
880
240
9
1018
OSal
Patient
Figure 11: Breakdown of anomaly cosine similarity per-patient. Maximum cosine similarity of MVPFormer's 's output with the true, random, and two-step targets while within an anomaly (seizure) the SWEC iEEG dataset. The data is shown aatientbyy-patind
True
Random
Two-step
0.8
Cosine similarity
0.6
0.4
0.2
2001
1000
100
1012
ID14
ID18
I018
ID20
1D24
1224
IDa
1D28
1030
122
ID24
ID36
1038
ID40
ID42
ID44
ID46
1D48
IDS0
Patient
Figure 12: Breakdown of boundary cosine similarity per-patient. Maximum cosine similarity of MVPFormer's output with the true, random, and two-step targets while in the onset and offset zones of seizures in the SWEC iEEG dataset. The data shown patient-by by-patient.
40

---

Published as a conference paper at ICLR 2026
G ADDITIONAL RESULTS
G.1 SEIZURE DETECTION
Table 17 reports summary of the seizure detection results across all datasets and architectures.
Table 17: Results on seizure detection. We compare MVPFormer with Brant-2, the current SOTA Transformer model for iEEG, and MV-Llama, our vanilla attention-bann bassed baseline.
SWEC
MAYO
FNUSA
Episodic
Raw
Raw
Raw
Model
Attention
Kappa
f1
sens
fp/h
f1
f1
spec
fl
sens
spec
MVPFormer
MVPA
0.61
0.59
0.72
0.15
0.51
0.36
0.38
0.91
0.46
0.94
0.10
MVPFormer-S
MVPA
0.57
0.53
0.12
0.49
0.88
0.46
0.99
Brant-2
Vanilla
0.06
0.01
0.01
0.11
0.00
0.19
1.00
0.18
0.46
0.99
0.02
BrainBERT
Vanilla
0.00
0.00
0.00
0.00
0.00
1
MV-Llama
Vanilla
0.11
0.01
0.01
0.02
0.00
1
G.2
BRAIN TREEBANK DECODING TASKS
Table 18 reports summary of the decoding tasks of the Brain TreeBank dataset.
Table 18: Results on Brain TreeBank IEEG tasks. We compare MVPFormer with multiple Transformer-based architectures on the four tasks of the Brain TreeBank dataset (Wang et al.
2024ab. The best results are bolded, while the results where the electrodess position beneficial are underlined.
Model
Attention
Electrode location Pitch
Volume
Onset
Speech
MVPFormer
MVPA
0.83 (0.02)
0.88 (0.01)
0.87 (0.02)
0.90 (0.02)
MV-Llama
Vanilla
No
0.62 (0.03)
0.77 (0.02)
0.80 (0.03)
0.81 (0.02)
Brant
Vanilla
No
0.61 (0.03)
0.74 (0.03)
0.80 (0.04)
0.80 (0.03)
PopT' w/o encoding
Vanilla
No
0.62 (0.07)
0.76 (0.07)
0.81 (0.09)
0.83 (0.10)
PopT (BrainBERT)
Vanilla
Yes
0.74 (0.03)
0.87 (0.03)
0.90 (0.01)
0.93 (0.02)
PopT (TOTEM)
Vanilla
Yes
0.64 (0.03)
0.79 (0.02)
0.90 (0.02)
0.88 (0.05)
41

---

Published as a conference paper at ICLR 2026
G.3 CONVENTIONAL EVALCATION
In addition our clinically motivated evaluation (see Section 5.2.1 we assess all our models using conventional machine learning metrics for seizure detection: F Fl-score, sensitivity, and false positive rate. These metrics are commonly used in benchmarking seizure detection models (Ziyabari al. 2017 Shah et al. 2020). and allow comparison with prior work. The full seizure detection results of M PFormer are shown in Table 17 and Figure
Ictal classfication scores
0.2
0.9
0.7
0.5
0.1
0.3
0.1
raw f1-score
episodic 11 -sccoore
opisodic sensitivity
fpm
egreement
7
Export agreament
Average
Number of patients
6
5
4-
2-
Kappa
Figure 13: Performance of MVPFormer on the classification task. (a) The F1-score, sensitivity, and fp/h are reported. Raw results are computed without any post-processing of MVPFormer's output, while episodic results follow common post-processing procedure which merge close ictal classifications. (b) Cohen's Kappa is used to measure agreement between the artificial assistant and the human expert. The average kappa is 0.61, competitive with the values obtained between human experts. The distribution of kappa values clearly indicates that minority of subjects are the source of most disagreement, consistent with the variability of inter-rater agreement among human experts.
We evaluate against two baselines: Brant-2 Yuan al., 2024a). SOTA Transformer model for iEEG, and MV-Llama, an ablation of MVPFormer-S that uses standard attention instead of MVPA (see Appendix G.5). Brant-2 fine-tuned with its published pre-trained weights and protocol. MV- Llama is trained identically to MVPFormer-S.
We report both raw and episodic metrics. Episodic metries reflect clinically meaningful detections by grouping predictions into events (Ziyabari et al. 2017). The detailed results are provided in Fables[25]and
The similarity between raw and episodic Fl-scores suggests that MVPFormer naturally learns to detect seizure episodes of realistic length and frequency. On the 50-subject SWEC test set, the false positive rate is 0.15 fp/h (0.12 for MVPFormer-S), comparable commercial EEG devices used in clinical practice (Van de Vel et al. 2014 Bruno et al., (220). As expected in real-world dataset, false positive rates vary considerably across subjects, with 81 % having fewer than 0.05 fp/h.
42

---

Published as a conference paper at ICLR 2026
Table 19: Details of seizure detection results of MVPFormer with 18 subject pre-training. Kappa is the inter-rater agreement. The classification metries report the raw and episodic met- rics relevant for the seizure classification task. The similarity reports the breakdown of the cosine similarity in each of the considered scenarios.
Clasiffictiin metrics
Similaritt
Raw
Lpisodic
True
Kandom
Two step
Subject
Kappa
95% C1
T1-score
Fl-scare
cansttivay
1pph
(cTa]
Dosi-ictll
Ictal
mon-ccil
ICTal
Roti-ictal
0.000
0.01
0:25
01.25
0.14
0.92
0.59
ID22
0.99
0.02
0.94
1.00
1,00
0.89
0.92
0.90
0.15
0.15
0.10
0.12
0.15
1025
0.12
0.0
0.21
0.11
0.005
0.00
0.88
0.95
0..4
0.22
0.22
0.20
0.20
0.21
0.20
1D24
0.92
0.01
0.91
0.91
0.93
0.00
0.80
0.93
0.89
0.13
0.13
0.09
0.14
0.11
0.14
ID25
0.00
0.00
0.00
01.00
01000
0184
0.30
0150
0.17
0.24
0.14
0.15
10.00
0.13
ID26
0.03
0.03
0.02
0.04
1,00
0.61
0.88
0.96
0.94
0.16
0.21
0.09
0.13
0.16
0.10
0.95
0.61
0.95
0.01
0.92
0.96
0.93
0.95
0.94
0.17
0.13
0.16
ID31
0.99
0.00
0.97
1.00
1,00
0.00
0.88
0.95
0.95
0.18
0.21
0.13
0.15
0.22
0.17
ID32
1.00
0.00
0.97
1.00
1,00
0.00
0,93
0.93
0.94
0.17
0.15
0.12
0.13
0.16
0.11
ID3S
0.07
0.00
0.00
0..00
0.15
01.55
0.9%
0195
0.39
0.38
0.31
00,99
10:25
0.46
TD34
0.99
0.01
0.97
1.00
1,00
0.00
0.89
0.94
0.92
0.14
0.14
0.09
0.14
0.12
ID35
0.95
0.96
1,00
0.97
0.99
0.99
1.00
0.77
0.15
0.13
0.13
0.14
ID39
0.99
0.02
0.92
1.00
1.00
0.00
0.83
0.70
0,83
0.16
0.12
0.10
0.14
0.12
0.10
ID<10
1.00
0.01
0.99
1.00
1,00
0.00
0.87
0.58
0.90
0.15
O.13
0.09
0.13
0.13
0.1]
ID41
0.15
0.03
0.18
0.15
1,00
0.41
01866
0.79
0.36
QL16
0.15
0.10
0.1S
10.15
0.10
ID42
0.96
0.03
0.86
1.00
1,00
0.00
0.82
0.80
0.79
0.15
0.14
0.07
0.13
0.15
0.07
0.45
0.44
ID47
U.77
0.09
0.00
0,50
1,00
0.00
0.84
0.81
0.83
0.16
0.15
0.12
0.14
0.16
0.00
ID-18
0.87
0.01
0.71
1.00
1.00
0.00
0.80
0.83
0.79
0.12
0.12
0.08
0.10
0.11
0.09
ID:9
0..25
0.04
0.20
0.36
1.00
0.15
0.89
0.79
0.36
0.23
0.14
0.35
0.25
1.17
0.53
ID50
0.97
0.05
0.82
1.00
0.54
0.24
0.38
IDS4
0.01
0.02
0.02
0.02
0.67
0.89
0.82
0.71
0.83
0.16
0.14
0.12
0.13
0.14
0.09
IDSS
1186
0.07
0.75
0.80
1.00
0.01
0.84
0.87
0.80
0.18
0.18
0.11
0.16
0.19
0.14
ID866
0.79
0.06
0.55
0.80
0.67
0.00
0.87
0.81
0.79
0.19
0.18
0.12
0.17
0.20
0.15
ID57
0.00
0.00
0.00
0.00
0.00
0.00
0.57
0.57
0.81
0.16
0.21
0.14
0.13
0.17
0.17
ID58
0.30
0.00
0.00
0.00
0.86
0.77
0.21
0.22
0.33
0.24
0.30
0.00
0.14
0.00
0.13
ID62
0.79
0.04
0.65
0.88
1.00
0.01
0.83
0.86
0.92
0.14
0.13
0.52
0.13
0.12
0.72
ID63
11766
0.06
0.52
0:77
0.02
0.00
0.82
0.86
0.86
0.15
0.16
0.09
0.13
0.15
0.06
1764
0.95
0.07
0.55
1.00
1.00
0.00
0.86
0.87
0.85
0.15
0.15
0.16
0.12
0.14
0.10
ID65
0.04
0.02
0.05
0.03
0150
1.14
01883
0.87
0.85
0.13
0.15
0.09
0.10
1.13
0.08
ID66
0.61
0.19
0.00
0.00
0.00
0.75
0.76
0.12
0.13
0.06
0.11
0.08
These results confirm that MVPFormer performs competitively on conventional seizure detection benchmarks, while also offering robust generalization to clinically realistic evaluation settings.
43

---

Published as a conference paper at ICLR 2026
G.4 EFFECTS OF THE SCALE OF THE MODEL
The performance improvements of LLMs as function of their model sizes have also been widely reported (Hoffmann training dataset is already not large enough to fully train MVPFormer-S (75M parameters), so we investigate whether larger model (MVPFormer-M, 1.2B parameters) can provide any improvement in performance.
2022 Kaplan et al. 2020). According Chinchilla's scaling law the
Figure 14 shows the seizure detection performance of MVPFormer-S on the SWEC iEEG dataset (see Table 20 As noted in the main results, MVPFormer-W marginally improves seizure detection results over MVPFormer-S. In particular, reaches higher Fi-score but higher fp/h rate well, with small net improvement. Therefore, we have shown that the amount of iEEG data currently available not sufficient to fully take advantage of the increase in model size of Transformers. We hope that making the SWEC iEEG dataset publicly available will increase overall availability and unlock further model scaling potential.
Ictal clessilication soores
0.9
0.8
0.7
0.5
0.1
0000
0.3
0.1
raw T1 r1-score
opisodic Ti-score
episadic sensitivity
fp/h
iiter-rater agreement
Expert agreement
Average
3-
2-
0
Figure 14: Seizure detection with 18 patient pre-training. (a) Seizure detection results of MVPFormer-S on unseen subjects: the Fi-score, sensitivity, and fp/h are reported. Raw results are computed without any post-proce post-processing of MVPFormer's output. while episodic results follow common post-processing procedure which merge close ictal classifications. (b) Cohen's 's Kappa is used to measure the agreement between MVPFormer-S and the human expert. The average kappa is 0.57, competitive with the values obtained between human experts. The distribution of kappa values clearly indicates that minority of subjects the source of most disagreement, consistent with the variability of inter-rater agreement among human experts.
44

---

Published as a conference paper at ICLR 2026
Table 20: Details of seizure detection results of MVPFormer-S. Kappa is the inter-rater agree- ment. The classification metrics report the raw and episodic metries relevant for the seizure classifi- cation task. The similarity reports the breakdown of the cosine similarity in each the considered
scenarios.
Classification metries
Similarity Random
Episodic
Twa-sop
Subject
Kappa
Sentativay
arerage
DOG-10L
average
non
prerage
ID19
0.01
0.00
0.02
0.33
0.50
0.25
0.27
0.21
0.20
0:25
0.21
0.20
0.23
1020
0.79
0.06
0.87
0.44
1.00
01002
0.95
0.96
0.95
0.18
0.21
0.12
0.15
0.17
0.12
1921
0.73
0.11
0.51
0,50
1.00
0.02
0.94
0.93
0.95
0.16
0.18
0.18
0.13
10.24
0.20
ID22
0.97
0.02
0.91
1.00
1.00
0.00
0.94
0.89
0.95
0.15
0.16
0.09
0:12
0.17
0.07
ID25
0.15
0.00
0.21
0.16
0.09
0.05
0.95
0.95
0.95
0.25
0.28
0.23
0.22
0.24
0.02
ID27
0.88
0.03
0.78
0.89
1.00
0.01
0.95
0.96
0.96
0.22
0.24
0.21
0.19
0.25
0.24
ID28
0.89
0.00
0.67
0.86
0.75
0.00
0.95
0.96
0.95
0.17
0.19
0.09
0.13
0.17
0.12
ID29
0.45
0.00
0.42
0:27
0.29
0.18
0.95
0.95
0.95
0.16
0.15
0.14
0.12
0,13
0.13
ID30
0.63
0.02
0.66
0.67
0.63
0.17
0.95
0.9%
0196
0.19
0.19
0.15
0.16
0.18
0.16
ID31
0.95
0.00
0.93
1.00
1,00
0.00
0.95
0.96
0.19
0.24
0.11
0.24
0.18
0.95
ID3S
0.77
0.33
0.79
0.80
0.86
0.02
0.92
0.94
0192
QU1Y
0.19
0.19
0.15
0.17
0.22
IT36
0.41
0.01
0.54
0.43
0.32
0.06
0.96
0.96
0.95
0.22
0.23
0.14
0.19
0.21
0.17
ID37
0.50
0.09
0.44
0,31
1.00
0.05
0.92
0.91
0.91
Q.16
O.15
0.33
0.14
0,17
0.89
ID38
11.94
0.02
0.94
0.91
1.00
0.01
0.93
0.95
0.91
0.14
0.15
0.11
0.11
1.16
0.08
0.89
1.00
0.93
0.16
ID43
0.922
0.0
0.29
1.00
1.00
0.00
0.93
0.90
0.95
0.14
0.14
0.10
0.11
0.16
0.12
ID44
0.90
0.22
0.32
0.67
0.50
0.00
0.93
0.95
0.92
0.12
0.12
0.51
0.09
0.10
0.64
ID:5
0.01
0.02
0.00
0.01
1,00
0.99
0.94
0.95
0.93
0.15
O
0.13
0.12
0,14
0.14
ID46
0.30
0.04
0.42
0:45
0.33
0.02
0.94
0.95
0.76
0.18
0.15
0.36
0.16
0.12
0.53
ID47
0.06
0.94
0.97
0.05
0.81
1.00
0.93
0.94
0.17
0.12
0.14
0.23
IDS1
0.70
0.12
0.55
0.33
1,00
0.02
0.93
0.91
0.93
0.23
0.15
0.20
0.23
0.28
0.25
IDS2
I.37
0.05
0.54
0.27
1.00
0.10
0.93
0.94
0.95
0.16
0.18
0.10
0.13
0.16
0.10
IDS3
0.47
0.08
0.40
0.25
1.00
0.04
0.93
0.90
0.93
0.19
0.17
0.13
D.15
0.16
0.12
ID:44
0.01
0.01
0.01
0.01
0.67
1.51
0.94
0.88
0.94
0.18
0.19
0.14
0.14
0.16
0.10
IDSS
0.29
0,20
0.29
0.00
0.11
0:20
0.24
IDS9
0.16
0.06
0.13
0.24
0.13
0.00
0.92
0.94
0.9
0.14
0.14
0.11
0.12
0.12
0.09
ID60
01.23
0.08
0.13
0.50
0.38
0.002
0.95
0.96
0198
0.14
0.14
0.61
0.11
0.12
0.57
ID61
11.36
0.09
0.08
0.29
I.17
0100
0.96
0.96
0196
Q.19
0.17
0.16
0.16
1.16
0.14
ID62
0.81
0.05
0.40
1.00
1.00
0.00
0.92
0.93
0.97
0.14
0.14
0.43
0.11
0.11
0.73
ID63
0.64
0.64
0.95
0,16
0.31
0.90
0.53
0.13
ID67
0.15
0.05
0.26
0.10
0.25
0.13
0.94
0.94
0.94
0.22
017
0.11
0.21
0.13
0.10
ID68
0.69
0.13
0.23
0.40
0.53
0.01
0.94
0.92
0.95
0.15
0.13
0.31
0.10
0.10
0.10
45

---

Published as a conference paper at ICLR 2026
G.5 EFFECTS OF THE ATTENTION MECHANISM
To assess the validity of our MVPA scheme, we train MV-Llama, a model almost equivalent to MVPFormer-S that uses vanilla attention instead of MVPA. While MV-Llama uses vanilla attention, is still based on the SOTA Llama2 architecture. We also re-use the vanilla positional encoding, with a simple adjustment to recover a one-to-one correspondence between the positional encoding and the position of the patch in the time-series (the Cantor pairing function, see App. A.I.
Table 17 the main text indicates that vanilla attention does not perform seizure detection level comparable to MVPA. In particular, the performance of MV-Llama is poor, indicating that it cannot generalize to this task. We argue this is due higher flexibility of the internal representations generated by MVPA, which better lend themselves to further tasks, such seizure classification.
Table 21: Details of seizure defection results of MV-Llama with 18 subject pre-training. Kappa the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task. The similarity reports the breakdown of the cosine similarity in each the considered scenarios.
Classifiation metries
Similarity Random
Raw
Episodic
Thue
Twirsspp
Subject
Kappa
95% Cl
T[-520re
T1-seare
setsstivay
1phh
Brerage
[ctal
noG-ictal
average
kctal
non-scial
Drernge
IcTal
Rot--ettal
ID19
0.00
0.00
0.01
0.12
0.05
0.32
0.229
0.23
0.23
0.25
0.25
0.22
0.22
0.00
0.95
0.96
0:25
ID22
0.34
0.12
0.00
0.00
0.00
0.95
0.78
0.95
0.15
0.13
0.08
0.12
0.16
ID25
0.00
0.00
0.00
0.00
0.00
0.00
0.96
0,96
0.96
0:25
0.26
0.20
0.22
0.24
0.24
ID24
0.00
0.00
0.00
0.00
0.00
0.00
0.94
0.94
0.93
0.19
0.20
0.19
0.16
0.18
0.17
ID25
0,022
0.06
0.00
0.00
01.00
0.17
0.95
0.96
0.96
0.17
0.19
0.16
0.14
1.18
0.15
ID26
0.00
0.00
0.00
0.00
0.00
0.00
0.96
0.97
0.96
0.18
0.22
0.10
0.14
0.18
0.12
0.00
0.00
0.95
0.96
0:13
ID330
0.01
0.06
0.00
0.00
0.00
0.96
0.96
0.96
0.19
0.50
0.15
0.16
0.19
ID31
0.000
0.00
0.00
0.00
0.00
0.00
0.96
0.96
0.96
0.28
0.24
0.16
0.16
0.24
0.18
ID32
0.12
0.09
0.00
0.00
0.00
0.05
0.95
0.58
0.96
0.18
0.16
0.16
0.15
0.18
0.12
ID33
0.000
0.00
0.00
0.00
0.00
0.00
0,93
0.95
0.92
0.23
0.24
0.19
0.21
10.25
0.20
ID34
0.00
0.00
0.00
0.00
0.00
0.00
0.95
0.95
0.96
0.15
0.16
0.12
0.11
0.15
0.12
0.00
0.00
0.00
0.92
0.87
0,16
ID38
0,00
0.00
0.00
0.00
0.00
0.00
0.93
0.95
0.91
0.14
0.16
0.10
0.11
0.16
0.07
1039
0.000
0.00
0.00
0.00
0.00
0.00
0.93
0.87
0.94
0L16
0.14
0.13
0.13
0.11
TD400
0.02
0.06
0.00
0.00
0.00
0.04
0.94
0.95
0.95
0.15
0.17
0.10
0.12
0.15
0.12
ID11
0.72
0.22
0.00
0.00
0.00
0.00
0.96
0.92
0.95
0.16
0.15
0.13
0.13
0,15
0.11
ID42
0.00
0.00
0.00
0.00
0.00
0.94
0.92
0.94
0.15
0.18
0.09
0.12
0.17
0.08
ID4S
0.42
0.14
0.00
0.00
0.00
0.94
0.93
0.15
0.12
0:12
0.14
ID46
0.00
0.00
0.00
0.00
0.00
0.00
0.94
0.96
0.80
0.18
0.14
0.28
0.16
0.12
0.53
ID47
1.29
0.12
0.00
0.00
0.00
0104
0.95
0.94
(1.95
0.18
0.20
0.14
0.15
0.18
0.13
ID48
0.00
0.00
0.00
0.00
0.00
0.00
0.93
0.94
0.93
0.12
0.12
0.07
0.09
0.12
0.00
ID49
8A00
0.03
0.00
0.00
01.00
0.01
0.95
0.93
0.92
0.11
0.14
0.29
0:20
0.19
0.46
ID500
0.06
0.07
0.00
0.00
0.00
0.01
0.94
0.96
0.94
0.17
0.21
0.12
0.23
0.11
0.000
0.00
8,00
0.00
0.00
0.00
0.95
0.94
0.11
0.16
0.17
IDS4
0.52
0,10
0.17
0.33
0.25
0.01
0.94
0.89
0.95
0.18
0.16
0.12
0.15
0.16
0.10
10SS
0.00
0.00
0.00
0.00
0.00
0.00
0.95
0.95
0.93
0.20
0221
0.13
0.17
0.20
0.17
ID56
0.68
0.21
0.00
0.00
0.00
0.00
0.94
0.91
0.89
0.19
0.18
0.15
0.16
0.18
0.15
ID57
0.099
0.08
0.00
0.00
01.00
(1.44
0.95
0.95
0.50
Q.14
0.13
0.10
0.10
D.09
0.14
ID58
0.00
0.00
0.00
0.00
0.00
0.00
0.94
0.94
0.88
0.21
0.22
0.15
0.19
0.23
0.17
1059
0.16
0.33
0.13
0.36
0.08
0.17
0.96
0.96
0.97
0.18
0.16
ID62
0.81
0.05
0.69
1.00
1.00
0.00
0.92
0.93
0.97
0.13
0.13
0.52
0.10
0.10
0.74
ID63
0.64
0.05
0.54
0.73
1.00
0.02
0.94
0.95
0.95
0.16
0
0.11
0.12
0.16
0.07
ID64
0.60
0.02
0.66
0.62
0.45
0.00
0.95
0.95
0.96
0.16
0.18
0.12
0.13
0.15
0.11
ID65
11.75
0.05
0.51
0.77
01022
(100
0195
0.95
0.95
0.14
0.15
0.11
0:11
1.14
0.09
ID66
0.90
0.10
0.53
0.80
1.00
0.00
0.88
0.93
0.92
0.12
0.14
0.08
0.09
0.13
0.08
0.15
46

---

Published as a conference paper at ICLR 2026
G.6
SEIZURE DETECTION WITH BRANT-2
Table 22 presents the detailed subject-by-subjeet breakdown of the performance of Brant-2.
Details of seizure detection results of Brant-2 with 18 subject pre-training, Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Table 22:
Classification metries
Raw
Episodic
Subject
Kappa
95% CI
---scoee
fl-score
sensitivity
fp/h
ID19
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
ID20
0.00
0.00
0.00
0.00
0.00
0.00
ID>21
0.00
0.00
0.00
0.00
0.00
0.01
ID22
0.00
0.00
0.00
ID23
0.00
0.00
0.00
0.00
0.00
0.00
ID24
-0.02
0.03
0.00
0.00
0,00
0.29
ID25
0.00
0.00
0.00
0.00
0.00
0.00
ID26
0.00
0.00
0.00
0.00
0.00
0.00
ID22
0.00
0.06
0.00
0.00
0.00
0.11
ID28
-0.01
0.02
0.00
0.00
0.00
ID29
0.00
0.00
0.00
0.00
ID30
0.00
0.00
0.00
0.00
0.00
0.00
ID31
0.00
0.00
0.00
0.00
0.00
0.00
ID32
0.00
0.00
0.00
0.00
0.00
0.00
ID33
0.48
0.14
0.00
0.00
0.00
0.02
ID34
0.17
0.09
0.00
0.00
0.00
0.05
0.07
0.00
ID36
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
ID37
0.00
0,00
0.00
0.00
0,00
0.00
ID38
0.48
0.15
0.00
0.00
0.00
0.01
ID39
0.07
0.08
0.00
0.00
0.00
0.07
ID40
0.32
0.12
0.00
0.00
0.00
0.03
ID41
0.00
ID42
0.00
0.06
0.00
0.00
0.00
0.14
ID43
0.00
0.00
0.00
0.00
0.00
0.00
ID44
0.00
0.00
0.00
0.00
0.00
0.00
ID45
0.00
0.00
0.00
0.00
0.00
0.00
ID46
0.00
0.06
0.00
0.00
0.00
0.02
ID47
0.00
0.00
0.00
0.00
ID48
0.00
0.00
0.00
0.00
ID49
0.16
0.09
0.00
0.00
0.00
0.02
IDS0
0.00
0.00
0.00
0.00
0.00
0.02
IDS1
0.00
0.00
0.00
0.00
0.00
0.87
IDS2
0.00
0.04
0.00
0.00
0.00
0.20
ID53
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
0.00
0.00
0.00
0.00
IDS5
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
ID56
0.01
0.06
0.00
0.00
0.00
0.16
ID57
0.00
0.03
0.00
0.00
0.00
0.01
IDS8
-0.01
0.05
0.00
0.00
0.00
0.12
ID59
0.00
0.05
0.00
0.00
0.00
0.10
ID60
0.29
ID61
0.69
0.20
0.00
0.00
0.00
0.01
ID62
-0.01
0.03
0.00
0.00
0.00
0.43
ID63
0.00
0.00
0.00
0.00
0.00
0.00
ID64
0.01
0.02
0.01
0.07
0.15
1.10
ID65
0.00
0.00
0.00
0.00
0.00
0.00
ID66
0.07
0.00
ID67
-0.01
0.04
0.00
0.00
0.12
ID68
0.34
0.12
0.00
0.00
0.00
0.02
47

---

Published as a conference paper at ICLR 2026
G.7 SEIZURE DETECTION WITH BRAINBERT
Table 23 presents the detailed subject-by-subjec breakdown of the performance of BrainBERT.
Table 23: Details of
seizure detection results of BrainBERT with 18 subject pre-training, Kappa
the inter-rater agreement. The classification metries report the and episodic metrics relevant for the seizure classification task.
Classification metries
Raw
Episodic
Subject
Kappa
95% CI
Fl T1-score
fl-score
sensitivity
fp/h
ID19
0.00
0.00
0.00
0.00
0.00)
0.00
ID20
0.00
0.00
0.00
0.00
0.00
0.00
ID21
0.00
0.00
0.00
0.00
0.00)
0.00
ID22
0.00
0.00
0.00
0.00
0.00
0.00
ID23
0.00
0.00
0.00
0.00
0.00
0.00
ID24
0.00
0.00
0.00
0.00
0.00
0.00
ID25
0.00
0.00
0.00
0.00
0.00
0.00
ID26
0.00
0.00
0.00
0.00
0.00
0.00
ID27
0.00
0.00
0.00
0.00
0.00)
0.00
ID28
0.00
0.00
0.00
0.00
0.00
0.00
ID299
0.00
0.00
0.00
0.00
0.00
0.00
ID30
0.00
0.00
0.00
0.00
0.00
0.00
ID31
0.00
0.00
0.00
0.00
0.00
0.00
ID32
0.00
0.00
0.00
0.00
0.00
0.00
ID33
0.00
0.00
0.00
0.00
0.00
0.00
ID34
0.00
0.00
0.00
0.00
0.00
0.00
ID35
0.00
0.00
0.00
0.00
0.00
0.00
ID36
0.00
0.00
0.00
0.00
0.00
0.00
ID37
0.00
0.00
0.00
0.00
0.00
0.00
ID38
0.00
0.00
0.00
0.00
0.00
0.00
ID39
0.00
0.00
0.00
0.00
0.00
0.00
ID40
0.00
0.00
0.00
0.00
0.00
0.00
ID41
0.00
0.00
0.00
0.00
0.00
0.00
ID42
0.00
0.00
0.00
0.00
0.00
0.00
ID43
0.00
0.00
0.00
0.00
0.00
0.00
ID44
0.00
0.00
0.00
0.00
0.00
0.00
ID45
0.00
0.00
0.00
0.00
0.00
0.00
ID46
0.00
0.00
0.00
0.00
0.00
0.00
ID47
0.00
0.00
0.00
0.00
0.00
0.00
ID48
0.00
0.00
0.00
0.00
0.00
0.00
ID49
0.00
0.00
0.00
0.00
0.00
0.00
IDS0
0.00
0.00
0.00
0.00
0.00
0.00
ID51
0.00
0.00
0.00
0.00
0.00
0.00
ID52
0.00
0.00
0.00
0.00
0.00
0.00
ID53
0.00
0.00
0.00
0.00
0.00
0.00
ID54
0.00
0.00
0.00
0.00
0.00
0.00
IDSS
0.00
0.00
0.00
0.00
0.00
0.00
ID56
0.00
0.00
0.00
0.00
0.00
0.00
ID57
0.00
0.00
0.00
0.00
0.00
0.00
ID58
0.00
0.00
0.00
0.00
0.00
0.00
ID59
0.00
0.00
0.00
0.00
0.00
0.00
ID60
0.00
0.00
0.00
0.00
0.00
0.00
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.00
0.00
0.00
0.00
0.00
0.00
ID63
0.00
0.00
0.00
0.00
0.00
0.00
ID64
0.00
0.00
0.00
0.00
0.00
0.00
ID65
0.00
0.00
0.00
0.00
0.00
0.00
ID66
0.00
0.00
0.00
0.00
0.00
0.00
ID67
0.00
0.00
0.00
0.00
0.00
0.00
ID68
0.00
0.00
0.00
0.00
0.00
0.00
48

---

Published as a conference paper at ICLR 2026
G.8 EFFECTS OF THE SELECTION CHANNELS
As discussed in previous sections, the number of channels can vary considerably across subjects.
We test three different scenarios:
Automatic channel selection (Appendix G.3
Manual channel selection
Evaluation with all channels
Beyond the automatic channel selection we use for the main results, we also select a subset of the channels (up to 50) which visually appear least noisy and most relevant. We also test the effect of including all channels, expecting to decrease both the speed and performance due to the decrease of the overall signal-to-noise ratio.
Figure 15 shows that performance decreases when we use a manual channel selection (for detailed breakdown Table 24). While this non-expert selection only has minor impact the overall performance, it still is notable that a standardized procedure produces better results.
Ictal clsssilicatio scores
F
0.8
0.8
0.4
0.2
0-
0
episodic
episodo sensltivity
1ph
Inter-rafer agreement
Expert Average
5-
patlents
4-
3-
0-
0.1
0.2
0.3 0.4
0.5
0.7
Kappa
Figure 15: Seizure detection with 18 patient pre-training and evaluation on manual subset of channels. (a) Seizure detection results of MVPFormer-S on unseen subjects evaluated on manual subset of channels: the FI-score, sensitivity, and fp/h are reported. Raw results are computed without any post-processing of MVPFormer's output, while episodic results follow common post- processing procedure which merge close ictal classifications. (b) Cohen's Kappa is used to measure the agreement between MVPFormer-S and the human expert. The average kappa is 0.54, lower than with the automatic channel selection routine. The distribution of kappa values clearly indicates that a minority of subjects are the source of most disagreement, consistent with the variability of inter- rater agreement among human experts.
Figure 16 that the performance decreases when we use all channels (for a detailed breakdown
Table 25 This is expected, the noise contained in the entire recording increases together
49

---

Published as a conference paper at ICLR 2026
Table 24: Details of seizure detection results of MVPFormer-S with evaluation on a manual subset of channels. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task. The similarity reports the breakdown of the cosine similarity in each of the considered scenarios.
Classification metries
Raw
Episodic
Subject
Kappa
95% CI
TT-score
[1-score
sensitivity
Ip/h
ID19
-0.05
0.02
0.00
0.00
0.00
0.97
ID20
0.98
0.03
0.87
0,80
1.00
0.00
0.92
0.86
0.80
1.00
0.00
ID22
0.98
0.02
0.89
1.00
1.00
0.00
ID223
0.26
0.03
0.17
0.14
0.09
0.16
ID24
0.91
0.01
0.90
0.93
0.93
0.02
ID25
0.35
0.12
0.00
0.00
0.00
0.03
ID26
0.08
0.06
0.03
0.11
1.00
0.18
ID27
0.99
0.93
1.00
1.00
0.00
ID28
0.71
0.04
0.68
0.75
0.75
0.01
ID29
0.25
0.05
0.29
0.20
0.13
0.03
ID30
0.92
0.01
0.88
0.93
0.93
0.05
ID31
0.98
0.00
0.95
1.00
1.00
0.00
ID32
1.00
0.02
0.92
1.00
1.00)
0.00
ID33
0.00
0.00
0.00
0.00
0.26
ID34
0.89
0.03
0.91
0.90
1,00
0.01
ID3S
0.85
0.03
0.87
0.82
1.00
0.03
ID36
0.50
0.03
0.62
0.51
0.37
0.03
ID37
0.74
0.09
0.69
0.50
1.00
0.02
ID38
0.94
0.02
0.95
0.91
1.00
0.01
ID39
0.92
0.05
0.89
0.80
1.00
ID40
0.95
0.95
0.91
1.00
0.00
ID41
0.61
0.05
0.65
0.67
1.00
0.04
ID42
0.86
0.05
0.84
0.80
1. 1.00
0.01
ID43
0.90
0.03
0.64
1.00
1.00
0.00
ID44
0.00
0.00
0.00
0.00
0.00
0.00
ID45
0.01
0.02
0.00
0.01
1.00
1.06
ID46
0.60
0.56
0.69
0.52
0.00
ID47
0.26
0.08
0.29
0.16
1.00
0.09
ID48
0.68
0.02
0.62
0.80
0.67
0.00
ID49
0.85
0.03
0.79
0.86
1.00
0.01
ID50
0.69
0.09
0.60
0.44
1 1.00
0.03
ID51
0.35
0.09
0.23
0.15
1.00
0.07
IDS2
0.23
0.35
0.18
1.00
0.16
ID53
0.12
0.06
0.21
0.09
1.00
0.16
IDS4
0.01
0.02
0.01
0.01
0.67
1.72
ID55
0.37
0.08
0.35
0.29
1.00
0.07
ID56
0.72
0.08
0.60
0.67
0.67
0.01
ID57
0.00
0.00
0.00
0.00
0.00
0.00
ID58
0.22
0.25
0.18
0.29
0.09
ID59
0.00
0.00
0.00
0.00
0.00
0.00
ID60
0.33
0.02
0.23
0.46
0.38
0.04
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.69
0.06
0.45
0.77
0.71
0.00
ID63
0.94
0.02
0.90
1.00
1.00
0.00
ID64
0.60
0.02
0.64
0.62
0.50
0.06
ID65
0.50
0.35
0.59
0.03
ID66
0.70
0.11
0.36
0.57
1.00
0.01
ID67
0.26
0.07
0.15
0.33
0.25
0.01
ID68
0.65
0.19
0.00
0.00
0.00
0.01
50

---

Published as a conference paper at ICLR 2026
with the information content. MVPFormer's ability to generalize is not affected by the number of channels, but the noise affects the performance. Therefore, the optimal real-world operation of MVPFormer is obtained by selecting subset of channels for detection.
disselication
0.2
0.9
0.8-
0.7-
0.5-
0.3-
0.1-
raw 11-4core
eptoodic 11-score
eploodic sensitiity
fph
agreement
6-
Average
Expert agreement
8-
Kappa
Figure 16: Seizure detection with 18 patient pre-training and evaluation all channels. (a) Seizure detection results of MVPFormer-S on the unseen patients evaluated on all channels: the F1-score, sensitivity, and fp/h are reported. The performance metrics reduced with respect the results obtained when selecting a subset of the channels. MVPFormer is not affected by the number of channels, but the increase of noise emerging from all the channels contributes to reduction in performance. (b) The average kappa is 0.36, reduced from the evaluation on a subset of channels.
51

---

Published as a conference paper at ICLR 2026
Table 25: Details of seizure detection results of MVPFormer-S with evaluation on all channels.
Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Classification metries
Raw
Episodic
Subject
Kappa
95% CI
fl score
fl fl-score
sensitivity
fp/h
ID19
-0.05
0.02
0.00
0.00
0.00
0.00
ID220
0.84
0.09
0.47
0.50
0.50
0.50
ID21
0.86
0.10
0.40
0.80
1.00
1.00
ID22
0.91
0.81
0.89
1.00
1.00
ID23
0.25
0.03
0.17
0.14
0.09
0.09
ID24
0.91
0.01
0.90
0.93
0.93
0.93
ID25
0.45
0.10
0.17
0.25
0.25
0.25
ID26
0.25
0.08
0.04
0.22
1.00
1.00
ID27
0.98
0.93
1.00
1.00
1.00
ID28
0.64
0.42
0.67
0.50
0.50
ID29
0.05
0.02
0.04
0.07
0.29
0.29
ID30
0.20
0.03
0.31
0.19
0.11
0.11
ID31
0.98
0.00
0.95
1.00
1.00
1.00
ID32
0.98
0.04
0.92
1.00
1.00
1.00
ID33
0.01
0.06
0.00
0.00
0.00
0.00
ID34
0.04
0.06
0.04
1.00
1.00
ID35
0.61
0.05
0.63
0.55
0.43
0.43
ID36
0.50
0.03
0.62
0.51
0.37
0.37
ID37
0.03
0.03
0.02
0.04
1.00
1.00
ID38
0.97
0.01
0.97
1.00
1.00
1.00
ID39
0.77
0.08
0.52
0.67
1.00
1.00
ID40
0.37
0.06
0.50
0.80
0.80
ID41
0.52
0.05
0.47
0.55
1,00
1.00
ID42
0.63
0.07
0.54
0.57
1.00
1.00
ID43
0.94
0.04
0.71
1.00
1.00
1.00
ID44
0.00
0.02
0.00
0.00
0.00
0.00
ID45
0.00
0.02
0.00
0.00
0.00
0.00
ID46
0.06
0.03
0.06
0.19
0.19
ID47
0.23
0.29
0.16
1.00
1.00
ID48
0.25
0.03
0.18
0.31
0.33
0.33
ID49
0.40
0.05
0.47
0.40
1.00
1.00
IDS0
0.05
0.04
0.05
0.06
1.00
1.00
ID51
0.06
0.06
0.05
0.07
1.00
1.00
ID52
0.00
0.01
0.01
0.01
1.00
1.00
ID53
0.13
0.21
0.09
1.00
1.00
ID54
0.01
0.02
0.01
0.01
0.67
0.67
IDS5
0.37
0.08
0.35
0.29
1.00
1.00
ID56
0.33
0.07
0.34
0.31
0.67
0.67
ID57
0.00
0.00
0.00
0.00
0.00
0.00
ID58
0.29
0.12
0.00
0.00
0.00
0.00
ID59
0.02
0.00
0.00
0.00
ID60
0.00
0.00
0.00
0.00
0.00
0.00
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.04
0.05
0.03
0.07
0.29
0.29
ID63
0.21
0.05
0.39
0.18
0.75
0.75
ID64
0.25
0.03
0.30
0.29
0.25
0.25
ID65
0.04
0.00
0.00
0.00
0.00
ID66
0.76
0.16
0.31
0.40
0.50
0.50
ID67
0.17
0.07
0.14
0.22
0.25
0.25
ID68
0.61
0.18
0.00
0.00
0.00
0.01
52

---

Published as a conference paper at ICLR 2026
G.9 EFFECTS OF THE SCALE OF THE PRE-TRAINING DATASET
The performance of LLMs as the size of their training dataset increases has been investigated quite thoroughly (Hoffmann al. 2022 Kaplan al. 2020), giving rise variety of scaling laws. Following Chinchilla's 8 scaling law, a model with 75 million parameters like MVPFormer-S should be trained with around 2 billion tokens, while we only have 400 million our disposal.
The architecture of the model and the nature of the training data, however, make it unclear whether such laws can be adopted for MVPFormer as well. We investigate this behavior by continuing the training of MVPFormer-S on 40 more subjects, to bring the total to 58 pre-training subjects for almost 7, 7,000 hours of iEEG recordings. In particular, MVPFormer is initially trained on 304 ictal events, and then further on 323 more. Therefore, we are left with 10 unseen subject to test the downstream seizure detection task.
Ictal classilication soores
0.2
0.9
0.7-
0.5
0.1
03
0.1
0
raw 11-score
opisodic 11 score
opisodic sensitiitt
1pmh
intor-rater agreement
Exper Spproctii
of
Seizure detection with 58 patient pre-training and manual channel selection. (a) Seizure detection results of MVPFormer-S on the 10 remaining unseen patients using manual chan- nel selection: the F1-score, sensitivity, and fp/h are reported. All performance metries improved with respect to the original MVPFormer model. The raw and episodic F1-scores are significantly different here, indicating that MVPFormer benefits of the episode merging effect of post-procsssing on these patients. The false positive rate has decreased further with the scale of the pre-training dataset. (b) Inter-rater agreement of MVPFormer with the human expert: Cohen's 's kappa is used to measure the agreement between the artificial assistant and the human expert. The average kappa is increased to 0.48. The distribution of kappa values again indicates that there is considerable vari- ability in the agreement.
Figure 17:
Figure 17 shows the performance of the 58-subject MVPFormer-S on the 10 unseen subjects using manual subset of channels (for a detailed breakdown see Table 26). On the other hand, Figure T8 shows the results the original MVPFormer-S model on those same 10 subjects (for a detailed breakdown see Table 27. All performance metrics improve with growing pre-training dataset size. although on small test cohort, indicating that increasing the number of subjects in the pre- training dataset has positive effect the downstream classification task.
53

---

Published as a conference paper at ICLR 2026
Ictal dlssilication
02
0.8
0.7-
0.6
04
02-
0-
raw score
optsodic
opisodic sensiiiity
fph
agreement
Expert agreement
Average
0.1
0.2 0.3 0.4
0.5
0.6
0.7 0.8 0.9
Kеppa
Figure 18: Seizure detection with 18 subjects pre-training on a selection of 10 unseen subjects and manual channel selection. (a) Seizure detection results MVPFormer-S on the subjects excluded from the 58- 58-subjects model using manual channel selection: the F1-score, sensitivity, and fp/h are reported. The raw and episodic Fl-scores are significantly different here, indicating that MVPFormer benefits of the episode merging effect of post-processing on these patients. These sults are subset of those presented in the Results section. (b) Inter-rater agreement of MVPFormer with the human expert: Cohen's 's kappa is used to measure the agreement between the artificial sistant and the human expert. The average kappa is 0.46, reduced from the overall results indicating that these subjects more difficult than average. The distribution of kappa values again indicates that there considerable variability the agreement.
54

---

Published as a conference paper at ICLR 2026
Table 26: Details of seizure detection results of MVPFormer-S with 58 patient pre-training and manual channel selection, Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Classification metrics
Raw
Episodic
Subject
Kappa
95% CI
fl-score
fl-score
sensitivity
fp/h
ID59
0.07
0.06
0.05
0.10
0.07
0.07
ID60
0.54
0.02
0.39
0.47
0.50
0.50
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.65
0.05
0.50
0.71
0.86
0.86
ID63
0.76
0.04
0.84
0.67
1.00
1.00
ID64
0.60
0.02
0.67
0.62
0.45
0.45
ID65
0.50
0.04
0.41
0.52
1.00
1.00
ID66
0.79
0.12
0.50
0.57
1.00
1.00
ID67
0.22
0.04
0.22
0.15
0.75
0.75
ID68
0.73
0.09
0.42
0.80
0.67
0.00
Table 27: Details of seizure detection results MVPFormer-S with patient pre-training on
selection of 10 subjects and manual channel selection. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Classification metrics
Raw
Episodic
Subject
Kappa
95% CI
fl-score
f1-score
sensitivity
fp/h
ID59
0.00
0.00
0.00
0.00
0.00
0.00
ID60
0.33
0.02
0.23
0.46
0.38
0.04
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.69
0.06
0.45
0.77
0.71
0.00
ID63
0.94
0.02
0.90
1.00
1.00
0.00
ID64
0.60
0.02
0.64
0.62
0.50
0.06
ID65
0.50
0.06
0.35
0.59
0.62
0.03
ID66
0.70
0.11
0.36
0.57
1.00
0.01
ID67
0.26
0.07
0.15
0.33
0.25
0.01
ID68
0.65
0.19
0.00
0.00
0.00
0.01
55

---

Published as a conference paper at ICLR 2026
G.10 PATIENT CLASSIFICATION DIFFICULTY
In the medical practice each patient has unique seizure presentations, though they might be broadly grouped into different categories (Shokooh al. 202T As an effect, some patients have seizures which can be considered more typical (Figure h.) and hence easier to detect, while others might have very atypical events (T). There might be broad disagreement among neurologists over these atypical seizures, and at the same time no disagreement at all over the typical patients (Gotman 2011
This phenomenon intuitively creates a difficulty scale among the patients, which also affects MVP- Former and contributes the spread of performance between the model and the human expert. To better assess the impact of this latent patient classification difficulty we performed multiple correlation analysis using the total recording length, the number of seizures, and the frequency of seizures to predict the kappa score, yielding an R of 0.054. The model performance is thus inde- pendent of the three variables, and we believe the difficulty might help explain most of the variance. The literature supports this hypothesis, the subjects themselves account for up to 65% of the variance (Grant et al.) 2014) while the clinical setup itself has no impact.
56

---

Published as a conference paper at ICLR 2026
G.11 CHANNEL CONNECTIVITY MAP
Generating future iEEG signal embedding implicitly places the greatest emphasis on the dimension of time, but do so it is necessary consider the interactions between channels well. MVP- Former thus takes into consideration all electrodes concurrently, as electric potentials flow across different areas and circuits in the brain following their intrinsic connections and constraints (Bet- zel ct al. 2019; Pang et al. 2023). The number of channels depends on the number of electrodes decided for specific patient and the clinical setup. As iEEG implantations are decided on a case-by- case basis by a physician, there is no uniform standard on where to place the electrodes, in contrast with the 10-20 system (Jasper 1958) for EEG. Therefore, we cannot give MVPFormer any a pri- ori knowledge of how the channels will interact in space, and the model has to learn it its own. MVPA enables our model to dynamically learn these connections to build an internal map of the flow, becoming independent from specific electrode configuration.
is well-known that two neighboring brain regions might not be as strongly connected as two faraway regions. The relationship between the electrodes (and hence the channels) mirrors this behavior. To truly understand the link between two channels the model must build a map of the connection strength between different brain areas and how these connections impact the diffusion of electric fields across channels. In MVPA, this understanding is the underpinning of the channel- based component, In particular, the complex interplay between the query, the channel codebook, and the channel attention, acts as the first level of processing. Further, the deep structure with multiple layers provides more representational power, as typical of deep models.
Figure shows that the channel- -ased MVPA component encodes form of the brain connectivity map. Initially, the map is random MVPA is randomly initialized. As training progresses, the atten- tion magnitude among the channels starts to differentiate, building map of the connection strength. The map dominated by the diagonal component, which indicates that in general neighboring chan- nels are more related than distant channels. However, it is possible to clearly distinguish clusters of strongly connected electrodes and also skipped connections, which possibly refer to strong connec- tions between distant regions. Since the channel distance is relative, it can apply to arbitrary clinical setups and not limited to already seen channels. Moreover, as the channel attention a function of both the query content and the channel distance, the combination of the two can effectively modulate the attention even on unseen subjects.
57

---

Published as a conference paper at ICLR 2026
Development channel attento
0.8
0.7
10
Query channdl
Queryhaam
12
Training
12
0.4
0.1
+
& +
+
Key
Development attention
10
10
0.9
30
0.7
40
Query time
40
0.8
Training
70
70
0.3
Key time
Key smo
Figure 19: Attention components before and after training. (a) The channel-based attention
component is randomly initialized. At the end of training, it shows the diagonal structure which indicates that the relationship between the channels is mainly one of proximity. This is expected, as nearby channels are expected be more closely related, and showcases MVPA's learning outcome. (b) The time-based attention component is also randomly initialized. At the end of training, shows that segments which are close in time are more related. Particularly, few closest segments are attended to more strongly, as the content-based attention's 's lookback windows is limited to few segments.
58

---

Published as a conference paper at ICLR 2026
G.12 RESULTS ON THE MAYO AND FNUSA DATASETS
We test MVPFormer additionally against Brant-2 on the iEEG MAYO and FNUSA datasets. Both datasets single-channel iEEG datasets containing both physiological and pathological activity. We use these datasets to evaluate MVPFormer's performance on extremely noisy data, and to com- pare its resilience with the SOTA.
The MAYO dataset contains 24 patients for a total of 130 hours of data. In particular, 36% of the data non-ictal, 9% is ictal, and 53% is noise. Moreover, 18 subjects contain no ictal activity and patients are fully noise, with 13 being majority noise. The FNUSA dataset contains 14 patients for a total of 160 hours. 48% of the dataset is non-ictal, 27% is ictal, and 23% noise. Moreover, the data of 2 subjects is completely noise, and the data of 3 other subjects is fully ictal. Both datasets are much smaller in scale than the SWEC iEEG dataset, and are heavily dominated by noise, both artifact and powerline. In particular, the SWEC iEEG dataset is almost 2,000 times larger and is also carcfully evaluated by an expert ncurologist to remove channels which contain too much noise or artifacts.
As we wish to assess MVPFormer in a realistic, real-world scenario, we do not remove any noise from the dataset but test them as-is. In particular, consider noise and physiological activity as one category, and pathological activity as another. However, kappa scores are not meaningful with such small datasets, therefore we provide the aggregate F1-score, and the average sensitivity and specificity. Specifically, given the fact that many patients do not contain ictal activity, we do not compute the average Fl-score across subjects, but pool together all subject's results and compute the aggregate Fl-score.
We use the same MVPFormer models pre-trained on our SWEC iEEG dataset, and train a specific classification head for either MAYO or FNUSA by fine-tuning on the first four patients. Then, we test on the remaining patients. We also use the Brant-2 model whose pre- pre-trained weights are publicly available, and fine-tune in the same manner as MVPFormer using the fine-tuning code provided by the authors.
The results can be found in Tables 28 and 29 datasets, overall performance affected. the FNUSA dataset, where the amount of noise is more moderate, all models perform similarly, with MVPFormer-M showing higher specificity. However, MVPFormer has a clear advantage on the MAYO dataset, with almost double the Fl-score with respect Brant-2. The difference between MVPFormer-S and MVPFormer-M is minimal, as the sizes the datasets involved are too small to fully train a very large model such as MVPFormer- M (see Appendix G.4 for more information).
Given the very low signal-to-noise ratio of both
Table 28: Summary of seizure detection results of all models the MAYO iEEG dataset. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Model
F1-score
Sensitivity
Specificity
MVPFormer-M
0.36
0.38
0.91
MVPFormer-S
0.35
0.41
0.88
Brant-2 0.19
1.00
0.18
Table 29: Summary of seizure detection results of all models the FNUSA IEEG dataset.
Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Model
Fl-score
Sensitivity
Specificity
MVPFormer-M
0.46
0.94
0.10
MVPFormer-S
0.46
0.99
0.03
Brant-2
0.46
0.99
0.02
59

---

Published as a conference paper at ICLR 2026
G.13 ABLATION OF THE PREDICTION TASK
We design MVPFormer with two-phase training regime. First, during the generative pre-training task MVPFormer learns predict the neuronal activity. Second, during the classification task needs to correctly classify ictal periods. To determine the significance of the generative task on the classification task, we train MVPFormer only on the classification task and compare its performance with the full architecture on the manual selection of channels. Figure 20 and Table 30 clearly indi- cate that the generative task is of fundamental importance the overall architecture, with Kappa score decrease to 0.52. This is below the original result of 0.54 and below the human agreement threshold. Moreover, the distribution of agreement has flattened, with an overall decrease of perfor- mance across the board and an increase of subjects with no agreement. This suggests that without pre- pre-traing training the generalization capability of MVPFormer suffers. Therefore, the generative task is necessary and is significant contributor to learning.
Ictal clasafication
0.2
0.9
0.8
0.6
0.5
0.1
0.3
0.1
sensitivity
fp/h
rater agreement
Average EExpert agreement
5
0.1 0.2 0.3 0.4
0.5
0.6
0.7
0.8
0.9
Kappa
Figure 20: Seizure detection with no generative pre-training and 18 subjects classification training on the manual selection of channels. (a) Seizure detection results of MVPFormer-S on 40 unseen subjects which are part of the training set for the 58-subjects model and manual se- lection channels: the Fl-score, sensitivity, and fp/h are reported. The raw and episodic F1-scores are notably lower here with respect to the 58-subjects model. This is expected given the 58-subject model pre-trained on these subjects. These results are subset those presented in the Results section. (b) Cohen's kappa is used to measure the agreement between the artificial assistant and human expert. The average kappa 0.56, competitive with expert agreement but, as expected, reduced from the 58 58-subjects pre-trained model.
60

---

Published as a conference paper at ICLR 2026
Table 30: Seizure detection with no generative pre-training and 18 subjects classification train- ing the manual selection of channels. Kappa is the inter-rater agreement. The classification metries report the raw and episodic metries relevant for the seizure classification task.
Classification metries
Raw
Episodic
Subject
Kappa
95% CI
TI score
fl-score
sensitivity
Tp/h
ID19
0.37
0.46
0.30
0.19
0.05
ID20
0.99
0.02
0.98
1.00
1.00
0.00
ID21
0.47
0.09
0.30
0.29
1.00
0.04
ID22
0.74
0.06
0.70
0.73
1.00
0.02
ID223
0.37
0.02
0.51
0.36
0.24
0.07
ID24
0.88
0.01
0.84
0.90
0.93
0.05
ID25
0.00
0.00
0.00
0.00
0.00
0.00
ID26
0.43
0.10
0.19
0.29
1.00
0.06
ID27
0.95
0.02
0.84
1.00
1.00
0.00
ID28
0.73
0.04
0.80
0.75
0.75
0.01
ID29
0.17
0.05
0.28
0.17
0.10
0.01
ID30
0.90
0.01
0.85
0.93
0.93
0.05
ID31
0.98
0.00
0.95
1.00
1.00)
0.00
ID32
0.98
0.05
0.93
1.00
1.00
0.00
ID33
0.00
0.00
0.00
0.00
0.00
0.00
ID34
0.78
0.03
0.84
0.82
1.00
0.02
ID35
0.71
0.03
0.77
0.74
1.00
0.05
ID36
0.59
0.03
0.68
0.53
0.37
0.01
ID37
0.66
0.10
0.61
0.50
1.00
0.02
ID38
0.59
0.05
0.64
0.59
1.00
0.04
ID39
0.76
0.09
0.71
0.57
1.00
0.02
ID40
0.67
0.05
0.77
0.57
0.80
0.02
ID41
0.75
0.05
0.72
0.75
1.00
0.02
ID42
0.97
0.04
0.74
1.00
1. 1.00
0.00
ID43
0.15
0.09
0.00
0.00
0.00
0.06
ID44
0.98
0.03
0.76
1.00
1.00
0.00
ID4S
0.03
0.03
0.04
1.00
0.33
ID46
0.60
0.03
0.62
0.68
0.62
0.03
ID47
0.79
0.08
0.72
0.55
1.00
0.02
ID48
0.88
0.01
0.76
1.00
1. 1.00
0.00
ID49
0.08
0.03
0.08
0.11
1.00
0.71
ID50
0.36
0.08
0.36
0.24
1.00)
0.07
IDS1
0.52
0.32
0.22
1.00
0.04
ID52
0.50
0.07
0.29
0,50
1.00
0.04
ID53
0.20
0.07
0.33
0.10
1,00
0.13
ID54
0.61
0.12
0.31
0.29
0.33
0.01
ID55
0.35
0.08
0.25
0.25
1.00
0.08
ID56
0.82
0.06
0.61
0.80
0.67
0.00
ID57
0.00
0.00
0.00
0.00
0.00
ID58
0.17
0.20
0.21
0.26
ID59
0.00
0.00
0.00
0.00
0.00
0.00
ID60
0.16
0.03
0.08
0.36
0.25
0.02
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.42
0.06
0.44
0.41
0.86
0.07
ID63
0.54
0.09
0.25
0.57
0.50
ID64
0.62
0.59
0.65
0.50
0.03
ID65
0.31
0.04
0.24
0.39
1.00
0.18
ID66
0.69
0.11
0.52
0.50
1.00
0.02
ID67
0.00
0.06
0.00
0.00
0.00
0.13
ID68
0.78
0.15
0.33
0.50
0.33
0.00
61

---

Published as a conference paper at ICLR 2026
G.14 RESILIENCE TO NOISE
A critical aspect of any clinical environment are the perturbations and noise in the recorder signal. The SWEC iEEG dataset is collected in real-world environment and is pre-processed following the indications of an expert neurologist, to guarantee a clinically-relevant scenario to test any model, including our MVPFormer against.
In addition to the inherent noise of the iEEG signal, we now further disturb the recordings using white gaussian noise evaluate the resilience of MVPFormer additional perturbations, Specifi- cally, we add noise to achieve an SNR of 30dB (Table 32). 40dB (Table 33 50dB (Table 34). and 60dB (Table 35
understand the behavior of MVPFormer.
Table 31: Results on seizure detection with noise. We evaluate MVPFormer's performance on the seizure detection task at varying levels of injected white gaussian noise.
Episodic
Raw
SNR
Kappa
fl
sens
fp/h
fl
None
0.61
0.59
0.72
0.15
0.51
60dB
0.58
0.54
0.71
0.12
0.49
50dB
0.54
0.50
0.72
0.13
0.47
40dB
0.36
0.34
0.74
0.46
0.31
30dB
0.12
0.12
0.71
1.22
0.10
A summary of the results is presented in Table 31 Remarkably, the performance remains above the expert-level threshold until 40dB of SNR, indicating that MVPFormer resilient noise. In particular, while sensitivity remains high. the number of false positives rapidly increases. This is consistent with the fact that seizures often appear as high-frequency activity which can mislead the model when in a noisy environment.
62

---

Published as a conference paper at ICLR 2026
Table 32: Details of seizure detection results of MVPFormer-S with 30dB SNR. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for seizure classification task. The similarity reports the breakdown of the cosine similarity in each of the considered scenarios.
Classification metries
Raw
Episodic sensitivity
Subject
Kappa
95%
fl-score
fl-score
fp/h
ID19
0.01
0.00
0.02
0.28
0.62
2.07
ID>20
0.05
0.04
0.09
0.04
1.00
0.36
ID21
0.01
0.03
0.01
0.02
1.00
0.78
ID22
0.08
0.04
0.06
0.12
1.00
0.36
ID23
-0.04
0.00
0.06
0.15
2.90
ID24
0.06
0.01
0.06
0.12
0.50
2.24
ID25
0.01
0.02
0.00
0.02
0.50
2.19
ID26
0.00
0.01
0.00
0.01
1.00
3.11
ID27
0.00
0.00
0.00
0.02
0.75
3.54
ID28
0.23
0.05
0.26
0.35
0.75
0.15
ID29
0.03
0.02
0.05
0.26
2.03
ID30
0.01
0.00
0.02
0.35
0.56
1.05
ID31
0.08
0.01
0.09
0.18
1.00
3.54
ID32
0.02
0.03
0.02
0.02
1.00
0.79
ID33
0.23
0.10
0.00
0.00
0.00
0.06
ID34
0.01
0.01
0.01
0.04
1.00
2.40
ID35
0.03
0.01
0.05
0.05
0.86
2.10
ID36
0.00
0.01
0.13
0.45
2.13
ID37
0.01
0.02
0.01
0.01
1.00
1.61
ID38
0.33
0.05
0.48
0.34
1.00
0.11
ID39
0.42
0.07
0.40
0.36
1 1.00
0.05
ID40
0.06
0.03
0.09
0.08
0.80
0.42
ID41
0.00
0.01
0.01
0.03
1.00
2.49
ID42
0.10
0.15
0.14
1.00
0.28
ID43
0.77
0.05
0.57
0.80
1.00
0.02
ID44
0.21
0.09
0.00
0.00
0.00
0.08
ID45
0.01
0.03
0.00
0.02
1.00
0.80
ID46
0.18
0.03
0.14
0.25
0.43
0.31
ID47
0.00
0.01
0.00
0.01
1.00
2.80
ID48
0.90
0.77
1.00
0.00
ID49
0.01
0.01
0.05
1 1,00
1.48
ID50
0.02
0.02
0.02
0.02
1.00
0.94
IDS1
0.22
0.08
0.11
0.11
1.00
0.11
ID52
0.04
0.02
0.05
0.03
1.00
0.99
ID53
0.01
0.02
0.01
0.01
1.00
2.18
ID54
0.00
0.00
1.00
4.02
IDS5
0.00
0.00
0.01
1,00
2.60
ID56
0.01
0.01
0.01
0.02
0.67
1.79
IDS7
-0.01
0.03
0.01
0.07
0.08
0.19
ID58
0.00
0.01
0.00
0.01
0.14
1.86
ID59
0.16
0.06
0.11
0.20
0.13
0.03
ID60
0.26
0.02
0.27
0.11
0.50
1.16
ID61
0.02
0.00
0.00
0.35
ID62
0.13
0.05
0.18
0.13
1.00
0.40
ID63
0.13
0.03
0.12
0.12
1.00
0,31
ID64
0.01
0.01
0.02
0.19
0.45
1.79
ID65
0.02
0.03
0.01
0.03
0.25
0.87
ID66
0.62
0.14
0.28
0.25
0.50
0.02
ID67
0.03
0.05
0.02
0.50
1.47
ID68
0.57
0.09
0.32
0.40
0.67
0.03
63

---

Published as a conference paper at ICLR 2026
Table 33: Details of seizure detection results of MVPFormer-S with 40dB SNR. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for seizure classification task. The similarity reports the breakdown of the cosine similarity in each of the considered scenarios.
Classification metries
Raw fl-score
Episodic sensitivity
Subject
Kappa
95%
fl-score
fp/h
ID19
0.01
0.00
0.02
0.28
0.44
1.24
ID>20
0.05
0.04
0.11
0.03
1.00
0.39
ID21
0.03
0.04
0.04
0.04
1.00
0.42
ID22
0.81
0.05
0.74
0.73
1.00
0.02
ID23
0.11
0.02
0.10
0.12
0.15
1.06
ID24
0.50
0.02
0.52
0.55
0.86
0.44
ID25
0.01
0.02
0.00
0.03
0.50
1.11
ID26
0.01
0.02
0.01
0.02
1.00
1.27
ID27
0.04
0.02
0.03
0.05
0.88
1.65
ID28
0.18
0.04
0.26
0.21
0.75
0.32
ID29
0.53
0.44
0.27
0.29
0.19
ID30
0.30
0.01
0.27
0.40
0.93
1.76
ID31
0.79
0.01
0.80
0.81
1.00
0.19
ID32
0.20
0.07
0.21
0.15
1.00
0.11
ID33
0.54
0.15
0.00
0.00
0.00
0.02
ID34
0.36
0.04
0.53
0.33
1.00
0.19
ID35
0.42
0.04
0.61
0.44
0.86
0.13
ID36
0.19
0.17
0.77
ID37
0.05
0.04
0.08
0.05
1.00
0.36
ID38
0.93
0.03
0.95
0.91
1.00
0.01
ID39
0.90
0.07
0.83
0.80
1 1.00
0.01
ID40
0.83
0.04
0.84
0.73
0.80
0.01
ID41
0.02
0.01
0.02
0.03
1.00
2.29
ID42
0.87
0.84
0.80
1.00
ID43
0.93
0.03
0.71
1.00
1.00
0.00
ID44
0.72
0.21
0.00
0.00
0.00
0.01
ID45
0.01
0.02
0.01
0.02
1.00
0.80
ID46
0.37
0.04
0.33
0.41
0.52
0.16
ID47
0.01
0.02
0.01
0.01
1.00
1.63
ID48
0.88
0.73
1.00
1.00
0.00
ID49
0.20
0.04
0.25
0.26
1,00
0.24
ID50
0.99
0.03
0.89
1.00
1.00
0.00
IDS1
0.38
0.10
0.22
0.18
1.00
0.06
ID52
0.09
0.03
0.18
0.06
1.00
0.57
ID53
0.03
0.04
0.07
0.03
1.00
0.50
ID54
0.00
0.00
0.02
1.00
1.85
IDS5
0.02
0.02
0.02
0.02
1,00
1.16
ID56
0.47
0.07
0.43
0.36
0.67
0.05
IDS7
0.00
0.00
0.00
0.00
0.00
0.00
ID58
0.07
0.06
0.07
0.07
0.14
0.15
ID59
0.18
0.06
0.10
0.24
0.13
0.00
ID60
0.32
0.02
0.18
0.33
0.38
0.14
ID61
0.24
0.17
0.36
0.13
ID62
0.78
0.05
0.60
0.93
1.00
0.00
ID63
0.64
0.05
0.67
0.73
1.00
0.02
ID64
0.56
0.02
0.52
0.62
0.60
0.19
ID65
0.39
0.06
0.35
0.43
0.62
0.07
ID66
0.79
0.13
0.38
0.50
0.50
0.00
ID67
0.03
0.05
0.02
0.50
1.43
ID68
0.52
0.11
0.14
0.25
0.33
0.02
64

---

Published as a conference paper at ICLR 2026
Table 34: Details of seizure detection results of MVPFormer-S with 50dB SNR. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for seizure classification task. The similarity reports the breakdown of the cosine similarity in each of the considered scenarios.
Classification metries
Raw fl-score
Episodic sensitivity
Subject
Kappa
95%
fl-score
fp/h
ID19
0.01
0.00
0.02
0.33
0.50
1.15
ID>20
0.76
0.10
0.82
0.33
1.00
0.01
ID21
0.50
0.09
0.38
0.31
1.00
0.04
ID22
0.97
0.03
0.89
1.00
1.00
0.00
ID23
0.11
0.03
0.15
0.09
0.06
0.19
ID24
0.91
0.01
0.87
0.93
0.93
0.02
ID25
0.13
0.06
0.05
0.15
0.25
0.07
ID26
0.03
0.04
0.02
0.05
1.00
0.47
ID27
0.68
0.04
0.66
0.73
1.00
0.04
ID28
0.86
0.03
0.87
0.86
0.75
0.00
ID29
0.52
0.49
0.31
0.29
0.13
ID30
0.74
0.01
0.73
0.76
0.78
0.17
ID31
0.98
0.00
0.92
1.00
1.00
0.00
ID32
0.87
0.09
0.76
0.67
1.00
0.01
ID33
0.50
0.15
0.00
0.00
0.00
0.02
ID34
0.79
0.03
0.85
0.75
1.00
0.03
ID35
0.71
0.04
0.75
0.75
0.86
0.03
ID36
0.46
0.58
0.43
0.04
ID37
0.46
0.09
0.42
0.29
1.00
0.05
ID38
0.90
0.03
0.95
0.91
1.00
0.01
ID39
0.99
0.03
0.89
1.00
1 1.00
0.00
ID40
0.82
0.04
0.88
0.80
0.80
0.00
ID41
0.23
0.04
0.29
0.22
1.00
0.25
ID42
0.99
0.92
1.00
1.00
0.00
ID43
0.93
0.03
0.71
1.00
1.00
0.00
ID44
0.89
0.15
0.32
0.67
0.50
0.00
ID45
0.01
0.02
0.00
0.01
1.00
0.99
ID46
0.40
0.04
0.42
0.45
0.33
0.02
ID47
0.04
0.04
0.07
0.05
1.00
0.38
ID48
0.86
0.70
1.00
1.00
0.00
ID49
0.44
0.05
0.49
0.55
1 1,00
0.07
ID50
0.98
0.05
0.81
1.00
1.00
0.00
IDS1
0.73
0.12
0.55
0.40
1.00
0.02
ID52
0.26
0.05
0.42
0.19
1.00
0.15
ID53
0.26
0.08
0.30
0.14
1.00
0.09
ID54
0.01
0.01
0.02
1.00
1.41
IDSS
0.12
0.06
0.12
0.12
1,00
0.20
ID56
0.72
0.07
0.54
0.67
0.67
0.01
IDS7
0.00
0.00
0.00
0.00
0.00
0.00
ID58
0.33
0.12
0.00
0.00
0.00
0.01
ID59
0.11
0.06
0.08
0.12
0.07
0.00
ID60
0.23
0.03
0.15
0.50
0.38
0.02
ID61
0.41
0.08
0.29
0.00
ID62
0.82
0.04
0.64
1.00
1.00
0.00
ID63
0.79
0.04
0.75
0.89
1.00
0.01
ID64
0.62
0.02
0.66
0.65
0.50
0.03
ID65
0.63
0.06
0.43
0.67
0.62
0.01
ID66
0.84
0.16
0.38
0.50
0.50
0.00
ID67
0.12
0.23
0.07
0.25
0.22
ID68
0.65
0.13
0.22
0.33
0.33
0.01
65

---

Published as a conference paper at ICLR 2026
Table 35: Details of seizure detection results of MVPFormer-S with 60dB SNR. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for seizure classification task. The similarity reports the breakdown of the cosine similarity in each of the considered scenarios.
Classification metries
Raw fl-sccre
Episodic sensitivity
Subject
Kappa
95%
fl-score
fp/h
ID19
0.01
0.00
0.02
0.33
0.50
1.15
ID>20
0.87
0.06
0.91
0.57
1.00
0.01
ID21
0.74
0.11
0.49
0.50
1.00
0.02
ID22
0.97
0.02
0.91
1.00
1.00
0.00
ID23
0.10
0.03
0.17
0.10
0.06
0.09
ID24
0.94
0.01
0.90
0.96
0.93
0.00
ID25
0.18
0.09
0.00
0.00
0.00
0.06
ID26
0.03
0.03
0.02
0.04
1.00
0.62
ID27
0.85
0.03
0.78
0.89
1.00
0.01
ID28
0.81
0.04
0.85
0.86
0.75
0.00
ID29
0.44
0.44
0.28
0.29
0.17
ID30
0.68
0.01
0.68
0.70
0.70
0.20
ID31
0.98
0.00
0.93
1.00
1.00
0.00
ID32
1.00
0.02
0.93
1.00
1.00
0.00
ID33
0.51
0.15
0.00
0.00
0.00
0.02
ID34
0.76
0.03
0.87
0.82
1.00
0.02
ID35
0.74
0.04
0.79
0.80
0.86
0.02
ID36
0.43
0.52
0.40
0.06
ID37
0.61
0.08
0.54
0.40
1.00
0.03
ID38
0.92
0.02
0.95
0.91
1.00
0.01
ID39
0.98
0.03
0.89
1.00
1 1.00
0.00
ID40
0.88
0.04
0.91
0.89
0.80
0.00
ID41
0.29
0.04
0.37
0.29
1.00
0.18
ID42
0.98
0.87
1.00
1.00
0.00
ID43
0.89
0.04
0.56
1.00
1.00
0.00
ID44
0.88
0.11
0.32
0.67
0.50
0.00
ID45
0.01
0.02
0.00
0.01
1.00
0.94
ID46
0.40
0.04
0.43
0.47
0.33
0.01
ID47
0.04
0.05
0.08
0.06
1.00
0.31
ID48
0.87
0.70
1.00
1.00
0.00
ID49
0.54
0.05
0.52
0.63
1 1,00
0.05
ID50
0.96
0.05
0.81
1.00
1.00
0.00
IDS1
0.83
0.13
0.67
0.50
1.00
0.01
ID52
0.35
0.05
0.52
0.25
1.00
0.11
ID53
0.46
0.09
0.40
0.22
1.00
0.05
ID54
0.01
0.01
0.67
1.49
IDS5
0.46
0.35
0.36
1,00
0.05
ID56
0.72
0.07
0.52
0.67
0.67
0.01
IDS7
0.00
0.00
0.00
0.00
0.00
0.00
ID58
0.30
0.12
0.00
0.00
0.00
0.01
ID59
0.20
0.06
0.13
0.24
0.13
0.00
ID60
0.26
0.03
0.15
0.50
0.38
0.02
ID61
0.35
0.08
0.29
0.17
0.00
ID62
0.84
0.04
0.67
1.00
1.00
0.00
ID63
0.70
0.05
0.67
0.80
1.00
0.01
ID64
0.65
0.02
0.67
0.67
0.50
0.00
ID65
0.70
0.06
0.51
0.77
0.62
0.00
ID66
0.88
0.09
0.53
0.80
1,00
0.00
ID67
0.15
0.24
0.10
0.25
0.14
ID68
0.76
0.13
0.26
0.40
0.33
0.01
66

---

Published as a conference paper at ICLR 2026
G.15 EVALUATION OF MAXIMUM PERFORMANCE
To better characterize MVPFormer's ability to generalize to unseen subjects, we perform the seizure detection task 40 subjects in two different scenarios. First, those 40 subjects (see Figure 2T and Table 36) to determine MVPFormer's maximum performance on the manual selection of channels. Second, we a model for which those 40 subjects are unseen (see Figure 22] and Table 37). As expected, with Kappa score of 0.73 the model trained on the testing subjects achieves superior agreement even to human experts, and can therefore be seen as having learned the training set. On the other hand, as seen with previous results as well, in the unseen subject scenario MVPFormer reaches a Kappa score 0.56, indicating a high degree of generalization.
model that is pre-trained
Ictal classlication
0.2
0.9
0.7
0.5
0.5-
0.3-
0.1
rater agrocment
Expert agreement
Average
pastents
4-
0.1
0.2
0.8
0.4
0.5
0.5
0.7 0.8
0.9
Kappa
Figure 21: Seizure detection with 58 subject pre-training and evaluation 40 pre-trained subjects on the manual selection channels. (a) Seizure detection results of MVPFormer-S on 40 previously seen subjects using the manual selection of channels: the Fl-score, sensitivity, and fp/h are reported. The performance metrics are notably improved due to testing on previously seen subjects. (b) Cohen's 's kappa is used to measure the agreement between the artificial assistant and the human expert. The average kappa is 0.73, notably improved from the baseline.
67

---

Published as a conference paper at ICLR 2026
R
0.9
0.7
0.6
0.4
0.2-
0.14
row f1-score
opisodic fl Scone
opisadic sensiivityy
fg/h
Inter-rater agreement
Expert agreement
Average
putients
3-
2-
1-
0.1 0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
Kappa
Figure 22: Seizure detection with 18 subjects pre-training selection of 40 unseen subjects and the manual selection of channels. (a) Seizure detection results of MVPFormer-S on 40 unseen subjects which are part of the training set for the 58-subjects model using the manual selection of channels: the Fl Fl-score, sensitivity, and fp/h are reported. The raw and episodic F1-scores are notably lower here with respect to the 58-subjects model. This is expected given the 58-subject model pre- trained on these subjects. These results are subset of those presented in the Results section. (b) Cohen's kappa is used to measure expert. The average kappa is 0.56, competitive with expert agreement but, expected, reduced from the 58-subjects pre- trained model.
agreement between the artificial assistant and the human
68

---

Published as a conference paper at ICLR 2026
Table 36: Details of seizure detection results of MVPFormer-S with 58 subject pre-training and evaluation on 40 pre-trained subjects using the manual selection channels. Kappa the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Classification metrics
Raw
Episodic
Subject
Kappa
95% CI
fl-score
fcore
sensitivity
fp/h
ID19
0.36
0.02
0.29
0.29
0.19
0,09
ID20
0.92
0.05
0.92
0.67
1.00
0.01
ID21
0.75
0.12
0.62
0.50
1.00
0.02
ID22
0.77
0.06
0.78
0.73
1.00
0.02
ID23
0.36
0.02
0.39
0.39
0.27
0.09
ID24
0.69
0.02
0.64
0.76
0.79
0.10
ID25
0.43
0.15
0.00
0.00
0.00
0.01
ID26
0.98
0.04
0.80
1.00
1.00
0.00
ID27
0.80
0.04
0.74
0.80
0.75
0.01
ID228
0.62
0.05
0.54
0.67
0.75
0.03
ID>29
0.24
0.05
0.30
0.22
0.13
0.01
ID30
0.85
0.01
0.79
0.87
0.89
0.10
ID31
0.97
0.00
0.94
1.00
1.00
0.00
ID32
0.99
0.03
0.92
1.00
1.00
0.00
ID33
0.00
0.00
0.00
0.00
0.00
0.00
ID34
0.88
0.03
0.88
0.90
1.00
0.01
ID35
0.92
0.02
0.89
0.93
1.00
0.01
ID36
0.26
0.04
0.42
0.27
0.17
0.02
ID37
0.93
0.07
0.85
0,80
1.00
0.01
ID38
0.98
0.01
0.95
1.00
1.00
0.00
ID39
0.87
0.06
0.84
0.80
1.00
0.01
ID40
0.77
0.05
0.82
0.62
0.80
0.02
ID41
0.99
0.02
0.94
1.00
1.00
0.00
ID42
0.99
0.01
0.95
1.00
1.00
0.00
ID43
0.83
0.05
0.79
0.80
1.00
0.02
ID44
0.98
0.02
0.87
1.00
1.00
0.00
ID45
0.91
0.13
0.67
0.67
1.00
0.01
ID46
0.80
0.02
0.76
0.86
0.76
0.00
ID47
0.93
0.07
0.84
0.86
1.00
0.00
ID48
0.97
0.01
0.93
1.00
1.00
0.00
ID49
0.67
0.05
0.74
0.67
1.00
0.04
ID50
0.81
0.08
0.75
0.67
1.00
0.01
ID51
0.71
0.12
0.53
0.40
1.00
0.02
ID52
0.36
0.05
0.63
0.24
1.00
0.12
ID53
0.99
0.04
0.91
1.00
1.00
0.00
ID54
0.55
0.09
0.35
0.36
0.67
0.03
ID55
0.98
0.03
0.85
1.00
1.00
0.00
ID56
0.85
0.05
0.87
0.75
1.00
0.02
ID57
0.20
0.01
0.38
0.11
0.83
1.74
ID58
0.40
0.04
0.41
0.46
0.86
0.09
ID59
0.00
0.00
0.00
0.00
0.00
0.00
ID60
0.16
0.03
0.08
0.36
0.25
0.02
ID61
0.00
0.00
0.00
0.00
0.00
0.00
ID62
0.42
0.06
0.44
0.41
0.86
0.07
ID63
0.54
0.09
0.25
0.57
0,50
0.01
ID64
0.62
0.02
0.59
0.65
0.50
0.03
ID65
0.31
0.04
0.24
0.39
1.00
0.18
ID66
0.69
0.11
0.52
0.50
1.00
0.02
ID67
0.00
0.06
0.00
0.00
0.00
0.13
ID68
0.78
0.15
0.33
0.50
0.33
0.00
69

---

Published as a conference paper at ICLR 2026
Table 37: Details of seizure detection results of MVPFormer-S with 18 subjects pre-training on selection of 40 unseen subjects using the manual selection of channels. Kappa is the inter-rater agreement. The classification metrics report the raw and episodic metrics relevant for the seizure classification task.
Classification metrics
Raw
Episodic
Subject
Kappa
95% CI
fl-score
fl-score
sensitivity
fp/h
ID19
-0.05
0.02
0.00
0.00
0.00
0.97
ID20
0.98
0.03
0.87
0.80
1.00
0.00
ID21
0.92
0.07
0.86
0.80
1.00
0.00
ID22
0.98
0.02
0.89
1.00
1.00
0.00
ID22
0.26
0.03
0.17
0.14
0.09
0.16
ID224
0.91
0.01
0.90
0.93
0.93
0.02
ID25
0.35
0.12
0,00
0.00
0.00
0.03
ID26
0.08
0.06
0.03
0.11
1.00
0.18
ID27
0.99
0.01
0.93
1.00
1.00
0.00
ID28
0.71
0.04
0.68
0.75
0.75
0.01
ID29
0.25
0.05
0.29
0.20
0.13
0.03
ID30
0.92
0.01
0.88
0.93
0.93
0.05
ID31
0.98
0.00
0.95
1.00
1.00
0.00
ID32
1.00
0.02
0.92
1.00
1.00
0.00
ID33
0.00
0.05
0.00
0.00
0.00
0.26
ID34
0.89
0.03
0.91
0.90
1.00
0.01
ID35
0.85
0.03
0.87
0.82
1.00
0.03
ID36
0.50
0.03
0.62
0.51
0.37
0.03
ID37
0.74
0.09
0.69
0.50
1.00
0.02
ID38
0.94
0.02
0.95
0.91
1.00
0.01
ID39
0.92
0.05
0.89
0.80
1.00
0.01
ID40
0.95
0.03
0.95
0.91
1.00
0.00
ID41
0.61
0.05
0.65
0.67
1.00
0.04
ID42
0.86
0.05
0.84
0.80
1.00
0.01
ID43
0.90
0.03
0.64
1.00
1.00
0.00
ID44
0.00
0.00
0.00
0.00
0.00
0.00
ID45
0.01
0.02
0.00
0.01
1.00
1.06
ID46
0.60
0.04
0.56
0.69
0.52
0.00
ID47
0.26
0.08
0.29
0.16
1.00
0.09
ID48
0.68
0.02
0.62
0.80
0.67
0.00
ID49
0.85
0.03
0.79
0.86
1.00
0.01
IDS0
0.69
0.09
0.60
0.44
1.00
0.03
ID51
0.35
0.09
0.23
0.15
1.00
0.07
ID52
0.23
0.05
0.35
0.18
1.00
0.16
ID53
0.12
0.06
0.21
0.09
1.00
0.16
ID54
0.01
0.02
0.01
0.01
0.67
1.72
ID55
0.37
0.08
0.35
0.29
1.00
0.07
ID56
0.72
0.08
0.60
0.67
0.67
0.01
ID57
0,00
0.00
0,00
0.00
0.00
0.00
ID58
0.22
0.07
0.25
0.18
0.29
0.09
70

---

Published as a conference paper at ICLR 2026
G.16 EFFECTS OF THE NUMBER OF CHANNELS ON THE BRAIN TREEBANK DATASET
To better evaluate the robustness of MVPFormer to an increasing number of channels, we evaluate the four tasks of the BrainTreelBank with range of 10 to 50 channels. The performance of MVP- Former moderately increases with no reduction with the channel number, as reported in PopTT (Chau al. 2025) as well, indicating that our model robust the number of channels.
Table 38: Effects of the number of channels on the four tasks of the Brain TreeBank dataset. Evaluation of the performance of MVPFormer with respect to number of channels for fine-tuning and testing.
Channels
Pitch
Volume
Onset
Speech
10
0.81 (0.01)
0.85 (0.01)
0.86 (0.02)
0.87 (0.02)
20
0.82 (0.01)
0.87 (0.01)
0.87 (0.02)
0.88 (0.02)
30
0.82 (0.02)
0.87 (0.01)
0.87 (0.02)
0.89 (0.02)
40
0.83 (0.01)
0.87 (0.01)
0.87 (0.02)
0.89 (0.02)
50
0.83 (0.01)
0.88 (0.01)
0.87 (0.02)
0.90 (0.02)
71

---

Published as a conference paper at ICLR 2026
G.17 EVALUATION ON TRADITIONAL LONG-TERM FORECASTING TASK
To provide comprehensive evaluation of MVPA, we compare MVPFormer with existing SOTA architectures classical long-term forecasting task. Table 39 reports the results of MVPFormer. the vanilla Transformer (Vaswani et al. 2017). PatchTST (Nie et al. 2023). TimesFM (Das et al.) 2024). TimeMixer (aag et al.] 2024b). and WPMixer (Chau et 2025) on the ETThl, ETTh2, and Weather datasets (Zhou et al. 2021 Wu et al., 2021). These datasets represent well-known benchmark that allows us decouple MVPFormer from the specific clinical setting. The lookback window is fixed to 96, while the forecast is performed at lengths of 96, 192, 336, and 720. These settings are well established in the literature (Wang et al. 2024b). MVPFormer notably surpasses the vanilla Transformer and is competitive with established architectures designed specifically for this task, achieving the best or second best result in most cases. Moreover, MVPFormer is on average 2x faster to train than the vanilla Transfomer and 1.4x faster than PatchTST still slower than TimeMixer, which a fully MLP-based model -, making an excellent choice in many scenarios. Therefore, have shown that MVPFormer and MVPA have wide applicability and transfer their performance from the clinical task for which they were designed to more general time-series tasks as well.
Table 39: Classical time-series forecasting benchmark. MVPFormer is compared with multiple SOTA architectures the time-series forecasting task using the ETTh1, ETTh2, and Weather datasets. The lookback window fixed at 96 and the forecasting length varies between 96 to 720. The vanilla Transformer also included point of comparison. In bold the best MSE results,
in italics are the second best. MVPFormer notably outperforms the vanilla Transformer and is
competitive with all baselines, having either the best or second best result in most cases.
Model
MVPFormer (ours)
Transformer
PatchTST
TimesFM
TimeMixer
WPMixer
Metric
Length
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.38
0.40
0.83
0.72
0.38
0.40
0.39
0.41
0.37
0.40
0.39
0.40
192
0.45
0.44
0.96
0.78
0.43
0.43
0.46
0.44
0.44
0.43
0.43
0.42
ETTh
336
0.49
0.46
1.04
0.83
0.49
0.45
0.49
0.45
720
0.49
0.48
1.16
0.86
0.52
0.51
0.50
0.48
0.49
0.48
0.49
0.47
96
0.30
0.35
2.64
1.30
0.31
0.35
0.30
0.45
0.29
0.35
0.30
0.35
ETTh2
192
0.37
0.40
3.48
1.47
0.38
0.40
0.37
0.40
0.37
0.39
0.37
0.40
336
4.07
1.62
0.43
0.44
0.42
0.43
720
0.43
0.45
3.28
1.52
0.44
0.45
0.47
0.47
0.45
0.46
96
0.17
0.22
0.33
0.38
0.17
0.22
0.17
0.21
0.16
0.21
0.17
0.21
Weather
192
0.21
0.26
0.51
0.50
0.22
0.26
0.22
0.26
0.21
0.25
0.22
0.25
336
0.62
0.56
0.28
0.30
0.26
0.30
720
0.35
0.35
0.91
0.70
0.36
0.35
0.35
0.35
0.35
0.35
0.35
0.35
72

---

Published as a conference paper at ICLR 2026
G.18 ABLATION OF THE THREE COMPONENTS TRADITIONAL LONG-TERM FORECASTING TASK
MVPA composed of three components (content, time, and channel attention) which process dif- ferent aspects of the time-series in parallel. We evaluate the impact of the components on the long- term forecasting tasks as above. Table 40 reports the results of full MVPA (all three components), content-only attention, time-only attention, channel-only attention, and no attention on the ETTh1, ETTh2, and Weather datasets (Zhou et al. 2021 Wu et al. 2021).
Full MVPA is consistently the better performer, except on the 96 and 720 lengths of the ETTh2 dataset. Notably, MVPA outperforms all variants by greater margin on the Weather dataset, which has the most number of channels (21 vs 7 of ETThl and ETTh2). This suggests that strongly multi- variate time-series provide a considerable advantage to MVPA, which is consistent with its design.
Table 40: Ablation the three components on the classical time-series forecasting benchmark. MVPA is compared with ablated variants of its three components the time-series forecasting task using the ETTh1, ETTh2, and Weather datasets. The lookback window is fixed at 96 and the forecasting length varies between 96 720. The vanilla Transformer is also included point of comparison. In bold are the best MSE results.
Model
MVPA
Content-only MAE
Time-only
Channel-only
None
Metric
Length
MSE
MAE
MSE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.38
0.40
0.39
0.40
0.39
0.40
0.39
0.40
0.41
0.40
192
0.45
0.44
0.45
0.44
0.45
0.44
0.45
0.45
0,46
0.45
ETTh1
336
0.49
0.46
0.49
0.46
0.49
0.46
0.49
0.46
0.50
0.49
720
0.49
0.48
0.49
0.48
0.49
0.48
0.49
0.48
0.49
0.48
0.30
0.35
0.31
0.35
0.29
0.35
0.30
0.35
0.30
0.35
192
0.37
0.40
0.37
0.39
0.38
0.40
0.37
0.40
0.38
0.39
ETTh2
336
0.42
0.43
0.43
0.43
0.42
0.43
0.42
0.43
0.43
0.43
720
0.43
0.45
0.42
0.44
0.43
0.44
0.42
0.44
0.47
0.47
96
0.17
0.22
0.19
0.23
0.19
0.23
0.18
0.22
0.19
0.23
192
0.21
0.26
0.23
0.26
0.23
0.26
0.22
0.26
0.23
0.26
Weather
336
0.28
0.30
0.29
0.30
0.29
0.30
0.28
0.30
0.29
0.30
720
0.35
0.35
0.36
0.35
0.36
0.35
0.35
0.35
0.36
0.35
73


Reviews:
[{"review_id": "SnKGa68cQw", "reviewer": "Reviewer_oKKd", "summary": "This paper introduces Multi-variate Parallel Attention (MVPA), which disentangles content, temporal, and spatial attention to handle heterogeneous time-series with varying channel configurations, and applies it to build MVPFormer, an iEEG foundation model achieving SOTA results on seizure detection and decoding tasks.", "questions": "What specific criteria define a foundation model in your view, and how does MVPFormer satisfy these conditions, given that it's trained on a single modality (iEEG) from one domain and requires fine-tuning for downstream tasks rather than demonstrating broad zero-shot generalization?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written with a clear motivation for the problem and an architectural approach.\n\n- The strong architecture design enables practical foundation models for clinical iEEG\n\n- MVPA's decomposition of attention into content, temporal, and spatial components is novel and specifically addresses the real-world challenge of heterogeneous channel configurations in clinical data.\n\n- The model demonstrates superior performance on seizure detection across three datasets and competitive results on Brain TreeBank decoding tasks and standard time-series benchmarks.", "weaknesses": "- The model is trained only on iEEG data, whereas foundation models typically leverage diverse datasets across multiple domains and modalities.\n- The model is fine-tuned on target tasks, so calling evaluation on \"unseen subjects\" zero-shot is inaccurate. Normally, a true zero-shot would require no task-specific training data. Should be called fewshots? \n- Section 5.3 abruptly shifts to generic time-series forecasting and classification tasks, creating a disjointed narrative that dilutes the paper's clinical focus.\n- The paper is missing an ablation study of the proposed attention and the tree components in the main body.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5zck2tkHvR", "reviewer": "Reviewer_cFg6", "summary": "The paper proposes Multi-Variate Parallel Attention (MVPA), an attention mechanism that splits attention into content, temporal, and channel components to better handle heterogeneous multi-channel time series. Building on MVPA, the authors train MVPFormer, a generative pre-trained foundation model for iEEG on a newly released “Long-term iEEG” corpus of about 10k recording hours. MVPFormer generalizes zero-shot across patients for seizure detection and performs competitively on four Brain TreeBank decoding tasks, while MVPA also matches or exceeds strong baselines on standard time-series forecasting and classification benchmarks. \n\nMain contributions include:\n1 MVPA: a self-attention variant that separately models content, time, and channel structure to support generalization under variable, heterogeneous channels.\n\n2 MVPFormer: a generative, MVPA-powered foundation model for electrophysiology that outperforms vanilla-attention baselines for seizure detection and improves over a matched discriminative model.\n\n3 Long-term iEEG dataset: release of a large public corpus, roughly 9,300 to 10,000 hours across 68 subjects, enabling open data, code, and weights for the community.", "questions": "1 Can you provide a fully automatic end-to-end inference pipeline with no manual channel selection, and report side-by-side results on the same test set? Use identical preprocessing, post-processing, and thresholding across all models, select thresholds on a shared validation split, and include per-subject distributions with 95% CIs plus clear leakage controls (window overlap, session boundaries). \n\n2 Can you give a rigorous derivation from dual-encoding attention to the sum of content, time, and channel terms, stating the conditions under which cross terms are dropped and providing error bounds? Please clarify how the summed logits are scaled or weighted (and whether weights are learnable). Show ablations that remove each term, swap relative for absolute encodings, and vary channel count; add a synthetic study to demonstrate when each component is required.\n\n3 Can you release reproducible runtime and memory curves versus sequence length T, channel count C, and local window L, reflecting the expected O(T·C·L) behavior for the content term? Provide controlled throughput comparisons on the same hardware and batch size against strong efficient-attention baselines (axial or factorized attention, linear-time variants, FlashAttention with GQA), with scripts and fixed seeds.\n\n4 Can you supply a same-dataset, same-protocol head-to-head with human experts, including inter-rater agreement, event-level error breakdowns, and statistical tests? Please adapt strong baselines for channel heterogeneity, enforce a unified evaluation protocol across datasets, and add few-shot curves to separate pretraining benefits from protocol artifacts. If the method still leads under these stricter comparisons, the “expert-level” statement becomes defensible.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1 MVPA factorizes self-attention into content, time, and channel to handle heterogeneous, variable-channel signals that standard attention struggles with. MVPFormer’s generative pretraining and the Long-term iEEG corpus add fresh angles on both model and data.\n\n2 The method is solid  and scalable, with a coherent pretraining recipe followed by light adaptation.\n\n3. The three MVPA components and their roles in the logits are well-explained with figures. Pretraining and fine-tuning protocols are modular, and the dataset description is specific enough to judge external validity.\n\n4  Addressing variable, heterogeneous channels is a core deployment blocker, and MVPA tackles it directly. The approach likely transfers beyond iEEG, while the large open corpus and a usable pretrained model can accelerate community progress.", "weaknesses": "1 Zero-shot tests use manual channel selection at inference, and preprocessing, post-processing, and thresholds are not harmonized across baselines. The reported gains may stem from pipeline differences rather than the core method.\n\n\n2 Overstated “expert-level” claim: A single Kappa threshold from prior work is used instead of a same-dataset, same-protocol human comparison. No per-subject confidence intervals or significance tests are reported.\n\n3 The MVPA decomposition lacks a rigorous derivation and error bounds. How summed logits are scaled or weighted is unclear, and notation/dimensions are inconsistent in places.\n\n4: The stated complexity does not square with a local-window content term, and there are no reproducible runtime or memory curves versus sequence length, channel count, or window size. Missing direct comparisons with other efficient attention variants.\n\n5: Pretraining choices (negative sampling strategy, temperature, hard negatives, sample count) are not systematically ablated. Cross-dataset protocols differ, and strong baselines are not adapted for channel heterogeneity, weakening SOTA claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wxrEQykYcC", "reviewer": "Reviewer_qWbc", "summary": "This paper presents a new method for learning representations of multi-channel intracranial activity. The input is one embedding vector per segment of activity, augmented with position vectors for time and for channel. Gains in efficiency can be had by reusing attention computations for channels that share the same time. During pretraining, a discriminative loss is used. Downstream evaluation is done on a seizure detection task and audio-linguistic tasks. Additionally, the paper releases 10K hours of iEEG recordings.", "questions": "●\tFor comparison, you can cite BrainWave: A Brain Signal Foundation Model for Clinical Applications https://arxiv.org/pdf/2402.10251. They have 35K hours of data. But most of it is EEG and most of it is private.\n●\tFigure 2: In the legend, what is \"two-step\"? It looks like this is defined in Fig 7 of the appendix, but it would be good to have that in the main text somewhere.\n●\tLine 329: \"We must also consider that our evaluation setup involves many more subjects and ictal events than are reported for human experts,\" — does this mean that the expert annotations contain false negatives? I.e., periods of activity that are not marked as seizures, but should be?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "●\tOverall, this paper represents a very strong contribution to the community.\n●\tReleases a large amount of data. I am personally not aware of a larger publicly available iEEG dataset. This is a big boon to the community. Especially so, since many other foundation models for intracranial signal train on private data, e.g., BrainWave.\n●\tEvaluation is thorough: the authors use their own seizure detection task as well as the Brain Treebank tasks.\n●\tPerformance on the epilepsy detection task exceeds the current state of the art", "weaknesses": "●\tAm I misunderstanding something? The paper refers to a \"generative\" objective, but the loss seems to be discriminative, i.e., an InfoNCE loss? The output of the model is in the embedding space, not neural activity, correct?\n●\tLine 364: The claim is that the choice of objective is justified by an ablation. But if I read appendix G.14 correctly, it seems that there is only justification for doing pretraining, not the specific type of pretraining, i.e., some other choice of pre-training objective. This is not a major weakness, but more precise wording is probably needed.\n●\tIn the related works section, it would be good to discuss various factored approaches that are proposed for spatio-temporal data. For example, for processing movies: https://arxiv.org/pdf/2106.05968.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Skyiz96g7c", "reviewer": "Reviewer_dxZg", "summary": "This paper introduces Multi-Variate Parallel Attention (MVPA), a novel self-attention mechanism that addresses the challenges of multi-variate time-series data with heterogeneous channel configurations. The MVPA mechanism is designed to handle time-series signals that vary across different subjects, particularly in clinical domains like intracranial electroencephalography (iEEG). The model efficiently separates attention into three components: content-based, time-based, and channel-based attention, enabling flexible processing of data without relying on fixed channel positions or global positional encodings. The authors apply MVPA to develop MVPFormer, a foundation model for human electrophysiology, which is trained on the Long-term iEEG dataset, the largest publicly available iEEG corpus. MVPFormer achieves superior generalization across subjects and outperforms state-of-the-art (SOTA) models in several clinical tasks such as seizure detection, while also excelling in general time-series forecasting and classification tasks.", "questions": "1. How does MVPA perform in scenarios with significantly different time series compared to iEEG, particularly in domains like financial time series or sensor data? Could the model’s flexibility be leveraged for other domains?\n\n2. While the results show strong generalization across subjects, could the model handle extremely varied electrode setups (e.g., patients with unusual electrode configurations)? How does MVPA cope with potential signal distortions caused by non-standard setups?\n\n3. With the Long-term iEEG dataset being very large (10,000 hours), what are the limitations in terms of processing and inference time when scaling to even larger datasets, especially in real-time clinical settings?\n\n4. Could MVPFormer's real-world clinical application be affected by variability in electrode placement (e.g., anatomical differences across patients)? Have you tested MVPFormer on data from patients who have had non-standard electrode placements due to medical conditions?\n\n5.While MVPFormer is open-source, how are you ensuring patient privacy and safety when providing access to the dataset and model weights? Are there any restrictions on data sharing due to privacy concerns that might hinder the broader clinical adoption of this model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of the Multi-Variate Parallel Attention (MVPA) mechanism is innovative and addresses the challenge of heterogeneous channel configurations in multi-variate time-series data. The way MVPA separates content, temporal, and spatial attention is novel and can be generalized to other time-series domains beyond iEEG.\n\n2. The MVPFormer model, powered by MVPA, shows impressive performance in several iEEG-related tasks, including seizure detection, outperforming existing models. The model demonstrates expert-level performance on the Long-term iEEG dataset and outperforms SOTA methods across various clinical benchmarks.\n\n3. The paper releases the Long-term iEEG dataset, the largest publicly available iEEG dataset to date, containing nearly 10,000 hours of recordings. This is a significant contribution to the research community, addressing the issue of data scarcity in iEEG research.\n\n4. The commitment to open-source the dataset, code, and weights is a major advantage for reproducibility and allows other researchers to build upon this work.\n\n5. The use of MVPA and MVPFormer is not limited to iEEG but is shown to generalize well to classical time-series forecasting and classification tasks, offering a broader impact in the field of time-series modeling.", "weaknesses": "1. While the results on seizure detection are impressive, the paper lacks a detailed real-world scenario evaluation, particularly regarding how the model would perform in a clinical setting with real-time data or noisy recordings. The authors should consider adding practical deployment considerations and edge-case performance, such as handling low-quality signals or data interruptions.\n\n2. MVPA introduces a level of computational complexity, especially in the time- and channel-based terms. While the paper provides solutions to mitigate this, it would benefit from more in-depth comparisons with simpler models that might achieve similar performance with less computational overhead.\n\n3. Although MVPFormer performs well in seizure detection and on Brain TreeBank tasks, the paper could benefit from broader evaluation across more clinical tasks (e.g., epilepsy classification or other cognitive tasks), especially those commonly encountered in clinical settings.\n\n4.While the Long-term iEEG dataset is a valuable contribution, it has limitations, such as lack of electrode location information, which may limit its utility in some clinical contexts. The paper mentions this but does not fully address how future versions of the dataset might overcome this limitation.\n\n5. While MVPFormer outperforms existing methods like Brant-2, BrainBERT, and others, a more detailed analysis of how these models compare in terms of generalization across different subjects and datasets would strengthen the argument for MVPA’s superiority. Specific examples of failure modes in the comparison would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)