EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping
Authors: Sam Gijsen
Kerstin Ritter
Venue: ICML 2025 poster (2025)
Keywords: multimodal
neuroscience
eeg
medical
pretraining
representation learning
Primary area: applications->neuroscience_cognitive_science
TL;DR: First application of multimodal language pretraining to the domain of medical EEG to improve clinical phenotyping.

Abstract: Multimodal language modeling has enabled breakthroughs for representation learning, yet remains unexplored in the realm of functional brain data for clinical phenotyping. This paper pioneers EEG-language models (ELMs) trained on clinical reports and 15000 EEGs. We propose to combine multimodal alignment in this novel domain with timeseries cropping and text segmentation, enabling an extension based on multiple instance learning to alleviate misalignment between irrelevant EEG or text segments. Our multimodal models significantly improve over EEG-only models across four clinical evaluations and for the first time enable zero-shot classification as well as retrieval of both neural signals and reports. In sum, these results highlight the potential of ELMs, representing significant progress for clinical applications.

EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping
Sam Gijsen 2 Kerstin Ritter 12
Abstract
2020; Yang et al., 2021), the temporal ordering of EEG data (Banville et al., 2021), subject identities of EEG crops (Gi- jsen & Ritter, 2025), as well as masking and reconstruction (Jiang et al., 2024). However, these are hindered by the difficulty of creating appropriate data augmentations and, especially reconstruction techniques, by low ägmallto-noisee Thus, progress in the medical context has lagged, likely fur- ther exacerbated by the modality displaying high similarity between pathologies.
Multimodal language modeling has enabled breakthroughs for representation learning, yet re- mains unexplored in the realm of functional brain data for clinical phenotyping. This paper pioneers EEG-language models (ELMs) trained on clinical reports and 15000 EEGs. We propose to com- bine multimodal alignment in this novel domain with timeseries cropping and text segmentation, enabling an extension based on multiple instance learning to alleviate misalignment between irrele- vant EEG or text segments. Our multimodal mod- els significantly improve over EEG-only models across four clinical evaluations and for the first time enable zero-shot classification well re- trieval of both neural signals and reports. In sum, these results highlight the potential of ELMs, rep- resenting significant progress for clinical applica- tions.
Meanwhile, important further progress was made in com- puter vision by leveraging natural language as a signal dur- ing pretraining (Radford et al., 2021). Specifically, con- trastive approaches which aim align embeddings of image- text pairs have shown to yield representations powerful for downstream tasks in radiology (Zhang et al., 2022a; 2023), Given that success in radiology is also believed to be bottle- necked by the availability of labeled data and the reliance on fine tne-grained information (Zhang et al., 2022a), this joint modeling approach is a particularly interesting and novel application for the challenging problem of medical EEG, Fortunately, this is made possible by the clinical reports of physicians which accompany hospital EEG recordings and contain information about the patient and recording itself (Obeid& & Picone, 2016).
Medical neuroimaging, including electroencephalography (EEG), has lagged behind other fields in leveraging the significant advancements of deep learning. While EEG sees widespread clinical use for clinical phenotyping such as pathology detection, in particular for epilepsy (Binnie & Stefan, 1999; Jing et al., 2020) as well as sleep disor- ders (Malhotra & Avidan, 2013), available annotated data is scarce. As the impressive scaling properties of deep learn- ing are now well described (Kaplan et al., 2020; Smith et al., 2023), sulf-super learning (SSL) is a promising direc- tion by enabling pretraining with unlabeled data and thereby increasing available training sample sizes (Hadsell et al.,, 2006; Chen et al., 2020). Various such approaches have shown initial success when applied to EEG. These include methods relying on data-augmeettations (Mohsenvand et al.,
However, language-EEa pretraining also entails unique challenges. First, datasets are generally smaller than those used in radiology and especially computer vision. Second, the clinical reports tend to be highly heterogeneous. While previous applications have paired natural and medical im- ages with short captions (Radford et al., 2021; Zhang et al., 2022a), EEG reports tend to span multiple paragraphs and include information irrelevant to downstream clinical tasks, potentially hindering the pretraining process. Moreover, they do not contain temporal information about when events occurred during the recording.
The current work presents the application of aligning func- tional brain data with medical textual information for the first time by training EFGG-language models (ELMs). To overcome the challenging formats of modalities, constitut- ing long timeseries and multiparagraph reports, we propose sub-unit alignment. To address inconsistent relevance of EEG-text pairs, additionally propose extension draw-
'Charité - Universitätsmmeizznn Berlin, Department of Psychia- try and Psychotherapy. Berlin, Germany 'Hertie Institute for Al in Brain Hcalth, University of Tübingen, Germany, Correspondence to: Sam Gijsen <sam. saam.gjjen@e @ charite.de>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

EEG-Language Pretraining for Clinical Phenotyping
Positive (e.J) MIL Sampling
Sub-unit Alignment Methods
Downstream Tasks
Painwise
Multiple Instances
Retrieval
ey
A/e MILL
eli MIL.
EEG Cropping
CLINICAL REPORT
e, MIL
Normal
CLENICAL BISTORY
ln
EO
E
2
C3
EM
g 8k]
Abnormal
Abnormality Classification
Loss:
Loss:
MEDICATIONS
is
2. Align+Orth MIL-IIffNCCE
1. InfoNCE
DN
CLINICAL CORR
Event Classification
positives
negatives
Report segmentation
Figure 1. Overview of the methodology. (Left) The ELM-MIL approach allows flexible multimodal alignment by cropping EEG and segmenting medical reports. We sample multiple positives in cross-modal fashion, such that each EEG crop can he aligned with any number of segments from the paired report (Ile). Vice versa, text can he aligned selectively to crops across the EEG recording (e I), illustrated by the differently shaded arrows. (Middle) An overview investigated methods by visualizing cross- moosll similarity matrices. (Right) To evaluate models, we perform bidirectional retrieval analyses and perform multiple pathology detection tasks.
ing on insights from the field of multiple instance learning (MIL). Furthermore, we investigate how best handle the heterogeneity of medical EEG reports. Specifically, we filter reports and perform content-based text segmentation, en- abling inference on the relative importance of the different information sources. Our approach allows us to provide the first evidence of considerable retrieval capabilities for clinical reports and EEG. We furthermore test downstream performance of ELMs on classifying pathological EEG, which is widespread clinical task, using four evaluations. These tests include zero-shot classification, leveraging lan- guage capabilities to show our approach's 's flexibility. Our results constitute considerable increases in pathology detec- tion performance, especially in scenarios with few labels. These are particularly relevant for clinical contexts, which tend operate with smaller datasets compared many common areas of deep learning applications.
the use of transformers (Yang et al., 2024; Jiang et al., 2024), with focus on scaling while adopting tokeniza- tion in attempt to improve the challenge of effective cross-dataset EEG training.
Using EEG for pathology detection. While SSL shows good performance for pathology detection, it is partic- ularly in contexts with little annotated data that per- forms well. When more labeled data is available, expert- based feature extraction combined with traditional ma- chine learning classifiers are competitive together with supervised deep learning (Roy et al., 2019; Gemein et al., 2020; Western et al., 2021; Kiessner et al., 2023; Darvishi- Bayazi et al., 2024). This trend has also been observed in other EEG applications (Schirrmeister et al., 2017: Lotte et al., 2018). This may indicate that inter-rater variability in EEG classification may create performance ceiling (Engemann et al., 2018; Gemein et al., 2020), highlighting the importance of improving classification with limited labels.
1. Related work
Self-supervised learning with EEG data. SSL with EEG data has been predominantly applied to emotion recognition (Zhang et al., 2022b; Wang et al., 2023), mo- tor imagery (Cheng et al., 2020; Rommel et al., 2022), sleep staging (Yang et al., 2021; Rommel et al., 2022), as well as pathology detection. For the latter applica- tion, the temporal order of EEG crops was used ini- tially to demonstrate label-efficient representation learn- ing (Banville al., 2021). Augmentation- trastive learning, combined with larger EEG encoders trained on multiple datasets, further improved pathology detection (Mohsenvand et al., 2020). Still, using subject identities as proxy labels during pretraining was found to meth- ods (Gijsen& & Ritter, 2025). Recent studies have explored
Medical multimodal language modeling. Medical vision-language modeling aims guide self-supervised pretraining on medical images using textual information in reports, with performance on a variety of downstream tasks benefiting as a result (Huang et al., 2021: Wang et al., 2022; Zhang et al., 2022a). Due to less available data in the medical domain, using a pretrained, frozen language encoder was found to boost downstream perfor- mance while considerably reducing computational cost (Liu et al., 2023a). Nevertheless, this line of work has focused mainly on the ECG, X-ray, CT images, and struc- tural MRI images (Chen et al., 2023; Lalam et al., 2023; Liu et al., 2023b). Recent advances outside the medical domain include multi-tast strategies, both during pretrain-
con-
improve disease detection over augmentation-ban
2

---

EEG-Language Pretraining for Clinical Phenotyping
ing by integrating contrastive learning and self-supervised losses (Tang et al., 2025; Tschannen et al., 2025), as well finetuning on multiple downstream tasks (Dai et al., 2023; Liu et al., 2024). Further exploration involves mov- ing compute from unimodal encoding to multimodal fu- sion (Kim et al., 2021).
(Xe, Xi) is projected into the vectors E Rd and R respectively. For every Xe: text of the associated report is sampled according to X z (x:), where represents the language sampling function detailed below. First, both the EEG crop X and text Y are encoded into vectors h. and hy For ELM,,,, we use projectors ge and 9i to yield vectors e and I, whereas for ELM, the text embeddings are not projected:
Multiple instance learning. MIL. has seen only limited exploration for EEG, Initial studies have investigated the framework by casting crops of EEG as instances and train- ing classifiers for emotion recognition (Caicedo-Acosse et al., 2019), motor imagery (Collazos-Huertal et al., 2020), mental disorders (Sadatnejad et al., 2019), and sleep apnea (Sadatnejad et al., 2019). Of these, only the
e = 9c (x.))
(1)
9i (fi (Ri))
EL..,. if ELM,
(2)
S (X1)
latter has relied on deep learning.
To enable multimodal pretraining, the projectors ge and gi map e and I to a shared latent space with identical dimen- sionality d. For ELMM, this is achieved by having Pe project to the native dimensionality of the text encoder ft.
2. Methods
2.1. Pretraining
1. BEG-LANGUAGE PRETRAINING
As paired medical EEG data and clinical reports scarce, training the text encoder function ft from scratch is unlikely to be successful. Furthermore, employing an existing lan- guage model and finetuning the model during multimodal pretraining can lead to training instability and collapse of the latent space (Jing et al., 2021; Liu et al., 2023a). To prevent resulting information loss, we follow the recommendations by Liu et al. (2023a) to use pretrained language model for fi and freeze its weights during training. For ELM, we adopt their proposed composite loss to learn fe and 9e:
Here we detail the setup for pretraining ELMs. Whereas vision-language models are typically trained by aligning a 2D image with a short caption (Radford et al., 2021; Zhang et al., 2022a), EEG- HEG-language modeling is confronted with long EEG time series and multi-pane paragripa medical reports. To overcome this, we employ text segmentation and time series cropping to create multiple ovneoaallap samples per modality and subject. Next, we propose sub-unit align- ment by pretraining on these cropped samples. In addition to considerably increasing sample size, this enables the ex- tension of successful approaches in vision-langee Asioonlanguage models. We initially describe two strategies for sub-unit alignment. First, EEG and text representations may be projected us- ing neural networks to new, shared latent space prior to alignment (as in CLIP; (Radford et al., 2021; Zhang al., 2022a)), denoted henceforth as ELM.... Alternatively, the EEG embeddings may be projected into the output space of the language model (as in M-FLAG by Liu et al. (2023a)), denoted as ELM; and trained using a bespoke loss function.
Lioual Latign + -oorth.
Latign = 11ē - -ill2 = 2 - 22
(4)
2
Lorth =
Σ 1 - [b_ hee)
(5)
j=1
+
Σ
(6)
jk
jz
where (j,k} E {1,..., dim (f.)) 12, he denotes a batch of fg-normalized EEG embeddings prior to projection and è denotes the normalized projected embeddings. Text embed- dings l are likewise normalized. Whereas Latign minimizes the difference between è and i, Corth promotes indepen- dence between latent dimensions of h2. More specifically, the latter is achieved by manipulating the empirical correla- tion matrix, where the diagonal and off-diagonal elements are pushed to and 0 respectively (Liu et al., 2023a).
This approach was found to reduce latent collapse in smaller data settings (Liu et al., 2023a). Following description of these models, we will introduce an extension based on MIL.
For EEG-language pretraining we assume the paired input X.,). Here Xz.i E R denotes one or a batch of crops of EEG signal with e channels and s time samples belonging to EEG recording i. Meanwhile, neural signals of recording as well as patient information described in X which represents a natural language text report. The main goal is to train the EEG encoder function Se. which projects crop of EEG signal into a vector of lower dimensionality. Following pretraining, this encoder function fe can be used for downstream applications such as pathology detection.
Mcanwhile, ELM,,1 relies on the cosine similarities between normalized EEG and text embeddings, 2521 = 0 T, and between text and EEG, !2e- = Jj with j = 1,,,,3 5.j
B for batch size B (Radford et al., 2021). The multimodal contrastive InfoNCE loss uses a temperature hyperparameter
Dropping the recording subscript for brevity, each pair
3

---

EEG-Language Pretraining for Clinical Phenotyping
T and is formulated as:
emphasize that "EEG-only" refers to pretraining without text, while ELMS use text solely during pretraining guide EEG representation learning. At test time, neither method uses clinical reports, ensuring alignment with standard EEG- based clinical practice. We provide further information on the following methods in Appendix B.5: Bootstrap- (8) Your-Own-Latent YBY: (Grill et al., 2020)), Variance- Invariance-Covarianee Regularization (VICReg: (Bardes et al., 2021)), Contrast with the World Representation (Con- traWR; (Yang et al., 2021)), Relative Positioning (RP; (Banville et al., 2021)), Temporal Shuffling (TS; (Banville et al., 2021)), Contrastive Predictive Coding (CPC; (Banville et al., 2021)).
exp
0021 (.kk
TT
CE2 j.k
=
log
2m=1 exp
(sf.m//)
12e jj
exp
log
2 exp J.ml
1
B
B
Lalign
C jik
+
2B
j=1
Multiple instance learning. While previous approaches aim to align text and EEG crops uniformly, certain text segments likely deseribe specific EEG sections more accu- rately than others. Therefore, we introduce MIL alignment strategy that builds on ELM,.1 and accommodates multiple positive samples, allowing for more nuanced multimodal relationships. Whereas MIL. approaches often rely on op- crations such as max-pooling to focus on single positive samples, we rely on insights from the video-text alignment approach (MIL-NCE) by Miech al. (2020). For a given text sample x,, we sample multiple positive EEG crops Xe from the paired recording to approximate the P(el() dis- tribution, while for an EEG crop, multiple text segments are sampled to model the P(]le) distribution. We combine these and sample positives for each EEG and text report respectively to approximate P(e,I) via bidirectional align- ment. This approach effectively relaxes the assumption of strong alignment for each individual (x) pair, instead assuming that, on average, positive samples should have higher similarity scores than negative samples. To this end, we extend the InfoNCE loss to multiple instances:
3. Experimental Setup
3.1. Datasets and evaluation tasks
TUEG. The Temple University Hospital (TUH) EEG Cor- pus is the largest available corpus of hospital EEG data with varying montages, channel counts, and sampling fre- quencies (n=26846 (Obeid & Picone, 2016)). For most of the dataset, no labels are available beyond patient age and sex. However, many EEG sessions are associated with a natural-language clinical report.
TUAB. The TUH Abnormal EEG corpus is subset of TUEG which was manually labeled by clinicians indicat- ing whether the EEG displays pathological abnormalities (Lopez et al., 2015). This enables the binary classification task of predicting the status of {normal, abnormal] on a recording-leved Following the literature, we use the pro- vided evaluation set the hold-out test set.
NMT. We leverage the NMT Scalp EEG Dataset (Khan et al., 2022) in order to validate our results for recording- level abnormality classification out-of-distribution. The NMT dataset deviates considerably from TUEG. Data was recorded from a South Asian population in Pakistan, using (11) a different EEG recording setup. Furthermore, the NMT
Bi log
Ejen, exp( B. L exp( (0p((12//T)
Cell_ =
P.
(10)
Bi
B.
1
Σ EjEQu (xp())
Clee_ 1
[2
log
Be k=1
Bi Σ exp(sff//T)
(12) participants are considerably younger, feature more males (66.6%), and their EEG recordings are labeled predomi- nantly normal (83.8% in the training set, while the test set is balanced). We use the provided train/test split.
with P& S N and |Qxl S M,
C² = -=1== call
1 ( + clle)
(13)
and where Ps and Qs are sets of positive EEG and text segments respectively. Be and Bi are the batch sizes for EEG and text respectively, for which we sample up to N EEG and M text segments for B subjects. We normalize N
To further evaluate learned representation, we use tasks requiring classification of single, short 5-second EEG crops.
TUSZ. The TUH Seizure Detection Corpus (Shah et al., 2018) is subset of TUEG which has sections labeled to contain either seizure or background activity. We perform binary classification using 5-fold cross validation on the provided train and dev sets (n=6491), while testing on the eval set.
using IQ&l or [Pxl to account for the varying number of crops across subjects.
2.1.2. EEG-ONLY SELF-SUPERVISER LEARNING
We compare the representations learned by EaG-lan pretraining to those obtained via EEG-only pretraining us- ing the same pretraining dataset and EEG encoder. We
TUEV. An Events Corpus (Obeid & Picone, 2016) con- taining annotated EEG with six classes, of which three are
4

---

EEG-Language Pretraining for Clinical Phenotyping
clinical (spike and slow wave, generalized periodic epilepti- form discharge, periodic lateralized epileptiform discharge) as well as eye movements, artifacts, and background activity. We only use the provided train set (5-fold CV) due to the test set not including the TUEG subject identifiers, which would have prevented the exclusion of these subjects from the pretraining data. For each 1-second event, we include two seconds of context before and after (Jiang et al., 2024).
3.2.2. EEG PROCESSING
EEG data received minimal preprocessing, with our ap- proach detailed in Appendix C.I. We describe the selection of our pretraining dataset in Appendix C.2, which avoids data leakage by excluding any data of subjects present in any of the evaluation data. The resulting sample sizes are shown in Table 1. To enable fair comparisons between meth- ods, the optimal crop-length for recording-level pathology detection was determined out of (5.10,20,30.60) seconds without data-leakage (Appendix B.2). yielding 20 and 60 second crops for EEG-only and EEG-language modeling respectively. For TUSZ and TUEV evaluations, we addition- ally pretrain models using 5-second crops and drop subjects which feature in either TUEV or TUSZ-test.
3.2. Preprocessing
3.2.1. TEXT PROCESSING
In order to categorize the textual content in the clinical reports, we employed regular expressions matching for commonly-occurrin headings (an overview is provided in Appendix E.4). These enabled the segmentation of indi- vidual reports into their respective headings with associated text paragraphs, providing insight into which information in physician reports is encoded in the EEG. We cluster head- ings into four categories, while filtering out headings which are irrelevant such as information on the EEG system, techni- cal issues, or general disclaimers. First, the clinical history cluster of headings contains demographic information in terms of patient age and sex, as well as a brief description of relevant current and/or past pathology. The record de- scription cluster includes the physician's observations of the EEG traces, which describes both normal and abnormal features, often in terms of oscillatory brain activity. The medication cluster contains the patient's 's current medication information. Finally, the interpretation cluster summarizes physician's 's thoughts, often including the impression of whether the EEG is normal or pathologically abnormal, as well as a clinical correlation. To investigate whether EEG- language models can learn richer representations by being exposed to a larger variety of text, we also train models by sampling text from these four aforementioned clusters.
3.3. Pretraining setup
We set the temperature parameter to 0.3 using evaluation on a holdout set (Appendix B.4). For ELM-MII, the number of sampled EEG and text crops per subject depend on the targeted downstream context. When using 60 second crops for subject-level prediction we set the number of sampled EEG crops N 32 and text crops M = 8 paragraphs as this covers the available samples for a majority of subjects. For tasks using single 5-second crops, we increase N to 120 and sample single sentences instead of paragraphs with M = 24 to accommodate the more finegrained nature of these tasks.
Language encoder. For we use a transformer model which was pretrained with medical data in contrastive manner on PubMed search logs (MedCPT; (Jin et al., 2023)). See Appendix B.3 for a comparison of language models. ELM; adopts the language model's 's native hidden dimen- sionality (768), while for ELM,, and ELM-MIL we project to a dimensionality of 256.
EEG encoder. For the EEG encoder f. use a randomly initialized residual convolutional neural network, with an identical backbone architecture across all comparisons. We use nonlinear MLPs with a single-hidden layer for ge and gt, as well as for the projector head in EEG-only self-supervised learning. More details are provided in Appendix B.2.
Due to the heterogeneity of the clinical reports, further test the utility of summarizing the pathological status indi- cated by the clinical report using a large language model (LLM). Due to the sensitive nature of the clinical reports, we use the Llama-3 8B model (Meta, 2024) locally and in- struet for the production of single-sentence summary of report, which should include whether the EEG was deemed abnormal and for which reasons (Appendix E.4).
4. Experimental Results
Language encoding, Given a sampled section from a clini- cal report or the LLM-generated summary, we encode this text by relying on the embedding of the [cls] token which aggregates the representations across all tokens. As such, given clinical report x,, the transformation function 2 corresponds to text segmentation or summarization yielding &; Following tokenization, we embed into the [cls] token using fi. The resulting text embedding h, may be used for multimodal pretraining.
4.1. Pretraining comparisons
Given the novelty of the studied domain, we extensively in- vestigate the information represented in the learned embed- dings resulting from EEG-language training as a function of both text selection and alignment strategy. To this end, we perform retrieval analyses as well as pathology detection using linear probes and zero-shot classification (Figure 2).
Retrieval analyses. Given a medical report describing
5

---

EEG-Language Pretraining for Clinical Phenotyping
ELM
ELM
EEG Retrieval Top-10
B
Report Retrieval Top-10
30
30
25
25
Accuracy (9)
20
20
Accuracy
15
15
10
10
5-
Linear Probing (1%)
D
Zero-Shot
84
Accuracy (8)
82-
80
Accuracy
78
60
50-
74
40
Cinciaal History
Medication
Deserption
Interreration
LLM Summary
clnnters
Cileball Hastory
edtin
Descciption
Intepontattonn
LLM Summary
Clastera
Chaters ey
Cuattrr fe
Clnten e./
Cleccal Histrr
Medication
Descriptoon
Interpreiation
LL Summnry
Clater
Cioiia History
Medication
Isteprotattin
Sunurary
Claxter
Clatert ofl
Clustrr Re Cisstrc
Desei
Figure 2. A,B) A set of EEG-Language models are evaluated on their retrieval ability using top-k accuracy out of 437 patients. C) Linear probing performance TUAB with 1% of labels and D) zero-shot classification. Error bars indicate standard deviations over five model training runs.
pretation, LLM summary) are most effective, highlighting pathology key source of variation. Combining multi- ple text clusters further enhances results, suggesting unique information capture and the model's ability to integrate di- verse patient data.
Table Dataset sample sizes. Crop lengths indicated for ELM/EEG-only models.
Data subset
EEG files
Reports
Crop Length
TUEG Pretrain
15144
11785
Variable
TUAB train
2712
Not used
60/20s
ELM,,1 models tend outperform ELM, models, particu- larly for report retrieval. This likely due to omission of a text projection head in ELM, which may therefore lack the flexibility to appropriately separate the EEG reports in latent space. Due to the benefit of pretraining when sampling from the four text clusters, we pretrain our ELM-MIL models in this manner only. We observe that our MIL extension further improves retrieval performance, benefiting from sampling multiple positives jointly (e, () and performing bidirectional aligmment. These results indicate for the additional flexibil- ity of a MIL-based approach to aid in multimodal alignment, supporting the hypothesis that not all EEG and text pairs are equally informative.
TUAB test
276
Not used
60/20s
Retrieval test
437
437
60/-
NMT Train
2216
60/20s
NMT Test
60/20s
TUSZ Train+Dev
6491
5/5s
TUSZ Test
865
5/5s
TUEV Train
359
5/5s
the patient and their EEG recording, we probe the abil- ity recover the patient's EEG by rank-ordering candi- date EEG based embedding similarity, as well as vice versa. We average embeddings of EEG and text segments within- within-modality, yielding one EEG and report embedding per recording, which we use for rank-ordering based on cosine similarity.
Clinical phenotyping, We study the learned representations in their relevance to clinical pathology by training linear probes TUAB (Figure 2C). Here we observe similar pat- terns for recording- classification, with bidirectional ELM-MIL scoring best. Interestingly, good classification was possible with many of the models, causing us to investi- gate whether the strategy of sub-unit multimodal modeling provides inherent benefits. We provide this additional set of analyses Appendix A.3, which indicates that our sub- unit alignment strategy promotes the encoding of between- subject information even in the absence of semantically
Top-K retrieval accuracy, assessing if the correct EEG or re- port ranks within the top K (Figure 2A.B), shows that many models perform well above chance. This indicates success- ful generalization of multimodal EEG-language learning. Text sampling significantly impacts report retrieval; clus- ters lacking direct EEG descriptions (clinical history, med- ication) score lowest. Including descriptive information improves retrieval, but pathology-relevant contexts (inter-
6

---

EEG-Language Pretraining for Clinical Phenotyping
Table Pathology detection via zero-shot (ZS) and linear probing 1%, 10%, and 100% labeled data of the TUAB training set. The (second) best scores are printed (underlined) bold. Standard deviations over five model training runs are included. Supervised serves as reference and refers to training end-to-end directly on labels.
Balanced Accuracy
AUROC
Method
ZS
1%
10%
100%
7S
1%
10%
100%
Supervised
71.36=1.10
81.0620.00
84.131029
1.38
89.23-0.51
91.83
BYOL
22.69
19.0
19.94+2214
7
-6.75
88.82+
VICReg
71.76=0.81
19.6=
82, 22.46-u956
78.7+11.11
86.04_0.00
88.78=104
ContraWR
73.30
80.72.1169
82.44
80.30
86.67=1.32
88,4411.20
RP
1..5
82.16
83. 83.36:10.42
82.63
89.78-0.43
91.43::0.4
TS
2.99
82, 0.6
84, 10:00.6
82.51 1:091
89,58_0.55
91.50
CPC
73.20=0099
19.95+149 78
81.48=1.0z
86.44=1.07
87.92
ELM-MIL l/e
68.864729
82.53 1.80
86.38 0..7
87.62 1..43
75.23
89.88 1.47
92.92=0.54
93.52
ELM-MIL e|t
79. 19210
83.7 1
84.37 +:097
85.65 1:0977
87.26
92.37
93.25 0.227
93,65
ELM-MII e,
84.31
83.10
84.21
87.11
91.56
91.54-1044
93.14-1024
93.91
by designing appropriate prompts. Having identified the requirements for stable alignment, we focus the ELM- MIL../ model for EEG-only baseline comparisons due to its consistent performance across tasks and seeds.
Table Linear probing for abnormality classification the NMT dataset using 1%, 10%, and 100% labeled training data.
AUROC
Method
1%
10%
100%
4.2. Baseline comparisons
BYOL
63.78
76.48 1:210
80.65
ContraWR
65.72
72.47:1:9
15.42+1.01
4.2.1. ABNORMALITY CLASSIFICATION
VICReg
61.57
14.19
78.50:1160
TS
64.90=2070
81.36-1..5
87.08=1.2
We compare our ELM-MIL models on the TUAB dataset to six EEG-only methods (Table 2). We pretrain using these methods on the same pretraining dataset as ELMs and, to- gether with the supervised baseline, also an identical EEG encoder architecture. This enabled accurate inference on the effectiveness of the pretraining strategy per se. We find ELMs yield large improvements for pathology detec- tion over EEG-only pretraining, with multimodal models being particularly effective at small sample sizes: at 1% of exposed labels, performance increases reach 8.7% bal- anced accuracy and 9.7% AUROC. To validate our results we evaluate learned representations on the NMT dataset without finetuning (Table 3). We find ELM-MILe, (henceforth 'ELM-MIL') to still perform well, especially with few labels, where they yield an improvement of at least 3.8% AUROC.
RP
64.92.0.8
80.4221.8
86.5012277
CPC
65.24
77.84+1.12
+..98+1660
ELM-MIL
69.49
81.4221.15
89.77.101
relevant text. This allows ELM,,1 to nearly match the best EEG-only pretraining strategy for pathology detection when reports are randomly shuffled.
Next, we investigate the unique ability of multimodal lan- guage modeling to leverage the language modality to per- form 'zero-shot' classification (Figure 2D). Without any explicit labels for downstream training, EEG may be clas- sified by computing its similarity in latent space to text prompts representing the candidate classes. As suggested by Radford et al. (2021), 21 variations of the phrasing "The EEG is normal, abnor- mal" (Appendix D). Despite a small dataset, EEG-language models can reach high levels of zero-shot pathology detec- tion if: 1) they include a text projector, 2) include at least the clinician interpretation, and 3) do bidirectional align- ment. These models, especially ELM-MIL,,.. score highly consistent across different weight initializations, suggest- ing sound alignment between modalities. Pretraining using exclusively one of the other text clusters yielded poor perfor- mance, which follows from these models not being exposed to the explicit phrasing indicating the EEG status as normal or abnormal per se. Their capability likely can be improved
create prompt ensemble over
Next, we further contextualize our results in the literature. By probing ELM-MII. representations for 10 second crop- level abnormality classification on TUAB with 80% labels able to compare to previously reported evaluations (Yang et al., 2024; Jiang et al., 2024; Dimofte et al., 2025). However, we note that methods use significantly different pretraining data and architectures, which complicates in- terpretation. Nevertheless, ELM-MIL scores highest and improves over L.BrraM-Huge +0.83% on average across scores (Table 4). This is despite LaBraM-Huge requiring finetuning, being pretrained on a large number of datasets (including TUEG), and featuring 369M parameters com-

---

EEG-Language Pretraining for Clinical Phenotyping
Table Crop-level performance on TUAB (80% labels) of super- vised and self-supervised methods using different training datasets and EEG encoders. Model sizes refer to trainable parameters for the EEG encoders.
Retrieual
7ero-Shot
innea Probe (100%)
37
60
100 A0000
Salanced accuracy
Methods
Fine tuned
Model Size
B. Acc.
AUROC
0.03 01 03 0.
1.0
1.5
0:03 0.1
03
0.6
14
1.5
01 0.03
03
06 10
SPaRCNet
Y
0.79M
78.96+0.38
86. 26176+
ContraWR
Y
1.6M
77.46+001
84,564a744
CNN-Transformer
Y
3.2M
77.77+0.2
84.61 +0.1
Figure 3. Model comparisons across EEG retricval and pathology detection under different values of the temperature parameter т.
FFCL
Y
2,4M
78.48
85.69
ST-Transformer
Y
3.5M
19.66100.3
87.07
BIOT
Y
3.2M
79.59
88. 49
We perform additional ablations to investigate crucial as- pects of the ELM-MIL e, model. First, we find that addi- tional positive EEG and text samples improve downstream performance (Table 5). We additionally ablate the aggre- gation method for positive samples and find MIL-InfoNCE to outperform considered alternatives. We compare to aligning only the most similar positive sample (denoted Max+InfoNCE), using attention to create a weighted av- erage across positive samples based on similarity values (Attn+InfNCC; (Ilse et al., 2018)), as well as taking the sum instead of mean across log-probabilitte
CEReBrO
Y
3.58M
79.40
CEReBrO
Y
40.0M
81.29=a.15
88.67 +:006
CEReBrO
Y
85.2M
81.67
89.16+0038
LaBraM-Base
Y
5.8M
=0.40
90.22:1009
LaBraM-Large
Y
46M
82.26
91.27
LaBraM- Huge
Y
369M
82.58
91.62:1016
ELM-MIL
N
0.93M
84.42=0.11
91.4420.11
pared to 0.9M of our EEG encoder. To further compare the quality of learned representations, we apply our linear prob- ing pipeline to the LaBraM-Base model (Appendix 7-9). We find gains for our ELM of up to 5.7-13.4% depending on the dataset, highlighting the improved semantic content in ELM representations. This stark discrepancy is likely due to the clinical specificity of ELMs, enabling much smaller models that do not need to be finetuned. As a result, our methodology is also significantly cheaper and faster to train (50 epochs in under 12 hours with 24GB of memory).
(Sum+InfoNCE). The latter does not account for the varying amount of text and EEG crops across subjects.
4.2.4. ALIGNMENT VISUALIZATIONS
For model interpretability, we visualize temporal multi- modal alignment in example hold-out recordings (Figure 4), We compute cosine similarity between all EEG crop embed- dings in a recording and an embedded snippet from its paired clinical report, then plot crops with the highest and lowest similarity. We observe distinct periods of stronger/weaker alignment, where strong alignment corresponds to relevant clinical events. This indicates our ELMS achieve temporally selective alignment of clinical EEG events without explicit temporal event information or labels, consistently across in- dependent pretraining runs. Additional examples, including failure cases, are in Appendix E.3.
4.2.2. SEIZURE AND EVENT CLASSIFICATION
We additionally investigated classification performance based on single, five second EEG crops. Although recording- level classification was the focus of our work, we find ELM to score better than the investigated BEG-only methods also on both seizure detection (TUSZ) and 6-class event detec- tion (TUEV; Table 6). We provide per-class performance in Appendix A.2.
5. Discussion
4.2.3. FURTHER ABLATION ANALYSES
This paper presents first application of multimodal pretrain- ing combining natural language and functional brain data in a medical context. The proposed methodology achieves significantly improved clinical representations compared to EEG-only SSL, while being inexpensive to pretrain. This improvement stems from our novel sub-unit alignment ap- proach in combination with MIL-InfoNCE to address in- herent data misalignment challenges. Notably, these mul- timodal models enabled zero-shot pathology detection and label-efficient linear probing with improvements up to 9.7%. We additionally show generalization of ELMs via external validation and clinical event detection tasks. Our pathology-
Whereas for InfoNCE the temperature parameter sets the relative focus across negative samples (Wang& & Liu, 2021), ) for MIL-InfoNCE it does so too for positive samples. We therefore test the sensitivity of our methods to the parameter (Figure 3). We find that MIL-InfoNCE more robust to changes of 7 for pathology detection, while retrieval perfor- mance can be further improved by lowering т. This may be explained as retrieval being subject-based rather than blasd (see Appendix 13). Moreover, performance in- creases from T < indicates the utility of this additional hyperparameter of InfoNCE, which is absent in NCE.
8

---

EEG-Language Pretraining for Clinical Phenotyping
Table Ablation studies (Means five training runs). Ret: EEG Retrieval (Top- accuracy), LP:: Linear Probe (Balanced accuracy
100%), ZS: Zero-sho classification (F1).
(b) Positive EEG Samples
(a) Aggregation
N
Ret.
LP
ZS
(c)
Positive Text Samples
Method
Ret.
LP
ZS
Ret.
LP
ZS
2
19.8
85.9
78.8
Max+InfoNCE
3.9
77.5
43.2
21.8
85.9
79.2
2
28.1
85.8
80,4
AtI++InfNNCE
8.3
84.9
17.5
8
25.3
86.5
80.0
4
27.1
86,0
80.4
Sum+InfoNCE
24.7
86.0
78.8
16
26.5
86.9
78.5
8
27.1
87.1
82.1
MII.InfoNCEE
27.1
87.1
82.1
32
27.1
87.1
82.1
"seizures arising from the right hemisphere"
"poly spike and slow wave"
Sim(text EEG)
M
0.5
M
0.6
0.0-
EEG Crops
BEG Crops
70
Figure Left, top: cosine similarity between EEG embeddings S-second crops and embedding the paired text "seizures arising from the right hemisphere" shows distinct periods of stronger and weaker alignment. Shading indicates the standard deviation across 5 random weight initializations. Green and purple dots mark two crops with highest and lowest similarity respectively. Left, bottom: EEG traces of the crops indicated by the coloured dots. Electrodes are colored based on their position (left: red, right: blue) and clearly reveal right-latere seizure activity. Right: As side- plot, but for a different recording with the text "poly spike and slow wake".
Some considerations of this study deserve mention. The current limitation of publicly available paired EEG-report datasets presents a challenge for scaling pretraining data. Future work could address this through the generation of synthetic text captions based clinical metadata. Further- more, due computational constraints, analyses on scaling model sizes are yet to be performed. However, we find highly favourable performance of ELMs compared to signif- icantly larger EEG foundation models pretrained on a large number of datasets. We provide code and pretrained models at https: //ght::///hhttt com/ jisnissen /ELM.
Table 6. Linear probing for seizure and event classification on the TUSZ and TUEV datascts respectively.
AUROC
TUSZ
TUEV
Method
1%
10%
100%
100%
Supervised
84.00=0.0
87.33
90.38
86,28
BYOL
77.45
86.60+..79
88.89
82.40
ContraWR
68.86=3.13
82.75
85,68 4.744
84. 1:66
VICReg
69.61=18t
82.01 1:091
86.
83.26 98
TS
78. 78.60+1..0
87.63-10.58
89. -0.77=0077
84.8
84.86
RP
64.41 =2.50
76. 1..7
79.52.21.8
1.36
78. 1..9
CPC
72.01+100
81.77+224
85.91 4:2000
81.94
A Acknowledgements
ELM-MIL
78.9845.18
88.98
91.31
87.69
This research was funded by the Deutsche Forschungs- gemeinschaft (DFG) through FOR 5187 (project number 442075332). Additional support was provided by the DFG through the following projects: CRC 1404 (project number 414984028), TRR 265 (project number 402170461), and
sensitive multimodal alignment is a critical step toward automated report generation (e.g. Biswal et al. 2020), ensur- ing EEG-text representations capture clinical information for future documentation tasks.

---

EEG-Language Pretraining for Clinical Phenotyping
RU 5363 (project number 459422098).
Biswal, S., Xiao, C., Glass, L. M., Westover, B., and Sun, J. Clara: clinical report auto-completion. In Proceedings of The Web Conference 2020, pp. 541-550, 2020.
The data used in this study was provided by the Neu- ral Engineering Data Consortium at Temple University. For further details about this data, please access the fol- lowing URL: https: / /tspp.ispp.. wes.sppppcccoo conepress com/ projects hetpcctttsttteeegeeeeee htmmm///
Caicedo-Acosta, J., Cárdenas-Peñaa D., Colrazos-Huurtas, D., Padilla-Burities J. I., Castano-Duque G., and Castellanos- Domnaanges,D G. Multiple-instance lasso reg- ularization via embedded instance selection for emotion recognition. In International Work- ConfeeCneeeecce on the Interplay Between Natural and Artificial Computation. pp. 241. Springer, 2019.
The authors declare no competing interests.
Impact Statement
Performance across both recording-level classification and event detection tasks suggests that our model learns clin- ically relevant features at multiple temporal scales. With further progress, this capability may support various future clinical applications, from rapid screening of prolonged recordings to real-time event detection. Whereas the present study focuses on establishing an initial application to ex- plore viability, future work may benefit from focusing on im- proving the interpretability of these representations through techniques such as channel-specife attribution. The multi- modal nature of our approach, by aligning EEG with clinical reports in pathology-sensitive manner, not only enhances detection but also lays an important foundation for auto- mated report generation. Specifically, such generation may greatly benefit from an aligned latent space which contains clinical information. This could facilitate clinical documen- tation by translating EEG signals into structured summaries. These can constitute highly valuable future efforts given the lime-intensive nature of manual reporting. Certain clinical limitations also deserve further attention, such careful study of how the frequency of specific pathology and clini- cal events in reports impacts model performance. Finally, biases present in language models may impact multimodal pretraining, which should be investigated future work.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Chen, Y., Liu. C.,, Huang, W., Cheng, S., Arcucci, R., and Xiong, Z. Generative text-guided 3d vision-language
pretraining for unified medical image segmentation. arXiv preprint arXiv:2306.048) 2023.
Cheng. J. Y., Goh, H., Dogrusoz, K., Tuzel, O., and Azemi, E. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.04871, 2020.
Collazos- Huertos, D., Caicedo-Acosta. J., Castaño-Duque, G. A., and Acosta-Medina, C. D. Enhanced multiple instance representation using time freeueeyyy atoms in motor imagery classification. Frontiers in neuroscience, 14:155, 2020.
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. InstructBLIP: Towards general-purpose vision-language models with instruc- tion tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: / //openreview net 2
References
Alsentzer, E., Murphy, R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M. Publicly available clin- ical bert embeddings. arXiv preprint 2019.
Darvishi-Bayazi, M.-J., Ghaemi, M. S., Lesort, T., Arefin, M. R., Faubert, J., and Rish, I. Amplifying pathological detection in eeg signaling pathways through cross-dataset transfer learning. Computers in Biology and Medicine, 169:107893, 2024.
Banville, H., Chehab, O., Hyvärinen, A., Engemann, D.-A., and Gramfort, A. Uncovering the structure of clinical eeg signals with self-supervised learning. Journal of Neural Engineering, 18(4):046020, 2021.
Dimofte, A., Bucagu, G. A., Ingolfsson, T. M., Wang. X., Cossettini, A., Benini, L., and Li, Y. Cerebro: Compact encoder for representations of brain oscilla- tions using efficient alternating attention. arXiv preprint arXiv: 10885, 2025.
Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance- regularization for self-supervised learning. arXiv preprint arXiv:2/05.0490 /05.04906, 2021.
Engemann, D. A., Raimondo, F., King, J.-R., Rohaut, B., Louppe, G., Faugeras, F., Annen, J., Cassol, H., Gosseries, O., Fernandez-Slezak, D., et al. Robust eeg- based cross-site and cross cross-protocol classification of states of consciousness, Brain,
Binnie, C. D. and Stefan, H. Modern electroencephalogra- phy: its role in epilepsy management. Clinical Neuro- physiology, 110(10):1671-1 16711)):11111999 1999.
2018
10

---

EEG-Language Pretraining for Clinical Phenotyping
Gemein, L. A., Schirrmeister, R. T., Chrabaszez, P., Wil- son, D., Boedecker, J., Schulze-Bonhage, A., Hutter, F., and Ball, T. Machine-learinng-oon diagnostics of eeg pathology. Neurolmage. 220:1 17021, 2020.
Jin, Q. Kim, W., Chen, Q. Comeau, D. C., Yeganova, L., Wilbur, W. and Lu, Z. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformat- ics, (9 1):tta6651, 2023.
Gijsen, S. and Ritter, K. Self-supervised learning for en- coding netween-subject information in clinical eeg. In Learning Meaningful Representations of Life (LMRL) Workshop at ICLR 2025, 2025. URL https:// openreview. wet.i//wweeeeeeee...e.mm..ee.e.e forum? dindmiddtggggsSSS
Jing, J., Herlopian, A., Karakis, I., Ng. M., Halford, J. J., Lam, A., Maus, D.. Chan, F., Dolatshahi, M.. Muniz, C. F., et al. Interrater reliability of experts in identify- ing interictal epileptiform discharges in electroencephalo- grams. JAMA neurology, 77(1):49-555 57, 2020.
Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., Goj, R., Jas, M., Brooks, T., Parkkonen, L., and Hämäläinen, M. S. MEG and EEG data analysis with MNE-Python. Frontiers in Neu- roscience, 7(267):1-13, 2013. doi: 10.3389/fnins.2013. 00267.
Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understand- ing dimensional collapse in contrastive self-supervised learning, arXiv preprint ar arXiv:2110.09348 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. 2020.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo. Z., Gheshlaghi Azar, M., al. Bootstrap your latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
arXiv preprint arXiv:2001.0888
Khan, H. A., UI Ain, R., Kamboh, A. M., Butt, H. T., Shafait, S., Alamgir, W., Stricker, D., and Shafait, F. The nmt scalp eeg dataset: open-source annotated dataset of healthy and pathological eeg recordings for predictive modeling. Frontiers in neuroscience, 15:755817, 2022.
Gu. Y., Timn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. Domain-specific language model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3-1), 2021.
Kiessner, A.-K., Schirmeisterr, R. T., Gemein, L. A., Boedecker, J., and Ball, T. An extended clinical eeg dataset with 15,300 automatically labelled recordings for pathology decoding. Neurolmage: Clinical, 39:103482, 2023.
Gutmann, M. and Hyvärinen, A. Noise-contrastive estima- tion: new estimation principle for unnormalized statisti- cal models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp- 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kim, W., Son, B., and Kim. I. Vilt: Vision-and-langee transformer without convolution or region supervision. In International conference on machine learning, pp. 5583- 5594. PMLR, 2021.
Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning invariant mapping. In 2006 IEEE computer society conference computer vision and pattern recognition (CVPR 06), volume 2, pp. 1735- 1742. IEEE, 2006.
Lalam, S. K., Kunderu, H. K., Ghosh, S., Kumar, H., Awasthi, S., Prasad, A., Lopez-Jimenez, F., Attia, Z. I., Asirvatham, S., Friedman, P., et al. Ecg representation learning with multi-modal ehr data. Transactions on Ma- chine Learning Research, 2023.
Huang. S.-C., Shen, L., Lungren, M. P., and Yeung. S. Glo- ria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 3942-3951, 2021.
Liu, C., Cheng, S., Chen, C., Qiao, M., Zhang, W., Shah, A., Bai, W., and Arcucci, R. M-flag: Medical vision- language pre-training with frozen language models and latent space geometry optimization. In International Con- ference on Medical Image Computing and Computer- Assisted Intervention, pp. 637-647 Springer, 2023a.
Ilse, M., Tomezak, J., and Welling, M. Attention- deep multiple instance learning. In International conference machine learning, pp. 2127-2136. PMLR, 2018.
Liu, C., Cheng, S., Shi, M., Shah, A., Bai, W., and Areucci, R. Imitate: Clinical prior guided hierarchical vision- language pre-training. arXiv preprint arXiv:2310.07355, 2023b.
Jiang, W.-B., Zhao, L.-M., and Lu12, B.-L. Large brain model for learning generic reperettntations with tremen- dous eeg data in bei. ICLR, 2024.

---

EEG-Language Pretraining for Clinical Phenotyping
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024.
Artificial Intelligence in Medicine, AIME 2019, Poznan, Poland, June 26-29, 2019, Proceedings 17, pp. 47-56. Springer, 2019.
Sadatnejad, K., Rahmati, M., Rostami, R., Kazemi, R., Ghidary, S. S., Müller, A., and Alimardani, F. Beg repre- sentation using multi-instance framework on the manifold of symmetric positive definite matrices. Journal of Neural Engineering, 16(3):036016, 2019.
Lopez, S., Suarez, G., Jungreis, D., Obeid, I., and Picone, J. Automated identification of abnormal adult eegs. In
2015 IEEE signal processing in medicine and biology symposium (SPMB), pp. 1-5. IEEE, 2015.
Lotte, F., Bougrain, L.., Cichocki, A., Clere, M., Congedo, M., Rakotomamonjy, A., and Yger, F. A review of classi- fication algorithms for eeg-based brain- computer inter- faces: a 10 year update. Journal of neural engineering, 15(3):031005 1005, 2018.
Schirrmeister, R. T., Springenberg. J. T., Fiederer, L.. D. J., Glasstetter, M., Eggensperger. K., Tangermann, M., Hut- ter, F., Burgard, W., and Ball, T. Deep learning with convolutional neural networks for eeg decoding and vi- sualization. Human brain mapping, 2017.
Malhotra, R. K. and Avidan, A. Sleep stages and scoring technique. Atlas of sleep medicine, pp. 77-99, 2013.
Shah, V., Von Weltin, E., Lopez, S., McHugh, J. R., Veloso, L.. Golmohammadi, M., Obeid, I,, and Picone, J. The temple university hospital seizure detection corpus. Fron- tiers in neuroinformatics, 12:83, 2018.
Meta. Llama 3. https://githie com/ meta-llaa ama / 1llama3/blll alaaaa//llbmmaai/////// CA, 2024.
Miech, A., Alayrac, J.-B., Smaira, I.,, Laptev, I., Sivic, J., and Zisserman, A. End-to-end learning of visual repre- sentations from uncurated instructional videos. Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9879-98878 2020.
Smith, S. L.., Brock, A., Berrada, I.., and De, S. Con- vnets match vision transformers at scale. arXiv preprint arXiv:2310. 16764, 2023.
Tang, Z., Lian, L., Eisape, S., Wang. X., Herzig, R., Yala, A., Suhr, A., Darrell, T., and Chan, D. M. Tulip: Towards unified language-image pretraining, 2025. Preprint.
Mohsenvand, M. N., Izadi, M. R., and Maes, P. Contrastive representation learning for electroencephalogramm classi- fication. In Machine Learning for Health, pp. 238-253, PMLR, 2020.
Tschannen, M., Gritsenko, A., Wang. X., Naeem, M. F.,
Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., al. Siglip 2: Multilingual vision-language encoders with improved semantic under- standing, localization, and dense features. arXiv preprint arXiv:2502. 14786, 2025.
Obeid, I. and Picone, J. The temple university hospital eeg data corpus. Frontiers in neuroscience, 10: 10:195498, 2016.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V.,
Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-282 2011.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Veloso, L., McHugh, J., von Weltin, E., Lopez, S., Obeid, I., and Picone, J. Big data resources for eegs: Enabling deep learning research. In 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1-3. IEEE, 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Wang. F. and Liu, H. Understanding the behaviour of con- trastive loss. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 2495- 2504, 2021.
Rommel, C., Paillard, J., Moreau, T., and Gramfort, A. Data augmentation for learning predictive models on eeg: a systematic comparison. Journal of Neural Engineering, 196):066020, 2022.
Wang. F., Zhou, Y., Wang, S., Vardhanabhuti, V., and Yu, L. Multi-granularity cross-modal alignment for generalized medical visual representation learning. Advances in Neu- ral Information Processing Systems, 35:33536-3356 2022.
Roy, S., Kiral-Kornkk, I., and Harrer, S. Chrononet: A deep recurrent neural network for abuormal eeg identification. In Artificial Intelligence in Medicine: 17th Conference on
12

---

EEG-Language Pretraining for Clinical Phenotyping
Wang, X., Ma, Y., Cammon, J., Fang, F., Gao, Y., and Zhang, Y. Self-supervised eeg emotion recognition mod- els based on cnn. IEEE Transactions Neural Systems and Rehabilitation Engineering. 31:1952-1962. 1962, 2023,
Wang, Y., Yu, Z., Wang, J., Heng, Q. Chen, H., Ye, W., Xie, R., Xie, X., and Zhang, S. Exploring vision-language models for imbalanced learning, International Journal of Computer Vision, (22(1):22422333 2024.
Western, D., Weber, T., Kandasamy, R., May. F., Taylor, S., Zhu, Y., and Canham, L. Automatic report-based labelling of clinical eegs for classifier training. In 2021 IEEE Signal Processing in Medicine and Biology Sympo-
sium (SPMB), pp. 1-6. IEEE, 2021.
Yang, C., Xiao, D., Westover, M. B., and Sun, Self- supervised eeg representation learning for automatic sleep staging. arXiv preprint arXiv:2 / 15278, 2021.
Yang. C., Westover, M., and Sun, J. Biot: Biosignal trans- former for cross-data learning in the wild. Advances in Neural Information Processing Systems, 36, 2024.
You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708. 03888, 2017.
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al. Biomed- clip: a multimodal biomedical foundation model pre- trained from fifteen million scientific image-text pairs. arXiv preprint ar arXiv:2303.00915. 2023.
Zhang, Y., Jiang, H., Miura, Y., Manning. C. D., and Lan- glotz, P. Contrastive learning of medical visual rep- resentations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022a.
Zhang. 7., Zhong, S.-h., and Liu, Y. Ganser: A self- supervised data augmentation framework for eeg-based emotion recognition. IEEE Transactions Affective Computing, 2022b.
13

---

EEG-Language Pretraining for Clinical Phenotyping
A. Additional Results
A.1. Linear probing comparisons with LaBraM
We apply our linear probing pipeline to the LaBraM- -aasc model and find significant performance drops, highlighting its requirement for downstream finetuning and the improved clinical representations of ELMs. We perform these analyses for NMT (Table 7), TUAB (Table 8), and TUEV (Table 9), while omitting TUSZ as this data included in LaBraM pretraining. Standard deviations indicate variability across five repetitions of 10-fold cross-validation. Whereas for ELM-MII. model trained with different weight initialization for each repetition, for LaBraM only one pretrained model is made available which we use for every repetition.
Table 7. Linear probing for abnormality classification comparing ELM-MIL and LaBraM on the NMT dataset using 1%, 10%, and 100% labeled training data.
Balanced Accuracy
AUROC
Method
1%
10%
100%
1%
10%
100%
LaBraM
1.07
69.12.0.73
67 12.1..111
82.02 69.49
78.03 1.717
ELM-MILL
60.60
68.57 #0.90
81.42 1.15
89.77
Table Linear probing abnormality classification comparing ELM-MIL and LaBraM on the TUAB dataset using Zero-Shot, 1%, 10%, and 100% labeled training data.
Balanced Accuracy
AUROC
Method
ZS
1%
10%
100%
ZS
1%
10%
100%
LaBraM
72.71
20.33
82.
18
89.61 80. 80.18
87.98_033
ELM-MIL
+057
83.10 +0.56
84.21 +0.82
87.11 97.11+033 +.366
91.56+131
91.54 +0.4:
91.91 1.9110.17 0.17
93.14
Table 9. Linear probing for abnormality classification comparing ELM-MIL and LaBraM on the TUEV datasct using 80% labeled training data.
BACC
AUROC
Method
80%
80%
LaBraM
83.22+1009 48.83++200
43.08:1165
ELM-MIL
87.69+101
A.2. TUEV:
Per-Class analysis
We additionally investigated per-class performance as TUEV includes distinctly different event categories (Figure 5). We observe that ELM-MIL scores well across the three clinical events (SPSW, GPED, PLED) with over 3.5% better average scores. However, the model underperformed on artifact and eye movement detection, which may indicate models may lose sensitivity events not described the text. Interestingly, portion of reports include sections on such technical problems, but these were segmented out for the current study. Follow-up research is needed to further investigate the effects of including such text.
A.3. Language-independerr effects of sub-unit alignment.
Language-independent effects of sub-unit alignment. Given the broad outperforming of ELMs compared to EEG-only models, especially for ELM we further investigate whether the general setup of multimodal pretraining provides inherent benefits. EEG recordings are split into multiple crops, which in turn are all aligned to the same clinical report during pretraining. It follows that EEG crops of single recording are indirectly aligned to one another to some extent (Figure 1C). We investigated this hypothesis by shuffling reports between patients prior to pretraining. We find that while embeddings of single EEG crops of an untrained encoder are only minimally more similar within-subject than between-subjer (ratio of
14

---

EEG-Language Pretraining for Clinical Phenotyping
TUEV Per-class Balanced Accuracy
TUEV Per-class AUROC
SPSW
SPSW
T5
BYOL
15
BrOL
RP
ViCReg
RP
VicReg
background
GPED
sackground
artifact
PLED
artitact
movement
movement
Figure 5. Per-class scores for TUEV show that ELM-MIL outperforms for the clinical events. SPSW: Spike and sharp wave, GPED:
generalized periodic epileptiform discharges, PLED: periodic lateralized epileptiform discharges.
~IIIx), this effect is much more pronounced after pretraining ELM,,1 on correctly paired reports (~6.3xx, and even more so after pretraining on shuffled reports (~~15.7x; figure 6). Linear probing reveals that training ELM,,1 on shuffled reports clearly boosts pathology detection over using untrained encoder and manages almost match EEG-only pretraining without the need for augmentations (mean accuracies of 73.70% 33.00%, 81.04%, 83.69%). On the contrary, the ratios for ELM, are close to after training using paired and shuffled reports, with the latter resulting in decreased pathology detection accuracy.
Conceptually, while shuffling reports destroys the semantic relevance reports, it still provides a unique subject-specific reference to which the EEG embeddings are aligned to. Pretraining then reduces to promoting invariance to within-subject information, as EEG crops patient are aligned to the same report. However, while ELM, these reports occupy arbitrary positions the latent language space due the absence the text projector, ELM,,/ exhibits additional dynamics. Namely, for given EEG crop (or text paragraph) in batch belonging subject (that is, id = i), nearly all negative contrastive samples will belong to a different patient (P(id = i) < P(id + i)). The negative contrast therefore largely amounts to minimizing similarity between patients. This can be viewed as encoding between-subject information and these results imply that training with this objective is a useful pretext task for EEG timeseries. Naturally, this will depend on the downstream tasks, but both retrieval and pathology detection require between suweeen--ubjett information. The advantage of retrieval and linear probing of ELM,, may thus be, at least in part, due to the inherent utility of our extension of multimodal language modeling to timeseries by using sub-unit alignment, independent of language. Still, pathology detection with only few annotations is considerably better using paired reports, indicating the importance of relevant clinical language for label-efficiency.
A.4. Post-hoc investigation of data leakage
To maximize the amount of data available this data-scarce setting, the TUAB training set was included during pretraining. We investigate whether this gave disproportionate advantage linear probes trained on ELM representations by repeating the "1% labels" context using unseen subjects as follows: Given only the TUAB test set, we train linear probes using 10-fold cross validation (times five random seeds), each time splitting 10-20-70% of the test set into rrain/vaiiatiinn/ess.. This gives the same labeled sample size as 1 1% the TUAB training set without relying on samples seen during pretraining. As seen in Table 10, results are highly similar, strongly suggesting that the advantage ELMs is not due to the inclusion of the TUAB training set in the pretraining set.
15

---

EEG-Language Pretraining for Clinical Phenotyping
Untrained
B
E Shuffled reports
ELM,,1 Paired reports
D
ELM Pathology delection
Uetween Sutjects
Subjects
Subjects
85
Within Subjects
Witrin Subjects
Within Subyects
fretuaiiy
Prequency
Presg
8
3
Untreined
Shuffled
Paired
70
0.3
10
10%
100%
Costne simllartty between crops
Cosine similaiiy between crops
Casine similerity traps
Labels
WS/BS Ratio
F
ELM, Shuffled reports
G
ELM Paired reports
ELM Pathology detection
Between Subjects
Betweer Subjects
84-
ELM,
Within Sutjects
Within Subjects
B Shuffed
Frequency
Freouency
Shuffled
10-
Untrained
Shufflad
Modes
-0.5
0.0
0.5
1.0 -1.0
-0.5
0.0
0.5
1.0
10%
100%
Casine similarity between EEG crops
Casine similarity between EEG crops
Labels
Figure A-C, E-G) We investigate the distributions cosine similarity values of EEG crop embeddings between- and within subjects (denoted BS and WS respectively). We plot these for an untrained encoder (one example run), well as EEG encoders of ELMs trained with paired or shuffled reports. We find that EL,M,,, produces dissimilar hetween-suhject EEG embeddings, while EI,M, does not. F) shows the ratio between WS and BS similarity values across five runs (with standard deviations). D,H) The downstream performance via linear probing is shown on the right, with error bars representing standard deviations across five training runs.
Table 10, Effect of overlap in subjects used for pretraining and linear probing. Higher standard deviations result from smaller test set.
Method
Overlap
Balanced Accuracy
TS
Yes
74.99
TS
No
74.56=1.12
ELMM,, 4 Clusters
Yes
82.642024
ELM 4 Clusters
No
82,28
B. Training Details
In this section, we provide further detailed information of the model training. Unless stated otherwise, ablation and hyperparameter analyses were performed on a data subset consisting of 5000 and 500 EEG recordings divided into a training and test set respectively. To prevent data leakage, this data had no overlap with the patients used for evaluation of the main results.
B.1. Optimization
All models are pretrained using the LARS optimizer (You et al., 2017) with cosine decay learning rate schedule over 50 epochs, with a warm-up of 4 epochs. The base learning rate is set to 0.3 for EEG-only, 0.01 for ELMs, and 0.06 for EL.--MIL, scaled with the batch size (BaseLR x BatchSize/256; (Grill et al., 2020)). When training on 5-second crops, we lower the learning rate for EEG-only to 0.1 and ELM-MIL to 0.02 to avoid instability. We use a weight-decay parameter of Models were trained on either an Nvidia Geforce GTX 3090 or Tesla V100 GPU and require less than 24GB of
1 x 10 -4 memory. Training took approximately 9 hours for EEG- LEG-aang modeling or 18 hours for EEG-only modeling due to data augmentations. We used CUDA v11.3 and PyTorc v1.12.1.
B.2. EEG Encoder
We use a CNN architecture with a residual stream as the EEG encoder for all analyses (Figure 7). The model uses parallel convolutions, involving reflection padding and 1D-convolutions with kernel sizes (4, 8, 16) with 32 filters each. These
16

---

EEG-Language Pretraining for Clinical Phenotyping
outputs are concatenated, resulting in 96 dimensional representation and 747K trainable parameters. We compare input lengths of EEG crops varying from 5 to 60 seconds. This presents trade-off where longer crops result in greater information content per crop, while reducing the total sample size. As EEG-only pretraining relies data augmentations, this introduces an additional influence of crop length. Specifically, longer crop lengths likely make the pretraining task easier, as augmentations introduce relatively lesser distortion due to the greater information content. We therefore compare performance different crop lengths for both EEG-language and EEG-only pretraining. As the EEG encoder progressively downsamples the signal, we adjust the pooling layers to the input length. These adjustments are shown Table B.2. For EEG-language pretraining we evaluate zero-shot pathology detection, while for EEG-only pretraining we are required to compare the performance of linear probe. Results shown in Figure 8. Due computational resources, we only compare crop lengths for BYOL and ELM; representations of EEG-only and EEG-language modeling. We observe that for EEG-only pretraining an intermediate crop-length of 20 seconds performs best, which matches the findings by Mohsenvand et al. (2020). Meanwhile, zero-shot pathology detection is found to be relatively insensitive to crop lengths of at least 10 seconds, with 60 second crops scoring highest, while the shortest crop length showed unstable learning. For TUSZ and TUEV evaluations, this was solved by lowering the learning rate.
For the EEG projector, we use linear layer with an output dimension of 512 followed by batch normalization, exponential linear units, and final linear layer with output size 256.
Encoder
backbone
ParateliConv
denotes:
ParalellConv
ReflectionPad
BatchNorm1D
Conv1D K4 D:32
Conv1D K:8 D:32
Conv1D K:16 D:32
ELU
Concatenate
ParalellConv
BetchNorm11D
Conv K:1 D:96
ParalellConv
BatchNorm1D
ELU
BatchNorm1D
repeat
MaxPool
N=4
AvgPool1D
Flatten
Figure 7. An identical EEG encoder architecture is used across all analyses. The size of the pool operation depends the input length. These are detailed in table B.2. K: Kernel size, D: Output dimensionality.
Table Multiple input lengths for the cropped EEG timeseries were compared, which included adjustments to pooling layer.
Model Setups
Batch Size
Input Dim.
Max Pool Size
Intermediate Dim.
EEG+Texi
EEG
500
[2,2,2,2]
[166, 55, 18, 6]
2048
2048
1000
[3,3,3,3]
[333, 111, 37, 12]
2048
2048
2000
[3,3,3.3]
[666, 222, 74, 24]
2048
1024
3000
[4,4,4.4]
[750, 187, 46, 11]
1024
800
6000
[4,4,4.4]
[1500, 375, 93, 23]
800
400
17

---

EEG-Language Pretraining for Clinical Phenotyping
BYOL (Linear probe at 10% labels)
ELM, (Zero-shot)
Balanced Accuracy (%)
Balanced Accuracy
80
78
70
77
60
500
1000
2000
3000
6000
500
1000
2000
3000
6000
Crop length (samples@100Hz)
Crap length (samples@100Hz)
Figure Comparison of pathology detection based EEG input crop length, ranging from 60 seconds, via averaged balanced accuracy scores. Error bars indicate the standard deviation across five random seeds.
B.3. Language Encoder
We compare three pretrained language models in their ability to perform zero-shot pathology detection following EEG- language pretraining (Table 12). We find that MedCPT performs best (Jin et al., 2023), which is trained using contrastive learning with 255 million user click logs from PubMed.
For the text projector of ELM,,I, units, and final linear layer with output size 256 and batch normalization.
linear layer with output size 1024 followed by batch normalization, rectified linear
Table 12. Zero-shot classification comparison between language models for ELM
Language Model
Balanced Accuracy
AUROC
BiRT (Gu et al., 2021)
78.61 +2.90
85.78+2.5
Bio-ClinicalBERT (Alsentzer al., 2019)
80.8621.19
87.331066
MedCPT (Jin al., 2023)
82.58
88.37
B.4. Temperature parameter
For ELM,,,, the softmax operation used in the loss computation includes temperature hyperparameter We compare zero-shot pathology detection for multiple values. We observe poor performance for low temperature values, but stable zero-shot classification for higher parameter values. We set 7 0.3 for all further analyses.
ELM (Zero-shot)
Balanced Accuracy (%)
80
60
50
0.01
0.03
0.1
0.3
Temperature
Figure 9, Comparison of temperature values for EL,M,, on zero-shot pathology detection. Error bars indicate the standard deviation across three random seeds.
B.5. EEG-Only Pretraining
We implement the following methods for EEG-only SSL.
18

---

EEG-Language Pretraining for Clinical Phenotyping
Bootstrap-Your-Owni Your-Own-Latent. BYOL relies on encoder models: an online and a target network (Grill et al., 2020). During pretraining, the online network is trained to predict the target model's output. Meanwhile, the weights of the target network are updated using& moving average of the weights of the online network, which has been empirically shown to prevent collapse of the latent space. For alignment, & normalization is applied to the EEG embeddings (h h" } and the mean square distance is minimized. We adopt the recommended parameter value for the exponential moving average (Grill et al., 2020). The projection head is a 2-layer non-linear MLP with a hidden dimension of width 256 and an output dimension of 32.
Variance- Invannine--aaieeeeeeee Covarriacc Regularization. VICReg allows for the use of a single encoder model and prevents collapse by applying two explicit regularization terms to each of the embedding batches {h h") (Bardes et al., 2021). The 'variance" term maintains the standard deviation (computed batch-wise) of every embedding dimension above threshold, thereby avoiding trivial solution. In addition, latent collapse is avoided through the 'covariance' term which decorrelates pairs of embedding dimensions. The method minimizes the mean square distance between (h',, h"). Hyperparameters are set to their recommended values (Bardes et al., 2021). The projection head is a 2-layer non-linear MLP with a hidden dimension of width 256 and an output dimension of 256.
Contrast with the World Representation. Co was proposed to improve augmentation-based SSL for EEG (Yang al., 2021). The method, which is contrastive in nature, maximizes similarity between {h', h") while preventing collapse by minimizing similarity with 'negative' samples. Contra WoRtraWRRFo forms a negative representation by aggregating across all negative batch elements, aiming to compensate for the low signal-to-noist of EEG data by creating a more reliable negative contrast. relies on a triplet loss based on Info-NCE (Gutmann & Hyvärinen. 2010). We also here set the hyperparameters to the values recommended by the authors (Yang et al., 2021). The projection head is a 2-layer non-linear MLP with a hidden dimension of width 256 and an output dimension of 32.
Relative Positioning. Pairs of EEG crops are sampled and assigned binary labels based on their temporal proximity (Banville et al., 2021). Crops close in time are labeled positive, while those far apart are labeled negative. We use the same EEG encoder as for all other methods to create representations and use the suggested contrastive module to compute the element-wise absolute difference between representations. A logistic regression model then predicts the label. The method is trained using binary logistic loss. For all methods by (Banville et al., 2021), we use the hyperparameters reported to work best on TUAB, including between-subject sampling of EEG crops.
Temporal Shuffling. An extension of Relative Positioning by sampling triplets of EEG crops. The task is to determine whether the crops in temporal order or shuffled (Banville al., 2021). The contrastive module concatenates absolute differences between representations. As with Relative Positioning, a logistic regression model is used for prediction, and the method is trained end-to-end using binary logistic loss.
Contrastive Predictive Coding. This method uses an autoregressive encoder to summarize a sequence of EEG crops into a context vector (Banville et al., 2021). The task is to predict which future crop actually follows the context, among negative samples. A bilinear model used for prediction at each future step. The method trained end-to-end using the InfoNCE loss.
B.5.1. DATA AUGMENTATIONS
For EEG-only pretraining, we adapt the data augmentations proposed by Mohsenvand et al. (2020), which were found to perform well on the TUAB dataset. For given EEG crop, we apply the same augmentation to each channel. Parameters are sampled independently for each EEG crop and uniformly from the ranges displayed in table 13. Augmentations are visualized for a single EEG channel in figure 10.
19

---

EEG-Language Pretraining for Clinical Phenotyping
Amplitude Scaling
Gaussian Noise
Augmented Original
DC Shift
Zero Masking
Bandstop Filter
Time Shift
Figure Data augmentations visualized for a single channel of EEG data.
Table 13. Data augmentation parameter ranges; adapted from Mohsenvand et al. (2020).
Data Augmentation
Min
Max
Amplitude Scale
0.5
1.5
Time Shift in samples
-60
60
DC shift in microvolts
-10
10
in samples
0
200
Additive Gaussian Noise (sigma)
0
0.2
Band-Stop Stop Filter (5Hz width, Hz)
2.8
47
C. Details on EEG Data
C.1. EEG preprocessing
From the EEG dataset, recordings longer than 2.5 hours were omitted to filter out a small subset of very long, potentially overnight recordings. For training efficiency, only the first 45 minutes recording were used. Any recording files shorter than 70 seconds were also omitted.
EEG data received minimal preprocessing (using MNE (Gramfort et al., 2013)). First, the initial 10 seconds were removed to reduce the impact of set-up artefacts. Afterwards, a bandpass filter of 0.1 Hz was applied and all recordings were resampled to 100 Hz. To reduce the impact signal artefacts, all EEG signals had their amplitude clipped to + 800 μVV. As a large majority of recordings used an average-reference (AR) or linked-eat reference (LE), we only used these recordings and standardized them via transformation to the 20-channel Temporal Central Parasagittal (TCP) montage.
20

---

EEG-Language Pretraining for Clinical Phenotyping
C.2. Subsampling
TUEG contains considerably more abnormal than normal EEGs, As vision-language models have been shown to be sensitive to imbalanced classes (Wang et al., 2024), we subsample the data to create approximately equal class representation. We rely on the LLM summaries of reports, which were more consistent in their phrasing regarding the normal or abnormal status. This allowed for a more reliable classification using regular expressions. All reports for which no clear classification was made were omitted. 5015 reports in the potential training set were classified as normal, which were associated with 7526 EEG recordings. For our 'pretrain' data subset, we subsampled the abnormal EEGs to match the amount of normal EEG recordings. This resulted in 7526 abnormal EEG recordings, with 6770 reports. Although only a minor subset of these preliminary classifications was manually verified, it is important to note that this process was solely to alleviate severe class imbalance and was not used for further analysis.
For EEG-language modeling, the pretrain subset was effectively smaller, report had to be omitted from pretraining when it did not contain at least one relevant heading. Out of the 15144 total EEG files, this resulted in pretrain sample sizes of: 1 14836 (clinical history cluster), 14320 (medication cluster), 1 14800 (description cluster), 14794 (interpretation cluster), and 14946 (four clusters).
To test for retrieval performance, we supplemented the TUAB test set with data from the TUH EEG Epilepsy Corpus (Veloso al., 2017) in attempt create larger, roughly balanced evaluation set of those with and without pathology. For this, we only selected the first recording of subject that multiple files from the same subject were present. Additionally, we only included reports which had at least one heading from each text cluster to allow for fair comparison.
D. Classification
To study the predictive capability of learned representations after pretraining, we train linear probes and perform zero-shot classification.
Linear probe
For linear evaluation, we train logistic linear regression models using 10-fold cross validation for each pretrained model using sklearn (Pedregosa et al., 10. We perform grid-search over 45 logarithmically-spaeed values for L2 regularization between 10 6 and 10S via a validation set.
Supervised Learning
For the supervised learning baseline, we use the identical EEG encoder backbone as used for all other analyses and 60 second crops. We add an MLP (hidden dimensionality of 256) with dropout p = 0.5 and output dimensionality equal to the amount of classes. The ADAM learning rate is set to 0.001 and we use the validation set to select weight decay out of [0.1, 0.01, 0.0001]. We use a batch size of 256 and train using the cross entropy loss. When using 100% labels, we first train on the training set for up to 50 epochs (with early stopping after 5 epochs without improvement) and select the epoch which resulted in the best validation loss. Subsequently, we continue training on the train and validation sets together until the loss has decreased below the best validation loss.
Zero-shot classification
For zero-shot pathology detection, we perform an ensemble over 21 binary prompts, listed in Table 14, Prompt ensembling was shown to improve performance (Radford et al., 2021), but we employ it here also as the limited data is likely to lead to less stable representations, which may lead to sensitivity to phrasing. To inspect whether results are sensitive to changes to the prompt set, we perform a post-hoc analysis using the held-out test set that iteratively leaves one prompt out of the ensemble (Figure 11). We observe that results are consistent across such reduced prompt sets, except for the ELM; model trained on the clinical history or interpretation clusters, although neither model reaches competitive performance. This set was only initially verified the training set to enable model- and parameter-comparisse Tuning is likely to enable further performance improvements, although the flexibility of the zero-shot approach may introduce severe risk of overfitting on the TUEG dataset.
using zero-shot performance.
21

---

EEG-Language Pretraining for Clinical Phenotyping
ELM, Prompts: leave-one-out
ELMp./ Prompts: leave-one-out
1.0
1.0
0.8
0.8
F1-score
F1-score
0.6
0.4
0.4
Clinical History
Interpretation
0.2
0.2
Medication
LLM Summary
Description
All Clusters
0.0
0.0
10
20
10
15
20
Removed prompt
Removed prompe
Figure 11. Analysis of the sensitivity prompts in the ensemble used for zero-shot classification. We plot the average F1-score across five random seeds. Note that for ELM, multiple models have a consistent Fl Fl-score of and are therefore not individually visible.
Table 14. Prompt ensemble used for zero-shot classification.
Normal EEG Prompts
Abnormal EEG Prompts
Normal EEG.
Abnormal EEG.
No pathology present.
Pathology present.
No abnormalities.
Abnormalitics observed.
Normal routine EEG.
Markedly abnormal EEG.
Normal awake record.
Abnormal awake record.
Normal EEG record.
Abnormal EEG record.
This EEG is normal.
This EEG is abnormal,
This is normal EEG.
This is an abnormal EEG.
This EEG within normal limits
This EEG is mildly abnormal.
Normal awake EEG.
Abnormal awake BEG.
Normal asleep EEG.
Abnormal asleep EBG.
Normal awake and asleep EEG.
Abnormal awake and asleep EEG.
Normal EEG in wakefulness and drowsiness.
Abnormal EEG in wakcfulness and drowsiness.
No pathology.
Abnormal EEG due to:
EEG shows no pathology.
Abnormal EEG subject of this age to:
No abnormalities.
Abnormalitics in the EEG.
No abnormalities observed.
Abnormalities observed.
EEG shows no abnormalities.
EEG shows abnormalitics.
No clinical events detected.
Clinical events detected.
No indications of pathology observed.
Indications of pathology observed.
The EEG normal.
The EEG is pathologically abnormal.
E. Additional visualisations
E.1. EEG embeddings of pathology
We provide t-SNE (complcxity=40, (Van der Maaten & Hinton, 2008)) visualisations of the averaged EEG embeddings per subject after pretraining. These are post-hoc plots for which we use models trained on the entire pretraining subset and display embeddings of hold-out TUAB patients. ELM,,1 and ELM-MIL show the clearest visual separation between abnormal and normal FEGs, which in line with the linear probing results.
22

---

EEG-Language Pretraining for Clinical Phenotyping
Untrained
BYOL
VICReg
ContraWR
Shuffled Reports
ELM,
ELM
ELM MIL e,
Figure 12. Example EEG embeddings averaged within-subject of pretrained models on the TUAB hold-out data (red: abnormal, blue: normal). The data projected using I-SNE. The 'untrained' and 'shuffled reports' plots feature the setup the ELM,,, model, with the latter being trained on reports randomly shuffled hetween subjects.
E.2. Within-subject EEG embeddings
We provide additional visualizations of t-SNE projections of EEG crops (Figure 13). Specifically, we compare ELM,,1 using InfoNCE and ELM-MIL using MIL-InfoNCE, across three temperature parameters [0.1, 1.0). To do so, randomly sample three normal (blue shades) and three abnormal (red shades) subjects. We observe that whereas both methods exhibit diminished subject clustering at higher temperature (r 1.0), at low temperatures (7 0.1) this only occurs for InfoNCE. Meanwhile, subject clustering gets more pronounced for MIL-InfONCE. This may explain the observation that retrieval performance increases by reducing т for MIL-InfONCE, which as a task requires subject rather than class separation per se.
InfoNCE =
InfoNCE T
InfoNCE =
MIL-InfoNCE r 0,1
MIL-InfoNCE r 0.3
MIL-InfoNCE T 1.0
Figure A comparison subject clustering using t-SNE projections of embeddings EEG crops. Red (blue) shades indicate three randomly sampled abnormal (normal) subjects.
23

---

EEG-Language Pretraining for Clinical Phenotyping
E.3. Alignment visualizations
"complex epilepticus"
eeg
Sinfiodt EIG)
0.5-
0.25
0.25
PFG Eraps
BEG Crops
105
Figure 14. Additional alignment visualizations in Figure
"focal eplieptifom activity"
"spike and wave"
Sinnoott EBS)
0.5
EEG Craps
264
BEG Crops
Figure 15. Additional alignment visualizations in Figure 4.
"mild ciffuse slowing"
"rare left frantotemporal spikes"
Siiffor EEG)
0.2-
Emps
Craps
240
Figure 16. Additional alignment visualizations as in Figure 4, but for two observations which were not well aligned. Both feature narrow range of similarity values, indicating lack temporal localization. Whereas mild diffuse slowing may have been pronounced enough, left frontotemporal spikes are very rarely described in the datasct.
24

---

EEG-Language Pretraining for Clinical Phenotyping
normal eeg"
"a normal eeg"
0.00
BIG Craps
EEG Crops
261
Figure 17. Additional alignment visualizations as in Figure but for two recordings which were classified normal by the physician.
E.4. Report and content segmentation
LLM report summarization
CLIMICAL REPORT
pethologgeewly
Lintt and will curINEL and peat
tocal: sharp Haves and slow waves likely oriçinating in anterior tesporal: region.
sysptorenologg.
MEDICATIONS:
Content Clusters
at of murment patient medizetion..
Clinical History
225 recorsing eguipeent..]
Headings Clinical History
TECHEICAL DIFFICULTIES:
Modications
doring
recording.]
Headings: Medications, AED
bOSSCRIPTION OF RBOORD:
Description of the Record
[dboervationa
seccading, Getoting nesral
Jas well potentially pathological features.]
Headings: Desorption of the Record, Events, Eplieptiion Activity...
(3812088::
Interpretation
[Nhethee selzures abserved.1
IMFRESS1ON:
Headings: Impression, Ciinical Correlation, Diagnoeis, Interpretation, All clusters
[Briet statexent concluding
is considered
isomsii abnorral,, way.)
CLINICAL CORRELATION:
Headings: All of above
symetomttiisy
land/or pathology..
Content-based segmentation
Sample single paragraph
X
Figure 18. An example set of headings which may make up clinical report. Paragraphs extracted from the reports into content-based clusters or an L.M-ggeerated summary.
25


Reviews:
[{"review_id": "yQWFQEx4xn", "reviewer": "Reviewer_Ln87", "summary": "The manuscript describes EEG-Language (CLIP-like) pretraining on medical EEG recordings and the accompanying textual medical reports. They used a pretrained medical langauge model and a from-scratch-trained EEG encoder to map temporal crops of EEG and subsections of medical reports to the same latent space, with some methodolical adaptations for the fact that multiple EEG crops belong to the same recording and multiple subsections belong to the same medical report. Experiments are performed across multiple medical EEG datasets. Results show that this yields good pathology detection accuracy in zero-shot settings, whihc can be further improved via linear probing on subsets of the training data and linear probing also works for seizure detection and event classification.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you sincerely for your positive feedback on our manuscript. We are grateful to hear about the value of our contribution to the research community, as well as the clarity of our manuscript.\n\nRegarding your question about $L_{orth}$, $h_e$ is indeed L2-normalized. We apologize for this omission in the paper. We have corrected the description and notation in Section 3.1.1 accordingly.\n\nThank you for helping us improve our manuscript.", "claims_and_evidence": "The claims of a EEG-language-model pretraining that yields good pathology detection from no or few labels seem well-supported to me.", "methods_and_evaluation_criteria": "The authors provide a sensible setup for the evaluation, multiple types of evaluation (retrieval, different types of downstream performance) on multiple datasets.", "theoretical_claims": "None", "experimental_designs_or_analyses": "See Methods and Evaluation Criteria", "relation_to_broader_scientific_literature": "EEG-Language pretraining with medical reports is an open area that seems to not have been tackled yet and for which any results are very valuable for the research community", "essential_references_not_discussed": "None that I am aware of", "other_strengths_and_weaknesses": "Figures look clean, writing seems easy to read", "other_comments_or_suggestions": "-", "questions_for_authors": "In p.3 $L_\\textrm{orth}$ is $h_\\textrm{e}$ normalized? I Assume so, because why otherwise is this correlation and not covariance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bHXOshepws", "reviewer": "Reviewer_hXr5", "summary": "This paper presents a multi-modality model that integrates EEG recordings and clinical reports for neural event detection. The proposed method segments an EEG recording and its corresponding report into sequences of epochs and words, then constructs epoch-word pairs and an alignment matrix for representation learning. The model employs both pairwise contrastive learning and multi-instance contrastive learning to enhance feature representation. Finally, it is fine-tuned for various downstream tasks. Experiments on multiple datasets demonstrate that the proposed approach outperforms several contrastive learning and EEG-based baselines.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your detailed and insightful review of our manuscript. We appreciate your feedback as it has helped us refine our presentation and clarify the motivations behind our work. We are encouraged by your recognition of EEG-text alignment as a promising research direction and would like to address your concerns.\n\n**EEG-Text Alignment**  \nWe acknowledge the point about the lack of explicit temporal correspondence between EEG epochs and clinical report segments, which indeed poses a challenge. However, our approach leverages the fact that concepts like “abnormal EEG” in reports are guaranteed to relate to multiple EEG crops from the same recording and, in expectation, more so than to unrelated (negative) crops. This principle mirrors successful contrastive learning in computer vision (e.g., noisy image-caption pairs) and video-text alignment (e.g., subtitles not always matching visuals), where pretraining remains effective despite imperfect correspondence. To illustrate this, we have added qualitative examples of our ELMs at this [[link]](https://docs.google.com/document/d/e/2PACX-1vQygdcAED1qMhVgFv5jU9TsclAyRxp-XKFiGwxK2pkxLSdrKAgyGVuAEYBVPnmQZeJDfIBVMLTbzTwG/pub) (Figures S1-5), showing similarity scores between text embeddings (e.g., “seizures arising from the right hemisphere”) and 5-second EEG crops for hold-out subjects. Visualizations of the highest- and lowest-similarity crops demonstrate that our method captures pathology-relevant alignment in the absence of explicit temporal information in the reports, reinforcing its practical utility. We add Figure S1 to the main text and Figure S2 through S5 to the appendix.\n\n**Downstream tasks and data costs**  \nRegarding the motivation for downstream tasks, we agree that EEG-only methods can address detection/classification. However, our multimodal approach enhances unimodal EEG encoder initialization, yielding better representations for these tasks. As shown in our manuscript, our method outperforms EEG-only baselines (e.g., +8.7% balanced accuracy at 1% labels on TUAB) and even large-scale models like LaBraM (Table 4, as well as Tables S1-3 with new additional linear probing results at the same [link](https://docs.google.com/document/d/e/2PACX-1vQygdcAED1qMhVgFv5jU9TsclAyRxp-XKFiGwxK2pkxLSdrKAgyGVuAEYBVPnmQZeJDfIBVMLTbzTwG/pub)), trained on many more EEG datasets. The pretrained EEG encoders we are releasing can be used for downstream EEG-only clinical tasks without additional complexity and at reduced cost due to the low parameter count.\n\nThis suggests that leveraging existing clinical reports—already available in large quantities in hospitals and with neglible costs to store compared to EEG data and especially compared to acquiring additional EEG data—offers a cost-effective way to improve performance without increasing data collection costs. Moreover, we like to clarify that our method repurposes existing clinical reports, and does not require new manual writing. These reports, a byproduct of standard clinical workflows, enhance model performance without additional expense.\n\n**Report generation**  \nFinally, we appreciate your suggestion of report generation as a valuable task, and we fully intend our work to pave the way for such applications, as noted in the manuscript. We believe that learning and evaluating pathology-sensitive representations is a critical first step; if alignment fails to capture clinical relevance in latent representations, subsequent text generation would lack grounding. Our focus on detection/classification validates this foundation, aligning with the broader utility of clinical reports. To emphasize how our results establish a stepping stone for future tasks like report generation, we now note this in the discussion and expand the impact statement to contextualize this trajectory.\n\n**Discussion addition [L422]:**\nOur pathology-sensitive multimodal alignment is a critical step toward automated report generation (e.g. Biswal et al. 2020), ensuring EEG-text representations capture clinical information for future documentation tasks.\n\n**Impact statement extension [L457]:**\nThe multimodal nature of our approach, by aligning EEG with clinical reports in a pathology-sensitive manner, not only enhances detection but also lays an important foundation for automated report generation. Specifically, such generation may greatly benefit from an aligned latent space which contains clinical information. This could facilitate clinical documentation by translating EEG signals into structured summaries. These can constitute highly valuable future efforts given the time-intensive nature of manual reporting.\n\nWe hope these clarifications address your concerns and demonstrate the clinical and scientific value of our approach. Thank you again for your constructive feedback, which we believe has significantly strengthened our manuscript.", "claims_and_evidence": "My main concern is the motivation behind aligning EEG recordings with clinical reports. EEG is a time-series signal that records a patient real-time physiological state, such as sleep stages, seizure events, and other neurological conditions. However, clinical reports typically consist of structured sections, such as an abstract and findings/details, which provide a summarized interpretation of the EEG recording. These reports often include patient information, the state during recording (e.g., awake, asleep, or under stimulation), and observations of seizure activity or specific waveforms, but they lack precise time indices, e.g., onset/offset of neural events.\n\nAuthors also point out this lack issue of time information of clinical reports but attempt to address it by using a neural network to learn representations of report segments and force-align them with temporal EEG epochs. However, this approach lacks clinical feasibility, as there is no inherent one-to-one correspondence between EEG epochs and report segments. Simply learning latent representations for alignment without considering the structured, non-temporal nature of clinical reports does not meet real-world clinical workflows.", "methods_and_evaluation_criteria": "Partially, the proposed method focuses on detection/classification tasks, but incorporating additional detailed clinical report, which require manual writing by doctor, significantly increases data costs. Moreover, such neural detection tasks can naturally be performed using EEG data alone, without the necessity of aligning with textual reports. The added complexity of learning from reports does not provide clear motivation and advantages for classification tasks.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Please kindly refer to Methods And Evaluation Criteria.\n\nA  meaningful task involving clinical reports would be report generation, as the authors also highlight in the Discussion and Impact Statement section. However, the current focus on detection and classification contradicts the intended use of clinical reports.", "relation_to_broader_scientific_literature": "EEG-Text alignment is a promising research direction. However, constructing a benchmark dataset, designing an effective alignment method, and establishing a meaningful evaluation framework remain open challenges. Addressing these aspects would be highly beneficial for advancing clinical tasks and BCI applications.", "essential_references_not_discussed": "CLARA: Clinical Report Auto-completion, WWW20.", "other_strengths_and_weaknesses": "Please kindly refer to the above comments", "other_comments_or_suggestions": "Please kindly refer to the above comments", "questions_for_authors": "Please kindly refer to the above comments", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGn9TNT5KY", "reviewer": "Reviewer_eiWw", "summary": "This paper introduces an approach for pretraining multimodal EEG-language models (ELMs) to improve pathology detection. The authors propose combining EEG data with clinical reports using a sub-unit alignment strategy, which involves cropping EEG time series and segmenting medical reports to create multiple non-overlapping samples. They further extend this approach with multiple instance learning (MIL) to address misalignment between EEG and text segments. The proposed model significantly improves pathology detection performance, especially in scenarios with limited labels. These results are particularly applicable to clinical settings, where datasets are typically much smaller than those in many common deep learning applications.\n\n## update after rebuttal\n\nThanks for the authors' rebuttals. The comments have addressed most of my concerns. I would keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your detailed and constructive review of our manuscript. We greatly appreciate your recognition of our approach’s innovation, performance improvements, and clinical relevance, as well as your thoughtful suggestions and questions, which have helped us strengthen our work.\n\n**Alignment and challenges**  \nWe appreciate your suggestion to further discuss our EEG-language pretraining including its potential challenges with alignment. While our MIL extension aims to mitigate inconsistencies, challenges may persist when reports contain information unrelated to clinical tasks. However, our results (e.g., robustness to additional text sections in Figure 2) suggest this is less impactful for common clinical concepts. We are pleased to provide new model intepretability figures which visualize alignment (Figures S1-5 available at this [link](https://docs.google.com/document/d/e/2PACX-1vQygdcAED1qMhVgFv5jU9TsclAyRxp-XKFiGwxK2pkxLSdrKAgyGVuAEYBVPnmQZeJDfIBVMLTbzTwG/pub)). While we show that our methodology is able to localize pathology despite a lack of explicit temporal information (Figures S1-S3), alignment for rare or subtle features remains challenging (Figure S4). These examples highlight both successes and shortcomings, setting the stage for future refinements. We add Figure S1 to the main text along with a short paragraph noting the successes and shortcomings, while adding Figure S2 through S5 to the appendix.\n\n**Multimodal literature**  \nRegarding literature on multimodal alignment, we are expanding the \"Related Work\" section to include further literature. In case the reviewer believes we are omitting important relevant work, we would be sincerely grateful for further suggestions.\n\nRelated Work - Medical multimodal language modeling. [L100] (...) *Recent advances outside the medical domain include multi-task strategies, both during pretraining by integrating contrastive learning and self-supervised losses [1,2], as well as finetuning on multiple downstream tasks [3,4]. Further exploration involves moving compute from unimodal encoding to multimodal fusion [5].*\n\n[1] Tschannen, Michael, et al. \"Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.\" arXiv preprint arXiv:2502.14786 (2025).  \n[2] Tang, Zineng, et al. \"TULIP: Towards Unified Language-Image Pretraining.\" arXiv preprint arXiv:2503.15485 (2025).  \n[3] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  \n[4] Dai, Wenliang, et al. \"InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.\" arXiv, 2023, arxiv.org/abs/2305.06500.  \n[5] Kim, Wonjae, Bokyung Son, and Ildoo Kim. \"Vilt: Vision-and-language transformer without convolution or region supervision.\" International conference on machine learning. PMLR, 2021.  \n\n**Logical structure**  \nWe appreciate the note about the logical structure and the placement of phrases such as “We set N=32 and M=8”. To improve the structure of the manuscript, we propose the following sections:  \n3. Methods (unchanged)  \n3.1 Pretraining (was: Experimental Setup)  \n3.1.1 EEG-language pretraining (unchanged)  \n3.1.2 EEG-only self-supervised learning (unchanged)  \n4. Experimental Setup (new)  \n4.1 Pretraining setup (new)  \n4.2 Datasets and evaluation tasks (was 3.2)  \n4.3 Preprocessing (was 3.3)  \n\nand move the mentions of hyperparameters settings (such as N, M, temperature, model dimensionality) to '4.1 Pretraining Setup'. This separates the description of the methodology and experimental details into distinct sections. We thank the reviewer for their suggestion.\n\n**Training scalability**  \nRegarding training scalability, our approach benefits from a small EEG encoder (0.9M parameters) compared to large-scale models like LaBraM (up to 369M), a frozen text model, and no finetuning requirement. This efficiency enables training on large batches with a single GPU (e.g., 9 hours on our dataset using a two-generation-old GPU) and thus scalability should remain very manageable.\n\n**Interpretability**  \nFor interpretability, we agree on its clinical importance and add the aforementioned alignment visualizations. This provides interpretability in the temporal domain and we hope the reviewer finds them valuable. While for the current manuscript we used an efficient CNN encoder to enable a clear focus on comparisons between pretraining strategies per se, we recognize that scaling to transformer architectures with attention visualizations can further enhance interpretability across the spatial domain. We leave these important encoder architecture explorations to follow-up work.\n\nWe hope these comments address your feedback effectively. Thank you again for your insightful comments, which have significantly improved our manuscript.", "claims_and_evidence": "The paper makes a clear claims with convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense.", "theoretical_claims": "The paper does not include a clearly defined theoretical proof section. However, in the methodology section, it provides a detailed description of how concepts such as multimodal alignment, time-series cropping, text segmentation, and multiple instance learning are applied to the pretraining of EEG and language models. The theoretical foundation of these methods primarily stems from existing research in contrastive learning and multimodal representation learning. For example, the paper mentions the InfoNCE loss function and MIL-InfoNCE loss function, both of which are based on the theoretical framework of contrastive learning and are used to learn aligned representations between EEG and text. These theoretical frameworks have been extensively studied and validated in other domains, making the proposed methods theoretically sound.\nHowever, the paper could further discuss the specific application and adaptability of these theoretical frameworks in EEG and language pretraining. For instance, while the multimodal alignment strategy and MIL extension are theoretically designed to mitigate inconsistencies between EEG and text segments, such inconsistencies may still persist in practical applications, especially when clinical reports contain a large amount of information unrelated to downstream clinical tasks.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and well-executed. The authors conduct extensive experiments to validate their approach, including:\nRetrieval Analysis: The authors evaluate the ability of ELMs to retrieve matching EEG recordings from clinical reports and vice versa, using top-K accuracy as the metric.\nPathology Detection: The authors compare ELM-MIL to EEG-only models on the TUAB dataset for binary classification of normal vs. abnormal EEG recordings. They also evaluate performance on the NMT dataset to assess generalization.\nZero-shot Classification: The authors demonstrate zero-shot classification performance using a prompt ensemble, showing that ELMs can leverage language embeddings for pathology detection without explicit downstream training.\nAblation Studies: The authors conduct ablation studies to investigate the impact of different components, such as the aggregation method for positive samples and the number of positive EEG/text samples.\nThe experimental designs are comprehensive and address various aspects of the proposed approach. The results provide clear evidence of the effectiveness of ELM-MIL in improving pathology detection.", "relation_to_broader_scientific_literature": "In the \"Related Work\" section, the authors thoroughly discuss the connections between their research and the broader scientific literature, highlighting key contributions in relation to previous studies. This section is divided into four subsections: Self-supervised learning with EEG data, Using EEG for pathology detection, Medical multimodal language modeling, and Multiple instance learning.\nThe proposed EEG-Language Models (ELMs) introduce multimodal pretraining by aligning EEG with text, significantly improving pathology detection performance, particularly in scenarios with limited labeled data. This represents a major advancement compared to previous self-supervised learning (SSL) approaches that rely solely on EEG data.", "essential_references_not_discussed": "In the paper, the authors have thoroughly discussed various works related to EEG-Language Models (ELMs), including the latest advancements in self-supervised learning, multimodal modeling, and EEG-based pathology detection. However, the related work section could be further expanded by including research on alignment strategies in multimodal learning.\nAlignment strategies are crucial in multimodal learning. While the paper mentions methods such as CLIP and M-FLAG, there have been recent advancements in multimodal alignment. These methods could provide new insights for improving EEG-text alignment.", "other_strengths_and_weaknesses": "Strengths:\nInnovative Approach: The paper presents a novel application of multimodal pretraining in the medical domain, combining EEG data with clinical reports in a meaningful way.\nSignificant Improvements: The proposed ELM-MIL model demonstrates substantial improvements in pathology detection, especially in scenarios with limited labeled data.\nZero-shot Classification: The ability to perform zero-shot classification using language embeddings is a unique and powerful feature of the proposed approach.\nWeaknesses:\nData Limitations: The availability of paired EEG-report datasets is limited, which may restrict the scalability of pretraining data. Future work could explore synthetic data generation techniques.\nThe related literature is insufficient. The related work section could be expanded by including research on alignment strategies in multimodal learning. Alignment strategies are crucial in multimodal learning. While the paper mentions methods such as CLIP and M-FLAG, there have been recent advancements in multimodal alignment. These methods could provide new insights for improving EEG-text alignment.\nThe paper lacks logical structure. The methods section should focus solely on describing the methodology. For instance, the statement “We set N=32 and M=8 as this covers all samples for a majority of subjects.” refers to specific experimental parameters and should be moved to the experiments section.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In practical applications, the dataset size may be much larger. In this case, would the model's training time and memory requirements increase significantly?\nThe multimodal model proposed in the paper performs excellently in pathology detection, but model interpretability is crucial for clinical applications. Has the author considered improving the model's interpretability through specific techniques, such as feature importance analysis or attention mechanism visualization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hn8viwWD28", "reviewer": "Reviewer_Qj1r", "summary": "This paper introduces EEG-Language Models (ELMs), a multimodal framework that integrates EEG signals with clinical text reports for various downstream tasks, including retrieval, abnormality classification, and event classification, across multiple datasets. The method employs time-series cropping, text segmentation, and a multiple-instance learning (MIL) variant of contrastive learning to address the alignment challenges between EEG signals and textual descriptions. Experimental results show that ELMs outperform EEG-only pretraining methods. Notably, the model exhibits zero-shot capability, further highlighting its adaptability across diverse downstream tasks. These findings underscore the potential of multimodal pretraining in medical applications, enabling richer and more effective representations for classification and retrieval tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough and supportive review of our manuscript. We are grateful for your positive assessment of our work’s novelty, methodological soundness, and comprehensive evaluation, as well as your recommendation to accept. We have used your constructive suggestions to refine our paper.\n\n**Baseline comparisons**  \nWe appreciate your comment regarding the baseline comparisons, particularly the limited inclusion of large-scale EEG models like LaBraM beyond Table 4. While we initially hesitated due to challenges in isolating the effects of data, encoder, architecture, parameter counts, or pretraining strategy, we recognize the value of broader comparisons. Following your suggestion, we have extended our evaluation to include LaBraM across additional datasets (TUAB subject level, NMT, and TUEV, while omitting TUSZ as they pretrain on this dataset) using the same evaluation strategy we used for all other methods. We adopted LaBraM’s preprocessing recommendations (resampling, bandpass filtering, notch-filtering, avoiding bipolar montages, and crop lengths of 10s for TUAB/NMT, 5s for TUEV) and obtained EEG embeddings. The updated results, visible in Tables S1-3 (available at this [link](https://docs.google.com/document/d/e/2PACX-1vQygdcAED1qMhVgFv5jU9TsclAyRxp-XKFiGwxK2pkxLSdrKAgyGVuAEYBVPnmQZeJDfIBVMLTbzTwG/pub)), show that ELM-MIL outperforms LaBraM across clinical contexts, with accuracy gains of up to 5.7–13.4% depending on the dataset. Recent literature such as LaBraM has predominantly focused on pretraining on many datasets (about 20 in their case) with large transformers to yield general representations. However, these representations are nonspecific and not ideal for downstream prediction tasks without fully finetuning the encoder, which is problematic when limited downstream data is available as in clinical contexts. In contrast, using medical text during pretraining helps ELMs learn relevant representations, to which we ascribe their strong performance.\n\n**Abbreviation**  \nAdditionally, thank you for noting the undefined abbreviation “SV” in Tables 2 and 6. We apologize for the oversight—“SV” refers to “Supervised”—and now provide the full term to ensure clarity.\n\n**Alignment visualisations**  \nFinally, we would like to kindly note the addition of alignment visualisations, which may be found in Figures S1-5 at the same [link](https://docs.google.com/document/d/e/2PACX-1vQygdcAED1qMhVgFv5jU9TsclAyRxp-XKFiGwxK2pkxLSdrKAgyGVuAEYBVPnmQZeJDfIBVMLTbzTwG/pub). These highlight the ability for our method to localize pathology, while also indicating shortcomings such as rarely mentioned features. We hope the reviewer finds these informative.  We add Figure S1 to the main text along with a short paragraph noting the successes and shortcomings, while adding Figure S2 through S5 to the appendix.\n\nWe hope these revisions address your suggestions effectively. Thank you once again for your insightful feedback.", "claims_and_evidence": "The experiments and analyses presented in the paper are sufficient to substantiate the authors' contributions.", "methods_and_evaluation_criteria": "The proposed methodology aligns well with the research problem and is well-motivated. The evaluation framework is comprehensive, with multiple benchmark datasets and appropriate performance metrics. The chosen baselines allow for a fair comparison, effectively demonstrating the advantages of the proposed approach.", "theoretical_claims": "The paper is primarily empirical and does not involve rigorous theoretical proofs.", "experimental_designs_or_analyses": "The experimental design is reasonable, with a diverse set of datasets and downstream tasks. The proposed method is compared against both supervised and self-supervised models, which strengthens the validity of the results. However, the paper only includes large-scale EEG models like LaBraM as a baseline in table 4, while the SSL baselines for other tables are relatively outdated. It is recommended to include more recent models and large-scale EEG models to provide a more comprehensive comparison.", "relation_to_broader_scientific_literature": "The paper presents a novel approach by integrating EEG with language modeling and multiple-instance learning. It builds upon prior works such as CLIP and M-FLAG while incorporating multiple-instance learning to better handle EEG-text alignment. The experimental results convincingly demonstrate the effectiveness of this framework, highlighting its potential for advancing EEG applications in the medical domain. The study makes a meaningful contribution by extending multimodal learning techniques to EEG-based medical analysis.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n•\tNovelty: The paper introduces an innovative approach by integrating EEG and language modeling with multiple-instance learning.\n•\tMethodological soundness: The proposed methodology is well-grounded and effectively addresses EEG-text alignment challenges.\n•\tComprehensive experiments: The study evaluates the model across multiple tasks, demonstrating strong performance.\nWeakness:\n•\tBaseline comparison: The paper only includes large-scale EEG models like LaBraM as a baseline in table 4, while the SSL baselines for other tables are relatively outdated. It is recommended to include more recent models and large-scale EEG models to provide a more comprehensive comparison.", "other_comments_or_suggestions": "In Tables 2 and 6, the abbreviation \"SV\" is not explicitly defined, which may lead to ambiguity. It would be helpful to provide the full term for clarity.", "questions_for_authors": "No major questions. The paper is well-structured, and the methodology is clearly explained.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a creative approach for pretraining EEG models by aligning them with medical text reports, leveraging a combination of time-series cropping, text segmentation, and multiple instance learning (MIL) variant of contrastive learning. The proposed EEG-Language Models (ELMs) demonstrate convincing performance in various downstream tasks, including retrieval, abnormality classification, and event classification, across multiple datasets.

The reviewers overall acknowledge an innovative approach, comprehensive experimental results, and the potential of multimodal pretraining in medical applications.

Although 1 reviewer out of 4 raises some concerns (e.g. the lack of clinical applicability of the method) the discussion during rebuttal period among reviewers did not affect the strong support for acceptance by the 3 positive reviewers.

I endorse this paper for publication at ICML, as it presents a creative and impactful approach that bridges ML research on text and neural signals with the potential to advance the state of the art in EEG analysis and medical applications.
Decision: Accept (poster)