ICML 2025

EEG-Language Pretraining for
Highly Label-Efficient Clinical Phenotyping
Sam Gijsen 1 2 Kerstin Ritter 1 2
Abstract
Multimodal language modeling has enabled
breakthroughs for representation learning, yet re-
mains unexplored in the realm of functional brain
data for clinical phenotyping. This paper pioneers
EEG-language models (ELMs) trained on clinical
reports and 15000 EEGs. We propose to com-
bine multimodal alignment in this novel domain
with timeseries cropping and text segmentation,
enabling an extension based on multiple instance
learning to alleviate misalignment between irrele-
vant EEG or text segments. Our multimodal mod-
els significantly improve over EEG-only models
across four clinical evaluations and for the first
time enable zero-shot classification as well as re-
trieval of both neural signals and reports. In sum,
these results highlight the potential of ELMs, rep-
resenting significant progress for clinical applica-
tions.
Medical neuroimaging, including electroencephalography
(EEG), has lagged behind other fields in leveraging the
significant advancements of deep learning. While EEG
sees widespread clinical use for clinical phenotyping such
as pathology detection, in particular for epilepsy (Binnie
& Stefan, 1999; Jing et al., 2020) as well as sleep disor-
ders (Malhotra & Avidan, 2013), available annotated data is
scarce. As the impressive scaling properties of deep learn-
ing are now well described (Kaplan et al., 2020; Smith et al.,
2023), self-supervised learning (SSL) is a promising direc-
tion by enabling pretraining with unlabeled data and thereby
increasing available training sample sizes (Hadsell et al.,
2006; Chen et al., 2020). Various such approaches have
shown initial success when applied to EEG. These include
methods relying on data-augmentations (Mohsenvand et al.,
1Charit´e – Universit¨atsmedizin Berlin, Department of Psychia-
try and Psychotherapy, Berlin, Germany 2Hertie Institute for AI in
Brain Health, University of T¨ubingen, ...