MindMix: A Multimodal Foundation Model for Auditory Perception Decoding via Deep Neural-Acoustic Alignment
Authors: RUI LIU
Zhige Chen
Pengshu
Wenlong You
Zhi-An Huang
Jibin Wu
KC Tan
Venue: ICLR 2026 Poster (2026)
Keywords: Electroencephalogram; Audio; Multimodal foundation model; Auditory decoding
Primary area: applications to neuroscience & cognitive science

Abstract: Decoding complex auditory experiences from non-invasive EEG is a rapidly emerging field that holds significant promise for advancing both fundamental neuroscience and human-machine interaction technologies. Recent developments in EEG foundation models have yielded powerful neural representations that are promising for auditory decoding. However, the effectiveness of these models remains fundamentally constrained by their limited integration with acoustic stimulus information. Specifically, the lack of deep coupling between neural signals and auditory inputs hampers the models’ ability to generalize effectively across diverse auditory tasks. To bridge this gap, we introduce MindMix, a multimodal foundation model designed to bridge the gap between unimodal EEG foundations and task-specific auditory decoders. MindMix employs a two-stage training strategy: first, a high-capacity EEG encoder is pre-trained on over 3,000 hours of EEG data to learn generalized EEG features that can transfer across tasks and subjects. Second, the model learns the neural-acoustic mapping using over 100 hours of paired data, facilitated by our novel Cross-Attention Low-Rank Alignment module, which facilitates fine-grained, cross-modal information integration. Experimental results demonstrate that MindMix substantially surpassing existing baselines across a range of auditory decoding tasks, including auditory attention decoding, auditory emotion recognition, and cross-modal retrieval. This work thus establishes a foundation for future research in multimodal brain decoding and auditory brain-computer interfaces. Our code is available at https://github.com/CookieMikeLiu/MindMix.

Published as a conference paper at ICLR 2026
MINDMIX: A MULTIMODAL FOUNDATION MODEL FOR AUDITORY PERCEPTION DECODING VIA DEEP NEURAL-ACOUSTI ALIGNMENT
Rui Liu', Zhige Chen', Shu Peng', Wenlong You', Zhi-An Huang", Jibin Wu'; Kay Chen Tan The Hong Kong Polytechnic University, "City University of Hong Kong (Dongguan) fruiliu, shupeng, wenlong, you, jibin. wu, zhige. chen@connect polyu. .k, huang aaleiityu.eu....
edu.hk.
ABSTRACT
Decoding complex auditory experiences from non-invasive EEG is rapidly emerging field that holds significant promise for advancing both fundamental neuroscience and numan--mchinn interaction technologies. Recent developments in EEG foundation models have yielded powerful neural representations that promising for auditory decoding. However, the effectiveness of these models re- mains fundamentally constrained by their limited integration with acoustic stim- ulus information. Specifically, the lack of deep coupling between neural signals and auditory inputs hampers the models' ability generalize effectively across diverse auditory tasks. To bridge this gap. we introduce MindMix, multimodal foundation model designed bridge the gap between unimodal EEG founda- tions and task-specific auditory decoders. MindMix employs two-stage train- ing strategy: first, a high-capacity EEG encoder is pre-trained on over 3,000 hours of EEG data to learn generalized EEG features that can transfer across tasks and subjects. Second, the model learns the neural-acoustic mapping us- ing over 100 hours of paired data, facilitated by novel Cross-Attention Low- Rank Aligmment module, which facilitates fine-grained, cross-modal information integration. Experimental results demonstrate that MindMix substantially passing existing baselines across a range of auditory decoding tasks, including auditory attention decoding, auditory emotion recognition, and cross-modal re- trieval. This work thus establishes foundation for future research in multimodal brain decoding and auditory brain-computer interfaces. Our code is available http:///ihbbcoo/CCCoooeiMieeeeee
INTRODUCTION
Auditory perception plays a central role in how humans interact with the world, shaping language understanding, environmental awareness, and social communication Opoku-Baah al. (20TT) De- coding the brain's 's representation of auditory experiences is a core pursuit in cogmitive neuroscience and a key capability for brain-computer interface (BCI) systems MMhrooz et al. (2024). Recent advances show that brain signals contain rich acoustic and semantic information, enabling the di- rect interpretation of internal auditory experiences from neural activity [Li et al.) (2023); Chen et al.)
(2024a): Mathis et al. (2024). Among available techniques, electroencephalographh (EEG) is widely used for its non-invasiveness and high temporal resolution Défossez et al. (2023): Li et al. (2024);
Liu et al. (2024). However, decoding rich, naturalistic auditory experiences is fundamentally hin-
dered by EEG's s inherent limitations: a low signal- toioo ratio and high inter-subject variability
astra al. 2021 Bonetti et al. 20024: Oxenham & Krefi (2066).
Historically, these challenges were compounded by task dask-specific modeling strategics that showed poor generalization across tasks and subjects Crosse al. (2016); Yan al. (2024b a). A recent paradigm shift towards EEG foundation models, such as EEGPT et al, (2024b). LaBraM al. (0024), and HEAR Chen et (2025), has begun address this by learning transferable repre-
"Corresponding authors

---

Published as a conference paper at ICLR 2026
sentations from massive unlabeled EEG datasets. However, their effectiveness in auditory decoding fundamentally limited by their unimodal nature. When trained exclusively on EEG signals, their representations are not optimized to align with the underlying structure of acoustic information, as they lack exposure to corresponding auditory stimuli. This highlights a critical research gap: the lack of a unified framework capable of learning well-aligned multimodal representations for robust and versatile auditory neural decoding Poziomska et al. (2024); Liu et al. (2025).
To bridge this gap, we introduce MindMix, the first multimodal foundation model specifically de- signed to learn a deeply aligned neural-acoustic representation from large-scale, paired EEG-audio data. The design of MindMix directly addresses the challenges of cross-modal learning. Its archi- tecture features two key innovations: (1) high-capacity EEG encoder, trained from scratch with a multi-task objective to robustly capture complex neural dynamics from noisy signals, and (2) novel Cross-Attention Low-Rank Alignment (CALRA) module, which enables fine-grained align- ment between neural patterns and acoustic features. CALRA moves beyond simple projection-base based alignment to facilitate deep interaction between modalities. The entire framework is optimized end- to-end via a contrastive learning objective on over 100 hours of paired data, which explicitly forces the model to map corresponding EEG-audio pairs to nearby points in shared embedding space. Our main contributions are summaris ed as follows:
We introduce MindMix, the first multimodal foundation model designed to learn fine- grained and deeply aligned neural-acoustic representations, enabling robust performance across diverse auditory decoding tasks.
We propose CALRA, novel neural architecture for cross-modal alignment that enables fine-grained and auditory-type-avee interaction between neural and acoustic modalities.
Extensive experimental results on MindMix demonstrate superior cross-modal alignment, leading to significantly improved neural decoding performance across range of auditory perception tasks, including auditory attention decoding, auditory emotion recognition, and cross-modal music retrieval.
2
RELATED WORK
2.1 AUDITORY PERCEPTION DECODING FROM BRAIN SIGNALS
Early work on auditory decoding focused on reconstructing speech features from brain signals us- ing linear models like regression or temporal response functions O'Sullivan al.) (2015); Crosse et al. (2016); Ferrante et al. (2024); Dahan et al. (225).. While effective in controlled settings, these methods struggle with naturalistic scenarios due their reliance on clean stimuli and long decision windows Mesgarani & Chang (2012). More recent deep learning models offer greater flexibility, with applications in auditory attention classification [Su et al. (2022), speech/music discrimination Wang et al. (002a);; Niu et al. (2024), and affective state recognition Hu et al. (2024). However, these models remain predominantly task-specific. They are typically trained and evaluated in iso- lation, exhibit poor generalization across datasets subjects Pozzomska et al. (2024); Chen et al. 2024c), and fail to scale to diverse, real-world listening conditions.
2.2 EEG FOUNDATION MODELS FOR NEURAL REPRESENTATION LEARNING
To address the limitations of task task-specific models, recent work has explored EEG foundation models, which learn general-purpose representations from large-scale datasets. Using Transformer-bas ar- chitectures and self-suuprrvsed objectives, models like EEGPT Wang et al. (2024b), LaBraM [Tang et al. (2024). Neuro-GPT Cui et al. (2024) and CBraMod Wang et al. (2025) have achieved strong performance on clinical benchmarks such as epilepsy detection or sleep staging. However, funda- mental limitation of these models for auditory decoding is their lack of exposure to auditory stimuli. Pretrained exclusively on EEG signals, their representations are not optimized to align with acoustic structures, resulting in poor transferability to auditory decoding tasks. MindMix is explicitly de- signed to bridge this gap: by incorporating paired EEG-audio data during pretraining, it learns a shared embedding space that effectively aligns these modalities.
2

---

Published as a conference paper at ICLR 2026
Muttimodill Alignment Training
Dowen war
BEG Proectton
Algned BEG Embeddiing
Clnssiiatton
BEG Signel
Foundation
Fine wuring
Cress Aninnoon Low Zark Afgiment
ArLL
5 Sgrial
Emerdan Armpux
Detection
Audio Emoocder
A
A
Av
Reconttruction
Audio Foundation Medel
Audio Projecioon
Algned Aucio Embedding
SEM Rudtory Type
Aigred PEGp
Audio Vell
Figure 1: Overview of the proposed MindMix framework, which consists of an EEG encoder trained from scratch, a pretrained audio encoder, and our proposed CALRA module for fine- fine-grained cross- modal alignment. Through large-scale pretraining with contrastive objective, MindMix learns unified EEG-audio representation space. This shared embedding facilitates strong generalization a wide range of downstream auditory decoding tasks.
METHODOLOGY
3.1 OVERVIEW
We introduce MindMix, multimodal foundation model that learns unified embedding space to align EEG signals with corresponding auditory stimuli. As illustrated Figure [ given an in- pair (SEEG. SAndio), MindMix uses dual-stream architecture with two modality-specific en- coders. These encoders produce feature embeddings (Epoj. Aguaj), which are then processed by our core innovation, the CALRA module. CALRA performs deep interaction between the modalities, conditioned on the auditory type (e.g., speech, music), to produce the final aligned embeddings (Faligned, Aatigned). The entire framework is optimized end-to-end via contrastive learning objec- tive, fCL [Chen et al. (2020), which maximizes the similarity between true (Faligned, Allignad) pairs while minimizing it for non-corresponding pairs within each training batch.
3.2
MODALITY-SPECIHC ENCODERS
To address the core challenges of EEG signals, high inter-subject variability and heterogeneous channel configurations, the EEG encoder, funG, is designed novel high-capacity architecture. As illustrated in Figure 2 this encoder is developed during the unimodal pre-training stage using multi-task, self-supervised objective.
EEG Encoder.
Our approach employs a channel-independenr patching strategy to robustly handle heterogeneous electrode configurations. Given a raw signal SEEG E RCx1 (where the channel count varies across datasets), segment each channel independently into K fixed-length temporal patches. These patches are passed through temporal ID convolution to obtain the initial embeddings X. Crucially, to learn discrete neural representations, we first quantize these initial embeddings X into discrete neural tokens v E V using a shared codebook. Following quantization, we construct the final input embedding Epatch by adding learnable positional information to these tokens:
Epatch + T +
where T represents the learnable temporal embedding, which is added to each patch to indicate its relative temporal position index (1 to K) within the epoch; and E represents the spatial (channel) embedding, implemented as Icarnable lookup table that maps standard 10-20 system electrode identities (e.g.. 'Cz', 'Pz') to unique vectors. This spatial embedding & allows the model to distin- guish the anatomical source of each patch regardless of the varying channel count C.
The main innovation of this stage is our unique pretraining methodology, which integrates two carefully designed self- superviee tasks. First, patch embeddings quantized into discrete neural tokens v E V using a shared codebook, which is optimized via a quantization loss Lg. Subsequently, compute two pretraining objectives:
3

---

Published as a conference paper at ICLR 2026
Codebook
Encading
E Couter
Tire
H+
moe sepock
Makel
Tamppral
Neural Reprasentation
Reconstrattoon
Figure 2: The multi-task pre-training architecture of the EEG encoder. The framework performs two tasks in parallel: one branch (top) reconstructs the Fourier spectrum from the full neural repre- sentations (Ls), while the main branch (bottom) performs masked token prediction (LM) to learn robust features.
Masked Token Prediction: A portion of the patch embeddings are randomly masked. A main Transformer encoder Vaswani et al. (2017) then predicts the original neural tokens of the masked patches from the visible ones, supervised by a masked modeling loss:
LM= = Σ
log (ivisble))))
(2)
JEM
where M is the set of masked patch indices.
Spectrum Reconstruction:
Concurrently, the unmasked patch embeddings are passed through separate, smaller Transformer encoder. Its output reconstructs the Fourier spec- trum (amplitude A and phase 1/) of the original patches, supervised by a spectrum predic- tion loss:
Ls B, |1 A1,1- 1155
+5yl]
The total pre-training loss is a weighted sum of these objectives. The main Transformer from the masking task serves the backbone for fEEG. For multimodal alignment, we apply mean pooling over its output sequence and project it to produce the initial EEG embedding, Epmn-
Audio Encoder. Motivated by the strong performance of seff-supervised pre-trained speech pro- cessing models |Wang et al. (202T): Kunešová et al.) (2024), we utilize the pretrained Wav2Vec 2.0 model Baevvki et al. (2020) as our audio encoder, fAudio- For each audio clip, we extract the final hidden state sequence from the Transformer, apply mean-pooling to obtain a single vector represen- tation, and pass it through linear projection layer to produce the initial audio embedding, Aproj.
3.3 CROSS- LOW-RANK ALIGNMENT
The primary motivation for CALRA is achieve deep, robust semantic alignment capable of handling the unique challenges of auditory decoding. This task faces two specific hurdles: (1) the low signal-to-noise ratio and high non-linearity of EBG-audio mapping, for which standard "shallow projections" " (like CLIP Radford et al. (2021)) are insufficient; and (2) the heterogeneity of stimuli (e.g., speech vs. music), where uniform mapping fails to capture distinct neural response patterns. While "early fusion" methods (e.g., concatenation) could model these interactions, they break the dual-stream architecture required for efficient retrieval.
To bridge this gap. we propose CALRA, a global feature refinement module that implements grefine-then-conragl strategy. Instead of directly contrasting raw projections, CALRA injects deep, context-aware interactions into the embeddings before the loss calculation. Uniquely, it is designed to overcome the limitations linear fusion by enforcing bilinear interactions in shared bottleneck, allowing the model to capture fine-grained multiplicative dependencies that simple concatenation or co-attention Liu et al. 2023) cannot effectively model. The module consists of three synergis- tic components: Type- sspe-specific Aligner handle stimulus heterogeneity, Bi-directional Cross- Alignment to
Attention mechanism for dynamic global context refinement, and a Shared enforce deep bilinear fusion, which detail below.
4

---

Published as a conference paper at ICLR 2026
EEG Projection
Type specific Aligner
Aligned EEG Embedding
proj
Milli Reaie
Add Nom
Food fonard
Eyroj
fi
Faligned
fz
TJ eule Fuce
Shared Low-Rank Alignment
C Auditory Type
mute ' PPv
manaa pan
ft
April
Autinnst
pri
Aligned Audio Embedding
Audio
Projection
Figure 3: Overview of the proposed CALRA module. Given paired EEG and audio embeddings, CALRA performs audisory- alignment using multi-headed cross-attention and feed- forward networks. A shared low-rank bottleneck module further aligns the two modalities compact semantic space. The resulting aligned embeddings are used for contrastive pretraining, enabling robust cross-man moosl representation learning across diverse auditory conditions.
Type-specific Aligner. Given neural responses vary significantly for different types of auditory stimuli (e.g., speech vs. music), our aligner routes initial projections (Fiproj Aprj) through learnable transformation fa corresponding to the auditory type label k:
(Emmj Amoj) = f& (Egroj Apray)
(4)
This allows the model to adopt optimal alignment strategies for different auditory stimulus types.
Bi-directional Cross-Attention Following type-specific alignment, CALRA utilizes a bi- directional cross-attention mechanism enable each modality to dynamically integrate comple- mentary global context from the other. Unlike standard local token matching, operate the global projected embeddings to enforce holistic alignment. Given the projected global vectors Exroj E R and A/moj RIxD (where D denotes the alignment dimension and the temporal dimension is aggregated via global pooling), this exchange occurs simultaneously:
Audio-to-EEC Alignment: The EEG sequence (Qæ) retrieves relevant information from the audio sequence (KA, VA):
Ey interacted MultiHeeaAAttennionnQQ:::: KA. VA)
(5)
EEG-to-Audio Alignment: Symmetrically, the audio sequence (Q.A) retrieves neural fea- tures from the EEG sequence (K Væ))
A'meracted MultiHeadAttennion((.. KE. ViE)
(6)
Following standard practice in Transformer architecture Vaswani et al. (2017), residual connections and Layer Normalization are applied, yielding representations hg and h These are then passed to Shared Low-Rank Alignment module.
Shared Low-Rank Alignment. To enforce semantic consistency, we employ shared low-rank fusion mechanism. Unlike standard CLIP Radford et al. (202I) which relies on a shallow linear dot-product, we aim to capture the complex, non-linear dependencies between neural and acoustic features. By projecting hgg and hA into shared bottleneck and fusing them via an element-wise product (6), this module enforces bilinear interaction:
Exeedback = WD WDeeeg (Hsnared (Wu,eeg (hg) O WV,andio (ha)))
(7)
Afeedbuuk W audio (H (W Ueeg (hx) ( 2 W auddioo (h))))

---

Published as a conference paper at ICLR 2026
where Wu.,, WD. are modality-specific projection layers, and shareed is a shared non- non-linear layer. The final aligned embeddings are obtained by integrating this feedback via a residual connection:
Ealigned, Aaligned = LayerNorm(hh + Etssstack) ), LayerNorm(hA + Assetboc).
We chose this low-rank structure because it efficiently approximates computationally expensive ten- sor fusion operations Liu et al. (2018): Yu et al. (2017), enabling the model to capture rich multi- plicative feature interactions Fukui et al. Zadeh et al. al. (2017). This is theoretically superior to simple linear combinations for disentangling the intricate correlations between brain signals and au- ditory stimuli. It also need to note that, this multiplicative fusion architecture differs fundamentally from parameter-efficien strategies like LoRA Hu et al. (022), which employ low-rank matrices for additive weight adaptation rather than for modeling the joint distribution of multimodal features.
Finally, we distinguish our design from recent works like MGCA Wang et al. (2022) and CARZero Lai et al. (2024), which primarily focus on fine-gre ane-ggained local token matching or modifying the similarity scoring function ZZannn et al. (2025). In contrast, CALRA operates as pre-loss refinement step for global representations. By enhancing the embeddings themselves (Faligned; Altigned) rather than altering the loss mechanism, it preserves the training stability of standard contrastive learning while capturing deep, context-aware dependencies that simpler projections miss.
3.4 PRE-TRAINING VIA CONTRASTIVE ALIGNMENT
We optimize the MindMix framework using a contrastive learning objective, inspired by CLIP Rad- ford et al. (202T), the final aligned embeddings. The goal is to maximize the cosine similarity of true EEG-audio pairs while minimizing it for incorrect pairs within a mini-batch. This is framed as directed prediction problem where, for each EEG embedding, the model must identify its correct audio counterpart from all available options. This EEG-to to-audio direction directly mirrors our down- stream decoding tasks and avoids the potential instability of aligning single audio stimulus with its many possible neural responses. We use the InfoNCE loss, which is equivalent to cross-entropy loss over the similarity scores, averaged over all samples in the batch:
1 N
N
exp(sim( Eatsgned, Aatigned,i)/)) log
La.
(10)
exp(sim( (Ealigeeddd Autigned,s)
where sim(u, v) is the cosine similarity and is learnable temperature. Minimizing this objective jointly trains the entire framework (fEEG, fAudio. and CALRA), forcing the model to learn seman- tically rich embedding space where neural activity is meaningfully aligned with auditory content.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Tasks. Our experiments follow three-stage pipeline: unimodal tre-training, multi- modal alignment, and downstream task fine-tuning. For pre-training, we leverage large-scale public corpora, including over 3,500 hours of general EEG data and over 100 hours of paired EEG-audio data (summarized in Table and detailed Appendix 1. We evaluate MindMix's generalization capabilities a diverse set of downstream auditory decoding tasks, including Auditory Attention Decoding (on KUL, DTU, and ESAA), Emotion Analysis (on PME4 and HR-EEGGEMOO), and Mu- sic Retrieval (on MAD-EEG). To ensure a fair evaluation, all downstream task datasets were held out from unimodal pre- treitrainin and multimodal alignment stages. Standardized data preprocessing protocols are detailed in Appendix A.2. For reproducibility, specific implementation details and the full hyperparameter configurations for the EEG encoder, CALRA module, and optimization process as well our rigorous negative sampling policy (in Appendix A.3).
Evaluation Protocol To ensure fair and rigorous comparison with existing SOTA methods, we adopt the widely established standard evaluation protocols. For all downstream tasks, we conduct cross-
experiments using subject-specific (within-subject protocol, implemented via validation scheme. In this setup, the data for each subject are randomly partitioned into 5 folds, using 70%/10%/00% split for training, validation and testing within each fold. We report the

---

Published as a conference paper at ICLR 2026
Table 1: Overview of all datasets used across three training stages in our study.
Stage
Dataset
Hours
Channels
Modallty
Paradigm
Stage
Unimodal Pre-freining
BCI-IV-2A Tangermann ctal.
2012
13.4h
22
EEG Signal
Motor Imagery
Schirminn vier et mi
28.7h
133
EEG Signal
Motor Imagery
Openkant
91. 6h
62
EEG Signal
Moter Imagery
EBG Avi 23 al
2019
2.4h
20
EEG Signa]
Workload Analysis
TUEP
TUEPNEavo Navo et
2010
631.8h
14
EEG Signal
Epilepsy Detection
TUEV
148.7h
EEG Signal
Event Classifiationn
HMCSteeAAAva AMD
5W2.5h
HEG Signal
Sleep Detection
CAPSkcp Terzano e a.
2001
1004.5h
20
EEG Signal
Sleep Detection
CMBMIT Shoeb 2019
1060.9h
EEG Signal
Sleep Detection
Totul Hours:
3564.5h
Stage 2: Multimudel A Alignment Training
ds0004356 Singh al. 2024
38.9h
34
EEG Audio
MusiciSpeech Listening
zenodo, 45187754 Aomooman
Audio
Speech AAD
senodo. 10260015
Tharnee et
2
12.0h
2
BEG + Audio
Specch AAD
Breanan_201 2018 Breinan Hale
2019
10.1h
61
EEG Audio
Story Listening
Broderick 2015 [Bhdeenck et
2018
19.1h
128
Andio
Story Listening
Le. Petit Prince Monoenan ot
2020
17.3h
64
EEG + Audio
Story Listening
Total Hours:
109.0h
Stage Downstream Fine-funing
MADDEEGiCantian et al. 2019
4.2h
20
EEG e Audio
Music Retrieval
KUL Sl 2016
19.2h
64
EEG Audio
Speech AAD
DTU Fughang 2017
15.0h
EEG Audio
Speech
ESAA et 2022
12:7h
64
EEG Audio
Speech AAD
PME4 Cazn of 2022
4.6h
8
EEG Audio
Emotion Analysis
HR-ERG4FRDD
2019
10.0h
128
EEG Audio
Fimetion Analysis
mean and standard deviation across these 5 folds. Crucially, all reported results utilize raw window- level metrics (accuracy per 2-second segment) rather than aggregated trial-level scores, providing conservative and fine-grate -r-iiriined assessment of decoding performance. Evaluation metrics include Balanced Accuracy and Weighted Fl-score for AAD and emotion analysis, and standard Duo/Trio Accuracy for music retrieval. The detailed evaluation metrics can be found in Appendix A.4.
However, specifically regarding the Speech AAD task, we acknowledge that the mainstream within- subject splitting may introduce potential data leakage risks due to temporal correlations, high- lighted by Puffay et al. (2023). To address this, we additionally introduce a rigorous between-trial evaluation protocol, where training and testing segments are strictly drawn from disjoint trials (e.g.. different stories or sessions) to prevent temporal overlap and artifact leakage. The detailed results of this robust evaluation are provided in Appendix A.5.
4.2 DOWNSTREAM EXPERIMENT RESULTS
MindMix was evaluated against strong baselines, including task-specific SOTA models (e.g.. DBP- Net, AADNet) and powerful unimodal EEG foundation models (e.g., LaBraM, EEGPT). The de- tailed information about the compared baseline is provided in Appendix A.6. As shown in Table MindMix substantially outperforms all basclines across all downstream tasks. achieves near- perfect performance in Speech AAD (e.g.. 99.82% on KUL.) and establishes new SOTA in other tasks with large margins (e.g.. over 10 percentage points on PME4), underscoring the effectiveness of our multimodal strategy. A deeper analysis of these results reveals two critical findings. First, the unimodal EEG foundation models, such as LaBraM and CBraMod, consistently underperform when compared to task-specific SOTA models like DBPNet and DARNet. For instance, on the KUL dataset, LaBraM and CBraMod achieve accuracies of only 63.30% and 68.42%. respectively, falling far short of the 94.81% 11% achieved by DARNet. This exposes a key limitation of current foun- dation models: they are predominantly pre-trained on non-auditory tasks, rendering their generic representations suboptimal for decoding auditory perception. Furthermore, these large models are often highly sensitive to the data format and preprocessing pipelines; any mismatch with their orig- inal training configuration can lead to high performance variance and poor fine-tuning results (as empirically quantified in Appendix A.2).
Second, and more importantly, our results highlight a crucial distinction in the effectiveness of multimodal integration. For other task-specific multimodal models like MusicAAD (94.87% KUL) and AADNet (93.18% on KUL), the performance improvement over their strong unimodal counterparts is relatively modest. In contrast, the performance leap demonstrated by MindMix to
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison of MindMix against SOTA baselines on various downstream tasks and datasets. The best values are highlighted in bold, and the second-best values in each block are underlined. Based on the paired t-test with p-value correction (a 0.05), the indicates the marked method is significantly better than the compared methods.
Sperch
Method
KUI
DTU
ESAA
Balanced Acc.
Weighted B1
Balanced Acc.
Weighte F1
Dalanced Acc.
Weighed E1
BRONG
2018
0.7G14 + 0.007
0.7510
0.097
0.0112 + 0.012
0.657' + 0.068
0.742 + 0.132
0.7915 + 0.111
DBPN
0.2857 + 0.042
0.9588 = 0.056
0.825] + 0.003
0.857 # 0.066
08818 + 0.121
0.7900 +: 163
DARNe
0.0881 + 0.035
0.9367
= 0,022
0,8391 2: 0.045
0.868 # 0.086
1 + 0.054
0.9890 + 0.042
MuRcA
AT MENDE Fi EX01
0,3018 2 0.018
0.8487
00016
0.8156-
è 0/38
08877 8 0.002
11815 32 0.012
AADNN
0.7258 0.057
0.7083
0.051
000513
11 0057
07312 0.055
08237 0.001
0.815 0.058
BENDB
Kostas ctal
0.5877 = 0.045
0.0083
0.080
0.6886
0.082
0.0280
0.040
0.7963
0.078
0.5245
0,5038 2: 0.051
0.8041
10047
0,0233
0,029
0.6588 0.002
0.7888 0.077
0.7896
BBGPT
0.042
0.7083
00045
0.6538
0,008
0.6789 0,042
0.071
CRraMin
0.058
0.001
0.532
MindMix (Ours)
0.992 0.005
0.991
0.004
0.9993 0.009
0.9996 0.005
1,0000 0.000
1,0000 0.000"
Tusk
Emotion Anabsis
Music Retrieval
Muthoel
PME4
DR-ELGAEMO
MAD-EEG
Balanced Ave.
Weighted P1
Balanced Acc.
Waighted F1
DLo Ace.
Tho Acc.
BBGNe Lawheme cral.
0.085
0.4920
D0045
0.6981 0.311
0.7081
0.071
0.025
DBFNe erere
0.082
0.5021
00053
0.8274 à 0.073
0.8458
0.004
0.7849 0.091
0.7152
DARNE TI
0.5725 + 0.025
0.5425
0.06.
0.081
081788
0.077
0.7544 0.080
0.7155
0.092
0,6345
10075
0,7648
0088
0.7822 0,009
0.022 TU8624 0.007
0.8722
0.033
AADNe er
2 0.077
00005
0,7344
00059
0,7832 0,004
0,8010 0L055
BENDR
clal.
0.5433 & 0.083
0,5218
10000
0UG45S
0.015
0.6836
0.017
0.348
0:6498
0J845
BIT
0.3224
0.071
10009
00225
0.6EST 0.019
BUG655 0.002
0.8798
0.049
BEGPT
0.3000
0.058
0.545
0.06L
0,7129
D002
0.7918
017887
0.0055
0.7582
0.008
Laltm
0.3818
01056
0,5095
0.052
0U72855
00882
0,7820
0.011
0.7582
0.082
0.72220
0.078
OHeaMiy 2075
0,0052
+ 0.072
0.5841
0.08%
037385
007S
0L7748 0.675
0.000 0.801
0.7051
0.057
MindMix (Ours)
6.7256 1 0.125
0.7089 1. 0.135
0.8678 0,045
0,8809 2: 9,046
0,9415
0.025
0.8824 2: 0.042"
99.82% is dramatic, which directly validates our central hypothesis: deep cross-modal alignment is paramount, and simply combining modalities is not enough. This substantial lead in performance underscores the efficacy of MindMix's architectural design, which integrates powerful modality- specific encoders with an effective alignment strategy. Our framework enables deep interaction to capture the fine-graine relationship between brain activity and complex audio signals, which is crucial for robust neural decoding.
4.3 ABLATION STUDY AND ANALYSIS
To validate our architectural choices and quantify the contribution of each component, we perform comprchensive ablation studies, the results of which are summarized in Table
Effectiveness of the CALRA Module. We first investigate the importance of our core contri- bution, CALRA, by comparing it against simpler alignment strategies. As shown in Table 3. re- placing CALRA with standard co-attention block or reverting to a simple CLIP-style projection ('w/o Alignment") leads to substantial performance degradation. Besides, to rigorously validate the structural advantage of our bilinear fusion, further compare against "Standard Concatenation- based Fusion (Concat-MLP)" baseline, which is the dominant strategy for vector-level integration. CALRA consistently outperforms this strong baseline (e.g., 0.8878 vs. 0.8574 on EEG4EMO). This empirical evidence confirms that the multiplicative interaction within CALRA captures complex cross-modal dependencies that simple concatenation cannot effectively model.
While Table 2 reports standard metrics for fair comparison, we further validated our model's ro- bustness using the strict between-trial protocol defined in Section 4. As detailed in Appendix A.5, MindMix maintains substantial and leading performance advantage under this challenging setting, although the absolute accuracy is lower as expected. This confirms that the model's superiority stems from genuine accrutccoustic alignment rather than the exploitation of trial-specific artifacts.
Impact of Modality Encoders. To validate our encoder choice, we substituted our specialized EEG encoder with several alternatives. Both SOTA foundation models (LaBraM and CBraMod) and the classic EEGNet resulted in significant performance drop, confirming the advantage of our custom pre-training strategy. Specifically, even when adapting the strong CBraMod backbone with our alignment module, the performance (96.37% on KUL.) still falls short of our full MindMix
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation studies on the main components of MindMix. All experiments are evaluated on two downstream tasks: emotion recognition on IR-EEG4EMO and auditory attention decoding on KUL. Note: For the "w/ LaBraM" and "w/ CBraMod" entries, we initialized the EEG encoder using their official pretrained weights and subjected them to our full multimodal alignment training before fine-tuning, ensuring a rigorous comparison of backbone capabilities.
Model Configuration
EEG Encoder
Audio Encoder
Aligument Module
Emotion Acc.
AAD Acc.
MindMix (Full Model)
Ours
Wav2Vec 2.0
CALRA (Ours)
0.8877 1 0.945
0,9962 0.006
Ablation Aligument
Co-Ateenion
Wer
Co-Altettonn
0.8629 0.055
0.0785
0.021
wf Cancat-MLPP
Our
WareVee War2Vee 2.0
Concat-MLP
0.8574 :: 0.035
0.0503 0.017
w/o Aligument
Ours
Standard CLIP
0.8483 1: 0.038
119653 = 0.015
Ablation on EEG Encoder
w/ LaRraM
LaBraM
Wav2Vee 2.0
CALRA (Chus)
0,8588 + 0.044
0.9744 + 0.012
w/ EEGNet
EDGNet
Wee 2.0
CALRA (Ouns)
0.8555 1 0.047
0.0 1 0.011
CBrrMoo
CBraMod
WayZVee
(Ouns)
0.8642 0.030
0.9637 0.010
Ablation w/ HuBERT
Encoder
HORERT
CALRA (Ouns)
0.8687 + 0.037
0.8893 0.010
w/ Mel-specteoguum
Curs
Mel-spectrogam
CALRA (Chrs)
0.8432 + 0.035
0.9448 + 0.016
Dissection the CALRA
who Type-specific Aligner
Ours
War2Wee 2.0
CALRA (Chrs)
0.8676 + 0.333
0,9853 + 0.010
Shared Low-Kank
War2Vee WavZVee
CALRA (Ours)
0.8557
1197722 0.012
Choss Artse-Aiteetionn
CALRA (Curs)
0.0435 0.013
model (99.82%). For the audio stream, substituting the powerful Wav2Vec 2.0 with traditional Mel- spectrogram features causes steep decline of up to 5.45% in AAD accuracy. This highlights that rich, pre-trained representations essential for both the neural and acoustic modalities.
Dissection of CALRA's Components. Finally, we dissect the CALRA module to quantify the contribution of its three key innovations. The bi-directional cross-attention mechanism proves to be the most critical element, as its removal ('wo Cross-Attention") causes the largest performance drop (up to 5.58% in AAD). The shared low-rank alignment also provides a vital contribution, with its removal ('wlo Shared Low-Rank') leading to significant drop. The w/o Type-specific Aligner ablation, which simulates the absence of auditory type information at test time, causes only a minor performance drop. This indicates that while the type-specific routing is beneficial, our model does not critically rely on it and remains highly effective even when stimulus type is unknown. Together, these results confirm that all three components of CALRA are integral to its success, working synergistically to achieve superior cross-moda. alignment.
Finally, we also investigated the trade-off between decoding accuracy and temporal resolution (win- dow size sensitivity), these additional results and detailed analyses are provided in Appendix A.7.
4.4 QUANTIFYING THE SYNERGY OF MULTIMODAL ALIGNMENT
To isolate and quantify the benefit of our multimodal approach, we conduct a critical analysis com- paring the full MindMix model against its EEG-only counterpart. The results, presented in Figure are striking and reveal deep synergy fostered by the alignment of brain signals and audio. No- tably, even this EEG-only counterpart is highly competitive its own, demonstrating performance comparable to the SOTA unimodal baselines reported in Table 2 For instance, it outperforms the LaBraM baseline on the ESAA and MAD-EEG tasks. Furthermore, to validate its broader gen- cralization capabilities, we benchmarked the encoder on standard non-auditory tasks (TUAB and BCIC-IV-2B). As detailed in Appendix A.8, our model achieves top-tier performance (e.g.. ranking 1st on BCIC-IV-2B among foundation models), confirming its robustness as a general-purpose EEG encoder. This demonstrates that MindMix's 's success stems not just from learning robust EEG repre- sentations but from learning the relationship between the neural signal and the auditory stimulus.
4.5 NEUROSCIENTIFIC INTERPRETATION OF CROSS-MODAL ALIGNMENT
To provide comprehensive assessment of the learned representations and validate their biologi- cal plausibility, we employ the "Stimulus Reconstruction" method Mesgarani & Chang 2012). a foundational approach in auditory neuroscience to quantify neural encoding. We adopt the pseudo- reconstruction framework from Défossez et al. (2023) to reconstruct audio Mel spectrograms from the aligned EEG embeddings (Faligee)) analyzing the results both qualitatively and quantitatively.

---

Published as a conference paper at ICLR 2026
10
Pleformence (Acoracy NC
Via
GAA
HMM
Speach AAD
Emotoo Anatydis
Nunko Rebleva
Figure 4: This figure compares the performance the full MindMix model with its unimodal (EEG- Only) counterpart to isolate the performance gain from our cross- cross-modal alignment strategy.
Thue Spectrogram
DTU
KUL 1b
II
Hc
Visualization of Mel Spectrogram Reconstruction
(b) Average Attention Weights KUL dataset
Figure 5: Neuroscientific Interpretability Analysis. (a) Mel Spectrogram Reconstruction. (b) Spatial Attention Topography.
As illustrated in Figure a MindMix reconstructions faithfully capture fine-grained harmonic structures. whereas these details are blurred in the LaBraM variant and the baseline method. Quanti- tatively, MindMix achieves PCC scores of 0.88 (DTU) and 0.91 (KUL), substantially outperforming baselines (e.g.. 0.67 and 0.61). Crucially, this high fidelity provides direct evidence that MindMix successfully encodes the spectro-temporal receptive fields of the auditory cortex, mapping neural activity accurately back to acoustic features.
To further investigate the physiological basis of these reconstructions, we visualized the spatial attention weights of the EEG encoder. shown Figure the model exhibits a distinct, high- intensity activation cluster in the left temporal region. This distribution is neuroscientifically signif- icant: it corresponds precisely to the primary auditory cortex and aligns with the well-established left- heliimeeisheee lateralization for speech processing Défossez al. (2023). The absence of high weights in the frontal pole further confirms that the model prioritizes genuine neural signatures over ocular artifacts.
CONCLUSION
This paper presents the first large-scale investigation into multimodal auditory brain decoding ing paired EEG and audio data. We introduce MindMix, a novel foundation model featuring our CALRA module. which enables deep aligmment between neural signals and sound. Our extensive experiments demonstrate that MindMix consistently and significantly outperforms SOTA basclines across a diverse set of downstream tasks, establishing a new and robust benchmark for the field. By successfully learning generalizable representations, this work significantly advances the capabilities of non-invasive BCIs and lays a critical foundation for understanding the interplay between neural and audio signals. While our results highlight the immense potential of this approach, we also un- derscore that the current scarcity of large-scale paired EEG-audio corpora is a primary bottleneck for the field, precluding full investigation into the scaling laws of such foundation models. Future research will focus on scaling the MindMix framework to leverage increasingly larger datasets, with the goal of further advancing this field.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was partially supported by the National Natural Science Foundation China (Grant No. 62306259 and 62572413), the Research Grants Council of the Hong Kong SAR (Grant No. C5052-23G, PolyU25216423, Poly114422 PolyU15217424, and SRFS2526-5504) The Hong Kong Polytechnic University (P0058445).
ETHIC STATEMENT
This work adheres the ICLR Code Ethics. In this study, no human subjects or animal expreci- sions were included. All data sets used were sourced in accordance with relevant usage guidelines, to ensure no privacy violation. We have taken care not to achieve bias discriminatory results in our research process. No personally identifiable information was used and no experiments were con- ducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. All code and model have been made publicly available in an anonymous repository to facilitate replication and verification, the experimental setup, including training steps, model configurations, and hardware details, described in detail in the paper, We have also provided full description of implementation description, assist others in reproducing
experiments.
Additionally, all EEG datasets used in the paper, such as KUL, DUT, and ESAA etc, are publicly available, ensuring consistent and reproducible evaluation results.
We believe these measures will allow other researchers to reproduce our work and further advance the field.
REFERENCES
Rijsman Roselyne M Alvarez, Diego. Inter-database validation of deep learning approach for automatic sleep scoring. PloS one, 16(8):e0256111 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances Neural Information Processing Systems, 33: 2e::12444-114466 2020.
Hanna Becker, Julien Fleureau, Philippe Guillotel, Fabrice Wendling, Isabelle Merlet, and Laurent Albera. Emotion recognition based on high-resolution eeg recordings and reconstructed brain sources. IEEE Transactions on Affective Computing, 11(2):24-2577, 2017.
Bonetti, Fernández-Rubic Carlomagno, M Dietz, D Pantazis, P Vuust, and ML Kringelbach. Spatiotemporal brain hierarchies of auditory memory recognition and predictive coding. Nature Communications, 15(1):3313, 2024.
Jonathan R Brennan and John Hale. Hierarchical structure guides rapid linguistic predictions during naturalistic listening. PloS One, 14(()=e0207741, 2019.
Michael Broderick, Andrew J Anderson, Giovanni M Di Liberto, Michael J Crosse, and Edmund C Lalor. Electrophysiological correlates of semantic dissimilarity reflect the comprehension of nat- ural, narrative speech. Current Biology. 28(5):803-809, 2018.
Giorgia Cantisani, Gabriel Trégoat, Slim Essid, and Gaël Richard. Mad-eeg: an eeg dataset for decoding auditory attention to a target instrument in polyphonic music. In Speech, Music and Mind (SMM), Satellite Workshop of Interspeech 2019, 2019.
Jin Chen, Tony Ro, and Zhigang Zhu. Emotion recognition with audio, video, eeg, and emg: a dataset and baseline approaches. IEEE Access, 10: 10229-2--13222, 2022.

---

Published as a conference paper at ICLR 2026
Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PmLR, 2020.
Xupeng Chen, Ran Wang, Amirhossein Khalilian-Gourtann, Leyao Yu, Patricia Dugan, Daniel Fried- man, Werner Doyle, Orrin Devinsky, Yao Wang, and Adeen Flinker. A neural speech decoding framework leveraging deep learning and speech synthesis. Nature Machine Intelligence, 6(4): 467-480, 2024a.
Zhige Chen, Rui Yang, Mengjie Huang, Fumin Li, Guoping Lu, and Zidong Wang. Eegprogress: A fast and lightweight progressive convolution architecture for eeg classification. Computers in Biology and Medicine, 107900 2024b.
Zhige Chen, Rui Yang. Mengjie Huang, Zidong Wang, and Xiaohui Liu. Electrode domain adapta- tion network: Minimizing the difference across electrodes in single-source to single-target motor imagery classification. IEEE Transactions on Emerging Topics in Computational Intelligence, (2):1994-2008, 2024c.
Zhige Chen, Chengxuan Qin, Wenlong You, Rui Liu, Congying Chu, Rui Yang, Kay Chen Tan, and Jibin Wu. Hear: An eeg foundation model with heterogeneous electrode adaptive representation. arXiv preprint arXiv:2510.122 12515, 2025.
Michael J Crosse, Giovanni M Di Liberto. Adam Bednar, and Edmund C Lalor. The multivariate temporal response function (mtrf) toolbox: matlab toolbox for relating neural signals to contin- uous stimuli. Frontiers in Human Neuroscience, 10:604, 2016.
Wenhui Cui, Woojae Jeong, Philiipp Thölke, Takfarinas Medani, Karim Jerbi, Anand A Joshi, and Richard M Leahy. Neuro-gpt: Towards a foundation model for eeg. In Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI), 2024. doi: 10. 10.1109/1BB15557022224 abs///bs///// 211.
10635453. URL WRRL/httts///arxxtt wwtp://.ggrb/bb/////o/oo//s
Simon Dahan, Gabriel Bénédict, Logan Zane John Williams, Yourong Guo, Daniel Rueckert, Robert Leech, and Emma Claire Robinson. Sim: Surface-based fmri analysis for inter-subject multi- modal decoding from movie-watching experiments. The Thirteenth International Conference on Learning Representutions, 2025.
Neetha Das, Wouter Biesmans, Alexander Bertrand, and Tom Francart. The effect of head-related filtering and ear-specific decoding bias on auditory attention detection. Journal of Neural Engi- neering. (3550005 2016.
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Réan-Rémi King. De- coding speech perception from non-invasive brain recordings. Nature Machine Intelligence, (10):1097-1107, 2023.
Xin Deng, Danni Li, Jianxun Mi, Fengxing Gao, Qiaosong Chen, Jin Wang, and Rui Liu, Motor imagery ecog signal classification using sparse representation with elastic net constraint. In 2018 IEEE 7th Data Driven Control and Learning Systems Conference (DDCLS), pp. 44-49, IEEE, 2018.
Matteo Ferrante, Tommaso Boccato, and Nicola Toschi. Towards neural foundation models for vision: Aligning eeg, meg and fmri representations to perform decoding, encoding and modality conversion. In ICLR 2024 Workshop on Representational Alignment, 2024.
Søren Asp Fuglsang, Torsten Dau, and Jens Hjortkjær. Noise-robust cortical tracking of attended speech in real-world acoustic scenes. Neurolmage, 156:435-444, 2017.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https:://
api. www.seeaeiiss..e. www.tticccchl.....oo www.o///ooooo. wwwp///r////ooooooo (CoppusID:22401999
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. The Tenth International Conference Learning Representations, 1(2):3, 2022.
12

---

Published as a conference paper at ICLR 2026
Yongquan Hu, Shuning Zhang, Ting Dang, Hong Jia, Flora D Salim, Wen Hu, and Aaron J Quigley. Exploring large-scale language models to evaluate eeg-based multimodal data for mental health. In Companion the 2024 on ACM International Joint Conference Pervasive and Ubiquitous Computing, pp. 412-417, 2024.
Weibang Jiang, Liming Zhao, and Bao liang Lu. Large brain model for learning generic represen- tations with tremendous EEG data in BCI. In The Twelfth International Conference Learning Representations, 2024. URL https: httpp:://ppeneeiee...ee htt.pppawaaaaaa.e.a..aooaoaaaaaa
Demetres Kostas, Stephane Aroca- teeeleeel,,,te, and Frank Rudzicz. Bendr: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15:653659, 2021.
Marie Kunešová, Zhyněk Zajfe, Luboš Šmídl, and Martin Karafiát. Comparison of wav2vee 2.0 models on three speech processing tasks. International Journal of Speech Technology, 27(4): 847-859, 2024.
Haoran Lai, Qingsong Yao, Zihang Jiang, Rongsheng Wang. Zhiyang He, Xiaodong Tao, and S Kevin Zhou. Carzero: Cross-attention alignment for radiology zero-shot classification. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11137- 11146, 2024.
Vernon J Lawhern, Amelia Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent Lance. Eegnet: compact convolutional neural network for eeg-based brain-computer interfaces. Journal of Neural Engineering, 15(5):056113, 2018.
Min-Ho Lee, Y-Yeon Kwon, Yong-Jeong Kim, Hong-Kyung Kim, Young-Eun Lee, John Williamson, Siamac Fazli, and Seong- Whan Lee. Eeg dataset and openbmi toolbox for three bei paradigms: An investigation into bci illiteracy. GigaScience, 8(5):iiz022, 2019.
Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, and Quanying Liu. Visual decoding and recon- struction via eeg embeddings with guided diffusion. In Advances Neural Information Process- ing Systems, volume 37, pp. 00822-112266 2024.
Peiwen Li, Enze Su, Jia Li, Siqi Cai, Longhan Xie, and Haizhou Li. Esaa: An eeg-speech auditory attention detection database. 2022 25th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Tech- niques (O-COCOSDA), pp. 1-6. IEEE, 2022.
Yuanning Li, Gopala K Anumanchipalli, Abdelrahman Mohamed, Peili Chen, Laurel H Carney, Junfeng Lu, Jinsong Wu, and Edward F Chang. Dissecting neural computations in the human auditory pathway using deep neural networks for speech. Nature Neuroscience, 26(12):2213- 2225, 2023.
Rui Liu, Zhi-An Huang, Yao Hu, Zexuan Zhu, Ka-Chun Wong, and Kay Chen Tan. Spatial-temporal co-attention learning for diagnosis of mental disorders from resting-state fmri data. IEEE Trans- actions on Neural Nenworks and Learning Systems,
2023.
Rui Liu, Yao Hu, Jibin Wu, Ka-Chun Wong, Zhi-An Huang, Yu-An Huang, and Kay Chen Tan. Dynamic graph representation learning for spatio-temporal neuroimaging analysis. IEEE Trans- actions on Cybernetics, 2025.
Xuan-Hao Liu, Yai Liu, Yansen Wang, Kan Ren, Hanwen Shi, Zilong Wang, Dongsheng Li, Bao-Liang Lu, and Wei-Long Zheng. Eeg2video: Towards decoding dynamic visual perception from eeg signals. Advances in Neural Information Processing Systems, 37:72245-72273, 2024.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshninarasimhan, Paul Pu Liang, Amir Zadeh, and Louis philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. ArXiv, Abs/1806.00064 2018. URL https: /tttps://ap.... http:://aa..semantisssss.s hor.cholrooooooo
CorpusID: 44131945
Mohammad H Mahrooz, Farrokh Fattahzadeh, and Shahriar Gharibzadeh. Decoding the debate: a comparative study of brain-computer interface and neurofeedback. Applied Psychophysiology and Biofeedback, 49(1):47-55 2024.
13

---

Published as a conference paper at ICLR 2026
Mackenzie Weygandt Mathis, Adriana Perez Rotondo, Edward F Chang, Andreas S Tolias, and Alexander Mathis. Decoding the brain: From neural representations to mechanistic models. Cell, (57(21):51145833) 2024.
Nima Mesgarani and Edward F. Chang. Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7397) 2012.
Mohammad Momenian, Zhengwu Ma. Shuyi Wu, Chengcheng Wang, Jonathan Brennan, John Hale, Lars Meyer, and Jixing Li. Le petit prince hong kong (Ipphk): Naturalistic fmri and eeg data from older cantonese speakers. Scientific Data, 11(1):992. 2024.
Abhijith Mundanad, Rob Zink, and Alexander Bertrand. Ultra high-density 255-channel ceg-aad dataset, February 2021. URL
5201//22//... 4518754
Nhan Duc Thanh Nguyen, Huy Phan, Simon Geimaert, Kaare Mikkelsen, and Preben Kidmose. Aadnet: an end-to-end deep learning model for auditory attention decoding. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2025.
Qinke Ni, Hongyu Zhang, Cunhang Fan, Shengbing Pei, Chang Zhou, and Zhao Lv. Dbpnet: Dual- branch parallel network with cemporal-frequeene fusion for auditory attention detection. Pro- ceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2024), 2024.
Yixiang Niu, Ning Chen, Hongqing Zhu, Jing Jin, and Guangqiang Li. Music-oriented auditory attention detection from clectroencephalogram. Neuroscience Letters, 818:137534 2024.
Collins Opoku-Baahe Baah, Adriana M Schoenhaut, Sarah G Vassall, David A Tovar, Ramnarayan Ra- machandran, and Mark T Wallace. Visual influences on auditory behavioral, neural, and per- ceptual processes: review. Journal of the Association for Research in Otolaryngology, 22(4): 365-386, 2021.
A. O'Sullivan, A. Power, N. Mesgarani, S. Rajaram, J. J. Foxe, B. G. Shinn-Cunningham, M. Slaney, S. A. Shamma, and E. C. Lalor. Attentional selection in cocktail party environment can decoded from single-trial eeg. Cerebral Cortex, 25(7):1697-1766, 2015.
Andrew J Oxenham and Heather A Kreft. Speech masking in normal and impaired hearing: Inter- actions between frequency selectivity and inherent temporal fluctuations in noise. In Physiology, Psychoacousties and Cognition in Normal and Impaired Hearing. pp. 125-132. Springer Interna- tional Publishing, 2016
Maria Carla Piastra, Andreas Nüßing, Johannes Vorwerk, Maureen Clere, Christian Engwer, and Carsten Wolters. comprehensive study on electroencephalograpyy and magnetoencephalog- raphy sensitivity to cortical and subcortical sources. Human Bruin Mapping, 42(4))978-992, 2021.
Martyna Poziomska, Marian Dovgialo, Przemystaw Olbratowski, Pawel Niedbalski, Pawel Og- niewski, Joanna Zych, Jacek Rogala, and Jaroslaw Zygierewicz. Quantity versus diversity: Influence of data on detecting eeg pathology with advanced ml models. arXiv preprint arXiv:2411.17709 2024.
Corentin Puffay, Bernd Accou, Lies Bollens, Mohammad Jalilpour Monesi, Jonas Vanthornhout, Tom Francart, et al. Relating eeg to continuous speech using deep neural networks: a review. Journal of Neural Engineering, 20(4):041003, 2023.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. International Conference Machine Learning (ICML), 2021.
Robin Tibor Schirmeister, Jost Tobias Springenberg. Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, (8(11))5331-544 2017.
Tong Shan, Madeline S. Cappelloni, and Ross K. Maddox. Subcortical responses to music and speech are alike while cortical responses diverge, 2024.
14

---

Published as a conference paper at ICLR 2026
Ali Hossam Shoeb. Application of machine learning to epileptic seizure onset detection and treat- ment. PhD thesis, Massachusetts Institute Technology, 2009.
Prajwal Singh, Dwip Dalal, Gautam Vashishtha, Krishna Miyapuram, and Shanmuganathan Raman. Learning robust deep visual representations from eeg brain recordings. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7553-7562, 2024.
Enze Su, Siqi Cai, Longhan Xie, Haizhou Li, and Tanja Schultz. Stanet: A spatiotemporal attention network for decoding auditory spatial attention from eeg. JEEE Transactions on Biomedical Engineering, 69(7):2233-22) 2242, 2022.
Michael Tangermann, Robertt Müller, Ad Aertsen, Niels Birbaumer, Christoph Braun, Clemens Brunner, Robert Leeb, Carsten Mehring, Kai J Miller, Gernot R Müller-Putz, et al. Review of the bei competition iv. Frontiers in Neuroscience, 6:55, 2012.
Mario Giovanni Terzano, Liborio Parrino, Adriano Sherieri, Ronald Chervin, Sudhansu Chokroverty, Christian Guilleminault, Max Hirshkowitz, Mark Mahowald, Harvey Moldofsky, Agostino Rosa, et al. Atlas, rules, and recording techniques for the scoring of cyclic alternating pattern (cap) in human sleep. Sleep Medicine, 2(6):537-553, 2001.
Michael Thornton, Tobias Reichenbach, and Danilo Mandic. emprriil ear-eeg dataset for au- ditory attention decoding, December 2023. URL
10260032
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30, 2017.
I. Veloso, J McHugh, Eva von Weltin, Sebas Lopez, I Obeid, and Joseph Picone. Big data resources for eegs: Enabling deep learning research. In 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1-3. IEEE, 2017.
Christopher Wang, Adam Yaari, Aaditya Singh, Vighnesh Subramaniam, Dana Rosenfarb, Jan De- Witt, Pranav Misra, Joseph Madsen, Scellig Stone, Gabriel Kreiman, et al. Brain treebank: Large- scale intracranial recordings from naturalistic language stimuli. Advances in Neural Information Processing Systems, 37:96505-99555 2024a.
Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi-gramulan cross-modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems, 35:33536-335-5 2022.
Guangyu Wang, Wenchao Liu, Yuhong He, Cong Xu, Lin Ma, and Haifeng Li. Eegpt: Pretrained transformer for universal and reliable representation of ceg signals. Advances in Neural Informa- fion Processing Systems, 37:39249 -92880, 2024b.
Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li, and Gang Pan. Cbramod: A criss-cross brain foundation model for eeg decoding. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba. A fine-tuned wav2vec 2. 2.0/hu- bert benchmark for speech emotion recognition, speaker verification and spoken language under- standing. arXiv preprint arXiv:21 1,02735, 2021.
Ruofan Yan, Shu Peng, Zhige Chen, Zhi-An Huang, Rui Liu, Kay Chen Tan, and Jibin Wu. Enhanc- ing spatio-temporal auditory attention decoding with st-aadnet. In 2024 IEEE 14th International Symposium Chinese Spoken Language Processing (ISCSLP), pp. 338-338. IEEE, 2024a.
Sheng Yan, Cunhang Fan, Hongyu Zhang, Xiaoke Yang, Jianhua Tao, and Zhao Lv. Damet: Dual attention refinement network with spatiotemporal construction for auditory attention detection. Advances in Neural Information Processing Systems, 37:31688-31707, 2024b.
Chaoqi Yang, M Westover, and Jimeng Sun. Biot: Biosignal transformer for cross- data learning in the wild. Advances in Neural Information Processing Systems, 36:78240-78260, 2023.
15

---

Published as a conference paper at ICLR 2026
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. 2017 IEEE International Conference on Computer Viston (ICCV), pp. 1839-1848, 2017. URL https:/ aattss:/app.... www.pp...mtttnccoooli.oooo org/Corpus ID: 3005731
Amir Zadeh, Minghai Chen, Soujanya Poria, E. Cambria, and Louis philippe Morency. Tensor fu- sion network for multimodal sentiment analysis. In Conference on Empirical Methods in Natural Language Processing, 2017. URL //api semant iommantiisse cscho ar. org/ www.o//rrrrsrpss...oooooo 950292
Shimin Zhang, Ziyuan Ye, Yinsong Yan, Zeyang Song, Yujie Wu, and Jibin Wu. Koopstd: Reli- able similarity analysis between dynamical systems via approximating koopman spectrum with timescale decoupling. In International Conference on Machine Learning, 2025.
Igor Zyma, Sergii Tukaev, Ivan Seleznov, Ken Kiyono, Anton Popov, Mariia Chernykh, and Oleksii Shpenkov. Electroenceehallorrms during mental arithmetic task performance. Data, 4(1):14, 2019.
A APPENDIX
A.1 DATASET DESCRIPTION
Stage 1: Unimodal Pre-training Corpus. For the initial unimodal pre-training stage, designed to build robust EEG encoder, assembled large-scale and diverse corpus totaling over 3,500 hours of EEG data from nine public datasets. These datasets span a wide range of BCI paradigms and clinical applications, ensuring that the learned representations generalizable and not biased towards a specific task. The collection includes:
BCI-IV-2A Tangermann et al. 2012 and HGD Schirrmeister et al. (2017) Widely used motor imagery datasets.
OpenBMI Lee al. 2019): A large-scale dataset for various BCI paradigms.
EEGMat Zyma et al.
2019): : A benchmark for mental workload analysis.
TUEP and TUEV Veloso et al. (017): The Temple University Hospital EEG Corpus,
containing extensive clinical data for epilepsy and event detection.
HMCSleep Alvarez (021). CAPSleep Terzano et al. (0001), and CMBMIT Shoeb
(2009): Large public corpora for sleep stage analysis.
The unimodal pre-training of our EBG encoder utilizes over 3,500 hours of data, establishing sub- stantial foundation in terms of scale. For context, this exceeds the pre-training corpus of several notable EEG foundation models, such as LaBraM ( 2,500 hours) and EEGPT ( 200 hours), under- scoring the large-scale nature of our unimodal representation learning.
Stage 2: Multimodal Alignment Corpus. In the second stage, we focused on learning the crucial alignment between neural and acoustic representations. For this, curated multimodal corpus of over 100 hours of paired EEG and audio data from seven distinct datasets. This corpus features diverse auditory stimuli, including music, attended speech, and naturalistic story listening. The datasets include:
ds004356 Shan et al. (2024) and zenodo_4 zenodo_451874 451875441 M7nddaad et al. (001):: : Publicly avail- able datasets featuring subjects listening to music or competing speech streams in AAD paradigms.
zenodo 10260082 Thortton et al. AAD tasks.
2023 : An additional EEG-audio dataset for speech
Brennan 2018 Brennan & Hale 2019 9 Broderick 2018 Broderick al. (2018), and Le
Petit Prince Momenian et al. (2024): Datasets containing EEG recordings subjects
listening to naturalistic stories, providing rich, continuous audio stimuli.
16

---

Published as a conference paper at ICLR 2026
Stage 3: Downstream Task Datasets.
We evaluate model three distinct task families:
Auditory Attention Decoding (AAD): Identifying which of two competing speech streams person is attending to. Datasets: KUL Das et al. (2016), DTU Fuglsang et al.
(2017), and ESAA Li et al. (2022).
Emotion Analysis: Recognizing emotional states from EEG while listening to affective stimuli. Datasets: PME4 Chen et al. (2022) and MA-EEGG4MM et al. (2017).
Music Retrieval: A cross-modal task designed to test the ability to identify the correct mu- sic piece corresponding to an EEG segment. Dataset: MAD-EEG Cantisani et al. (2019).
We strictly enforced subject independence across all three stages of our training pipeline to prevent any form of data leakage. The subjects in the downstream test sets (e.g., KUL, DTU, ESAA) were never seen during the Unimodal Pre-training (Stage 1) or Multimodal Alignment (Stage 2) stages. The datasets used in Stages and 2 contain entirely distinct participant cohorts from those in the downstream evaluation datasets.
A.2 DATA PREPROCESSING
A standardized preprocessing pipeline applied across all datasets for consistency. The specific steps and justifications are detailed below:
EEG Data Preprocessing:
Filtering: Raw EEG signals were mandpass-filteree between 1.0 Hz and 40.0 Hz using a zero-phase
4th-order Butterworth filter to isolate neural activity in relevant frequency bands and reduce high-frequency noise and slow drifts.
Resampling: The filtered signals were then downsampled to a uniform sampling rate of 200 Hz.
Epoching: Continuous data were segmented into non-overlapping 2-second epochs. Normalization: Each 2-second epoch was normalized independently on a per- channel basis using z-score standardization (subtracting the epoch's 's mean and dividing by its standard deviation). This segment-wise normalization prevents any data leakage between training and test sets.
Artifact Handling: We relied on the dataset-level denoising (e.g., EOG/EMG artifact removal) provided by the original authors of the public datasets we used (see Table 1) and did not apply additional artifact removal algorithms (e.g., ICA).
Bad- Madd-Chnnel Policy: We used all channels provided in the original datasets. Our channel-indeperne
patching strategy combined with learnable spatial embeddings allows the model handle heterogeneous channel configurations robustly, without the need for explicit bad-channel detection or interpolation.
Audio Data Preprocessing:
Resampling: Raw audio waveforms were resampled to 16 kHz (mono).
- Epoching & Normalization: To match the EEG segments, audio was segmented into 2-second epochs. Each audio epoch was normalized by its peak absolute value to ensure consistent amplitude scale.
We intentionally chose the 1-40 Hz bandpass to isolate relevant neural components. As shown in Table A1 applying this pipeline to foundation models pretrained on different distributions causes significant performance drop. Thus, to ensure rigorous comparison:
Justification for Pipeline:
Foundation Model Baselines (e.g., LaBraM, CBraMod) were fine-tuned using their native preprocessing pipelines (e.g., 0.1-70 Hz for LaBraM, 0.3-75 Hz for CBraMod) to avoid unfair penalties from distribution shift.
MindMix and Task-Specific Models (e.g., DARNet, DBPNet), which are trained from scratch or adapted to our domain, utilized our standardized 1-40 Hz pipeline.
17

---

Published as a conference paper at ICLR 2026
Table Al: Pipeline Sensitivity Analysis. Comparison of bascline performance when fine-tuned using our preprocessing pipeline (1-40 Hz) versus their native pipelines. The drop in performance using our pipeline confirms the distribution shift, justifying our choice use native pipelines for baselines in the main comparison.
Model
Dataset
Acc. (Our Pipeline)
Acc. (Native Pipeline)
Task: Emotion Analysis (Metric: Accuracy)
BENDR BIOT
EEG4EMO
0.6051 + 0.019
0.6458 0.015
EEG4EMO
0.5615 + 0.027
0.6352 0.023
EEGPT CBraMod
EEG4EMO
0.6527 = 0.064
0.7129 = 0.072
EEG4EMO
0.6680 +: 0.083
0.7295 + 0.082
LaBraM
BEG4EMO
0.6701 + 0.085
0.7285 + 0.078
Task: Music Retrieval (Metric: Duo Accuracy)
BENDR
MAD-EEG
0.5844 + 0.054
0.6235 + 0.048
BIOT
MAD-EEG
0.6042 + 0.042
0.6485 + 0.052
EEGPT
MAD-EEG
0.7311 + 0.074
0.7887 + 0.065
CBraMod
MAD-EEG
0.7007 = 0.075
0.7582 0.082
LaBraM
MAD-EB
0.7554 + 0.066
0.8011 + 0.069
A.3 IMPLEMENTATION DETAILS
Implementation Details. All experiments were conducted in PyTorch on a cluster of 8 NVIDIA A6000 GPUs We used the Adam optimizer (3) = 0.9, 32 = 0.95, weight decay=0.05) with a cosine learning rate schedule and 10-epoch linear warmup. The peak learning rate was set to 1 x 10-A for the pre-training and alignment stages, and x 10-5 for the downstream fine-tuning stage. For complete reproducibility, detailed hyperparameter configurations for the EEG encoder, CALRA module, and the optimization process are listed in Table A2. Batch sizes for the three stages were 512, 256 and 64, respectively; The temperature parameter T in contrastive loss was a learnable logit scale, initialized to correspond to t 0.07. All models were trained until convergence based on the performance of the validation set. To assess practical feasibility, we also benchmarked model complexity and inference latency, summarized in Table A3.
Computational Cost Analysis. We detail the pre-training budget to address concerns regarding resource usage. Our training was conducted on NVIDIA RTX A6000 GPUs. The total computa- tional budget for the foundation model phases was approximately 240 GPU hours:
Stage 1 (Unimodal Pre-training): Utilized 8 GPUs for ~0 hours (== 160 GPU hours).
* Stage 2 (Multimodal Alignment): Utilized GPUs for ~20 hours (== 80 GPU hours).
For comparison, the recent strong baseline CBraMod Wang et al. (2025) reports using 4 NVIDIA RTX A5000 GPUs for approximately days, equating to ze 480 GPU hours. MindMix achieves superior performance while requiring only ~50% of the pre-training duration of this SOTA baseline, demonstrating significant training efficiency.
Negative Sampling Policy. We uniformly utilize In-Batch Negative Sampling throughout our training. However, the composition of batches differs between stages to address specific goals:
Stage 2 (Multimodal Alignment): We utilize Global Shuffling across the entire 100+ hour corpus. This means that within any given batch, the negatives (other samples in the batch) are naturally drawn from all subjects and all distinct auditory materials. This strictly satisfies your criteria during the alignment phase, forcing the model to learn robust, subject- invariant features.
Stage 3 (Downstream Fine-tuning): We construct batches using data from the same sub- ject but across different trials. This allows the model to optimize for individual neural distributions. Even here, the negative sampling is rigorous: for Retrieval tasks, negatives
18

---

Published as a conference paper at ICLR 2026
Table A2: Detailed Hyperparameter Contiguration. Specific architectural and training hyperpa- rameters used for MindMix fine-tuning.
Category
Hyperparameter
Value
Transformer Layers
12
Embedding Dimension
200
EEG Encoder
Attention Heads
10
Feed-forward Dimension
800
Type
3 3-layer 1D CNN
Patch Encoder
Patch Dimension
200
Output Channels
Input/Output Dimension
256
Low-Rank Dimension
128
CALRA Module
Attention Heads
4
FFN Hidden Dimension
512
Type
A
Fine- Fine-tuming Learning Rate
x 10-5
Optimizer
Weight Decay
0.01 (0.9, 0.95) 3
Adam Betas
Warmup Epochs
Table A3: Efficiency Analysis. Model complexity and inference latency benchmarked on a single NVIDIA A6000 GPU (Batch Size=l).
Model
Params (M)
FLOPs (G)
Latency (ms)
EEGNet
0.003
0.01
1.9
LaBraM-Base
5.8
0.83
10.4
MindMix (Ours)
97
7.71
39.6
are drawn from different trials (different stories/music); for the AAD task, the negative is implicitly the simultaneous unattended stream (a "hard negative" with identical recording conditions but different semantic content).
Model Complexity and Stage-wise Training Dynamics. To provide transparency regarding model scale and computational cost, we detail the parameter breakdown of MindMix across its three training stages in Table A4.
Stage (Unimodal Pre-training): We focus exclusively on training the EEG Encoder (s= 6M params) to learn generic neural representations.
Stage 2 (Multimodal Alignment): This stage involves end-to-end fine-tuning of the en- tire framework (EEG Encoder CALRA Audio Encoder), bringing the total trainable parameters to ~ 97M. This allows for deep adaptation of both modalities.
Stage (Downstream Fine-tuning): We perform comprehensive fine-tuning of the full model (including the Task Head) to ensure optimal adaptation to specific downstream tasks.
A.4 EVALUATION METRICS
In this section, we introduce the details of the metrics used in evaluation.
19

---

Published as a conference paper at ICLR 2026
Table A4: Parameter Breakdown and Stage-wise Training Strategy. The table details the static
parameter count for each module and specifies the trainable status across the three pipeline stages.
Module
Component
Statie Params
Stage I
Stage
Stage 3
EEG Encoder
12-layer Transformer
6 M
Trained
Fine-tuned
Fine-tuned
CALRA
Alignment Module
1 M
N/A
Trained
Fine-tuned
Audio Encoder
Wev2 2.0 (Base)
95 M
N/A
Fine-tuned
Fine-tuned
Task Head
Classifier
< 1 M
N/A
N/A
Trained
Total Trainable
2 6 M
97 M
~
97 M
Balanced Accuracy is performance metric suitable for imbalanced datasets, defined as the average of recall (sensitivity) obtained on each class. We use it for classification tasks (AAD and Emotion Analysis).
Weighted F1-score the weighted average of the Fl-score for each class, where the score for each class is weighted by the number of true instances for that class. This metric accounts for class imbalance.
Duo/Trio Accuracy is used for the music retrieval task. It measures the standard classi- fication accuracy in a forced-choice task where the model must select the correct audio stimulus from two (Duo) or three (Trio) options.
A.5 ROBUSTNESS ANALYSIS: RIGOROUS BETWEEN-TRIAG EVALUATION
Motivation and Protocol. While the standard within-tria evaluation protocol (randomly splitting segments from the same trial) is widely used for benchmarking Yan et al. (2024b); Ni et al. (2024), recent studies suggest it may introduce data leakage due to temporal correlations in EEG signals Puffay et al. (2023). To rigorously evaluate the robustness of MindMix and rule out potential over- fitting to trial-specific artifacts, the training and testing sets are constructed from disjoint trials (e.g., different stories or recording sessions), ensuring zero temporal overlap.
implemented strict Between- Protocol. In this setting,
Results and Analysis. We re-evaluated MindMix and key baselines under this rigorous protocol on the KUI,, DTU, and ESAA datasets. The results are summarized in Table A5. As expected, the absolute performance metrics for all models decrease compared to the within-trial setting (Table 2) due to the increased difficulty of generalizing to unseen trials. However, MindMix consistently maintains significant performance advantage over all baselines. For instance, on the KUL dataset, while the strong baseline drops to ~60 70%, MindMix achieves 96. 13%, demonstrating that our model's 's superiority stems from genuine neuro-acoustic alignment rather than artifact exploitation. Below, we provide three lines of evidence to substantiate the validity of these results and the source of our performance gain.
1. Data Independence Verification. We re-examined the metadata to verify split integrity. For DTU and ESAA, we confirm that under our between-trial protocol, the datasets are strictly content- disjoint and temporally disjoint; the audio stories used in the test set never appear in the training set. Yet, MindMix still achieves ~95% accuracy. proving that high performance replicable even when content repetition leakage is impossible. While KUL (with only 4 stories) inherently limits strict content independence, the consistent results on DTU/ESAA validate generalizability. Furthermore, we observe revealing contrast in baseline performance. Task-specific SOTA models (DBPNet, DARNet), explicitly designed for the within-subject classification regime, suffer "collapse" in performance (dropping ~30%) under this stricter protocol. This suggests they overfit to temporal leakage patterns in the standard setting. contrast, foundation models (LaBraM, CBraMod, Mind- Mix) demonstrate greater robustness. MindMix stands out further by leveraging the multimodal edge: unlike unimodal baselines blind to auditory content, it utilizes the pre-aligned Audio Encoder to perform semantic matching, maintaining high accuracy where others fail.
2. The "Sanity Check": Ruling out EEG Leakage. To definitively rule out hidden leakage or artifact learning in the EEG pipeline (e.g., normalization bugs EMG), conducted controlled
20

---

Published as a conference paper at ICLR 2026
Table A5: Robustness Evaluation under Between-Trial Protocol. Comparison of MindMix against SOTA baselines using strict trial-disjoint splitting to prevent data leakage. Despite the chal- lenging setting, MindMix maintains superior performance across all datasets.
KUL
DTU
ESAA
Method
Balanced Ace.
Weighted F1
Balanced Ace.
Weighted F1
Balanced Acc.
Weighted F1
DBPNet
0,6829 0.092
0.6620 10
0.6141 + 0.074
0.5887
0.077
0.3758 0.071
0.5220 + 0.075
DeRNet
0.6536 0.037
0.6167
0.112
0.5918 + 0.080
0.5420
0.104
0.5676 + 0.076
0.5454 + 0.078
MusicAAD
0.7936 0.098
0.7601
0.101
0.7679 1 0.082
0.7421
0.080
0.7639
0.082
0.7915 1 0.070
AADNet
0.7736 0J887
0.7312 = 0.132
0.6513 : 0.078
0.6377
0.081
0.7125 2 0.072
0,7741 2: 0.075
LaBraM
0.7521 0.085
0.7293
0.096
0.6475 1 0.092
0.6214
0.085
0.6789 1 0.082
0.6918 1 0.072
CBraMod
0.7701 0.091
0.7356
0.101
0.6321 + 0.097
0.6079
0.009
0.6932 4 0.091
0.6901 - 0.095
MindMix (Encoder-only)
0.74125 0.082
0.7221 = 0.088
0.6557 + 0.082
0.6279
0.089
0.6815 0.078
0.6911 +: 0.081
MindMix (Full)
0.9876 0.049
0.9613
0.054
0.9543 1: 0.033
0.9351 1: 0.032
0.9774 0.072..0.025
0.9719 1 0.031
comparison. First, our MindMix (EEG-Only) model drops to realistic accuracy of 65.57% (DTU) and 68.45% (ESAA) under this protocol, similar to other unimodal baselines. This serves as defini- tive proof: if there were leakage in the EEG data itself, the Encoder-Only model would also achieve inflated scores. The fact that it does not confirms our EEG pipeline clean. Conversely, other multimodal variants (e.g., MusicAAD) maintain much higher performance (~76%) than unimodal ones. This effectively isolates the Audio Modality and our alignment mechanism as the key drivers of the performance jump.
Paradigm Shift to Alignment. The exceptional performance of Mind- Mix compared traditional classifiers stems from a fundamental Task Mass-Obeective Alignment. Tra- ditional baselines treat AAD as binary classification problem, struggling to learn decision bound- aries from scratch with limited data. In contrast, MindMix reformulates AAD as a Neural-Acoustic Matching problem: calculating which audio embedding is closer to the EEG embedding. This task mathematically identical our Stage contrastive pre-training objective. Thus, the model is not learning a new task but applying pre-leamned, robust alignment capability. While this align- ment paradigm offers a structural "head start," it represents a significant scientific advantage over classification. The fact that MindMix maintains exceptional performance even under strict content- disjoint protocols demonstrates the immense potential of this contrastive matching approach for high-precision, real-time AAD applications in real-world scenarios.
3. Mechanism of Success:
A.6 BASELINE MODELS
Here, we introduce the details of the baselines for performance evaluation. We include both task- specific state-of-the-ar models and sfate-of-the-art unimodal EEG foundation models.
EEGNet Lawhern et al. 2018) compact convolutional neural network for EEG-based BCIs, utilizing depthwise and separable convolutions for efficient feature extraction.
DBPNet 2024 is a dual-branch parallel network designed specifically for auditory attention detection, fusing temporal and frequency features.
DARNet Yan et al. (2024b) is dual attention refinement network with spatiotemporal
construction for auditory attention detection.
MusicAAD Niu et al.
(2024) is recent model designed for music-oriented auditory atten- tion detection from EEG.
AADNet Nguyen et al. (2025) is an end-to-end deep learning model specifically proposed for the auditory attention decoding task.
BENDR Kostas et al. (202T) is an early EEG foundation model that uses a Transformer architecture and a contrastive self-supervised learning task.
BIOT Yang et al. (2023)
biosignal Transformer model for cross-data learning, pre- trained on a diverse set of biosignal datasets.
EEGPT Wang et al. (2024b) is a pretrained transformer for universal representation of
EEG signals based on a masked reconstruction objective.
LaBraM Diang et al. (2024) is a large brain model that learns generic representations by predicting neural tokens of masked EEG patches.
Miang
21

---

Published as a conference paper at ICLR 2026
Table A6: Window Size Sensitivity Analysis. Performance comparison across different decision window lengths (Is, 2s, 5s). Longer windows generally improve performance due to increased context, but the window offers the best -fficiency-accuracy trade-off.
Dataset
Window Size
Balanced Accuracy Standard Deviation
second
0.8535
0.099
HR- HR-EEE44MM
2 seconds
0.8878
i 0.045
5 seconds
0.8917
i 0.062
second 2 seconds
0.6998
0.107
PME4
0.7256
0.123
5 seconds
0.7290
+ 0..12
CBraMod Wang al. (2025) criss-cross brain foundation model for EEG decoding that models spatial and temporal dependencies separately.
A.7 WINDOW SIZE SENSITIVITY ANALYSIS
Our main results utilize a 2-second decision window, chosen to balance decoding accuracy with temporal resolution. To validate this choice, performed sensitivity analysis window lengths of 1s, 2s, and 5s.
As shown in Table A6, performance consistently improves with longer windows (integrating more context) and decreases with shorter ones. While windows yield marginally higher accuracy, we retain the 2s window the optimal trade-off for system responsiveness.
A.8 GENERALIZATION ON NON-AUDITORY TASKS
To verify the generalization capability of our EEG-Only encoder beyond the auditory domain, we benchmarked two standard non-auditory BCI tasks: TUAB Veloso et al. (2017) (abnormal de- tection) and BCI Competition IV-2b Tangermann et al. (2012); Deng et al. (2018) (motor imagery). We followed the standard evaluation protocols for both tasks and compared model against ported SOTA baselines.
Results and Analysis. The results are summarized in Table A5.
On the TUAB dataset, our encoder achieves Balanced Accuracy of 0.8545, securing the second-best performance among all comparison models, surpassing LaBraM (0.8210) and CBraMod (0.8289).
On the BCIC-IV-2B dataset, our encoder achieves the highest performance (0.6943) among all listed foundation models, outperforming EEGPT (0.6893) and CBraMod (0.6910).
These results confirm that while MindMix is specialized for auditory decoding, its underlying EEG encoder learns highly robust and generalizable features effective for diverse BCI paradigms.
A.9 ROBUSTNESS AND EFFICIENCY ANALYSIS
To comprehensively assess the practical feasibility of MindMix in real-world scenarios, ducted two critical analyses: (1) Cross-Dataset Generalization to evaluate robustness against se- vere domain shifts, and (2) Data Efficiency Analysis to determine performance stability in low-data regimes.
Moos--Dtaseet Generalization (Zero-shot Transfer). To assess robustness to domain shifts with- out task-specific adaptation, we performed a cross-dataset evaluation like Chen et al. 2024bb.. Specifically, we trained the model on the KUL dataset (Dutch stimuli) and evaluated it directly on the DTU dataset (Danish stimuli) without any fine-tunin sonettunngg.. This represents an extremely challenging setting involving shifts in subjects, acquisition devices, and languages.
22

---

Published as a conference paper at ICLR 2026
Table A7: Generalization on Non-Auditory Tasks. Performance comparison on TUAB (Abnormal
Detection) and BCIC-IV-2B (Motor Imagery). MindMix (Encoder-only) demonstrates SOTA-level generalization capabilities.
Model
Dataset
Balanced Acc.
Weighted F1
BENDR
TUAB
0.7915 0.007
0.8522 0.004
BIOT
TUAB
0.7844 + 0.005
0.8854 = 0.003
EEGPT
TUAB
0.8833 0.002
0.9432 + 0.001
LaBraM
TUAB
0.8210 + 0.003
0.8979 = 0.002
CBraMod
TUAB
0.8289 = 0.005
0.9018 = 0.002
MindMix (Encoder-only)
TUAB
0.8545 0.004
0.91 + 0.005
BENDR
BCIC-IV-2B
0.6806 0.007
0.6801 + 0.007
BIOT
BCIC-IV-2B
0.5524 + 0.010
0.5516 + 0.010
EEGPT
BCIC-IV-2B
0.6893 + 0.009
0.6890 = 0.009
LaBraM
BCIC-IV-2B
0.6610 0.011
0.6608 + 0.011
CBraMod
BCIC-IV-2B
0.6910 0.008
0.6898 + 0.008
MindMix (Encoder-only)
BCIC-IV-2B
0.6943 + 0.010
0.6921 + 0.010
The results are summarized in Table A8. As expected, all models exhibit performance drop com- pared within-dataset training. However, MindMix achieves an accuracy of 56.55% which sig- nificantly higher than the random chance level (50%) and outperforms both the EEGNet (53.16%) and LaBraM (51.16%) baselines under the same protocol. This indicates that MindMix learns more robust and transferable neuro-acoustic representations than unimodal baselines, likely benefiting from its explicit spatial encoding strategy (E) that effectively handles heterogeneous electrode vari- ations.
Table A8: Cross-Datased Generalization (KUL - DTU). Zero-shot transfer performance where models trained on KUL are tested on DTU without fine-tuning. MindMix demonstrates superior robustness to domain shifts compared to baselines.
Model
Transfer Task
Accuracy
Score
EEGNet
KUL + DTU
0,5316
0.5281
LaBraM-Baas -ase
KUL DTU 0.5116
0.4987
MindMix (Ours)
KUL - DTU
0.5655
0.5492
Data Scaling and Efficiency Analysis. To evaluate data efficiency, we analyzed performance degradation on the HR- HR-EEG4EMO dataset by varying the training set size from 25% to 100%, while keeping the validation and test sets fixed. We employed rigorous subject-specific stratified sam- pling protocol: for every subject, we randomly sampled subsets (25%, 50%, 75%) of their specific training trials.
As shown in Table A9, MindMix demonstrates exceptional data efficiency. Notably, with only 50% of the training data, MindMix (0.6942) effectively matches the full-data (100%) performance of EEGNet (0.6981). With 75% data, MindMix (0.7855) significantly surpasses the full-data perfor- mance of the strongest unimodal baseline, LaBraM (0.7295). This flatness confirms that the robust cross cross-modal priors learned during our alignment stage significantly reduce the dependency on large-scale subject-specific calibration data.
degradation curve
B THE USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used to aid in the writing and polishing of the manuscript. Specifically, used an LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
23

---

Published as a conference paper at ICLR 2026
Table A9: Data Efficiency Analysis on HR-EEG4EMO. Performance (Balanced Accuracy) with varying percentages of per-subject training data. MindMix outperforms full-data basclines even with significantly reduced training samples.
Training Data %
EEGNet
LaBraM
MindMix (Ours)
25%
0.6245 + 0.146
0.6184 + 0.126
0.6307 + 0.109
50%
0.6429 0.131
0.6296 0..13
0.6942 0.127
75%
0.6875 + 0.120
0.6769 = 00.114
0.7855 + 0.121
100%
0.6981 0.111
0.7295 0.082
0.8878 0.045
It is important to note that the LLM was not involved in the ideation, research methodology. or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement the scientific content or data analysis.
The authors take full responsibility for the content of the manuseript, including any text generated or polished by the LLM. We have ensured that the L.g-geerrteed text adheres to ethical guidelines and does not contribute to plagiarism scientific misconduct.
24


Reviews:
[{"review_id": "MlANQGvPTu", "reviewer": "Reviewer_FBHr", "summary": "This paper introduces MindMix, a multimodal EEG–audio foundation model designed to decode auditory perception via deep neural-acoustic alignment. The key architectural innovation is the Cross-Attention Low-Rank Alignment (CALRA) module, which enables fine-grained token-level interactions between EEG and audio embeddings. The training follows a two-stage process: (1) large-scale unimodal EEG pretraining (3,500+ hours) using masked token prediction and spectral reconstruction objectives, and (2) contrastive multimodal pretraining (100+ hours of EEG–audio pairs) using CLIP-style alignment. Experiments show strikingly high performance (up to 0.999 accuracy) on several auditory decoding benchmarks, reportedly surpassing both task-specific and unimodal EEG foundation model baselines.\n\nHowever, **empirical results are too strong to be credible without verification** (code link does not work). Combined with presentation ambiguities and overlapping ideas with prior cross-attention-based alignment methods, it is difficult to assess soundness and novelty with confidence.\n\nNevertheless, the direction is highly promising, and if the authors can:\n\n1. Provide working, verifiable code and confirm the data split strategy,\n2. Clarify the architectural details and novelty of CALRA, \n3. Justify the plausibility of the near-perfect accuracies, and \n4. Answer the questions regarding proper comparison with prior models,\n\nthen this work could become a significant contribution to multimodal brain decoding.\n\nAt the current stage, I remain cautiously skeptical—the ideas are intriguing, but the evidence and clarity are insufficient for confident acceptance. However, if the concerns outlined in the Weakness section are resolved, I am willing to raise the score.", "questions": "All the questions are listed in the Weakness section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Proposes a new **two-stage multimodal foundation model** framework for EEG–audio alignment.\n- Introduces a novel **CALRA module** that combines type-specific alignment, bidirectional cross-attention, and shared low-rank bottlenecks.\n- Achieves very high performance in auditory tasks.", "weaknesses": "- **Reproducibility / Verification, with Scores maybe too good to be true**:\n    \n    The code link currently does not work. Reported accuracies are extremely high (approaching (>0.999) or reaching **1.0** with zero standard deviation on noisy, cross-subject EEG data), which raises doubts about correctness or possible data leakage (e.g., temporal overlap between training/test). Access to working code is essential to verify claims. Also please describe in more detail the data-split procedure in the Appendix if possible (beyond the “strict subject independent evaluation protocol with 70%, 10%, 20% split …”)\n    \n- **Architectural description clarity**:\n    \n    The description of the EEG-only pretraining (Sec. 3.2) is confusing:\n    \n    - Eq. (1) defines $E_{\\text{patch}} = X + T + E$, but Fig. 2 seems to show that quantization happens before addition (i.e. $E_{\\text{patch}} = \\text{quantizer}(X) + T + E$)—please clarify the correct order.\n    - Please clarify the “unique pretraining methodology” (line 153) of yours relative to **LaBraM** (which also uses masked patch prediction and spectral reconstruction). What, specifically, is new about this pretraining? Also please cite the relevant papers that motivated your architecture choice (e.g., LaBraM motivated your tokenizer usage?)\n    - Dimensional consistency is unclear: $S\\in C\\times T$ implies $E_{\\text{patch}}\\in C\\times N\\times D$ ($N$ being the number of temporal patches), yet $E’_{\\text{proj}}\\in N\\times D$ (lines 204–206). How is the channel dimension aggregated? (It seems $f_k$ is used? Can you elaborate on what exactly this is?) Also how does it handle varying number of channels during training?\n- **Missing implementation details**:\n    \n    Important training configurations (number of layers, embedding dimension, optimizer hyperparameters, fine-tuning LR, rank for CALRA, etc.) are missing. \n    \n- **Novelty concerns for CALRA**:\n    \n    I am not an expert in CLIP style training, but it seems the CALRA module resembles prior cross-attention alignment mechanisms used in multimodal CLIP-style works (e.g., *CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification*, *Multi-Granularity Cross-Modal Alignment*). Please differentiate CALRA from these or acknowledge conceptual overlap, and add a related works for CLIP modality alignment especially using CALRA like modules.\n    \n- **Justification and motivation**:\n    \n    Beyond performance gains, please explain in more detail *why* CALRA works with proper references to previous CLIP works if possible—e.g., what failure modes of simpler co-attention or projection methods it solves. In its current form, the architecture seems to have come out of nowhere with no context explained.\n    \n- **Presentation issues**:\n    - Define T and E in Eq. (1) explicitly (positional vs. channel embeddings? Is it cosine positional embedding?).\n    - Clarify Eq. (10): what is E_i? E_{\\text{aligned}} or E_{\\text{proj}}?\n    - Combine results from Table 2 and Figure 4 for easier comparison (currently scattered).\n    - Explain what “token-level interaction between modalities” (line 134) precisely means.\n    - In Table 3, “Ablation on EEG Encoder”, when the author mentions “w/ LaBraM”, does this mean using the pretrained weights of LaBRaM or using their architecture only initialized from scratch?\n- Miscellaneous but important points :\n    - The author claims in line 351 that “… these large models are often highly sensitive to the data format and preprocessing pipelines..”, but do not provide any reference or supporting experimental evidence, and whether their model is superior.\n    - The author **claims in line 417 that their EEG-only counterpart often outperforms SOTA**. However, when comparing Table 2 with Figure 4 we find that **CBraMod is beaten by the EEG-only model only in the ESAA task**. Please clarify this discrepancy.\n- Proper comparison with previous models.\n    - The model uses data that was preprocessed using a bandpass filter of **1–40 Hz**, whereas baselines you used such as LaBraM was pretrained on data bandpassed at **0.1–70 Hz**. Were preprocessing pipelines harmonized across comparisons? (Or preprocessing set differently based on the basemodel used?) **Differences here can cause distribution shifts that unfairly penalize other pretrained models.**\n    - The author claims in line 417 that their EEG-only counterpart often outperforms SOTA. It might beneficial to verify this with tasks other than auditory tasks. (More general tasks used in the previous models’ papers)\n    - Given that CBraMod still outperforms your EEG-only model (as pointed out by the \n    ”Miscellaneous but important points” I wrote above), have you tried CLIP-style pretraining on top of CBraMod to isolate CALRA’s contribution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RaNTMm0HyP", "reviewer": "Reviewer_gpHw", "summary": "MindMix is a multimodal foundation model that learns a shared space between EEG and audio. It trains in stages: first, a large EEG encoder is pretrained on diverse, unlabeled EEG to capture robust temporal, spectral, and spatial patterns; next, paired EEG–audio data are used to align the two modalities with CALRA, a block that mixes type-aware routing, bidirectional cross-attention, and a shared low-rank fusion bottleneck under a contrastive objective; finally, the aligned model is fine-tuned for specific tasks.\n\nOn benchmarks, MindMix reports near-perfect speech auditory attention decoding and strong gains on affective recognition and music retrieval compared with task-specific and unimodal EEG baselines. Ablations indicate that both cross-modal alignment and the low-rank fusion are essential, and performance improves as pretraining and alignment data grow. The key contributions are a unified EEG–audio representation learning pipeline, the CALRA module for efficient coupling, and evidence that a single pretrained model can transfer across multiple auditory cognitive tasks.", "questions": "1 Can authors provide runnable split manifests with fixed seeds that are subject-disjoint and content-disjoint, guarantee no temporal overlap across splits, and rerun AAD under this protocol? Include shuffled-stimulus and shuffled-subject controls. If results stay near perfect, pls show an error analysis.\n\n2 Please resolve the pooling-vs-sequence contradiction. Specify the tokenization for EEG and audio, sequence lengths, and the exact tensor shapes that CALRA consumes and emits.\n\n3 could authors define the negative policy? Ensure negatives come from different subjects and different stories. Report sensitivity to 1 s, 2 s, and 5 s windows, and report both raw window-level metrics and any aggregated or majority-vote metrics.\n\n4 When audio is unavailable, how is the type label k obtained? Report performance with known k, with a learned EEG-only predictor for k, and with no type routing. This will clarify reliance on dataset priors.\n\n5 could authors release a minimal repo with preprocessing scripts, split files, training configs, and eval scripts that reproduce a subset of the tables. Fix table typos, add statistical tests and confidence intervals, and report parameters, FLOPs, and inference latency alongside key baselines.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "MindMix offers a clear two-stage recipe: large-scale unimodal EEG pretraining to learn general neural patterns, followed by EEG–audio alignment with the CALRA block. CALRA’s type-aware routing plus bidirectional cross-attention through a shared low-rank bottleneck is a thoughtful way to couple modalities while keeping capacity focused.\n\nQuality:\nThe empirical study spans three task families (speech auditory attention decoding, emotion recognition, music identification) and shows consistent gains over both task-specific and unimodal EEG baselines. Ablations replace the EEG encoder and systematically modify CALRA, and the resulting performance shifts align with expectations, which supports the design choices.\n\nClarity:\nThe training pipeline is easy to follow: pretrain the EEG encoder on broad corpora, then align EEG and audio with CALRA, then fine-tune for tasks. The role of each CALRA component is explained through targeted ablations, which helps readers map architecture pieces to observed effects.\n\nSignificance:\nBy learning a shared EEG–audio space that transfers across multiple auditory cognition tasks, the work pushes toward a reusable foundation for brain–audio applications. The parameter-efficient cross-modal coupling suggests practical value for future BCI systems where compute and data are constrained.", "weaknesses": "1 AAD results: Near-perfect scores (0.998–1.000) are not plausible without airtight leakage controls.\n\n2: The paper lacks concrete safeguards against content leakage (same story/music across splits), window overlap across splits, and other split-level confounds.\n\n3 Representation inconsistency: Section 3.2 pools to a single vector, while Section 3.3 treats those embeddings as sequences for cross-attention, leaving CALRA under-specified.\n\n4 The text claims a symmetric contrastive objective, but the equations show only EEG→audio.\n\n5 The type-specific aligner assumes access to stimulus type k at test time, which may leak dataset priors when audio is unavailable.\n\n6  Artifact removal (EOG/EMG), filtering, bad-channel policy, and whether z-score stats are fit only on training data remain unclear.\n\n7 Using 2-s windows with negatives drawn from the same recording can inflate performance; the negative sampling policy is unspecified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2Zf7P7Ftl", "reviewer": "Reviewer_bKp3", "summary": "This paper introduces MindMix, the first multimodal foundation model specifically designed for auditory perception decoding from non-invasive EEG signals through deep neural-acoustic alignment. The work addresses a critical limitation of existing EEG foundation models (such as EEGPT, LaBraM, and CBraMod), which are pre-trained exclusively on EEG signals without exposure to auditory stimuli, resulting in representations that are not optimized to align with acoustic structures and thus perform poorly on auditory decoding tasks. MindMix employs a two-stage training strategy: first, a high-capacity EEG encoder is pre-trained on over 3,500 hours of diverse EEG data using a novel multi-task self-supervised objective that combines masked token prediction and Fourier spectrum reconstruction, enabling the model to learn robust neural representations that can transfer across tasks and subjects. The encoder uses a heterogeneous patching strategy that integrates temporal and spatial encodings to handle varied electrode layouts, and quantizes patch embeddings into discrete neural tokens via a shared codebook for the masked prediction task. Second, the model learns neural-acoustic mapping on over 100 hours of paired EEG-audio data through a novel Cross-Attention Low-Rank Alignment (CALRA) module, which facilitates fine-grained, token-level cross-modal information integration. CALRA consists of three key components: a type-specific aligner that routes projections through learnable transformations conditioned on auditory type labels (speech vs. music), bi-directional cross-attention that enables each modality to retrieve complementary information from the other, and a shared low-rank alignment mechanism that enforces semantic consistency by projecting both modalities into a compact shared space, modeling their interaction via element-wise product, and integrating feedback through residual connections. The entire framework is optimized end-to-end via a symmetric contrastive learning objective (InfoNCE loss) that maximizes cosine similarity between true EEG-audio pairs while minimizing it for non-corresponding pairs.\n\nThe experimental evaluation demonstrates substantial improvements over existing methods across diverse auditory decoding tasks. MindMix is evaluated on six downstream tasks using strict subject-independent protocols with 70%/10%/20% train/validation/test splits: auditory attention decoding (AAD) on three datasets (KUL, DTU, ESAA), emotion analysis on two datasets (PME4, HR-EEG4EMO), and cross-modal music retrieval on MAD-EEG. Results show near-perfect performance on speech AAD (99.82% balanced accuracy on KUL, 99.93% on DTU, 100% on ESAA), dramatically outperforming both task-specific state-of-the-art models like DARNet (94.81% on KUL) and unimodal EEG foundation models like LaBraM (63.30% on KUL) and CBraMod (68.42% on KUL). On emotion analysis, MindMix achieves 72.56% on PME4 and 88.78% on HR-EEG4EMO, showing improvements of over 10 percentage points compared to the best baselines. Comprehensive ablation studies reveal that each component contributes significantly: removing CALRA entirely causes performance drops up to 4.58% in AAD accuracy, replacing the custom EEG encoder with LaBraM reduces performance by 2.38%, and within CALRA, removing cross-attention has the largest impact (5.58% drop), followed by shared low-rank alignment (2.40% drop) and type-specific aligner (1.29% drop). A critical analysis comparing the full model against its EEG-only counterpart quantifies a massive \"synergy gap\" ranging from 11.26% on ESAA to 36.32% on DTU, demonstrating that the performance gains stem from learning the deep relationship between neural signals and auditory stimuli rather than simply having better EEG representations. Qualitative visualization using pseudo-reconstruction of Mel spectrograms shows that MindMix achieves substantially higher Pearson correlation coefficients (0.88 on DTU, 0.91 on KUL) compared to the LaBraM variant (0.72, 0.85) and baseline methods (0.67, 0.61), with visual inspection confirming that MindMix faithfully captures fine-grained harmonic structures that are blurred or lost in other methods.", "questions": "Please see your weaknesses and I will adjust the final score based on your answers.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper presents a well-motivated and technically sound approach to auditory perception decoding from EEG signals. The core contribution—the CALRA module enabling deep token-level neural-acoustic alignment—represents a genuine architectural innovation beyond simple projection-based fusion methods. The experimental design is rigorous, employing strict subject-independent evaluation protocols across six diverse downstream tasks, and the results are compelling, with near-perfect AAD performance and consistent improvements of 10+ percentage points over strong baselines. The ablation studies systematically validate each component's contribution, while the quantified \"synergy gap\" analysis (11-36% performance gain) effectively demonstrates that gains stem from multimodal alignment rather than merely improved unimodal representations. The paper maintains appropriate scientific rigor with transparent limitations regarding data scarcity, clear implementation details supporting reproducibility, and thoughtful architectural choices such as the two-stage training strategy that pragmatically addresses limited paired data availability.", "weaknesses": "The paper's experimental evaluation, while showing impressive performance gains, raises several methodological concerns that limit confidence in the generalizability of the findings. First, the near-perfect accuracy achieved on speech AAD tasks (99.82%-100%) is unusually high and deviates dramatically from established performance ranges in the literature, suggesting potential data leakage or overfitting issues despite the claimed subject-independent protocol. The authors do not provide sufficient detail about how subject independence was strictly enforced across the three training stages, and it remains unclear whether any subjects appearing in the downstream test sets were also present in the 100+ hours of multimodal alignment training data, which could artificially inflate performance. Second, the baseline comparisons appear somewhat unfair: unimodal foundation models like LaBraM and EEGPT are evaluated directly on auditory tasks for which they were never designed, while MindMix benefits from explicit auditory-specific pretraining. A more appropriate comparison would include these baselines after additional fine-tuning on auditory data, or alternatively, evaluate MindMix on non-auditory tasks where it received no specialized pretraining. The paper also lacks critical analyses such as cross-dataset generalization (training on one AAD dataset and testing on another without fine-tuning), performance degradation curves with varying amounts of training data, and computational cost comparisons, all of which are essential for assessing practical deployment feasibility. Additionally, the massive performance gap between MindMix and baselines raises questions about whether the improvements stem primarily from the proposed architecture or simply from having access to substantially more paired training data during the alignment stage.\nBeyond experimental concerns, the paper suffers from limited novelty in its core technical contributions and insufficient contextualization within the broader neuroscience literature. The CALRA module, while effective, essentially combines standard architectural components (cross-attention mechanisms and low-rank factorization) that are well-established in multimodal learning literature, with the primary novelty being their specific arrangement and the addition of type-specific routing. The paper does not adequately justify why this particular architecture is theoretically well-suited for EEG-audio alignment compared to alternatives, nor does it provide ablations comparing against other fusion strategies beyond simple co-attention. More critically, the work lacks neuroscientific grounding: there is no discussion of which brain regions or neural signatures the model learns to leverage, no analysis of temporal alignment between EEG and audio features (whether the model captures known neural lags in auditory processing), and no interpretation of what the learned embeddings represent in terms of underlying neural computations. The visualization of Mel spectrogram reconstructions is superficial and does not demonstrate that the model has learned physiologically plausible or neuroscientifically meaningful representations. The framing as \"the first multimodal foundation model\" for auditory decoding also overstates the contribution, as the distinction from existing multimodal EEG-audio methods appears to be primarily one of scale rather than fundamental approach. Yet the paper provides no systematic comparison of model scale effects, no evidence of zero-shot transfer capabilities to genuinely new auditory tasks, and no analysis of what makes this a \"foundation\" model beyond simply being larger and trained on more data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kzIRWWkggR", "reviewer": "Reviewer_NCy7", "summary": "This paper proposes **MindMix**, a multimodal foundation model designed to achieve deep EEG–audio alignment for auditory perception decoding. The approach follows a three-stage training pipeline: (1) large-scale unimodal EEG pretraining, (2) neural–acoustic alignment using the proposed Cross-Attention Low-Rank Alignment (CALRA) module, and (3) fine-tuning on downstream auditory tasks including AAD, emotion recognition, and music retrieval. Experimental results show strong improvements over prior works.", "questions": "1. Please provide **detailed parameter counts** for each major module and stage. How does model size relate to performance? Any preliminary scaling analysis?\n\n2.  Key details are missing (Pretraining duration GPU hours / computational cost Comparison to baselines under equal compute).\n\n3. Can you clarify **why training a LaBraM-style encoder from scratch** is better than using existing pretrained EEG encoders? What inductive bias or architectural novelty is gained?\n\n4. Can you provide **cross-trial** / **cross-subject** AAD performance on KUL/DTU/ESAA? The current within-trial only protocol does not sufficiently measure practical generalization.\n\n5. Have you compared CALRA with **multimodal LoRA** as an alignment bottleneck? Is there a reason LoRA-style low-rank adaptation was not considered?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The paper addresses a clear limitation in current EEG pretraining, which lack of grounding with auditory stimuli, and proposes a principled cross-modal alignment strategy.\n\n2) MindMix achieves substantial gains across diverse tasks and datasets, signaling strong generalization capabilities.\n\n3) The component-wise analysis provides reasonably convincing evidence for the effectiveness of CALRA and the training pipeline.\n\n4) Multiple datasets, tasks, and baseline categories are included, strengthening the validity of the comparison.", "weaknesses": "1. The paper does not provide **parameter counts** for different model components (EEG encoder, CALRA, stage-wise variations). Without reporting efficiency and compute scaling trends, claims of “foundation model” scalability feel incomplete.\n\n2. The **three-stage pipeline** is resource-intensive, yet key details are missing (Pretraining duration GPU hours / computational cost Comparison to baselines under equal compute).\n\n3. The architecture appears **highly similar to LaBraM** (masked token prediction + quantized neural tokens). What specifically differentiates this encoder from LaBraM beyond being trained from scratch?\n\n4. The improvement (+ ~16% over strong baselines) is unusually large given DTU’s difficulty. Whether **within-trial only** evaluation inflates performance Need **cross-trial** and **cross-subject** results for consistency with prior work\n\n5. CALRA uses shared low-rank fusion. but no comparison to multimodal LoRA or other parameter-efficient alignment adapters is provided  (especially since LoRA is widely adopted in multimodal foundation models)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)