Title: Is Limited Participant Diversity Impeding EEG-based Machine Learning?
Venue: NeurIPS 2025
Accepted: yes
Keywords: EEG

electroencephalography

scaling behaviour

distribution shift

augmentation

self-supervised learning
Abstract: The application of machine learning (ML) to electroencephalography (EEG) has great potential to advance both neuroscientific research and clinical applications. However, the generalisability and robustness of EEG-based ML models often hinge on the amount and diversity of training data. It is common practice to split EEG recordings into small segments, thereby increasing the number of samples substantially compared to the number of individual recordings or participants. We conceptualise this as a multi-level data generation process and investigate the scaling behaviour of model performance with respect to the overall sample size and the participant diversity through large-scale empirical studies. We then use the same framework to investigate the effectiveness of different ML strategies designed to address limited data problems: data augmentations and self-supervised learning. Our findings show that model performance scaling can be severely constrained by participant distribution shifts and provide actionable guidance for data collection and ML research. The code for our experiments is publicly available online.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how participant diversity impacts the performance and generalisation of machine learning models trained on EEG data. It frames EEG datasets as multi-level data structures, where large numbers of samples are derived from a small number of participants via windowing. The authors conduct large-scale empirical studies to disentangle the effects of total sample size versus number of participants. They demonstrate that increased participant diversity consistently leads to more robust generalisation, even when total sample size is held constant. The paper also evaluates common ML strategies like pretraining, domain adaptation and model architecture choices under varying diversity conditions to show that some methods are disproportionately sensitive to participant homogeneity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper systematically varies total data volume and number of unique participants to allow for controlled experiments that isolate the effects of diversity on model generalisation. This is very useful and not common in EEG literature.\n\n- The paper assesses pretraining, domain adaptation, self-supervised learning and architecture variants across multiple datasets with a well-rounded perspective on how methods scale with diversity.\n\n- By explicitly thinking of EEG datasets as nested hierarchies (participants -> sessions -> segments), the paper gives a helpful abstraction that clarifies why naïve sample count comparisons can be misleading.\n\nWeaknesses:\n\n- The authors claim in line 83 that their work is distinct because it investigates the utility of data augmentations under limited participant counts and limited data per participant. However, this distinction is unclear and weakly justified. Most augmentation methods in EEG, and in machine learning in general, are motivated precisely by data scarcity and generalisation issues. Without a clearer articulation of what makes their analysis novel in this space, the claim reads more as a restatement of standard goals rather than a meaningful point of differentiation.\n\n- The claim that LaBraM is a \"representative example\" of self-supervised learning approaches (line 93) is non-trivial and unsubstantiated. LaBraM is just one specific method, and the field of self-supervised learning in EEG is diverse and covers paradigms like contrastive learning, masked prediction and generative approaches. The authors provide no justification for why LaBraM adequately captures the range of behaviors or limitations of these different methods. This weakens the argument that the findings generalise to self-supervised learning as a whole. A claim of representativeness must be supported by either a taxonomy of the space or comparative results, neither of which are provided.\n\n- The claim in line 239 (doubling participants would yield “roughly 1%” improvement) is presented without any accompanying justification, reference to specific experiments or quantitative derivation. The number appears arbitrary. Given the paper’s central argument about the value of participant diversity, unsupported numerical claims like this undermine the credibility of the analysis. \n\n- The question \"Can contemporary EEG data augmentations effectively increase performance across different data regimes?\" is framed too broadly given the narrow scope of the study. The authors test augmentation methods that are all relatively standard (e.g., Gaussian noise, cropping). No principled selection is made and there is no attempt to explore the space of more recent or task-specific augmentations, nor any ablation or combination strategies. As such, the findings are limited to a small subset of possible techniques and cannot support general conclusions about the effectiveness of augmentations across EEG tasks or regimes. The question overstates what the study can validly answer.\n\n- The paper poses the broad question of whether self-supervised pre-training can improve performance across different data regimes, but only evaluates this using a single pre-trained model, LaBraM. This is insufficient to support a general conclusion. Evaluating just one model, trained on one dataset and applying it to a few target conditions does not constitute a rigorous or comprehensive assessment. The conclusion lacks external validity and is not grounded in enough empirical breadth to answer the question as posed.\n\nQUESTIONS:\nHow is your analysis of augmentations under limited participant counts fundamentally different from standard augmentation studies aimed at data scarcity?\n\nOn what basis do you consider LaBraM indicative of broader self-supervised learning approaches in EEG?\n\nProvide the derivation or source for the \"~1% improvement\" figure in line 239; is this drawn from your own results, external literature or an estimation?\n\nExplain the rationale behind the choice of augmentation techniques. Why these four and what criteria were used to select them over others?\n\nClarify the scope of your conclusions on self-supervised learning. Are your findings intended to generalise beyond LaBraM and if so, how is that generalisation justified?\n\nLIMITATIONS:\nThe authors do include a limitations section but it is in the appendix and not integrated into the main discussion. The listed limitations are reasonable. They acknowledge the limited number of datasets and models used and the difficulty of decoupling all confounding factors in EEG data. However, these caveats are not reflected in the framing of their main conclusions, which sometimes overstate generality (e.g., claims about augmentation or self-supervised learning). These limitations should be surfaced earlier and more clearly tied to the scope of their findings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this article, the authors investigate scaling laws for EEG-based decoding of normality prediction, dementia diagnosis, and sleep staging. In particular, they investigate how decoding performance scales with the number of training subjects and with the number of examples per subject. Additionally, they investigate if data augmentation techniques and SSL pre-training improve performance.\n\nSTRENGTHS AND WEAKNESSES:\nThe figures and diagrams are clear and of good quality (vectorized).\nThe research questions (RQs) are relevant. In particular, answers to RQ 2 (Is collecting more data per participant a reliable way to compensate for a lack of participants?) could be used to guide data collection, known to be costly, in the future.\n\nHowever, **the experiments conducted by the authors do not allow to answer RQ2**. Indeed, to simulate a *scarce data regime*, the authors randomly sample examples from the train sets (lines 126 and 127). Because they do not provide more details, I assume that they are sampled using a uniform distribution over the available samples. This method is often used when benchmarking decoding algorithms. However, it can not be used to answer RQ 2 because it omits the long-term non-stationarities inherent to EEG recordings. These non-stationarities include slow drifts in the signal, loss of contact with the electrode leading to increased noise, subject fatigue, change in the mental state of the subject, eventual drug intake, etc. When the authors randomly sample n examples among all, they have a large diversity of examples regarding the different non-stationarities aforementioned, larger than if the n examples were consecutive, which will lead to overestimated generalization abilities.\n\nAdditionally, four RQs is probably too much content for one conference article. This article could easily be split into three separate articles: RQs 1+2, RQ3, and RQ4.\nFinally, this article might out of the scope of NeurIPS as it does not introduce any novel method. However, with a corrected evaluation procedure to properly answer RQ 2, ths article would be of great interest for the BCI comunity.\n\nQUESTIONS:\nCan you confirm that the examples were sampled using a uniform distribution over the available samples?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the impact of participant diversity and data volume on the performance of machine learning models used for EEG analysis. The authors use various EEG datasets and tasks and analyze the effectiveness of data augmentation and self-supervised learning as strategies to address data limitations. Their findings suggest that participant diversity (i.e., in 2/3 tasks including more data per participant was not effective to compensate for small participant numbers) often limits model generalization, highlighting the need for methods that specifically tackle inter-individual distribution shifts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors distinguish between overall sample size and participant diversity (i.e., number of participants and segments per participant). This level of experimental control enables a careful disentanglement of their respective effects.\n- The paper is clearly written and well-structured.\n- The study presents a systematic comparison across models and datasets. The use of multiple datasets and model types strengthens the generalizability of the findings. It is particularly valuable that the authors directly compare the effects of self-supervised pre-training and data augmentation.\n\nWeaknesses\n- The discussion of limitations is confined to the appendix. These should be addressed in the main text to provide a more balanced and transparent assessment of the work.\n\nQUESTIONS:\n1. What was the train/test/validation split used for each dataset?\n2. The paper shows that common EEG data augmentation methods failed to improve performance and, in some cases, even decreased the performance. While the authors mention that PhaseRandomization can destroy wave morphology and cross-channel correlation, which are critical for sleep stages, could the authors elaborate why they believe that AmplitudeScaling and FrequencyShift did not show an improvement in accuracy? Additionally,  what type of augmentations would mitigate participant differences?\n\n\nSmaller improvement:\n- Figure 5 caption: Capitalise pre-training. The sentence should be: \"...baseline models for the same seed. Pre-training consistently improves...\"\n\nLIMITATIONS:\nThe paper includes a dedicated section for \"Limitations and Future Work\" in Appendix A.1, however for transparency, this should be moved to the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the impact of both sample size and participant diversity for model generalisability and robustness when applying machine learning to EEG data. By viewing EEG generation as a multi-level process, the authors empirically analyse how model performance scales with data quantity and participant diversity. They further evaluate the impact of data augmentation and self-supervised learning in scenarios with limited data. This study reveals that distributional shifts among participants can significantly limit performance gains and offers practical recommendations for data collection and machine learning methodologies. The experimental code is also made publicly accessible to advance research within the community.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n\n1.\tThe paper presents what appears to be the first systematic investigation into how the number of data samples and subjects influences the performance of EEG-based machine learning models.\n2.\tThe experimental design is thorough and methodologically sound, incorporating both supervised and self-supervised learning approaches across datasets from diverse domains.\n\n**Weaknesses**:\n\n1.\tThe main conclusions—particularly the answers to the four questions posed in Section 4—are largely intuitive and may be considered unsurprising by readers familiar with the field. As a result, the paper offers limited new insights or conceptual advancement.\n2.\tThe role of the analysis in Section 3 is unclear. It is not evident how this section connects with the core experiments or contributes directly to the study’s central conclusions. A more explicit explanation of its relevance would strengthen the narrative.\n\nQUESTIONS:\n1. The main answers to the proposed four questions in Section 4 seem quite intuitive and might not offer much novelty to readers already knowledgeable in this area. Could the authors clarify what new insights or conceptual advancements their work brings beyond what is already expected?\n\n2. The purpose and significance of the analysis presented in Section 3 are somewhat ambiguous. How does this section specifically relate to the core experimental results or support the main conclusions of the paper? Providing a clearer explanation of its role would help improve the coherence and impact of this study.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper systematically investigates the impact of participant diversity versus overall sample size on the performance of EEG-based machine learning models. The findings highlight that participant diversity can be a critical factor, often more so than simply increasing the total number of segments, and demonstrate the potential of self-supervised pre-training to mitigate data scarcity.

STRENGTHS AND WEAKNESSES:
# Strength
1. Good writing and organization
2. The paper provides a valuable and systematic empirical analysis of the relative importance of "participant diversity" and "overall sample size" in EEG machine learning, a crucial aspect often overlooked in prior work.
# Weakness
1. The experiments in this paper are not sufficient the number of tasks, models, and datasets used is too small, which reduces the persuasiveness of the results.
2. The manuscript lacks a conclusion section.
3. While the manuscript provides a detailed analysis of the experimental phenomena, it offers limited discussion on the underlying causes behind these observations.

QUESTIONS:
1. Why were TCN, mAttn and LaBraM chosen? Can these three models cover the commonly used method categories in EEG-based tasks? It is suggested to explain it in the paper.
2. In each task domain (normality prediction, dementia diagnosis, sleep staging), what augmentations are commonly adopted in prior literature? Are amplitude scaling, frequency shifting and phase randomization considered best-practice baselines? If not, how can the universality of your Q3 conclusion be claimed?

LIMITATIONS:
YES
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29