The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning
Authors: Dulhan Jayalath
Gilad Landau
Brendan Shillingford
Mark Woolrich
Oiwi Parker Jones
Venue: ICML 2025 poster (2025)
Keywords: neural decoding
speech decoding
neuroscience
Primary area: applications->neuroscience_cognitive_science
TL;DR: We propose neuroscience-inspired self-supervised objectives that enable learning from heterogeneous and unlabelled neural recordings, unlocking the potential for training speech decoding models with significantly more existing MEG data.

Abstract: The past few years have seen remarkable progress in the decoding of speech from brain activity, primarily driven by large single-subject datasets. However, due to individual variation, such as anatomy, and differences in task design and scanning hardware, leveraging data across subjects and datasets remains challenging. In turn, the field has not benefited from the growing number of open neural data repositories to exploit large-scale deep learning. To address this, we develop neuroscience-informed self-supervised objectives, together with an architecture, for learning from heterogeneous brain recordings. Scaling to nearly **400 hours** of MEG data and **900 subjects**, our approach shows generalisation across participants, datasets, tasks, and even to *novel* subjects. It achieves **improvements of 15-27%** over state-of-the-art models and **matches *surgical* decoding performance with *non-invasive* data**. These advances unlock the potential for scaling speech decoding models beyond the current frontier.

The Brain's Bitter Lesson:
Scaling Speech Decoding With Self-Supervised Learning
Dulhan Jayalath
Gilad Landau'
Brendan Shillingford
Mark Woolrich Oiwi Parker Jones
Abstract
G supervisdd
The past few years have seen remarkable progress in the decoding of speech from brain activity, primarily driven by large single-subject datasets. However, due to individual variation, such as anatomy, and differences in task design and scan- ning hardware, leveraging data across subjects and datasets remains challenging. In turn, the field has not benefited from the growing number of open neural data repositories to exploit large- scale deep learning. To address this, we develop neuroseience-infomee
Te (f
Unlabelled MEG (abundann))
T(x)
Dupersiool
Labeleed MEG (scarce)
supe-super objectives, together with an architecture, for learning from heterogeneous brain recordings. Scaling nearly 400 hours of MEG data and 900 subjects, ap- proach shows generalisation across participants, datasets, tasks, and even to novel subjects. It achieves improvements of 15-27% % over state-of- the-art models and matches surgical decoding performance with non-invasive data. These ad- vances unlock the potential for scaling speech decoding models beyond the current frontier.
Figure 1: Leveraging unlabelled data using pretext tasks for speech decoding. We pre-train neural network using tasks that generate implicit labels from abundant unlabelled MEG neuroimaging data, permitting learning from large heterogeneous datasets. The tasks apply a randomly selected neuroscientifically relevant transformation T the data and the network predicts the transformation. We then train linear probe top of the pre-trained model, which remains frozen, with labelled data, achieving superior generalisation owing to the strength of the representation.
1. Introduction
Recent work towards brain-computer interfaces (BCIs) have tried to scale up labelled datasets for individual subjects, using either invasive (Moses et al., 2021; Willett et al., 2023) or non-invasive brain recordings (Tang et al., 2023), map- ping these to transcripts of attempted or imagined speech. Yet, a number of obstacles to scale remain, especially in magnetooencepaalography (MEG) data. Current speech de- coding models rarely train multiple subjects, combine datasets, utilise data from diverse tasks. Thus the size of training data has been limited to how much can be ac- quired for a single subject, and data from other subjects, or from the growing number of public data repositories, has not been leveraged. There are many reasons for these limitations; individual brains and data from different neu- roimaging scanners differ, for example. But overcoming these limitations holds the promise of training models on collective, internet-scale data.
In his Bitter Lesson, Richard Sutton argues that major conclusion of 700 years of AI research is that general methods exploiting large-scale computation will outperform model- based approaches as the availability of compute increases (Sutton, 2019). The ability of deep learning to learn from ever-larger datasets has enabled seemingly arbitrary scaling with computation, leading to astounding advances across diverse set of domains (Jumper al., 2021; Caron al., 2021; OpenAI, 2023; Radford et al., 2023).
In the domain of brain data, and of tasks like speech de- coding, the bitter lesson has not yet been fully assimilated.
'PNPL & OOHBA, University of Oxford Gooole DeepMind. Correspondence to: <(dulhan, oiwi) | @ @wwbots.ox..oooo wx.ac.uk...........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
sttpp::/pppl) robots .ox ac. www.ac.ub/bbb

---

The Brain's Bitter Lesson
Decoding methods for MEG need to be highly data-efficient. While electroencephalography (EEG) data abundant, MEG provides richer signals for decoding (Lopes da Silva, 2013; Hall et al., 2014) but are comparatively rare. Given the scarcity of speech-labelled MEG and the larger proportion of other MEG data, self-supervised learning (SSL) appears promising as it is an avenue for domains where labels are rare or hard to obtain (Balestriero et al., 2023). To data- efficiently learn from unlabelled MEG, we propose pretext training with neuroscience-informes input transformations that benefit downstream tasks. We use this for learning from unlabelled brain data (Figure 1) through an architecture for processing continuous multi-sensor neuroimaging signals. Our method provides a unified approach that enables lever- aging data from other experiments that do not have the same labels (by treating them as unlabelled) and that come from different subjects and neuroimaging scanners. We evaluate representations learned with our approach on heard speech datasets acquired with MEG, setting the baselines for speech detection and voicing classification on this data.
(e.g. Anumanchipalli al. (2019) and Makin et al. (2019) in surgical settings and Csaky et al. (2022) non-invasively). Défossez et al. (2023) show cross-subject generalisation for segment identification task from participants listening to connected speech. However, they do not demonstrate gener- alisation to novel subjects and retrain their model for new datasets rather than being able to generalise across datasets or pool them. Their method is also unable to incorporate data without corresponding audio labels and so does not scale with other kinds of tasks.
In general, speech decoding has centred on different kinds of speech: listening, imagining, speaking out loud, and, for paralysed patients, attempting to speak aloud. We focus on listening because it easier to decode than imagined speech (e.g. Martin et al. (2014)). There is also evidence of func- tional overlap between listening and imagined speech rep- resentations in the brain (Wandelt et al., 2024), though the question of overlap has been contested (Langland- Hassan & Vicente, 2018). While some work on decoding text di- rectly from heard speech tasks with MEG and EEG ex- ist, is unclear whether these methods perform any better than a baseline that provides pure noise inputs to the model (Jo et al., 2024). Non-invasive speech decoding remains a highly challenging and unsolved domain.
Our main contributions are:
A domain--peccfic self-supervision method and a neu- ral architecture for representation learning from MEG that unlock scaling speech decoding over multiple sub- jects, multiple studies, and unlabelled data;
Self-supervised pretext tasks have been successful in com- puter vision (Agrawal et al., 2015; Doersch et al., 2015; Noroozi & Favaro, 2016; Larsson et al., 2016; Zhang et al., 2016; Gidaris et al., 2018) but rarely applied to brain de- coding. There are, however, methods that leverage unla- belled brain data in other ways (Banville et al., 2019; Kostas et al., 2021; Le & Shlizerman, 2022; Zhang et al., 2023; Yi et al., 2023; Cai et al.. 2023; Ye et al., 2023; Yuan et al., 2024; Chen et al., 2024b). Unfortunately, most of this lit- erature is unable to scale and harmonize heterogeneous non-invasive data. The most notable of these works include Wang et al. (2023), who learn contextualised embeddings of time-frequen frenuefreuency input representations through masked spec- trogram in-filling. Their impressive speech detection results were achieved with invasive neural recordings, which are comparatively rare and thus have less potential to scale than non- non-invasive data. Another, BIOT (Yang et al., 2023), learns from generic heterogeneous bio-signals with a contrastive pre- pre-training objective, rather than masking, and applies it to ECG/EEG data. Notably, none of these methods optimise their objectives for speech decoding focus on MEG.
Achieving 15-27% gains over state-of-the-an self- supervised models, matching surgical self-supervised decoding non-invasively, and showing novel subject generalisation for the first time in MEG; and
Demonstrating evidence for scaling laws arising from pre-training with unlabelled MEG recordings using multiple times the volume of data in prior work.
2. Related Work
Prior work in speech decoding has focused almost entirely on supervised learning with decoding models that typically do not generalise across participants experiments. This is true both in recent state-of-the-an invasive studies (Moses et al., 2021; Metzger et al., 2023; Willett et al., 2023; Chen et al., 2024a) and non-invasive studies (Tang et al., 2023). These prior works have scaled up the experimental data col- lected within individual subjects, but are unable to leverage data from other subjects and experiments. Nevertheless, the method developed by Tang et al. (2023) is remarkable for showing an ability to generalise across labelled task data. They do not however, use unlabelled data or show cross-subject generalisation.
3. Method
We introduce a neural architecture to embed heterogencous brain signals. Then, we leverage this architecture for self- supervised learning from unlabelled MEG data using set of pretext tasks designed to generate generalisable brain representations for speech decoding.
Specific studies into the limitations of generalising models between subjects show that while performance decreases on average when subjects are pooled, there are exceptions
2

---

The Brain's Bitter Lesson
3.1. Network Architecture
Table I: Functional frequency bands in brain activity.
Our neural network architecture has two stages (Figure 2): pre-training with pretext tasks on unlabelled data, and train- ing a linear probe with labelled data for downstream tasks.
Band
Hz
Association
Delta (6)
.1-4
Rhythmic structure of heard speech (Luo et al., 2010) Tracking (Luo & Poeppel, 2007) and phase-locking to the ampli- tude envelope of heard sentences (Peelle et al., 2012)
We divide recordings into windows of length w seconds or I samples. At train time, each batch of windows is standard- ised such that each sensor has zero mean and unit variance. The network takes as input the standardised sample win- dows. To combine heterogeneous datasets, which have vary- ing numbers of sensors S, we apply dataset-condittioal linear layer to the sensor dimension, projecting the signal into shared space with dimension dahared. Then, to en- code the signal, we construct a wave-to-wave convolutional encoder architecture, the cortex encoder, inspired by work in neural audio codecs (Zeghidour et al., 2022; Défossez et al., 2022). Specifically, our convolutional encoder adapts the SEANet architecture (Tagliasacchi et al., 2020) used in Défossez et al. (2022) which we describe here and as part of Figure 2. As these codecs typically operate on mono audio signals in 1xxt, while our signals are in Rdsshared xt we increase the convolutional channel dimension from 1 to match dshared while also inflating the channel dimension of subsequent convolutions. We refer to the output dimension of embeddings from this backbone dhackbone- Thus, the backbone takes as input a window in R5xt and encodes this into 7 embeddings, each of dimension dbackhone (i.e.
Theta (0)
4-8
Alpha (a)
8-12
Attentional processes and the in- hibition of irrelevant information
(StrauB et al., 2015)
Beta (8)
12-30
Top-down predictive coding (Bressler & Richter, 2015)
which affects lexical processing (Weiss & Mueller, 2012)
Gamma (7)
30-70
Higher cognitive functions (c.g. memory, learning, reason- ing, and planning) (Fries, 2009;
Buzsáki & Wang, 2012) Speech detection (Hamilton et al., 2018) and phonemic feature classification in the STG (Mesgarani et al., 2014) and the ventral sensorimotor cortex (vSMC) (Cheung et al., 2016)
High
70+
Gamma (?hish)
an RR dhaackkoone XT output).
embeddings are flattened into a tensor in Rhhackhonee XT rep- resenting the entire window. This is the input to the voicing classifier and is referred to as full epoch decoding in neu- roimaging literature (Csaky et al.. 2023).
In the speech recognition literature, models include speaker conditioning to account for vocal and prosodic differences (Gibiansky et al., 2017). Just as speakers have different voices, neural responses between subjects have different characteristics. Consequently, individual variation leads to models that do not generalise well across subjects (Csaky et al., 2022). We address this with a similar approach to the speech literature by introducing subject conditioning linear modulation (FiLM) (Perez et al., 2018). As Zeghidour et al. (2022) find that conditioning as equally effective the encoder bottleneck as in other stages of the model, we also condition at the bottleneck.
3.2. Pretext Tasks
To use our architecture for pre-training, we construct pretext objectives for unsupervised learning of generalisable speech decoding features. These objectives are inherently agnostic to the sensor count because they operate on properties that are independent of the specific sensor arrangement. This key design choice enables the tasks to work seamlessly across datasets with varying numbers of sensors- a critical requirement for combining heterogeneous brain data.
Following Balestriero et al. (2023, Section 3.2), we use two-layer projector to alleviate misalignment between our pretext and downstream tasks in the representation. After the projector, linear classifiers make predictions for each of the pretext tasks. When fine-tuning, we train a linear decoder, for a downstream task, on top of the pre-trained representation, which remains frozen. Thus, we backpropa- gate only through the classifier. A trainable -ptaset-specific linear layer can be introduced for a novel dataset.
Band prediction. In the literature, neural responses can be segmented into functional frequency bands (Giraud & Poeppel, 2012: Piai et al., 2014; Mai et al., 2016) (Table 1). Sensitivity to these frequencies would bring about functional separability in the representation space. Thus, learn such representations,
train the network to classify rejected bands. As High Gamma is a relatively wide band we split it into two sub-bands: Lower High Gamma high 770 Hz) and Upper High Gamma (* high waves (100- 150 Hz). Our task applies a band-stop filter for a randomly selected band w to the sample æ, passes the filtered sample
For speech detection, our classifier makes a prediction for each individual embedding. For voicing classification, where there only one label for each sample window, the
(Thower) waves
3

---

The Brain's Bitter Lesson
Lineer Classifiere
Pre-training
Band Predictor
Conv1D
Phase Shift Prodictor
Linear
Conv Block
Projector
Ampiltude Scale Predictor
MEG Inputs
Linear
Conv Block
Speech Datector
Linear
Fine-tuning
Volcing Classifier
LSTM
Conditional
Layer
Cortex Encoder
Subject Conditoning
Figure 2: Architecture overview. Inputs are projected into a shared dimension by the dataset-conditional layer, then encoded. In pre-training, all weights trainable except for modules in light-red, while in fine-tuning, modules with light-blue borders frozen and modules with light-red borders unfrozen. Dashed borders indicate optional components.
xtd through the network backbone g and the corresponding linear predictor Sband, requiring the network to classify which frequency band w was rejected. This yields the loss
Amplitude scale prediction. MEG and EEG signals use an array of sensors at different spatial locations, capturing different signal sources more intensely. Representing the rel- ative amplitude difference between sensors could be impor- tant for differentiating between neural responses originating from distinct parts of the brain. Within speech, Hamilton et al. (2018) find that localised regions of the STG respond to sustained speech and speech onsets. Differentiating be- tween neural responses from this region and others may be essential for decoding speech perception.
Lhand Σ Chanaaand w), ZEB
where B a mini-batch of samples, w J. and Lce is the cross-entropy loss this is multi-class classification task.
E
{6, a, B, 7, Nlwer hhgh
high
Phase shift prediction. Phase coupling between networks of neuron populations is necessary for coordinating brain activity (Fries, 2005; Vidaurre et al., 2018) and so phase coupling between spatially distant sensors likely be useful feature. Supporting this insight, recent work (Jiang al., 2024) also finds phase be an essential component of the signal.
Thus, this pretext task focuses on learning representations that encode relative sensor amplitude differences. Similar to the phase shift task, we select random proportion of the sensors P E [0, 0.5] and apply discrete random amplitude scaling coefficient A E [-2, 2]. discretised into 16 scaling factors, to the signal. The objective is to predict the scaling factor, leading to the loss
To learn representations that encode phase differences be- tween brain areas, this task applies discrete uniform ran- dom phase shift & E {0, Σ x. 3n a 5x 3x 7m 1 to a uni- 8 3' 8: 2 8.
Lamplitude
Σ CCE(Fampltuue ()(x A), JEB
(3)
formly randomly selected proportion p 0.5) of the sensors. Applying this shift to random sensors is critical since sensors are placed in different positions, capturing different regions of the brain. Uniform random selection ensures differences between any two regions of the brain are represented. The objective of this task is to predict the phase shift. This leads to a similar loss
where x is the signal scaled with A.
Combined tasks. These pretext tasks capture complemen- tary time- and frequency-domain properties of the signal. Hence, during pre-training, we combine them, creating an augmented version of the input for every pretext task by ap- plying the matching transformation. We feed the augmented inputs through the network backbone and apply the corre- sponding classifier to predict the transformation, summing the weighted losses such that our final pre-training loss is
Ephase
Σ (prseeeeelgg d). FEB
where d describes the signal with phase shift ø applied to proportion of the sensors. We use discrete number of possible phase shifts, treating it as a multi-class task rather than a regression task, to ease the difficulty of the problem as MEG scanners typically have large number of sensors.
LSsL. = Whh Lbsan + W2 + W3 Camplitude,
(4)
where w, is constant coefficient for each self-supervised loss.

---

The Brain's Bitter Lesson
4. Experiments
on speech detection. This task determines whether speech occurs in the auditory stimulus using the neural response. This is a fundamental task in understanding speech percep- tion and is one of the few tasks that so far show statistically significant results in highly noisy MEG signals. It also has direct applications to BCIs as it can be used to segment words or sentences for decoding and activate a speech BCI when a patient wishes to communicate. Secondarily, we also study voicing classification to demonstrate the versatility of our representations for general speech decoding tasks, Given data aligned at the onset of phoneme, the task is to recognise whether the phoneme is voiced or voiceless, where voicing is a phonetic feature that categorises whether speech sound is associated with vocal cord vibration. This task is also directly relevant to a speech BCI as it involves classifying phonemes which can be used to decode words.
We evaluate our self-supervised representations by mea- suring how they scale with unlabelled data and generalise across datasets, subjects, and tasks. We focus our evaluation on MEG data as the signal is rich, with better spatial resolu- tion than EEG (Lopes da Silva, 2013) and faster sampling rates than iMRI (Hall et al., 2014). We pre-train all models to completion and then train a linear probe on labelled data for each task. In all tables and figures, we quote the receiver operating characteristic area under the curve (ROC AUC) where chance is always 0.5 regardless of the class distribu- tion. We show the test ROC AUC at the best validation ROC AUC (carly stopping) and quote uncertainty the standard error of the mean over up to five seeds.
4.1. Experimental setup
Datasets. In total, we use almost five times the volume of data in prior MEG work, totalling approximately 400 hours with nearly 900 subjects across pre-training and down- stream training. Unless specified otherwise, we pre-train with Cam-CAN (Shafto et al., 2014; Taylor et al., 2017) as an unlabelled representation learning dataset. This is study containing 641 subjects with resting and sensorimotor tasks, totalling approximately 160 hours of MEG recordings. When aggregating datasets, we also pre-train with MOUS (Schoffelen et al., 2019), which contains 204 subjects and another 160 hours from visual and auditory tasks. Down- stream, we use labelled heard speech MEG datasets where participants listen to short stories or audiobooks. We mainly focus on Armeni et al. (2022) which contains 3 subjects who listen to 10 hours of recordings each (30 hours total). We also analyse Gwilliams et al. (2023) which has 27 subjects, each recorded for 2 hours (54 hours total). The latter dataset is particularly difficult to decode from as there is very little within-subject data and it did not enforce the use of head casts to immobilise participants. Nevertheless, given it has many more subjects, we use this dataset to study subject generalisation.
4.2. Learning Generalisable Representations Using Pretext Tasks
Table 2 shows that our approach achieves two key feats: outperforming comparable state-of-the-an self-supervised methods by 15-27% (part C), and matching the performance of prior self-supervised methods with surgical data (11) while using only non-invasive data. Since non-invasive data has a much lower signal-to-noise ratio than surgical data, this is quite unprecedented. the rest of this section, we analyse this table in detail.
In part B, we show the results of pre-training models with each pretext task independently, together, and without any pre-training at all. Using pretext tasks (3, 4, 5) outperforms no pre- trainini (2). Interestingly. the combination of all pretext tasks (5) leads to better generalisation than any task on its own (the improvement over (3) is statistically signifi- cant). We conjecture that this is because our pretext tasks capture complementary properties in time- and frequency- space, ensuring that the representation includes more salient features than any individual task could encode. Finally, we apply Gaussian filtering to the predictions (7), smoothing out anomalies in the predicted speech envelope.
Preprocessing, Each recording is in SXT where S is the number of sensors and T' is the number of time points sam- pled by the scanner. To eliminate high-frequency artifacts, we apply a low-pass filter at 125Hz as well as a high-pass fil- ter at 1.1 to remove dlow-drift artifacts. Since the datasets were recorded in Europe, where the electric grid frequency is 50Hz, we apply notch filter at 50Hz and its harmonics to account for line noise. Treating the low-pass filter thresh- old as the Nyquist frequency, we downsample the signal to twice that at 250Hz, avoiding aliasing within our band of interest. Finally, we detect sensor channels with significant noise and artifacts using a variance threshold and replace them by interpolating the spatially nearest sensors.
Next, we turn to the baselines, starting with Table part A. Our method outperforms random selection (0) and training linear layer with the MEG signal directly (I). The latter even has substantially more trainable parameters because the imput dimension is larger without an encoder. In part C, we compare our approach to two ofate-of-the-arr self-supervised methods. In each experiment, we apply Gaussian filtering as in (7). Although better than random, BrainBERT (9) does not generalise as well as our method (11). BrainBERT em- ploys generic masked spectrogram in-filling pre-training objective. While it is intended for speech decoding tasks, the pre-training objective is not designed to specifically capture features salient to neural speech processing. Furthermore,
Downstream tasks. We evaluate methods primarily
5

---

The Brain's Bitter Lesson
Table 2: Our approach surpasses baselines in speech detection by up to 27% and matches surgical decoding. For linear, we train a supervised linear classifier on the MEG signals. For ours and BrainBERT, we train a linear layer on top of a backbone pre-trained on CamCAN, with the rest of the model frozen. For BIOT, we use their pre-trate pre-trained weights. In the no pre-training baseline, the backbone uses randomly initialised and frozen weights. In the surgical context, we quote the result from Wang et al. (2023, Table 2). With all pretext tasks, losses are weighted equally.
ent tasks and datasets. For speech detection on Armeni et al. (2022), we observe logarithmic scaling in log-space (log- log scaling), suggesting diminishing but continued returns with increased data. For other tasks, ROC AUC improves log-linearly, indicating robust scaling potential. Importantly, even our smallest pre-training dataset beats chance perfor- mance, while our largest (160 hours) continues to show gains without plateauing. Notably, we have scaled far be- yond the data regime of prior surgical and non-surgical work and yet performance has continued to scale. Thus, our self- supervision approach may remain useful as the volume of open data in the field continues to rapidly increase.
Part / ID
Model
ROC AUC
0
Random select
.500
Our results also reveal several new and notable phenom- ena. We scaled up the pre-training dataset by increasing the number of subjects. Since this led to consistent and almost monotonic improvements in downstream accuracy, our method is an exception to the consensus that pooling subjects worsens generalisation. As we pre-trained our model with a different dataset to those we fine-tuned on, our representation shows cross-datass generalisation. This is surprising as the Armeni et al. (2022), Gwilliams et al. (2023), and our pre-training dataset all use different scan- ners entirely. Performing well across these datasets indicates that, together, our architecture and pretext tasks successfully generate representations that are generalisable across het- erogeneous scanners. Finally, we note that our pre-training dataset contained no language data whatsoever yet still im- proved downstream accuracy on language tasks. Remark- ably, this shows that unlabelled brain data collected from any task (including those that are not linguistic) can be used to improve speech decoding performance.
A
1
Linear
.5394.002
2
Ours
No pre-train.
.519+.002
3
+ linear
Amp 0.2)
.624+.001
4
Phase 0.5)
.615+.001
B
5
Band
.5881.001
6
All tasks
.630g.000
7
+ smoothing
700+.002
8
BIOT' + linear
.615..002
C*
9
BrainBERT + linear
.556..007
10
EEGPT linear
.602...06
11
Ours (best) + linear
1.05..003
12
BrainBERT + lin. (surgical) 1.06
Yang et al. (2023) Wang et al. (2023) 'Wag et al. (2024)
their method is less data-efficient because in-filling is harder generative task compared to classification and while their methodology was developed for relatively high-fidelity intracranial recordings, the inherently lower signal-to-noist -oise ratio of MEG presents an even greater challenge.
Since the results show improvements on both downstream tasks, this indicates that our pretext tasks are sufficiently generic to produce representations that work with multiple speech decoding tasks while still generalising well on each task individually. This is generally a challenging trade-off to manage. However, we notice that in both tasks, the base accuracy is higher and the improvement in ROC AUC is steeper for Armeni et al. (2022). This is likely to be because this dataset has more within-subject data. The weaker results for Gwilliams et al. (2023) may be a consequence of shorter intra-subject recordings, greater subject variation, and the lack of head casts in data collection. These observations support the findings of work such as Csaky et al. (2022).
Our last baselines are BIOT (8) and BEGPT (10). We lever- age publicly released weights pre-trained with thousands of hours of EEG. However, both still fall short of our pre- training method for reasons which we believe are similar to BrRinBERTTthei objectives do not leverage neuroscien- tific understanding of speech processing.
Finally, and quite remarkably, our best result matches the AUROC quoted in Wang et al. (2023, Table 2) who use intracranial data from heard speech (12). We achieved this score with non-invasive data which is typically substantially more difficult to decode due to the low aggal-to-noiss ratio.
4.4. Scaling Unlabelled Data Improves Generalisation to Novel Subjects
4.3. Scaling Speech Decoding With Unlabelled Data
Figure 3 shows that performance scales predictably with unlabelled data volume, following distinct patterns for differ-
In neuroimaging, brain data is generally highly variable across participants, leading to difficulty transferring mod- els to novel subjects (Csaky et al., 2022). Whilst we have shown generalisation across subjects, here, we investigate whether we can generalise to novel subjects an even more
pattems
'Due to the large computational cost of processing embeddings for MEG data in BrainBERT, we restrict pre-training of experi- ments in part C to approximately 30 hours and use only subject 001 of the downstream dataset for training and evaluation.
6

---

The Brain's Bitter Lesson
Armeni
Gwilliams
Speech detection
Voicing classification
0.55
0.625
0.54
0.600
ROC AUC
Défossez al. (2023)
ROC AUC
0.575
0.53
0.550
0.52
0.525
Moses al. (2021)
0.51
0.500
10
10a
102
10
10a
102
Unlabelled data (hours)
Unlabelled data (hours)
Figure 3: Scaling unlabelled data improves generalisation. We pre-train the model increasing amounts of unlabelled data from Cam-CAN (Shafto et al., 2014; Taylor et al., 2017). The solid lines are linear fits and the dashed lines show the volume of data used in prior surgical (Moses et al., 2021) and non-invasive (Défossez et al., 2023) work. Unlike Table (7) we do not apply Gaussian filtering to the predictions for simplicity. The best improvements are statistically significant.
difficult challenge. This critical in order to widely deploy speech BCIs for new patients. In this experiment, we use Gwilliams et al. (2023) our downstream dataset because of the large number of participants, holding out three sub- jects with which we evaluate novel subject generalisation.
Table 3: Aggregating unlabelled datasets outperforms single studies in speech detection. For the first time with MEG, we show that unlabelled pre-training data from mul- tiple studies with different hardware profiles can be aggre- gated while gaining the benefits of scaling. Combining data leads significant (p < 0.05) improvement.
Figure 4 shows that scaling up the amount of unlabelled data used in pre-training not only improves accuracy on subjects previously seen, but also demonstrates positive log-linear trend in performance for novel subjects. This indicates that scaling our method is encouraging direction for resolving the challenges of subject variance faced by prior work. As far as we are aware, this is the first result to demonstrate novel subject generalisation in speech decoding from MEG.
Pre-training Data
Hours
ROC AUC
CamCAN',2
159
63001.0001
MOUS
160
.614
CamCAN 12 MOUS
319
.6384.0002
Shafto et al. (2014) TTyloo et al. (2017)
Schoffelen al. (2019)
4.5. Aggregating Unlabelled MEG Datasets
Given the promising scaling results with single datasets, a natural question arises: can we achieve even better per- formance by combining multiple MEG datasets? This is particularly challenging since datasets often use different scanning hardware and experimental protocols. Thus, it has so far not been shown in MEG.
pre-training on Cam-CAN was better than pre-training on MOUS given that MOUS and the downstream dataset both used speech tasks and were acquired on the same MEG scanner. Cam-CAN, by contrast, did not speech task and was acquired sise that the better results for Cam-CAN are due to it being cleaner. During our experiments, we found that data quality, even among unlahelled data, can have a significant effect as artefacts in recordings disrupt learning.
different MEG scanner. We hypothe-
As a preliminary investigation, we combine two of the largest public MEG datasets: MOUS (Schoffelen et al., 2019) and Cam-CAN (Shafto et al., 2014; Taylor et al., 2017). In this section, we investigate how pre-training with these combined datasets affects downstream performance using the same experimental setup as Figure
While the combination of the two datasets includes far more hours of data than any prior work on deep learning with MEG, further work needs to be done aggregate more datasets. Here, we were limited by compute budget and data availability. Increasing the number of datasets (c.g., by including EEG too) could further improve results. Just as increasing the number of subjects (rather than only within- subject data) improves novel subject generalisation, larger
The results in Table 3 show, for the first time, that combining datasets can improve performance on downstream speech decoding tasks. It leads to better performance compared to pre-training on either dataset alone. is surprising that

---

The Brain's Bitter Lesson
In-Distribution
Out-of-Distrbutiion
Speech detection
Voicing classification
0.55
0.530
0.54
0.525
ROC AUC
0.53
Défessez et al. (2023).
ROC AUC
0.520
0.52
0.515
0.51
0.510
0.50
Moses (2021)
0.505
0.49
10
10
10a
10c
10a
102
Unlabelled data (hours)
Unlabelled data (hours)
Figure 4: Scaling unlabelled data improves novel subject generalisation. We train linear probe Gwilliams et al. (2023). When in-distribution, we evaluate on held-out sessions; when nit-o---ditrrbbttonn we evaluate on held-out subjects. The lines represent the same as in Figure 3. The best improvements are statistically significant.
number of datasets may be key to scaling results when datasets are aggregated in pre-training.
Perhaps the biggest limitation of the present work is that, while surpasses the amount of data used other studies, it remains to be how much speech decoding tasks can be improved by scaling up the number of datasets used in training. In sharing this work now, we believe that the current proof of concept will be sufficiently impactful to the field as we continue to actively scale up the datasets that we can leverage.
4.6. Limitations
Although our results are significant in demonstrating a vi- able path forward scale up speech BCIs, there remain number of limitations to the present work. We focused here on two downstream tasks: speech detection and voicing clas- sification. Ultimately, we would like to expand this work to predict full transcripts from brain recordings (i.e. brain- to-tex(). This has been achieved with surgical data (Moses et al., 2021; Willett et al., 2023) but not yet convincingly with non-invasive methods like MEG EEG (Jo et al., 2024). Speech detection has played an important role in the development of full brain-to-text in surgical context (Moses al., 2021) and we hope may play similar role for non-invasive methods. In future work, we would also like to expand classification to all English phonemes as a step towards full transcript decoding.
5. Conclusion
Speech decoding from the brain has been limited by the field's inability to scale up data to leverage deep learning. Prior methods have been unable to aggregate data across different datasets, labels, or subjects to scale up because of heterogeneity in recording hardware, experiment design, and participants. A handful of studies have shown weak signals towards alleviating these issues. But until now, no one has developed general solution. We present uni- fied solution through data-efficient, self-supervised pretext tasks that overcome these fundamental scaling challenges. Our experiments demonstrate not just scaling with heteroge- ncous data, but generalisation across datasets, subjects, and tasks. They also show significant improvements of up to 27% compared to the prior state-of-the-e and even provide evidence of matching surgical decoding performance. Our method unlocks the potential of the bitter lesson, providing general method to exploit more computation by using more data. We implore the research community to employ the vast quantities of data and compute available to realise this potential. If scale all you need in speech decoding, then the bitter lesson may be bitter.
Additionally, while we have been able to demonstrate the utility of few pretext tasks, we do not claim to have ex- hausted the full set of useful tasks. Rather, that more useful pretext tasks remain to be found and be- lieve useful avenue of research will be into other input representations for brain recordings. For example, this pa- per did not make use of spatial features when the geometry of a scanner's sensor configuration is strongly correlated with the area of the brain from which the signal is derived. Another limitation is emphasis heard speech over other types of speech, such as attempted or imagined speech. We hypothesise that the same methods presented here will generalise to these other varieties of speech, though this has yet to be shown.
conjecture
8

---

The Brain's Bitter Lesson
Acknowledgements
could enable unauthorized access to neural information, requiring robust safeguards against exploitation.
We would like to thank Botos Csaba for many early insight- ful discussions which helped shaped the direction of this work. In alphabetical order, thanks also to Mats W.J. van Es for technical assistance with the OSL library, Yonatan Gideoni for advice on data splits, Minqi Jiang for an encour- aging conversation on scaling unsupervised representation learning, Brian Liu for technical contributions which did not reach the final paper, and Miran Özdogan for reviewing a draft of this work. Finally, we thank the rest of the PNPL. group for their continued and unwavering support.
Widespread adoption could affect pri- vacy norms around inner speech, while unequal access could exacerbate existing inequalities.
Societal Impact:
We focus specifically on decoding heard speech rather than inner speech, limiting potential misuse. Nevertheless, we recognize that advances in heard speech decoding contribute to the broader development of neural decoding technology. We encourage the research community to actively engage with these ethical considerations as the field progresses.
The authors would like acknowledge the use of the Uni- versity of Oxford Advanced Research Computing (ARC) fa- cility in carrying out this work. ht http:: //xx://xx// doi.org/ 10. 1281/ zenodo. 22558.
References
Agrawal, P., Carreira, J., and Malik, J. Learning to see by moving. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiugo, Chile, De- cember 7-13, 2015, pp. 37-45. IEEE Computer Soci- ety, 2015. doi: 10.1109/CCC.2005.133. URL https: //dpi.org///aao http://....111/11. www.cccc......... 20.
DJ is supported by an AWS Studentship from the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (AIMS) (EP/S024050/1). GL is
supported by EPSRC Studentship. MW is supported by the Wellcome Trust (106183/2/14/2, 215573/2/19/2), the New Therapeutics in Alzheimer's Diseases (NTAD) study supported by UK MRC, the Dementia Platform UK (RG94383/RG89702) and the NIHR Oxford Health Biomed- ical Research Centre (NIHR203316). The views expressed those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. OPJ is supported by the MRC (MR/X00757X/1), Royal Society (RG) RI R1 241267). NSF (2314493), NFRF (NFRFT-2022- 00241), and SSHRC (895-2023-1022).
Anumanchipalli, G. K., Chartier, J., and Chang, E. F. Speech synthesis from neural decoding of spoken sentences. Na- ture, 568:493 498, 2019.
Armeni, K., Güçlü, U., van Gerven, M., and Schof- felen, J.-M. A 10-hour within--prricipant magne- toencephalography narrative dataset to test models of language comprehension. Scientific Data, 9(1): 278, June 2022. $41597-022-01382-- URL https: www.w...w....w.tw.twttwwwww....w
ISSN 2052-4463. doi: 10.1038/
Impact Statement
This work uses publicly available datasets from human stud- ies (Armeni et al., 2022; Gwilliams et al., 2023; Shafto et al., 2014; Taylor et al., 2017; Schoffelen et al., 2019), each with their own ethical approvals and documentation available in their respective publications.
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F,, Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A. G., Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., and Goldblum, M. A cookbook of self-supervieed learning. CoRR, abs/2304. 123304.1220 12210, 2023. doi: 10.48550/ARXIV. http://dd...og/..../// 48550/ arXiv. 23044 12210.
Neural speech decoding research has transformative po- tential for healthcare and assistive technology. Advances could help paralysed patients communicate freely and as- sist those with communication difficulties. By developing non- invasive methods, the field opens up the possibility of broader access to these technologies without the risks of surgical implants.
2304. URL tttpp::///.......
Banville, H. J., Moffat, G., Albuquerque, I., Engemann, D., Hyvärinen, A., and Gramfort, A. Self-supervised representation learning from electroencephalograply sig- nals. In 29th IEEE International Workshop on Ma- chine Learning for Signal Processing, MLSP 2019, Pitts- burgh, PA, USA, October 13-16, 2019, pp. 1-6. IEEE, 2019. doi: 10. 10.1109/ 091.1119..99999999 89693 URL https: /
However, we acknowledge potential societal risks as this technology matures:
Privacy and Data Protection: Brain signals contain highly sensitive personal information, raising concerns about data security and individual privacy.
/0..0000111 1 1109111100011 /LSSP 2019 8918693.
Bressler, S. L. and Richter, C. G. Interareal oscillatory syn- chronization in top-down neocortical processing. Current Opinion in Neurobiology, 31:62-66 66, 2015.
Consent and Misuse: Advanced decoding capabilities

---

The Brain's Bitter Lesson
Buzsáki, G. and Wang, X.-J Mechanisms of gamma os- cillations. Annual Review of Neuroscience, 35:203-225, 2012.
2022. doi: 10448550ARRXVV:2100.1552 URL https: 22338.
/ o/db..org//1..45550/0///
Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In 2015 IEEE International Conference on Computer Vi-
Cai, D., Chen, J., Yang, Y., Liu, T., and Li, Mbrain: multi-channel self superviie learning framework for brain signals. In Singh, A. K., Sun, Y., Akoglu, L.,, Gunop- ulos, D.. Yan, X., Kumar, R., Ozcan, F., and Ye, J. (eds.), Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pp. 130-141. ACM, URL https:
sion, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1422-1430. IEEE Computer Society, 2015. doi: 10. 10.1109/CCV.2215...6.. URL //ttps://doi... 10. wtt.0///....1......
Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O., and King, J.-R. Decoding speech percep- tion from non-invasive brain recordings. Na- ture Machine Intelligence, 5(10):1097-1107, Octo- ber 2023. ISSN 2522-5839.
2023. doi: 10.1 10.1145/3580005335990000 //doi. org/ 100 10. 11..111444 1145/118030000 3599426.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bo- janowski, P., and Joulin, A. Emerging properties in self- supervised vision transformers. In 2021 IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9630- 9640. IEEE, 2021. doi: 10.1009/CCCV489922221.000
doi: 10.1038/
$42256-023-00714-33 URL ctmm//atticclees22255--0-----000
Fries, P. A mechanism for cognitive dynamics: neu- ronal communication through neuronal coherence. Trends in Cognitive Sciences, 910):474-480, October 2005. ISSN 1364-6613. doi: 10.10166j.tics.2000 011. URL httppppswwwwwwweeeeeeeeeeeeeeeee
URL https: http://....og/111111//////////// 2021. 00951.
Chen, X., Wang, R., Khalilian-Gourtani, A., Yu, L., Dugan, P., Friedman, D., Doyle, W., Devinsky, O., Wang. Y., and Flinker, A. A neural speech decod- ing framework leveraging deep learning and speech synthesis. Nature Machine Intelligence, pp- 1-14, April 2024a. ISSN 2522-5839. doi: 10.1038/ +42256-024-008824-8. http:t com/ con/articlest 0tt.a//eee/s/22222--00-00033
sci actftenc//arrrrteeee///////66666
Fries, P. Neuronal gamma-bant synchronization as a funda- mental process in cortical computation. Annual Review of Neuroscience, 32:209 224, 2009.
Gibiansky, A., Arik, S. ö., Diamos, G. F,, Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y. Deep voice Multi-speaker neural text-to-speech. Advances in Neu- ral Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 2962- 2970, 2017
Chen, Y. Ren, K., Song, K., Wang. Y., Wang. Y., Li, D., and Qiu, L. Fegformer: Towards transferable and interpretable large-scale EEG foundation model. CoRR, abs/2401. 102788 2024b. doi: 10.48550/ARXIV. 2401.10278. URL hwwph:tt/////....tm //ts:////oo..ee http///o.oooo/////////////////// /atp//....555///
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised rep- resentation learning by predicting image rotations. In 6th International Conference
Cheung. C., Hamilton, L. S., Johnson, K., and Chang, E. F. The auditory representation of speech sounds in human motor cortex. eLife, 5:e12577, 2016.
Learning Representations, ICIR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: http://ppeeeeeeeeeeeeee/eeeeeeee net n id=S1v4N210-.
Csaky, R., van Es, M. W. J., Jones, O. P., and Woolrich, M. W. Group-level brain decoding with deep learning. Human Brain Mapping, 44:6105 6119, 2022. URL https:/ //oii//ii///////////..// org/ 10. 1002 hhm. ht..mmm.55... 26500.
Giraud, A.-L. and Poeppel, D. Cortical oscillations and speech processing: emerging computational prin- ciples and operations. Nature Neuroscience, 15(4): 511-517, April 2012. ISSN 1546-1726. 1038/nn.3063. URI. https: //tp:///wwwww// www. nature.comm articles/nn. 3063.
Csaky, R., van Es, M. W., Jones, O. P., and Wool- rich, M. Interpretable many-class decoding for MEG. Neurolmage, 282: November 2023. ISSN 10538119. doi: 10. 10010066i.neurommaee22000 120396. URL URL URChhcpss//Lnniiinnhb..eeeeeee com/ htap:/ecrriseeep1/11111111122200
doi: 10.
Gwilliams, L., Flick, G., Marantz, A., Pylkkänen, L., Poep- pel, D., and King, J.-R. Introducing MEG-MASC a high-quality caanttto-nncephllrrpphyy dataset for eval- uating natural speech processing. Scientific Data 10(1):
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fi- delity neural audio compression. CoRR, abs/2210. 13438,
10

---

The Brain's Bitter Lesson
862, December 2023. ISSN 2052-4463. doi: 10.1038/
Le, T. and Shlizerman, E. STNDT: modeling neural popu- lation activity with spatiotemporal transformers. In Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA,, USA, November 28- December 9, 2022, 2022.
$41597-023-02752-5. URL commarticces/s41555555 02001002727222-2-2--
Hall, E. L., Robson, S. E., Morris, P. G., and Brookes, M.
The relationship between MEG and fMRI, Neurolm-
age, 102:80-91, 2014. URL https:ttt///c/ /tt.di..r////// 1016 www.oinauoimeee... 2013.11. 005.
Lopes da Silva, F. EEG and MEG: Relevance to Neuroscience. Neuron, 80(5):1 December 2013. ISSN 0896-6273, doi: 10.1016/j.neuron2201... 017. URL htttps::/www..ciienncc.. htlaanc//aatiiclee//l/////99/299
Hamilton, I.. S., Edwards, E., and Chang, E. A Spatial Map of Onset and Sustained Responses to Speech in the Human Superior Temporal Gyrus. Current Biology, 28( 28(12)1860-18771c4, June 2018.
con/
ISSN 09609822. doi: 10. 10.10166j.cub22018044...3. URL httpss//LLLiiinggbbb.eeeeeeeerr com/ ret dttrrieve/p11/50009222222
Loshchilov, and Hutter, Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: // wwwpoeeeee....e.eeeeeee.ee.eeeee htttt/fouummiico Bi..:iiidd-..iiigg....
Jiang. W., Zhao, L., and liang Lu, B. Large brain model for learning generic representations with tremendous EEG data in BCI. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ /// openreview net/ 20rumm?1==0ITpIRVV...
Luo, H. and Poeppel, D. Phase patterns of neuronal re- sponses reliably discriminate speech in human auditory cortex. Neuron, 54(6): 2007.
Luo, H., Liu, Z., and Poeppel, D. Auditory cortex tracks both auditory and visual stimulus dynamics using low- frequency neuronal phase modulation. PLOS Biology, (8)ee1000445 2010.
Jo, H., Yang, Y., Han, J., Duan, Y., Xiong, H., and Lee, W. H. Are EEG-to-text models working? arXiv, 2024. doi: hutps:/aaxiix.or/////2444...44.
Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M.. Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Ziclinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Sil- ver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021. URL https: http:///oo.rr///110033303 241586-021-0388
Mai, G., Minett, J. W., and Wang. W. S. Y. Delta, theta, beta, and gamma brain oscillations index levels of auditory sen- tence processing. Neurolmage, 133:516-528, June 2016. ISSN 1053-8119, doi: 10. 064. URL http:::://wwwwwssseteees sctence/arttiileep////////
com/
Makin, J. G.. Moses, D. A., and Chang, E. F. Machine trans- lation of cortical activity to text with an encoder-decoler framework. Nature Neuroscience, 23:575 582, 2019. URL https: /atp://appia/ http://app..ss......./////mm. wcchiolcchollcc lar. org/ http/CoopSII::199539999
Kostas, D., Ouelleee S. T., and Rudzicz, F. BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15, 2021.
Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., Schalk, G., Knight, R. T., and Pasley, B. N. Decoding spectrotemporal features of overt and covert speech from the human cortex. Frontiers in Neuro- engineering, 7:14, 2014.
Langland- Langland-Hassan, Hassan, P. and Vicente, A. Inner Speech: New Voices. Oxford University Press, 2018.
Mesgarani. N., Cheung. C., Johnson, K., and Chang, E. F. Phonetic feature encoding in human superior temporal gyrus. Science, 343(61 (011111))::00010000 2014. doi: DOI: 10.1126/ciiecc
Larsson, G., Maire, M., and Shakhnarovich, G. Learning representations for automatic colorization. In Leibe, B., Matas, J.. Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016 14th European Conference, Ams- terdam, The Netherlands, October 1 11-14, 2016, Proceed- ings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp- 577-593. Springer, 2016. doi: 10.1007/ 978- 778-3-319-46493-0..25 35. URL attps: //ddo..oorg 10. 10.1007/978-3--00
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., Zhuravleva, I., Tu-Chan, A., Ganguly, K., Anumanchipalli, G. K., and Chang, E. F. A high- performance neuroprosthesis for speech decoding and avatar control. Nature, 620:1037-1046, 2023.

---

The Brain's Bitter Lesson
Moses, D. A., Metzger, S. L.,, Liu, J. R., Anumanchipalli, G. K., Makin, J. G., Sun, P. F., Chartier, J., Dougherty, M. E., Liu, P. M., Abrams, G. M., Tu-Chan, A., Gan- guly, K., and Chang, E. F. Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria. New Eng- land Journal of Medicine, 385(3):217-227, July 2021. ISSN 0028-4793. doi: 10. 100600000000025400 URL https: / //oii/oii.o//// org/ 10. 1056 1ttp:/1..11///////222200000 /NE.M0022222250.
Schoffelen, J.-M., Oostenveld, R., Lam, N. H. L., Uddén, J., Hultén, A., and Hagoort, P. A 204- subject multimodal neuroimaging dataset to study language processing. Scientific Data, 6(1):17,
April 2019. ISSN 2052-4463. doi: 10.1038/
$41597-019-0020-y, URL https://www..... nature. com/artiicles//41155
Shafto, M. A., Tyler, L. K., Dixon, M.. Taylor. J. R., Rowe, J. B., Cusack, R., Calder, A. J., Marslen- Wilson, W. D., Duncan, J. S., Dalgleish, T.,, Henson, R. N. A., Brayne, C., and Matthews, F. E. The Cambridge centre for ageing and neuroscience (Cam-CAN) study protocol: cross-sectional, lifespan, multidisciplinaty examination of healthy cognitive ageing. BMC Neurology, 14, 2014.
Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In Leibe, B., Matas, J.. Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, volume 9910 of Lecture Notes in Computer Science, pp. 69-84. Springer, 2016. doi: 00.10077978-33-11946666644 URL https://dci.
Strauß, A., Henry, M. J., Scharinger, M., and Obleser, J. Alpha phase determines successful lexical decision in noise. Journal of Neuroscience, (177)33256-3622 2015,
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774,
Sutton, R. The bitter lesson. Incomplete Ideas (blog), 2019. URL http: http://wwww.n..oo/ www.oinneemeeeeeeses.........ee. net/IncTdeas/ wetttreeessooon. html.
2023. doi: 10.48550/ARXX1.2300..355 URL https: //doi. org/10, 111111..8855000
08774.
Peelle, J. E.,, Gross, J., and Davis, M. H. Phase-locked responses to speech in human auditory cortex are en- hanced during comprehension. Cerebral Cortex, 23(6): 1387- 2012.
Tagliasacchi, M., Li, Y., Misiunas, K., and Roblek, D. SEANet: A multi-modal speech enhancement network. In Meng, H., Xu, B., and Zheng, T. F. (eds.), In- terspeech 2020, 21st Annual Conference of the Inter- national Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 1126- 1130. ISCA, 2020. doi: 10.21457/NTERSPEECCH 2020-1563. URL https: /ooi .ttg/0..111333333 Interapeech.co 2020 1163.
Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. C. FiLM: Visual reasoning with a general conditioning layer. In Mcllraith, S. A. and Weinberger, K. Q. (eds.), Proceedings of the Tecooy AAAI Con- ference on Artificial Intelligence, (AAAI-18), the 30th in- novative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3942-3951. AAAI Press, 2018. doi: (001669/AAAAA.V221111166 URL https: wttp://aaa..v.ii1...............
Tang, J., LeBel, A., Jain, S., and Huth, A. G. Seman- tic reconstruction of continuous language from non- invasive brain recordings. Nature Neuroscience, 26(5): 858-866, May 2023. ISSN 1546-1726. doi: 10.1038/ s4 $41593-023-0110449, URL ht potttpss/// www.//wwww......mmmm/////a///...
Piai, V. Roclofs, A., and Maris, E. Oscillatory brain responses in spoken word production reflect lexical frequency and sentential constraint. Neu- ropsychologia, 53:146-156, January 2014. ISSN
Taylor, J. R., Williams, N., Cusack, R., Auer, T., Shafto, M. A., Dixon, M., Tyler, L. K., Group, C.-C., and Hen- son, R. N. A. The Cambridge centre for ageing and neuroscience (Cam-CAN) data repository: Structural and functional MRI, MEG, and cognitive data from a cross- sectional adult lifespan sample. Neuroimage, 144:262 269, 2017.
0028-3932. doi: 10. 10.1006f.eenmppyyhhoeea....1...1 014. URL httppp://ww..sceenneiiiiieeeeeee com/ sci httnc//articcee/i////ee//ee
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 28492-288 28518. PMLR, 2023. URL https: httpa/:://///peppeeeeeee mlr. mlr.press// 2202/radford22 html.
Vidaurre, D.,, Hunt, L. T. Quinn, A. J., Hunt, A. E., Brookes, M. J., Nobre, A. C., and Woolrich, M. W. Spontaneous cortical activity transiently organises into frequency specific phase- coupling networks. Nature Communications, 9(1): 2987, July 2018. ISSN 2041-1723. doi: 10.1038/ $41467-018-00316-z. URL https: http://wttw......////////////... nature. com/articles//411446
05316-2.
12

---

The Brain's Bitter Lesson
Wandelt, S. K., Bjånes, D. A., Pejsa, K., Lee, B., Liu, C. Y., and Andersen, R. Representation of internal speech by single neurons in human supramarginal gyrus. Nature human behaviour, 2024. URL https://///..... bttps://doi.org/ /oi. 10. 038/s415560 024 01867-y.
Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Yuan, Z.., Shen, F., Li, M., Yu, Y., Tan, C., and Yang. Y. BrainWave: A brain signal foundation model for clinical applications. 2024. URL https:
Wang, C., Subramaniam, V., Yaari, A. U., Kreiman, G., Katz, B., Cases, I., and Barbu, A. Brainbert: Self- supervised representation learning for intracranial record- ings. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023. URL https: // //openrevien net
267740511.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neu- ral audio codec. IEEE ACM Trans. Audio Speech Lang- Process., 30:495-507, 2022. doi: 10.1109/ FASLP.2021.3129990 URL https:////// ptops:://oiii org/00. 1109 1109/TASLP 2021. 3129994..
Wang, G., Liu, W., He, Y. Xu, C., Ma, L., and Li, H. EEGPT: pretrained transformer for universal and reli- able representation of EEG signals. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U.. Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Infor- mation Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Zhang, D., Yuan, Z., Yang, Y., Chen, J., Wang. J., and Li, Y. Brant: Foundation model for intracranial neural signal. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Weiss, S. and Mueller, H. M. "Too many betas do not spoil the broth": the role of beta brain oscillations in language processing. Frontiers in Psychology, 3, 2012. doi: http://o...orr111133383///...../ 2012.00001.
Zhang, R., Isola, P., and Efros, A. A. Colorful image col- orization. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11- 14, 2016, Proceedings, Part III, volume 9907 of Lecture Notes in Computer Science, pp. 649-666. Springer, 2016. doi: 111100//977883111119444448110119 URL https: 1018887-90 9_40.
Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F. Glasser, M. F, Hochberg, I. R., Druckmann, S., Shenoy, K. V., and Henderson, J. M. A high-perffrmancc speech neuroprosthesis. Nature, 620(7976):0331-036, August 2023. ISSN 1476-4687. doi: 10.1038/ $415886002-006377-x URL https::tt//..... //wp://wwww/../...o//... http///wwttr...../../////a./ com/articles//4411
023 06377-x.
Yang, C., Westover, M. B., and Sun, J. BIOT: biosignal transformer for cross-data learning in the wild. Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Ye, J., Collinger, L., Wehbe, L., and Gaunt, R. Neural data transformer 2: Multi-context pretraining for neu- ral spiking activity. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
Yi, K., Wang, Y., Ren, K., and Li, D. Learning topology- agnostic EEG representations with geometry-aware mod- eling. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on
13

---

The Brain's Bitter Lesson
A. Experiment Details
Pre-training data. We pre-train with non-overlapping sample windows from all subjects and sessions. We adjust the amount of unlabelled data used from Cam-CAN by increasing the number of subjects in the sequence 1, 4. 8, 17. 36, 74, 152, 312, and 641, successively randomly selecting more subjects to include. Each seed uses different set of subjects to reduce negative effects from outlier subjects.
Labelled training data. When training with Armeni et al. (2022), we hold out session 009 for validation and 010 for testing. Similarly, when fine-tuning with Gwilliams et al. (2023), we hold out task from subjects 23, 24, 25, 26, and 27, using these sessions for evaluation only. As there is limited within-subject data in the latter dataset, we did not hold out a session from all subjects as before. For our novel subject experiments, we hold out subjects 1, 2, and entirely and use the data for these subjects during evaluation. In Gwilliams et al. (2023), we note that they use four different tasks for each subject and their order is randomized between subjects. Both sessions for each task are repeats of the task. This means that while the recording itself is unseen, in this dataset, it is possible that held-out sessions use stimuli that may be shared.
dapting models for more sensors. As BrainBERT is designed for single-electrode representations, for fair comparison, to take into account all sensors, we ensured our linear classifier is applied to a concatenated embedding over all sensors. This a large vector and leads to very computationally expensive training and is the reason we had to reduce the pre-training and downstream data in part C of Table 2. We similarly concatenate sets of 19 sensors when evaluating EEGPT.
Statistical testing. Significance was determined using one-sided I-tests with p < 0.05 the threshold for significance.
B. Hyperparameters
We conducted a search over hyperparameters of interest to optimise our self-supervised objectives and neural architecture. While these ablations indicated a theoretically ideal architectural configuration, in practice we altered our final experimental architecture due to instabilities during training when data was scaled up. Our final architecture hyperparameters achieve a balance between the best values from our hyperparameter search and stable training. These values are detailed in Table
Table 4: Experimental hyperparameters.
Hyperparameter
Value
Window length (s)
0.5
P (phase)
0.5
p (amplitude)
0.2
(wi, W2, w3)
(1.0, 1.0, 1.0)
dahared
512
dbackbone
512
SEANet convolution channels
(512, 512, 512, 512) (5, (5,5,11 1)
SEANet downsampling ratios
FiLM conditioning dimension
Subject embedding dimension
16
Pre-training epochs
200
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Learning rate
0.000066
Train ratio
0.8
Validation ratio
0..
Test ratio
0.1
Why p? The choice of the proportion of sensors to apply transformations to, 0.5 for phase shift prediction and 0.2 for amplitude prediction, were determined through a hyperparameter search. It is important to note that >= .5 leads to the same effect as 1 P for the complementary amplitude or phase shift. We conjecture that a smaller is optimal for amplitude scale prediction since this leads to representations that are especially strong at discriminating amplitude differences among
14

---

The Brain's Bitter Lesson
small groups of sensors. Perhaps this makes it easier distinguish between neural responses from distinct parts of the brain such as the STG, which is associated with speech onset (Hamilton et al., 2018). In contrast, larger for phase shift prediction could lead to representations that better discriminate neural synchrony information which is distributed across the brain rather than localised. As result, a large proportion of the sensors in a MEG scanner should encode information about this feature.
15

---

The Brain's Bitter Lesson
C. Compute Resources
All experiments were run on individual NVIDIA V100 and A100 GPUs with up to 40GiB of GPU memory on a system with up to ITiB of RAM. Each pre-traiming run with the maximum amount of pre-training data took approximately 200 hours (8.3 days). Fine-tuning following pre-training took up to another 12 hours. We estimate that we used approximately 3000 hours of compute for the final experimental runs, including hyperparameter searches. In total, over the course of developing this work from idea to final paper, we used around 10,000 hours of GPU compute.
D. Licences For Datasets And Code
The Armeni al. (2022) dataset is distributed under CC-BY-4.0 while the Gwilliams et (2023) dataset distributed under the CCO 1.0 Universal licence. The Schoffelen et al. (2019) dataset is distributed with RU-DI-HD-L.0 licence from the Donders institute. The licence for the Cam-CAN (Shafto et al., 2014; Taylor al., 2017) dataset is unknown. The SEANet code adapted from Défossez al. (2022) is distributed under the MIT licence, and the OSL library, which use for preprocessing, is under the BDD-3-Clause licence.
16


Reviews:
[{"review_id": "KSq8E5tcdE", "reviewer": "Reviewer_aJaK", "summary": "Current speech decoders are generally trained individually per subject and only on task-specific data. The authors propose an MEG-specific self-supervised learning objective to build representations from a vast quantity of unlabeled MEG data from several subjects and tasks. They then built decoders that used these representations to detect phoneme voicing or the presence of speech. The authors claim that these decoders beat previous state-of-the-art self-supervised methods, with the advantage of generalizing to unseen subjects.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a review. We are glad you found the paper to be well-written and the experiments and comparisons to be sound. Please find below, our responses to your questions and concerns:\n\n>  Most tables mention a single ROC AUC score [...] I am assuming that all scores are for speech detection\n\nYes, where it is not specified the score is for speech detection. Thanks for highlighting that this is unclear. We will add the specific score to the table captions where relevant.\n\n> could the correct amplitude not be ambiguous?\n\nYou are absolutely correct. Thank you for identifying this and demonstrating it with an example. There is indeed an ambiguity here and we must restrict rho to be < 0.5 in order to have an unambiguous class. We conducted our ablation with values between 0 and 0.5 as we were aware of this, but as you have identified, we did not make this clear in the writing. We will add a note about this to the revised PDF. This should similarly apply in the phase shift task.\n\nThe boundary effect hypothesis is very interesting. Could you please explain why you believe this influences the selection of a smaller rho?\n\n> [...] as the authors mention in Sec. 4.6, existing BCIs can decode more useful features like phoneme (or even word) identity\n\nSo far, only existing *invasive* BCIs have been able to decode more complex features convincingly [A, B, C]. Attempts with non-invasive signals for sentence decoding have not yet produced results statistically significant beyond chance level, failing replication studies [D]. Nevertheless, as you point out, our voicing results do imply better-than-chance phoneme recognition results. In our preliminary experiments, while the results were better than chance, they were not much beyond that. We are actively working on improving the decoding of phonemes (focusing on acoustic features) and of words (focusing on semantic features) in a follow-up work.\n\n> it may be helpful to visually differentiate Figures 3 and 4\n\nYes, we agree. Thank you for noting this. We will change the colours in Figure 4.\n\n> Does this specifically refer to a difference in quality between Cam-CAN and MOUS?\n\nYes. We used a method to automatically detect corrupted channels and remove them using a variance-based threshold (known as autoreject in neuroimaging [E]). The percentage of corrupted channels in sessions from MOUS was higher than in Cam-CAN suggesting that the quality of the data was not as good in general. Better data filtering is likely to be an important future direction to continue getting improvements from additional datasets.\n\nThank you again for your review. You have highlighted several important points which will help improve our paper. Do you have any further questions or concerns?\n\n[A] Moses, D.A., Metzger, S.L., Liu, J.R., Anumanchipalli, G.K., Makin, J.G., Sun, P.F., Chartier, J., Dougherty, M.E., Liu, P.M., Abrams, G.M. and Tu-Chan, A., 2021. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine, 385(3), pp.217-227.\n\n[B] Willett, F.R., Kunz, E.M., Fan, C., Avansino, D.T., Wilson, G.H., Choi, E.Y., Kamdar, F., Glasser, M.F., Hochberg, L.R., Druckmann, S. and Shenoy, K.V., 2023. A high-performance speech neuroprosthesis. Nature, 620(7976), pp.1031-1036.\n\n[C] Card, N.S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F.R., Kunz, E.M., Fan, C., Vahdati Nia, M., Deo, D.R. and Srinivasan, A., 2024. An accurate and rapidly calibrating speech neuroprosthesis. New England Journal of Medicine, 391(7), pp.609-618.\n\n[D] Jo, H., Yang, Y., Han, J., Duan, Y., Xiong, H. and Lee, W.H., 2024. Are eeg-to-text models working?. arXiv preprint arXiv:2405.06459.\n\n[E] https://autoreject.github.io/stable/explanation.html", "claims_and_evidence": "The authors were thorough in their evaluation, but I believe it's not always clear which task the decoder is being evaluated on. Most tables mention a single ROC AUC score, whereas Figures 3 and 4 mention separate scores for speech detection and voicing classification. I am assuming that all scores are for speech detection unless otherwise noted (based on \"We evaluate our methods primarily on speech detection\"), but clarification is needed.", "methods_and_evaluation_criteria": "1. The individual terms of the self-supervised loss are well motivated, but I am somewhat confused by one aspect of amplitude scale prediction. Based on the choice of $\\rho$, could the correct amplitude not be ambiguous? For example, suppose $\\rho=0.5, A=2$. Then either of these could be true:\n\n    A. $x_s^A = 2x_s$ for each sensor $s$ that _was_ randomly sampled\n\n    B. equivalently, $x_s^A = 1/2 \\cdot x_s$ for each sensor that was _not_ randomly sampled. So it could be that $A = 1/2$.\n\n    Does this term strictly require $\\rho < 0.5$ to have an unambiguous class? I also wonder if there's a boundary effect near 0.5, and this is why a smaller $\\rho$ like 0.2 was chosen (as discussed at the end of Appendix B).\n\n2. The two decoding tasks (speech detection & voicing classification) seem like a good place to start evaluating the quality of the SSL method. But as the authors mention in Sec. 4.6, existing BCIs can decode more useful features like phoneme (or even word) identity. Have the authors looked at decoding higher-level features like these? (I expect that phoneme error will at least be above chance given the phoneme voicing detection results.) Based on Appendix C, it looks like >90% of compute was dedicated to pre-training the network. Since that component is task-agnostic, I don't think adding another decoding task like phoneme classification would be too computationally difficult, but it could make the overall contribution much more substantial.", "theoretical_claims": "No theoretical claims were made in the paper.", "experimental_designs_or_analyses": "The evaluation method seemed sound -- in particular, it seemed the authors were careful not to contaminate the test set for within- or across-subject evaluations (except for the stimulus as noted in Appendix A, which I think is not a concern for these decoding tasks.)", "relation_to_broader_scientific_literature": "The authors demonstrate that their self-supervised pre-training method outperforms existing methods for MEG by making use of existing publicly-available data. In particular, their non-invasive decoder shows comparable performance to invasive ones, and they show for the first time generalization to unseen subjects. They also show that scaling increases log-linearly, suggesting there is still room for improvement.", "essential_references_not_discussed": "I am not aware of any missing references.", "other_strengths_and_weaknesses": "The paper is written well, and the ablations and comparisons to other models are done well.", "other_comments_or_suggestions": "Minor, but it may be helpful to visually differentiate Figures 3 and 4 more -- they're easy to mix up at a glance. Changing the line colors in Fig. 4 would likely be sufficient, since currently, the same dark blue means different things: \"Gwilliams\" in Fig. 3 and OOD in Fig. 4.", "questions_for_authors": "1. In Sec. 4.5, the authors write:\n    > During our experiments, we found that data quality, even among unlabelled data, can have a significant effect as artefacts in recordings disrupt learning.\n    \n    Does this specifically refer to a difference in quality between Cam-CAN and MOUS? If so, how was data quality judged? What kinds of artefacts were found in MOUS but not in Cam-CAN?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1qCfog8Zg1", "reviewer": "Reviewer_32uy", "summary": "This paper proposes a framework about how to leverage self-supervised learning (SSL) to improve the decoding of speech from brain activity. The authors propose a approach that utilizes large-scale unlabeled MEG data to train models, thereby addressing challenges posed by individual differences and dataset heterogeneity. The method demonstrates the generalization capabilities across multiple datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a detailed review. We are glad you found the paper easy to follow. Please find our responses below:\n\n> significant progress has already been made in Brain-to-Text tasks by numerous studies [1][2][3]\n\nWhile [1] and [2] are impressive, they address a different context and problem with invasive data and [3] suffers fundamental limitations to scaling.\n\nSpecifically, [1] and [2] work on surgical data which is much easier to decode due to the better signal-to-noise ratio. We focus on non-invasive decoding as it avoids the risks and complexity of surgery in implanting BCIs. Additionally, [1] and [2] do not show scaling, do not work on novel subjects, and for [2], do not leverage pretraining. Thus, although they are important works, they are not relevant to our problem context.\n\n[3] is the only milestone work so far in non-invasive speech decoding from MEG. We discuss in lines 58-63 (column 2) that “they do not demonstrate generalisation to novel subjects and retrain their model for new datasets rather than being able to generalise across datasets or pool them. Their method is also unable to incorporate data without corresponding audio labels and so does not scale with other kinds of tasks.” As a result, they are fundamentally limited in scaling up data.\n\nThank you for noting these papers as they are still important and interesting works. We will ensure to cite [1] and [2] in our revised PDF. \n\n> I believe that the limited scope of downstream tasks somewhat diminishes the contribution of this work.\n\nOur scope is similar to that of comparable leading work [3] who use a similar but ultimately less practical task to speech detection. Their speech segment identification task matches arbitrary 3-second segments of audio and brain and could be picking up on lots of unknown features (e.g. patterns of silence). Because it requires paired audio and brain data, it is unlikely to be a useful task for future BCIs. We go beyond a single task by also looking at voicing classification and in contrast to [3], speech and voicing classification are also directly relevant to speech BCIs for segmenting phrases and distinguishing phonemes for word decoding.\n\nWhile we agree that the ultimate goal is full phoneme, word, and sentence decoding, our work provides essential building blocks that non-invasive approaches have failed to achieve so far. [A] demonstrate that current EEG/MEG approaches haven't exceeded chance for full speech decoding. Our approach establishes prerequisite foundations by solving generalisation and data efficiency problems. We are developing more complex tasks in follow-up work.\n\n> it only marginally surpasses the fine-tuning performance of previously proposed general foundational models such as BIOT and BrainBERT\n\nOur improvements are 15-27% over BIOT and BrainBERT. They are also highly statistically significant (p<<.05). We would classify this as substantial as it is even sufficient to bring non-invasive decoding up to BrainBERT’s surgical decoding accuracy. The improvements are also larger than those quoted in BIOT over its own baselines.\n\n> lacks comparisons with neural large-scale models like LaBraM [1] and EEGPT [2]\n\nWe have now added this comparison. Thank you. Our method outperforms EEGPT:\n\n| Method | AUROC |\n| :---- | :---- |\n| EEGPT | .602 \\+/- .006 |\n| Ours | .705 \\+/- .003 |\n\nAs the largest EEGPT model supports up to 54 channels and our dataset uses 269 sensors, we concatenate the embeddings of consecutive chunks of 54 channels so that we can fairly take into account information from all sensors before linearly probing. We will add this result to Table 1.\n\nThank you again for noting this. We have already cited LaBraM and will do the same with EEGPT.\n\n> the methodology presented in this paper is overly simplistic, as the use of unlabeled data for pretraining and fine-tuning on downstream tasks is not a novel insight.\n\nOur primary contribution isn't just applying pre-training and fine-tuning, but developing novel neuroscience-informed objectives (amplitude scaling, phase shifting, and band rejection) that take steps to address three fundamental challenges:\n\n1) Novel- and cross-subject generalisation (historically a major blocker for MEG research [B])\n2) Data-efficient and scalable self-supervised learning from heterogeneous datasets (no other work in MEG has done this); and\n3) Generalisation across tasks and datasets\n\nIn practice, our results also show a significant jump over baselines. We will make sure to emphasise this more in the revised PDF.\n\nThank you again for your efforts in reviewing. You have helped us clarify several important aspects of our paper. Do you have any further concerns?\n\n[A] Jo, H., Yang, Y., Han, J., et al., 2024. Are eeg-to-text models working?. arXiv preprint arXiv:2405.06459.\n\n[B] Csaky, R., Van Es, M.W., Jones, O.P. et al., 2023. Group‐level brain decoding with deep learning. Human Brain Mapping, 44(17), pp.6105-6119.", "claims_and_evidence": "No.", "methods_and_evaluation_criteria": "The model architecture used in this study does not differ fundamentally from those in prior work. The paper primarily focuses on unsupervised pretraining for MEG and speech task decoding. However, pretraining before downstream tasks is not a novel concept, and the lack of comprehensive experimental evaluation may undermine the claimed contributions of the proposed method.\n\nThe downstream tasks in this paper are focused on Speech Detection and Voicing Classification. However, significant progress has already been made in Brain-to-Text tasks by numerous studies [1][2][3]. I believe that the limited scope of downstream tasks somewhat diminishes the contribution of this work.\n\nReferences\n\n[1] Zheng H, Wang H, Jiang W, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. Advances in Neural Information Processing Systems, 2024, 37: 79996-80033.\n\n[2] Chen X, Wang R, Khalilian-Gourtani A, et al. A neural speech decoding framework leveraging deep learning and speech synthesis[J]. Nature Machine Intelligence, 2024, 6(4): 467-480.\n\n[3] Défossez A, Caucheteux C, Rapin J, et al. Decoding speech perception from non-invasive brain recordings[J]. Nature Machine Intelligence, 2023, 5(10): 1097-1107.", "theoretical_claims": "The methodological framework proposed in this paper is relatively straightforward, involving pretraining through the setup of three proxy tasks and fine-tuning specific modules. However, in terms of performance, it only marginally surpasses the fine-tuning performance of previously proposed general foundational models such as BIOT and BrainBERT. Additionally, it lacks comparisons with neural large-scale models like LaBraM [1] and EEGPT [2].\n\nReferences\n\n[1] Jiang W, Zhao L, Lu B. Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI[C]//The Twelfth International Conference on Learning Representations.\n\n[2] Wang G, Liu W, He Y, et al. Eegpt: Pretrained transformer for universal and reliable representation of eeg signals[J]. Advances in Neural Information Processing Systems, 2024, 37: 39249-39280.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "See Theoretical Claims.", "essential_references_not_discussed": "See Methods And Evaluation Criteria.", "other_strengths_and_weaknesses": "The writing in this paper is straightforward and easy to follow, with clear descriptions of figures and tables. However, the lack of robust downstream task experiments and comprehensive baselines weakens the impact and contribution of this work.\n\nI must point out that the methodology presented in this paper is overly simplistic, as the use of unlabeled data for pretraining and fine-tuning on downstream tasks is not a novel insight. It is recommended to emphasize the originality of the proposed method and its exceptional performance across a variety of downstream tasks.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47N6seipB3", "reviewer": "Reviewer_Bk17", "summary": "This paper presents a unified solution through data-efficient, self-supervised pretext tasks to improve the speech detection and voicing Classification tasks. The experiments demonstrate significant gains from self-supervised pre-training. The method surpassed the baselines and is comparable to the model trained with surgical data. The data ablation (data size and data source) provides interesting insights to the community.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our work. We are glad that you found the paper well written and organized, and that the experiments validated our hypothesis. Below, we have provided our responses:\n\n> The designed pre-text tasks might be suitable for simple tasks only. The gain might be limited on transcript tasks.\n\nYes, thank you for highlighting this consideration. It is an important tradeoff. If the pre-training task is overly specific, then it will likely be easier to learn but the representation’s utility will be limited for more general and complex downstream tasks. Ultimately, in the extreme, the most general pre-training task is next-step prediction. However, unlike some of the baselines we compare to (e.g. BrainBERT), we did not pursue this as the data scale in MEG is limited and, given most of the signal is noise, it is a very difficult pre-training task to learn well. Instead, we aimed for balance by designing pre-text tasks that are more specific than next-step prediction, through targeting useful features in brain activity, but general enough that they work well with downstream tasks related to neural speech decoding. This balance led us to achieve better downstream performance than comparable methods while generalising across multiple tasks, datasets, and subjects.\n\nWe do share your interest in extending this work to full speech transcription. Jo et al. 2024 [A] demonstrate that current EEG/MEG approaches haven't exceeded chance for full speech decoding, highlighting the fundamental challenges in this space. Our approach specifically addresses prerequisites for more complex decoding by addressing generalisation and data efficiency problems. We're actively developing these extensions in follow-up work to further support our hypothesis that some of these pretext tasks will also enable more complex decoding.\n\n> Data scaling might be over-claimed as 1000h is a relatively small data size as compared to the speech recognition tasks.\n\nWhile this may seem relatively small in comparison to modalities such as audio now, data scaling should be considered within the context of MEG, which is the focus of our work. Here, we have scaled well beyond the volume of data used in prior MEG work e.g. Défossez et al. 2023 [B]. We are approaching the scale of early work in the adjacent field of deep learning speech recognition from audio e.g. Speechstew [C]. Thus, with these efforts we hope to similarly scale up MEG to progress speech recognition from brain activity in the near future. We will make this point clearer to ensure that we are not over-claiming. Thank you for noting this.\n\n> Have the authors considered about data filtering method?\n\nThank you for the suggestion. So far, we have only explored detecting and rejecting corrupted channels through a variance-based threshold (known as autoreject in neuroimaging [D]). However, you are right that there are perhaps better ways to deal with data filtering as some artefacts can be global (across all channels) e.g. heart beats, muscle spikes, breathing, etc. One way we could attempt to address this in future work is to use signal-space projection (SSP) [E] if the network is not learning to ignore these types of artefacts.\n\n> How about applying the proposed method for surgical as what was done for BrainBERT2+linear?\n\nThis is an interesting suggestion. In our work, we opted to apply BrainBERT+linear to non-invasive data as that is what we are mainly concerned with. While surgical data is out-of-scope for our current work, it is certainly of interest to the community and something we may explore in the future.\n\nThank you again for your review. You have helped elucidate some critical points in our work. Do you have any further questions or concerns?\n\n[A] Jo, H., Yang, Y., Han, J., Duan, Y., Xiong, H. and Lee, W.H., 2024. Are eeg-to-text models working?. arXiv preprint arXiv:2405.06459.\n\n[B] Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O. and King, J.R., 2023. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10), pp.1097-1107.\n\n[C] Chan, W., Park, D., Lee, C., Zhang, Y., Le, Q. and Norouzi, M., 2021. Speechstew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint arXiv:2104.02133.\n\n[D] https://autoreject.github.io/stable/explanation.html\n\n[E] https://mne.tools/stable/auto_tutorials/preprocessing/50_artifact_correction_ssp.html", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "They make sense.", "theoretical_claims": "This is not a theoretical paper.", "experimental_designs_or_analyses": "Experimental design is valid, and the analyses are good.", "relation_to_broader_scientific_literature": "This is an incremental work by studying more pretext tasks and show its effectiveness on down-stream tasks.", "essential_references_not_discussed": "The references look good.", "other_strengths_and_weaknesses": "The strengths of this paper:\n\n1) The paper is well organized and written.\n2) The experiments are sufficient to validate the hypothesis.\n\nThe Weaknesses of the paper:\n1) The designed pre-text tasks might be suitable for simple tasks only. The gain might be limited on transcript tasks.\n2) Data scaling might be over-claimed as 1000h is a relatively small data size as compared to the speech recognition tasks.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Several minor questions:\n\n1. In Table 3, it shows that the cleaner data helps more on the finetuning tasks (CamCAN better than MOUS). The addition of MOUS data on CamCAN does not seem to improve a lot actually. Have the authors considered about data filtering method? Some data in MOUS might have negative effect on the pretraining tasks.\n\n2. I am not familiar with surgical shown in Tabel 2. How about applying the proposed method for surgical as what was done for BrainBERT2+linear?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2LwMeanAk", "reviewer": "Reviewer_RHbR", "summary": "This paper introduces a pre-training strategy for MEG recordings, which consists in neuroscientifically-grounded pretext tasks.\nIt shows scaling laws on two downstream tasks on two different datasets.\n\n## update after rebuttal\nSince the authors did not provide additional results or modifications along my suggestions, my score remains the same", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a review. We are glad that you found the paper easy to follow and that our experiments were sound and well-executed. Please find below, our responses to your question and concerns:\n\n> authors do not report anywhere [...] results obtained when finetuning the full model\n\nThanks for pointing this out. You’re right that we have not provided results when fine-tuning the full model. Here, we focused mainly on demonstrating that our self-supervised representation is highly generalizable. The simplest way to do this is through linear probing as it allows us to cleanly separate the contribution of self-supervision to the contribution of supervised learning. Fine-tuning the full model, would, however, be more thorough and likely provide better results. We prioritised demonstrating the SSL method over this given time constraints. We are working on collecting this result and others for posterity.\n\n> the downstream tasks chosen are rather easy tasks\n\nWe appreciate this perspective, but want to clarify that these tasks are surprisingly challenging in MEG. The signal-to-noise ratio in non-invasive neural recordings is exceptionally poor. Recent work in [A] demonstrated that more complex decoding tasks (e.g. transcript decoding) perform no better than chance with current methods. Similarly, so far, no work that we are aware of has demonstrated strong results for speech detection or voicing classification on MEG which we believe highlights the difficulty of the challenge even here. State-of-the-art works in MEG speech decoding have also used similar tasks such as [B] and their segment identification task. Theirs is also less relevant for a speech BCI as it’s based on paired audio and brain data.\n\nYou are right that ultimately the objective is to reach more complex tasks in order to achieve full speech BCIs. To this end, we are actively working on full phoneme and word classification as follow up work.\n\n> [...] which task Table 2 reports results for\n\nThank you for noting this. It is for speech and we will clarify this in the captions for Table 2 and 3. \n\n> I find it confusing that the plots in figures 3 and 4 look very similar\n\nYes, we agree that this is confusing. We will update figure 4 to use different colours. Thanks for bringing this to our attention.\n\n> Why not use classification of global rotation of sensors as another pretext task\n\nIndeed, as we discuss in the limitations section, we did not pursue any spatial pretext tasks. Your suggestion of a global sensor rotation task could be particularly useful in learning to account for head position changes. Thank you. Beyond movement compensation, phoneme perception also seems to have a strong spatial signature in brain activity [C, Fig. 4] and could further benefit from your suggestion. We are working on a small-scale experiment to test this.\n\nOn a related note, some MEG datasets (which use scanners built by Electa) have MaxFilter [D] programs which can be applied to automatically compensate for head movements. However, this is not general enough for scaling as not all datasets will use these scanners.\n\nThank you once again for highlighting some important points that have helped to improve our paper. Do you have any further questions or concerns?\n\n[A] Jo, H., Yang, Y., Han, J., Duan, Y., Xiong, H. and Lee, W.H., 2024. Are eeg-to-text models working?. arXiv preprint arXiv:2405.06459.\n\n[B] Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O. and King, J.R., 2023. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10), pp.1097-1107.\n\n[C] Joan Orpella, Francesco Mantegna, M. Florencia Assaneo, David Poeppel; Decoding imagined speech reveals speech planning and production mechanisms; bioRxiv 2022.05.30.494046; doi: https://doi.org/10.1101/2022.05.30.494046\n\n[D] https://ohba-analysis.github.io/osl-docs/pages/docs/preprocessing-maxfilter.html", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes. The authors follow the standard procedure of evaluating their pretrained model on downstream tasks, and provide comparisons with two baselines and two competing methods.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design makes sense to me", "relation_to_broader_scientific_literature": "In the neuroscientific literature, there has been a relatively limited amount of work on pre-training deep learning models compared to other fields, in part due to the difficulty to handle cross-dataset distribution shifts.\nExisting approaches (e.g. BENDR, EEG2Rep, BIOT) typically involve self-supervised losses, where the input is masked or perturbed in some way and must be reconstructed, and only consider EEG recordings (rather than MEG).\nThis paper takes a novel approach which consists in using neuroscientifically-grounded pretext tasks, and is (to the best of my knowledge) the first to consider MEG recordings.", "essential_references_not_discussed": "None that come to mind", "other_strengths_and_weaknesses": "Strengths\n- Novelty: novel approach with the pretext tasks (differs from the usual self-supervised losses), and first to consider MEG\n- Soundness: the experiments presented are sound and well executed (although not very exhaustive)\n- The paper is well-written and easy to follow\n\nWeaknesses:\n- Missing results: I may have misunderstood the sentence \"the backbone uses randomly initialised and frozen weights\", but it looks to me as if the authors do not report anywhere two important points of comparison: (i) results obtained when finetuning the full model (not just a linear layer on top) on the labeled data of the downstream tasks, and (ii) the same but training the full model from scratch (no pretraining). \n- The results are clean, but not particularly impressive: the downstream tasks chosen are rather easy tasks (binary classification). I appreciate that the authors wanted to display very clear scaling laws, and that has certainly been achieved, but it would be nice to also consider some more challenging tasks such as phoneme classification (even if results are poor). Similarly, the results on CamCAN+MOUS hardly outperform pretraining on CamCAN alone: although the gap might be statistically significant, I'm not sure how relevant it is; this section seems preliminary to me.\n- Clarity: although the paper is well-written, there are some missing details. (i) After going back and forth several times, I could not clarify which task Table 2 reports results for: is it speech classification, voicing classification, or something else? Similarly for table 3. Please clarify this in the caption of the tables. (ii) I find it confusing that the plots in figures 3 and 4 look very similar, but in fig 3 the colors represent the datasets and in fig 4 they represent in-distribution vs out-of-distribution. Please change colors or find a different way of plotting", "other_comments_or_suggestions": "None", "questions_for_authors": "“The latter dataset is particularly difficult to decode from as there is very little within-subject data and it did not enforce the use of head casts to immobilise participants” --> Why not use classification of global rotation of sensors as another pretext task (similar to random rotation augmentations used in computer vision)? It seems as if this could alleviate the aforementioned issue", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel contribution to the field of machine learning, introducing an innovative self-supervised learning (SSL) approach for neural signals collected using magnetoencephalography (MEG). The proposed method demonstrates improved decoding performance across participants, which is essential for developing generic models that can be transferred effectively across individuals.

The reviewers acknowledge the novelty of this approach, particularly in its use of unique pretext tasks that deviate from traditional self-supervised losses. The execution of these tasks is deemed sound and well-implemented. While all reviews are generally supportive of the paper, one reviewer suggested extending the comparison with recent EEG models, which could provide additional insights. The authors have addressed this request during the rebuttal period.

Based on the overall positive feedback from the reviewers, I endorse this paper for publication at ICML.
Decision: Accept (poster)