BrainBERT: Self-supervised representation learning for intracranial recordings
Authors: Christopher Wang
Vighnesh Subramaniam
Adam Uri Yaari
Gabriel Kreiman
Boris Katz
Ignacio Cases
Andrei Barbu
Venue: ICLR 2023 poster (2023)
Keywords: neuroscience
language models
self-supervision
transformer
decoding

Abstract: We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings. Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal. Just like in NLP where one can study language by investigating what a language model learns, this approach opens the door to investigating the brain by what a model of the brain learns. As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain. To construct these representations, we combine a technique for producing super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.  

Published as a conference paper at ICLR 2023
BRAINBERT: SELF-SUPERVISED REPRESENTATION LEARNING FOR INTRACRANIAL RECORDINGS
Christopher Wang 12 Vighnesh Subramaniam Adam Yaari
Gabriel Kreiman 2.3, Boris Katzl.? Ignacio Cases Andrei Barbu
MIT CSAIL 2CBMM 3 Boston Children's 's Hospital, Harvard Medical School
[czww, vsub851, yaari, boris, cases, abarbu) 0mit.edu gabriel. Setoriil.keeimmntttre. kreiman@tch. harvard.eds
ABSTRACT
We create reusable Transformer, BrainBERT, for intracranial field potential recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on large corpus of unannotated neural recordings. Our approach generalizes to new subjects with electrodes new positions and to unrelated tasks showing that the representa- tions robustly disentangle the neural signal. Just like in NLP where can study language by investigating what a language model learns, this approach enables investigating the brain by studying what model of the brain learns. As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain. To construct BrainBERT, we combine super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.
INTRODUCTION
Methods that analyze neural recordings have an inherent tradeoff between power and explainability. Linear decoders, by far the most popular, provide explainability; something is decodable, it is computed and available in that area of the brain. The decoder itself is unlikely to be performing the task we want to decode, instead relying on the brain to do so. Unfortunately, many interesting tasks and features may not be linearly decodable from the brain for many reasons including paucity of annotated training data, noise from nearby neural processes, and the inherent spatial and tempo- ral resolution of the instrument. More powerful methods that perform non-linear transformations have lower explainability: there is a danger that the task is not being performed by the brain, but by the decoder itself. In the limit, one could conclude that object class is computed by the retina using CNN-based decoder but it is well established that the retina does not contain explicit in- formation about objects. Self-supervised representation learning provides a balance between these two extremes. We learn representations that are generally useful for representing neural recordings, without any knowledge of a task being performed, and then employ a linear decoder.
The model we present here, BrainBERT learns complex non-linear transformation of neural data using a Transformer. Using BrainBERT, one can linearly decode neural recordings with much higher accuracy and with far fewer examples than from raw features. BrainBERT is pretrained once across a pool of subjects, and then provides off-the-shelf capabilities for analyzing new subjects with new electrode locations comparison to other machine learning settings, making additional sample efficiency critical. Other applications, such as brain-computer interfaces can also benefit from shorter training regimes, as well as from BrainBERT'ss significant performance improvements. In addition, the embeddings of the neural data provide new means by which to investigate the brain.
when data is scarce. Neuroscientific experiments tend to have little data in
hhtpps://gi http://ggttttut..e thub. com/ ez1 wang/ Bra nHERT

---

Published as a conference paper at ICLR 2023
Electrade Placements
Using BrainBERT: Feeture extraction, Intrinsic dimensionality analysis
Training BrainBERT Self-sugervised pre-troining with reconstruction loss
Classification
Intrinsic dimansion
collection
Stimulus
Classification
PCA
Spectrogr Prediction Flead
BrainBERT
BrainBERT
BrainBERT
Visuel Stimulus
300
Voluage ETT IeREtraac
Time Frequincyy Sepresent (Superin))
Masked Time Pronuurcyy Depresentaation (Supereet)
Figure 1: (a) Locations of intracranial electrodes (yellow dots) projected onto the surface of the brain across all subjects for each hemisphere. (b) Subjects watched movies while neural data was recorded (bottom, example electrode trace). (c) Neural recordings
converted to spectrograms which are embedded with BrainBERT. The resulting spectrograms are useful for many downstream tasks, like sample-efficient classification. BrainBERT data is available, by fine-tuning for each subject and/or task. (d) During pretraining, BrainBERT is optimized produce embeddings that enable reconstruction of masked spectrogram, for which must learn to infer the masked neural activity from the surrounding context.
be used off-the-shelf, zero-shot, if
BrainBERT provides contextualized neural embeddings, the same way that masked language modeling provides contextual word embeddings. Such methods have proven themselves in areas like speech recognition where a modest amount of speech, 200 to 400 hours, leads to models from which can linearly decode the word being spoken. We use comparable amount of recordings, 43.7 hours across all subjects (4,551 electrode-hours), of unannotated neural recordings to build similarly reusable and robust representations.
To build contextualized embeddings, BrainBERT borrows from masked language modeling (Devlin al., 2019) and masked audio modeling (Baevski al., 2020; Liu et al., 2021). Given neural activ- ity, as recorded by a litreoo-lkccteeneephlogggaap
(SEEG) probe, we compute a spectrogram per electrode. We mask random parts of that spectrogram and train BrainBERT produce embeddings from which the original can be reconstructed. But unlike speech audio, neural activity has fractal and scale-free characteristics (Lulzenberger et al., 1995; Freeman, 2005), meaning that similar pat- terns appear at different time scales and different frequencies, and identifying these patterns is often challenge. To that end, we adapt modern neural signal processing techniques for producing super- resolution fime-frequency representations of neurophysiological signals (Moca et al., 2021). Such techniques come with a variable trade-off time-frequency resolution, which we account for in our adaptive masking strategy. Finally, the activity captured by intracranial electrodes is often sparse. To incentivize the model to better represent short-bursts of packet activity, we use a content-aware loss that places more weight non-zero spectrogram elements.
Our contributions are:
1. the BrainBERT model reusable, off-the-shelf. subject-agnostic, and electrode-agnoste model that provides embeddings for intracranial recordings,
2. demonstration that BrainBERT systematically improves the performance of linear de- coders,
3. demonstration that BrainBERT generalizes to previously unseen subjects with elec- trode locations, and
4. novel analysis of the intrinsic dimensionality of the computations performed by different parts of the brain made possible by BrainBERT embeddings.
2 METHOD
The core BrainBERT is a stack of Transformer encoder layers (Vaswani et al., 2017). In pre- training. BrainBERT receives unannotated time-frequency representation of the neural signal as input. This input is randomly masked, and the model learns to reconstruct the missing portions.
2

---

Published as a conference paper at ICLR 2023
Short-time Fourier Transform (STFT)
Short-time Fourier Transform with mask
150
100
50
20
Superlet
Superlet with adaptive mask
Fi 1175
150
100
50
Time
Time
Figure 2: BrainBERT can be trained to either use spectrograms computed by traditional method, such as the short-time Fourier Transform (top left), or modern methods designed for neural data, such as the superlet transform (bottom left). Shown above are spectrograms from single elec- trode over a 5s interval. Superlets provide superresolution by compositing together Morlet wavelet transforms across a range of orders. As in Liu al. (2021), we mask multiple continuous bands of random frequencies and time intervals (top right, red horizontal and vertical rectangles). Since the temporal resolution of superlets falls off as the inverse function of frequency (bottom right), adopt a masking strategy that reflects this.
The pretrained BrainBERT weights can then be combined with classification head and trained on decoding tasks using supervised data.
Architecture Given the voltage measurements x Th' of a single electrode sampled at rate r for i seconds, we first find the time-frequency representation ((x) Y E T which has frequency channels and m time frames. BrainBERT is built around a Transformer encoder stack core (Vaswani et al., 2017) with N layers, each with H attention heads and intermediate hidden dimension dy. The inputs to the first layer are non-contextual embeddings for each time frame, En = (W P), which are produced using weight matrix Win Rd and combined with a static positional embedding P (Devlin et al., 2019). Each layer applies self-attention and a feed forward layer to the input, with layer normalization (Ba al., 2016) and dropout (Srivastava al., 2014) being applied after each. The outputs E of the j-th layer, become the inputs the (j [++]) 1)-th layer. The outputs of BrainBERT are E the outputs at the N-th layer.
During pretraining, the hidden-layer outputs from the top of the stack are passed input to spectro- gram prediction head, which is stacked linear network with single hidden layer. GeLU activation (Hendrycks Gimpel, 2016), and layer normalization.
Time-frequency representations BrainBERT can take two different types of time- tene-freuency rep- resentations as imput: the Short-Time Fourier Transform (STFT) and the superlet transform (Moca et al., 2021), which is a composite of Morlet wavelet transforms; see appendix A.
Due to the Heisenberg- Gabor uncertainty principle, there exists an inherent trade-off between the time and frequency resolution that any representation can provide. The most salient difference between STFT and superlet transform the way they handle this trade-off. For the STFT, resolution is fixed for all frequencies. For the superlet transform, temporal resolution increases with frequency. This is a well-motivated choice for neural signal, where high frequency oscillations are tightly localized in time.
For both types of representations, the spectrograms are z-scored per frequency bin. This is done in order to better reveal oscillations higher frequencies, which usually hidden by the lower frequencies that typically dominate in power. Additionally, the z-score normalization makes Brain- BERT agnostic to the role that each frequency band might play for different tasks. Specific fre- quency bands have previously been implicated in different cognitive processes such as language (Babajani-Feremn et al., 2016), emotion (Drane al., 2021), and vision (Jia et al., 2013). By inten- tionally putting all frequency bands on equal footing, we ensure that BrainBERT embeddings will be generic and useful for a wide variety of tasks.
Pretraining During pretraining, a masking strategy is applied to the time-frequency representa- tion Y E R*x", and an augmented view of the spectrogram, Y, is produced. Given Y, BrainBERT creates representations for spectrogram prediction network, which produces reconstruction Y of the original signal; see fig. 1.d.
3

---

Published as a conference paper at ICLR 2023
For the STFT, adapt the masking strategy of Liu et al. (2021), in which the spectrogram is cor- rupted at randomly chosen time and frequency intervals. The width of each time-mask is a randomly chosen integer from the range [stepmm. steppar] Following Devlin al. (2019), intervals selected for masking are probabilistically either left untouched (probability PID), replaced with a random slice of the same spectrogram (probability Prepluce), filled in with zeros otherwise; see appendix D) for the pseudocode summary and parameter settings. The procedure for masking frequency intervals is similar, but the width is chosen from the range [stepmin freq
stepmax fneg]
For the superlet model, we use an adaptive masking scheme that reflects the variable trade-off in time-frequency resolution made by the continuous wavelet transform. The temporal width of the time mask increases with the inverse of frequency. Similarly, when masking frequencies, more channels are masked at higher frequencies; see appendix C for parameters.
The model optimized according to a novel content aware loss. For speech audio modeling, typical to use an LI reconstruction loss (Liu et al., 2020; 2021): Yzgl
LL
Σ
Y
M
(1J)MM
where M the set of masked spectrogram positions.
But intracranial neural signal is characterized by spiking activity and short oscillation bursts. Fur- thermore, since the spectrogram z-scored along the time-axis, approximately 68% of the 7-scored spectrogram is O or < 1. So, with just bare reconstruction loss, the model tends to predict 0 for most of the masked portions, especially in the early stages of pretraining. To discourage this and to speed convergence, add term to the loss which incentivizes the faithful reconstruction of the spectrogram elements that are far away from O, for some threshold y.
Lc H(i,)) Yis T
Σ
This incentivizes the model to faithfully represent those portions of the signal where neural processes are most likely occurring.
Then, our loss function is:
L, aLc
(3)
Fine-tuning After pretraining, BrainBERT can be used as feature extractor for a linear classifier. Then, given an input spectrogram Y E RR the features are E BrainBERT(()) For a window size k, the center 2k features are W - and the input the classification network is the vector resulting from taking the mean of W along the time (first) axis. We use k = 5, which corresponds with time duration of x 244ms. During training, BrainBERT's weights can either be frozen (no fine-tuning) or they can be updated along with the classification head (fine-tuning.. Fine-tuning uses more compute resources, but often results in better performance. We explore both use cases in this work.
Data Invasive intracranial field potential recordings were collected during sessions from 10 subjects (5 male, 5 female; aged 4-19, 11.9, 4.6) with pharmacologically intractable epilepsy. Approximately, 4.37 hours of data were collected from each subject; see appendix J. During each session, subjects watched feature length movie in quiet room while their neural data was recorded at rate of 2kHz. Brain activity was measured from electrodes on perrooodectooenccphalloggggppppe (SEEG) probes, following the methods of Liu et al. (2009). Across all subjects, data recorded from a total of 1,688 electrodes, with mean of 167 electrodes per subject. Line noise was removed and the signal was Laplacian re-referenced (Li et al., 2018). During pretraining, data from all sub- jects and electrodes is segmented into 5s intervals, and all segments are combined into a single training pool. The complete preprocessing pipeline is described in appendix F.
EXPERIMENTS
For pretraining purposes, neural recordings from 19 of the sessions (Aduration 2.3hrs) was selected, and the remaining 7 sessions (Jduration 2.55hrs) were held out to evaluate performance on de- coding tasks; section 3 Tasks. All ten subjects are represented in the pretraining data. In total,
4

---

Published as a conference paper at ICLR 2023
Sentence onset
Speech/Non- Pitch speech
Volume
Task Avg.
Linear (.25s, time domain)
.54 = _04
.52 .03
.48 + .09
.54 .09
.52 - .07
Linear (5s, time domain)
.63 = _04
:58
.58
:56
.59 = 111
Linear (.25s, STFT)
.60 = _04
.53
.51
.06
.54 = .66
Linear (.25s, superlet)
.59 = .03
.53
.52
.08
.54 = .06
Deep NN (5s, 5 FF layers)
.72 10
.67 .08
57=.00
.54 .11
.63 = .12
BrainBERT (STFT)
82 = .07
.93 + .03
.75 .03
.83
= .09
.83 = .09
random initialization
.68 - 10
.59 .11
+0++00
.61 +111
.60 + .12
without content aware loss
.81 = .07
.90 .12
.68 + .06
.84 .04
.81 +111
BrainBERT (superlet)
.78 = .08
.86 .06
62
.70
+10
112
random initialization
.66 = .99
.04
.22
.60
.58 .09
without content aware loss
.74 = .12
.19
.59
.05
700
.13
.71 : .14
without adaptive mask
.78 = .08 .86
.70
004
.76
00
.77 = .08
Table 1: BrainBERT improves the performance of linear decoders across a wide range of tasks.
The tasks vary from low-level perceptual tasks like determining the volume of the audio that the subject heard, mid-level tasks like determining if the subject is listening to speech, and high-level tasks like determining if the subject heard the start of a new sentence as opposed audio of speech. Five baselines (top): four linear decoders with varying receptive fields (taking as input either 0.25s and of the neural recordings) and varying input modalities (the time-domain signal or spectorgrams computed by STFT or Superlets). BrainBERT with either STFT or Superlets (bottom). A linear decoder trained on top of the embeddings from BrainBERT, jointy tuning both the parameters of BrainBERT and the linear decoder. See Table 2 for results where BrainBERT is held fixed. Results are reported on the 10 berf-pernogming electrodes (as measured by AUC) selected with the linear model (5s, time domain) model. This ensures that results are biased away from BFainBERT and toward the linear decoders. Despite this, BrainBERT significantly outperforms any other decoders, often by very large margins. Five different ablations are described in the text.
some other
there are 21 unique movies represented across all sessions - the set of movies used in pretrain- ing does not overlap with the held-out sessions. From this held-out set. for each task, session, and electrode, we extract annotated examples. Each example corresponds with time-segment of the session recordings with time stamp Depending on the classifier architecture, the input for each example is either 0.25s or 5s of surrounding context, centered on f. The time-segments in each split are consistent across electrodes.
We run two sets of experiments, pretraining BrainBERT separately on both STFT and superlet repre- sentations. Pretraining examples are obtained by segmenting the neural recordings into 5s intervals.
Tasks To demonstrate BrainBERT's abilities to assist with wide range of classification tasks, we demonstrate it on four such tasks. Tasks range from low-level perception such as determining the volume of the audio the subject is listening to, to mid-level tasks such as determining if the subject is hearing speech or non-speech, as well as the pitch of the overheard words, and higher-level tasks such determining if the subject just heard the onset of sentence as opposed non-speech sounds. BrainBERT pretrained without any knowledge of these tasks.
Volume of the audio computed automatically from the audio track of the movie. Segments of neural recordings that corresponded to where audio track that was one standard deviation above or below the mean in terms of volume were selected. Speech ally annotated. An automatic speech recognizer first produced a rough transcript which was then corrected by annotators in the lab. Pitch was automatically computed for each word with Librosa (McFee et al., 2015) and the same one standard deviation cutoff was used to select the data. Sentence onsets were automatically derived from the transcript. In the case of the sentence onset task and the speech vs. non-speech task, the two classes were explicitly balanced. See appendix J for the exact number of examples used per class. All results are reported in terms of the ROC (receiver operating characteristic) AUC (area under the curve), where chance performance is 0.50 regardless of whether the dataset were balanced or not. The data was randomly split into a 80/10/10 training/validation set/test set.
non-speech examples were manu-
To establish baselines on supervised performance, we train decoding networks both on the raw neural signal and on the time-frequency representations, ((x). We train two linear classifiers, at two different time scales, which take the raw neural signal as input. One network re- ceives 250ms of input, which is approximately the same size (244ms) as the window of BrainBERT
Baselines
5

---

Published as a conference paper at ICLR 2023
Sentence onset
Speech/Non- speech
Pitch
Volume
Task Avg.
BrainBERT (STFT)
.66 + .03
.63 + .05
.51 + .07
.60
- .05
.60
+ .08
random initialization
.62 = .04
.57 .04
=52
.59
=.0
.57 :06
without content aware loss
.65 -: .04
.07
.60
.05
.60
.08
BrainBERT (superlet)
.71 + .06
.69 6996006
.33
.60
.00
.63
1 .10
random initialization
.62 = .03
.56 = .05
.52
.59
.08
.57 007
without content aware loss
.68 .06
.67 .07
.37
.60
.07
.62 .09
without adaptive mask
.67 .06
.66 = .06
544
006
.60
.07 .62 .08
Table 2: BrainBERT can be used with fine-tuning or without by freezing its weights. In Table
1, we showed results that include fine-tuning, jointly tuning the linear decoder and BrainBERT. Here, we show AUC with frozen weights. BrainBERT with frozen parameters and just a linear decoder has the same performance as the deep NN shown in Table 1. We get the best of both worlds: performance of deep networks and explainability of linear decoders. Note that superlet- based BrainBERT generalizes better without fine-tuning than the STFT-based BrainBERT.
embeddings received by the classifier network. The other linear network receives 5s of raw signal, which the size of the receptive field for the BrainBERT embeddings. These networks provide baselines on the amount of linearly decodable information available in the raw signal. To get a sense of what is possible with more expressive power, we also train a deep network, which has 5 stacked feed forward layers.
Because BrainBERT receives a frequecyyy representation, ((xr), as input, we also train two lin- ear networks that take time-frequency representations (either STFT or superlet) as input. Complete descriptions of each network architecture is given in appendix E.
Evaluation is performed per task and per electrode. In order to keep computational costs reasonable, we only evaluate on subset of electrodes. Our comparison to the baselines thus proceeds in two stages. First, the Linear (5s time domain input) network is trained once per electrode, for all data recorded in the held-out sessions; see section 3. Then, we select, per task, the top 10 electrodes, Susk. for which the linear decoder achieves the best ROC-AUC. For each task, this set Sesk is held fixed for all baselines, ablations, and comparisons to our model.
4 RESULTS
BrainBERT pretrained as described above in an unsupervised manner without any manual anno- tations. Our goal is for BrainBERT to be usable off-the-shelf for new experiments with new data. To that end, we demonstrate that it improves decoding performance on held-out data and held-out electrodes. Next, we show that its performance is conserved for never-before-seen subjects with novel electrode locations. After that, we demonstrate that a linear decoder using BrainBERT em- beddings outperforms the alternatives with 1/5th as much data. Finally, BrainBERT embeddings extremely useful practically, they open the doors to new kinds of analyses like investigating properties of the computations being carried out by different brain regions.
show that not only are
Improving decoding accuracy We compared a pretrained BrainBERT against the baseline models described above (variants of linear decoders and 5 reed-forward layer network); see table 1. Using the linear decoder (5s time domain input) on the original data, without BrainBERT, we chose the 10 best electrodes across all of the subjects. This puts BrainBERT at a disadvantage since its highest performing electrodes might not be included in the top 10 from the linear decoder. Then we inves- tigated if BrainBERT could offer a performance improvement when performing the four tasks with data from these 10 electrodes. Decoding BrainBERT embeddings resulted in far higher performance on every task. On average, linear decoder using BrainBERT had an AUC of 0.83. The baseline deep network had an AUC of 0.63. A linear decoder on the unprocessed data had AUC of 0.59. This difference is very meaningful: takes a result which is marginal with a linear decoder and makes it unqualified success using BrainBERT.
Decoding accuracy without fine-tuning In the previous experiment, we fine-tuned BrainBERT in conjunction with the linear decoder. In table 2, report results without fine-tuning, only updating the parameters of the linear decoder. BrainBERT without fine-tuning has an average AUC of 0.63, on par with the baseline deep network. This is what we mean by BrAinBERT being the best of both worlds: it provides the performance of a complex network with many non-linear transformations

---

Published as a conference paper at ICLR 2023
ROG-AUC
00
Linear decoding
Linear decoding with BrainBERT
Figure 3: Using linear decoder for classifying sentence onsets either (left) directly with the neural recordings (right) with BrainBERT (superlet imput) embeddings. Each circle denotes different electrode. The color shows the classification performance (see color map on right). Electrodes are shown on the left right hemispheres. Chance has AUC of 0.5. Only the 947 held-out electrodes shown. Using BrainBERT highlights far more relevant electrodes, provides much better decoding accuracy, and more convincingly identifies aentuage-related regions in the superior temporal and frontal regions.
0.70
0.65
0.60
0.55
0.50
Figure 4: BrainBERT can be used off-the-shelf for new experiments with new subjects that have new electrode locations. The performance of BrainBERT does not depend on the subject data being seen during pretraining. We show AUC averaged across the four decoding tasks (table 1), in each case finetuning BrainBERT's weights and training linear decoder. Ten held-out electrodes were chosen from the held-out subject's data. As before, these electrodes have the highest linear decoding accuracy on the original data without BrainBERT. The first two columns each group show Brain- BERT decoding results when given subject is included in the pretraining set (bluc), and when that subject held out (orange). The performance difference between the two is negligible, and both significantly outperform the linear decoding baseline (green), showing that BrainBERT is robust and can be used off the shelf. Error bars show a 95% confidence interval over the ten electrodes.
while only tuning a task-specific linear decoder. Note that STFT provides the best performance when fine-tuning BrainBERT while superlets appear to be more robust when not fine-tuning.
Ablations BrainBERT might work only because of its structure; even a random network can disentangle part of its input. To verify that the pretraining is useful, in both table and table 2 we show that Bra with randomly initialized weights is considerably worse at increasing decoding accuracy than the pretrained BrainBERT. For example, BrainBERT with STFT and fine- tuning gains on average 0.23 AUC over its randomized ablation on average (table 1). In BrainBERT, the weights matter, not just the structural prior.
BrainBERT has two other novelties in it: the content aware loss and the superlet adaptive mask. We evaluate the impact both in table and table 2. The content aware loss has mild impact on performance, e.g., BrainBERT with superlets and fine-tuning gains 0.01 AUC, increasing perfor- mance in 3 out of tasks by about 5%. The adaptive mask has similar impact, e.g., on the same model it also gains 0.01 AUC increasing performance on 2 out of tasks by about 5%. Cumulatively these changes have significant impact on performance, although second order one compared to BrainBERT itself.
Generalizing new subjects Since BrainBERT is meant be used off-the-shelf to provide performance boost for neuroscientific experiments, it must generalize to never-before-sean subjects with novel electrode locations. To test this, we perform hold-one-out analysis. For given subject, we train two variants of BrainBERT with superlet input, one that includes all subjects and one where the subject is held out. We evaluate the fine-tuning performance on all of the four tasks described above with both variants. In this analysis, we include only the held-out subject's best 10 electrodes (while previously we had included the best 10 electrodes across all subjects). We find that the performance difference while generalizing to new subject with new electrode locations is negligible; see fig. 4. For all cases, BrainBERT still gains a large boost over a linear decoder on the original data. BrainBERT generalizes extremely well to subjects.

---

Published as a conference paper at ICLR 2023
0.90
BrainBERT suparfet (w/s fine-tune)
BrainDER STFT
0.85
Deep
BrainBENT superlet
0.80
Uiner
0.75
BrainBERT STFT (wlo fine-tune)
0.70
0.65
300
150
200
230
200
400
750
Number training exampres
Figure 5: BrainBERT not only improves decoding accuracy, but it does so with far less data than other approaches. Performance on sentence onset classification is shown for an electrode in the superior temporal gyrus (red dot brain inset). Error bars show standard deviation over 3 random seeds. Linear decoders (blue) saturate quickly; deep neural networks (green, 5 FF layers, details in text) perform much better but they lose explainability. BrainBERT without fine tuning matches the performance of deep networks, without needing to learn new non-linearities. With fine-tuning, BrainBERT significant outperforms, and it does so with 1/5th as many examples (the deep NN peak at 1,000 examples is exceeded with only 150 examples). This is a critical enabling step for other analyses where subjects may participate in only a few dozen trials as well as for BCI.
Improved data efficiency Not only does BrainBERT increase performance, does so with far fewer training examples than a linear decoder or the baseline deep network require. In fig. 5, we compare the decoding AUC of baseline models and BrainBERT variants as a function of training examples. All models are trained on the data of a single electrode in the left superior temporal gyrus, which was selected from full-brain analysis (see fig. 3) to find the electrode with the best combined ranking of linear decoding (5s time duration) and BrainBERT performance. BrainBERT has much steeper learning curve: it achieves the performance of the best baseline model with 1,000 examples with only 150 examples. In addition, if more examples are available, BrainBERT is able to use them and further increase its performance.
Intrinsic dimension BrainBERT's embeddings provide value beyond just their immediate uses in increasing the performance of decoders. We demonstrate a first use of these embeddings to investigate a property of the computations in different areas of the brain without the restrictive lens of a task. We ask: what is the representation dimensionality (ID) of the neural activity? In other words, what is the minimal representation of that activity? Work with artificial networks has shown that this quantity can separate input and output regions from regions in a network that perform computations, such that high ID regions reflect increased intermediate preprocessing of the data (Ansuini et al., 2019).
Given a pretrained BrainBERT model, estimate the intrinsic dimension at each electrode by way of principal component analysis (PCA): see appendix I. For each electrode, we find the number of PCs required account for 95% of the variance in the signal (see fig. 6). We find that results are locally consistent, and that intrinsic dimension seems to vary fairly smoothly. To determine where high II) processing is located, we take the electrodes which fall in the top 10-th percentile. These electrodes mainly fall in the frontal and temporal lobes. Among these electrodes, the regions with the highest mean ID are the supramarginal gyrus, which is involved with phonological pro- cessing (Deschamps et al., 2014), the lateral obitofrontal cortex, involved with sensory integration (Rolls, 2004), and the amygdala, involved with emotion (Gallagher & Chiba, 1996). The intrinsic dimension is computed without respect to any decoding task, e.g. speech vs. non-speech volume classification, and provides a novel view of functional regions in a task agnostic way.
Investigations into the embedding space of models of neural data like BrainBERT are in their in- fancy. They could in the future lead to an understanding of the relationship between brain regions, the general flow of computation in the brain, and brain state changes over time such during sleep.
RELATED WORK
Self-supervised representations Our work builds masked-speettoogram modeling approaches from the field of speech processing (Liu et al., 2021; 2020; Ling & Liu, 2020), which use a re- constructive loss to learn representations of Mel-scale audio spectrograms. In other fields, self- supervised models have been used to produce representations for variety of downstream applica-
8

---

Published as a conference paper at ICLR 2023
BrainBERT
250
* Electrones
Intrinsic Dim.
125
20
40
60
150
Randomly initielized
250
50
71 90 :10
Intrinsic Dimension
Figure 6: Given neural recordings without any annotations, we compute the intrinsic dimensonality (ID) of the BrainBERT embeddings at each electrode. (a) These embeddings lie lower dimen- sional space (top) than those produced by a randomly initialized model (bottom). (b) The electrodes with the highest poral lobes, and demonstrate that electrodes that participate in similar computations on similar data will have similar ID, providing new data-driven metric by which to identify functional regions and the relationship between them.
(top 10-th percentile; circled red) can be found mainly in the frontal and tem-
tions (Oord et al., 2018), including language (Devlin et al., 2019), audio (Baevski et al., 2020; Hsu al., 2021), and images (Chen et al., 2020). The representations learned by these models in turn be used to investigate their input domains. Language models have been used to study language processing by examining the semantic and syntactic content of the embeddings (Tenney al., 2019; Roger et al., 2022) and the dimensionality of the input space (Hernandez & Andreas, 2021). Our work opens the doors for similar studies of neural processing. Here we investigate the intrinsic dimensionality of neural computation.
Neural decoding Intracranial probes, such as SEEG, have been proposed key components in the next-generation of brain machine interfaces (Jackson & Hall, 2016; Herff et al., 2020). Our work the first to provide modern representation learning approach to feature extraction for intracra- nial signals. Previous work gives approaches for other modalities, including IMRI data (Malkiel et al., 2021), and EEG (Kostas et al., 2021; Banville et al., 2021). But the signal obtained for these modalities have fundamentally different characteristics. Importantly, they lack the temporal reso- lution and spatial precision of intracranial electrodes, which directly measure eloctrophysiological brain activity from a small population of neurons.
CONCLUSION
We present self-supervised approach for learning representations of neurophysiologica data. The resulting representations have three chief benefits. First, they improve the accuracy and data- efficiency of neural decoding. This is small matter: many core results in neuroscience hinge on whether a linear decoder can perform a certain task. Morcover, neuroscience tends to be in the small data regime where gathering data for even small experiments is significant investment; here, a large reduction in the amount of data required to decode something can make a large difference. Such decoding is also critical to building the next generation of brain machine interfaces. Second, they provide this performance improvement while maintaining the benefits of linear decoders, i.e., explainability, with the performance of complex decoders. This is direct upgrade of one of the most popular techniques in neuroscience. Finally, by providing task-agnostic embeddings that can then be analyzed, they open the door for new investigations of cognitive processes. The changes in representations across time may reveal mechanisms and dynamics underlying sleep and other brain state changes in a completely data-driven manner. To further refine the representations that Brain- BERT builds, in the future, we intend to train much larger variants on continuous 24/7 recordings from numerous subjects. Fields like NLP have been revolutionized by such models; we hope that neuroscience will see similar benefits.
BrainBERT is available as a resource to the community along with the data and scripts to repro- duce the results presented here. It should be noted that the pretrained weights that release have been trained on activity induced by a particular stimulus, i.c. passive movic viewing. the down- stream use case departs significantly from this setting, then recommend pretraining BrainBERT from scratch to improve performance. Otherwise, most groups investigating intracranial recordings should be able to use BrainBERT drop-in addition to their existing pipelines.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF- 1231216, the NSF award 2124052, the MIT CSAIL Systems that Learn Initiative, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the SEMMM-ieemen Grad- uate Fellowship, the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program, the DARPA Knowledge Management at Scale and Speed (KMASS) program, the United States Air Force Research Laboratory and United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number 148750-19-2-1000 the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-001 and the Office of Naval Research under award number N00014-20-111 1-2589 and award number N00014-24 11114--0---6-44 1-2643. The views and conclusions con- tained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copy- right notation herein We would also like to thank Colin Conwell and Hector Luis Penagos- Vargas for helpful comments and discussion.
ETHICS STATEMENT
Experiments that contributed to this work were approved by IRB. All subjects consented to partici- pate. All electrode locations were exclusively dictated by clinical considerations.
REPRODUCIBILITY STATEMENT
Code to train models and reproduce the results was submitted as part of the supplementary materials and can be accessed here: https: / htt///gtttt thub. wtwphcoooooo...cccooo.ooc.oeeeee cz/ wang Braa nBERT.
REFERENCES
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Sys- fems, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16Xiv: 2016.
Abbas Babajani-Feremi, Shalini Narayana, Roozbeh Rezaie, Asim F Choudhri, Stephen Fulton, Frederick A Boop, James W Wheless, and Andrew C Papanicolaou. Language mapping using high gamma electrocorticooraphy, fMRI, and TMS versus electrocortical stimulation. Clinical Neurophysiology, 2016.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 2020.
Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis Alexander Engemann, and Alexandre Gramfort. Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of neural engineering. 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 2020.
Isabelle Deschamps, Shari Baum, and Vincent L. Gracco. On the role of the supramarginal gyrus in phonological processing and verbal working memory: evidence from rTMS studies. Neuropsy- chologia, 53:39 46, 2014.
Rahul S. Desikan, Florent Ségonne, Bruce Fischl, Brian T. Quinn, Bradford C. Dickerson, Deborah Blacker, Randy L. Buckner, Anders M. Dale, R. Paul Maguire, Bradley T. Hyman, Marilyn S. Albert, and Ronald J. Killiany. An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neurolmage, 2006.
10

---

Published as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Daniel L Drane, Nigel Pedersen, David S Sabsevitz, Cady Block, Adam S Dickey, Abdulrah- man Alwaki, and Ammar Kheder. Cognitive and emotional mapping with SEEG. Frontiers in Neurology, 2021.
Mingyu Fan, Nannan Gu, Hong Qiao, and Bo Zhang. Intrinsic dimension estimation of data by principal component analysis. arXiv preprini urx.v:002.2000 2010.
Bruce Fischl, André van der Kouwe, Christophe Destrieux, Eric Halgren, Florent Ségonne, David H. Salat, Evelina Busa, Larry J. Seidman, Jill Goldstein, David Kennedy, Verne Caviness, Nikos Makris, Bruce Rosen, and Anders M. Dale. Automatically Parcellating the Human Cerebral Cortex. Cerebral Cortex, 2004.
Walter J Freeman. tel--theoretic approach to understanding scale-free neocortical dynamics. Biological cybernetics, 2005.
Michela Gallagher and Andrea A Chiba. The amygdala and emotion. Current opinion in neurobi- ology, 6(2):221-227, 1996.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.
David M. Groppe, Stephan Bickel, Andrew Dykstra, Xiuyuan Wang, Pierre Mégevand. Manuel R. Mercier, Fred A. Lado, Ashesh D. Mehta, and Christopher J. Honey. iELVis: An open source matlab toolbox for localizing and visualizing human intracranial electrode data. Journal of Neu- roscience Methods, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 2016.
Christian Herff, Dean J Krusienski, and Pieter Kubben. The potential of stereotactic-eeg for brain- computer interfaces: current progress and future directions. Frontiers in neuroscience, 2020.
Evan Hernandez and Jacob Andreas. The low-dimensional linear geometry of contextualized word representations. In Proceedings of the 25th Conference on Computational Natural Language Learning, 2021.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
Andrew Jackson and Thomas M Hall. Decoding local field potentials for neural interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2016.
Xiaoxuan Jia, Seiji Tanabe, and Adam Kohn. Gamma and the coordination of spiking activity in early visual cortex. Neuron, 2013.
A. Joshi, D. Scheinost, Okuda, D. Belhachemi, Murphy, L. H. Staib, and Papademetris. Unified framework for development, deployment and robust testing of neuroimaging algorithms. Neuroinformatics, 2011.
Demetres Kostas, Stephane Aroca-Ouellttte, and Frank Rudzicz. BENDR: using transformers and contrastive self-supervised learning task learn from massive amounts of EEG data. Frontiers in Human Neuroscience, 2021.
Guangye Li, Shize Jiang, Sivylla E Paraskevopoulou, Meng Wang, Yang Xu, Zehan Wu, Liang Chen, Dingguo Zhang, and Gerwin Schalk. Optimal referencing for stereo- electreencephalggaapiie (SEEG) recordings. Neurolmage, 2018
11

---

Published as a conference paper at ICLR 2023
Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659, 2020.
Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsu- pervised speech representation learning with deep bidirectional transformer encoders. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
Andy T. Liu, Shang Wen Li, and Hung Yi Lee. TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech. IEEE/ACM Transactions on Audio Speech and Language Processing, 2021.
Hesheng Liu, Yigal Agam, Joseph Madsen, and Gabriel Kreiman. Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron, 2009.
Werner Lutzenberger, Hubert Preissl, and Friedemann Pulvermüller. Fractal dimension of electroen- cephalographic time series and underlying brain processes. Biological Cybernetics, 1995.
Itzik Malkiel, Gony Rosenman, Lior Wolf, and Talma Hendler. Pre-training and fine-tuning trans- formers for fMRT prediction tasks. arXiv preprint arXiv:21 2021.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt Mccicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in Python. In Proceedings of the 14th Python in Science Conference, 2015.
Vasile V Moca, Harald Bårzan, Adriana Nagy-Dābåcan, and Raul C Muresan. Time-frequency super-resolution with superlets. Nature communications, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint 100.0038,, 2018.
Elise Roger, Sonja Banjac, Michel Thiebaut de Schotten, and Monica Baciu. Missing links: The functional unification of language and memory (LuM). Neuroscience and Biobehavioral Reviews, 2022.
Edmund T Rolls. Convergence of sensory systems in the orbitofrontal cortex in primates and brain design for emotion. The Anatomical Record Part A: Discoveries in Molecular, Cellular, and Evolutionary Biology: An Official Publication of the American Association of Anatomists, 281 (1):1212-1225 2004.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 2014.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv: 0505,0550, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- fion processing systems, 2017.
A. I. Yang, X. Wang, W. K. Doyle, E. Halgren, C. Carlson, T. L. Belcher, S. S. Cash, O. Devinsky, and T. Thesen. Localization of dense intracranial electrode arrays using magnetic resonance imaging. Neuroimage, 2012.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. arXiv preprint arXiv: 1904, 00962, 2019.
12

---

Published as a conference paper at ICLR 2023
A SUPERLETS
Superlet transforms are technique for time-frequency analysis introduced by Moca et al. (2021). The superlet transform is formed from composite Morlet wavelet transforms. In contrast to tra- ditional tine-frequency techniques, such as the Short-time Fourier Transform (STFT) which has a fixed time-frrqueee freeeefrruenny resolution trade-off for all frequencies, the Morlet wavelet has lower temporal resolution at lower frequencies. This variable trade-off allows the resulting representation to better capture the dynamics of neural signal, for which high frequeney oscillations often occur in short bursts and low frequency oscillations persist for longer durations.
However, the Morlet transform is not Pareto optimal, and the superlet transform makes improve- ments to both time and frequency resolution simultaneously in order to better capture the self-similar oscillations across different frequencies which are often present in neural signal. In this section, we briefly summarize the content of Moca et al. (2021) relevant to our work.
Consider the following formulation of the Morlet wavelet for given frequeney of interest f:
1
12
(1,,(1)
exp
exp (j2m
(4)
B.V2
2B
B
(5)
Here, is the number of cycles, which is a parameter that controls the time-frees timeffreuency resolution trade-off. The expression for the wavelet the right side of the equation eq. (4) can be understood as being composed of two terms: the first term can be thought of as complex wave function (Fuler's formula), and the second term can be understood the amplitude of the wave. Note that the width a of the windowing function falls off with frequency.
Gaussian windowing function that modulates
For given frequency f, the superlet transform of order o is the geometric mean of Morlet transforms Hee. for range of values с E [1, o). The superlet representation at a frequency of interest f for signal x is then:
0
II V2 x Vo.f i=1
where * is the complex convolution operator. Although, compared to the Morlet wavelet transform, the overall resolution is improved, the f in the denominator of eq. (5) means that there continues to be relatively lower time resolution at lower frequencies. In other words, there is more temporal smearing at the bottom of the spectrogram than at the top. To account for this, we use an adaptive masking strategy (see section 2: Pretraining).
B TIME-FREQUENCY REPRESENTATION PARAMETERS
For the STFT, we use window size of 400 samples (z= 200ms) with overlap of 350 samples (+= 175ms) with frequency channels evenly spaced from 0 to 200Hz.
For the superlet, from 0.1 to 200Hz. To match the somp-sanp rate of the STFT, the superlet representation is decimated by a factor of 50. Finally, for both representations, remove 5 columns from each array on either side (+x 250ms) to account for edge-efficcss
e₁ 1 with orders 30. The frequencies of interest are evenly spaced
Superlets take the composite of many Morlet wavelets. Thus, superlet is simply collection of Morlet wavlets with different c:
SL. = (10 (c,,.., Go.
where o the order of the superlet.
For this work, we use the multiplicative, adaptive superlet transform. In the multiplicative version of the transform, G = C1 And in the adaptive version, the order of the superlet changes according to frequency:
f Jmnn
Omin + (omac
Omen)
(8)
smin -f fman
13

---

Published as a conference paper at ICLR 2023
Adaptive Temporal Masking
Adaptive Frequency Masking
Frequency (Hz)
200
4.0
Mask Width
30-
0000000000
100
25
75
15
10
05
10
3.5 20 25
100
150
Time (s)
Frequency (Hz)
Figure 7: Adaptive masking (Left) The adaptive temporal mask, for which the width increases with the inverse of frequency. (Right) The width of the frequency mask, which specifies the amount of masked frequency channels, increases as function of frequency.
C MASKING TIME AND FREQUENCY
receives an augmented version of the spectrogram, from which random time and frequency bands have been removed. In this work, use two masking strategies: static masking, which is suited for spectrograms fixed time- frequenyy resolution, and adaptive masking, which is appropriate for spectrograms with variable trade-off in time-frequee time-freuuency resolu- tion. Static masking For our static masking procedure, we adapt the work of (Liu et al., 2020; 2021). The width of the temporal mask is randomly selected from the range [1, 5]. and the width of the frequency mask randomly selected from the range [1,2] 2.
During pre-training, BrainBERT
Adaptive masking For the superlet transform, The temporal width of the adaptive mask as a function of frequency f is given by:
adaptive temporal and frequency masks.
200
wf()) max m. 20 +
where is the minimum width of the mask, which is randomly selected from (1,2) per example.
See fig. for visualization.
The width of the frequency masks is given function of frequency f:
wr(s) = max (1,
4.9
(10)
250
The parameters of these equations selected to roughly match the width of the masks used for the STFT.
D MASKING PROCEDURE
The masking procedure is described algorithm (see section Pretraining). Note that intervals are masked without overlap. The procedure is adapted from Liu al. (2021) and Devlin et al. (2019). During pre-training, we use Pmask 0.05, PID = 0.1, and Preplace = 0.1. The purpose of letting some segments go unaugmented is to make the distribution seen at train time and test time more similar. The purpose of replacing some segments with random signal is to prevent the model from simply learning the identity function.
14

---

Published as a conference paper at ICLR 2023
Algorithm T'mme-maaee masking procedure
Y < m spectrogram
i --0 0
while < m do
~ 0n)
p < Pmask then
1 ~~ Unif( stePmax 1)J
g ~ Unif(0. 1)
if 9 PID then
pass
else if PiD S < PID Preplace then
+ Unif (0. m
Y 1] + [ijjj+t
else
Y {iiit+0
end if
i i i+1
end if end while
BASELINES
Linear baselines (time domain) We train two feed forward networks with layer dimensions (Aipput) 1] and sigmoid activations. The first model, Linear (5s time domain), takes 58 of time domain input, sampled at 2048 Hz, so dinput 10,240. 240. The second model, Linear (0.25s time domain), takes 0.25s of input, so dinput 512.
Deep neural network (time domain) We train deep neural network, Deep NN (5s time domain), which consists of stacked feed forward layers with dimensions [dmputt 1024, 512, 256, 128] and ReLU activations on the hidden layers and sigmoid activation on the output layer. The network takes 5s of time domain input, sampled at rate of 2048 Hz, so dinpar 10, 240.
Linear baselines (time-frequency domain) We train two feed forward linear networks with di- mension [dinpa: 1] and sigmoid activations that take the time-frequency representations as input. Linear (.25s STFT) and Linear (.25s superlet) receive the time-frequency representation, averaged across time in 244ms interval, centered the example. More precisely, given time-frequency representation Y E 2 the model receives the vector obtained by averaging Y.u-s.44 across the time axis. There are = 40 frequency bands, dinput 40. This is the same form as the input that our classification network receives from BrainBERT (see section 3).
F PREPROCESSING
Data was high-pass filtered at O.I Line noise at 60Hz and its harmonies were removed. Each electrode was re-referenced by subtracting out the mean signal from the two adjacent electrodes on the same shaft (Laplacian re-referencing). This decreases the cross-correlation between electrodes (Li et al., 2018). Only electrodes which can be Laplacian re-referenced, i.c., have neighbors, included in the pretraining data. Finally, signals from all electrodes are visually inspected, and recordings which show obvious signs of corruption removed. In total, data from 1,249 electrodes (Rz 74% of total electrodes; 4,551 hours) are used for pretraining.
G PRETRAINING PARAMETERS
All layers (NV = 6) in the encoder stack are set with the following parameters: d, = 768, = 12, and Paropout = 0.1. We pretrain the BrainBERT model with the LAMB optimizer (You et al., 2019) and Ir 1e - 4. We use a batch size of Reacc = 256, train for 500k steps, and record the validation performance every 1,000 steps. Then, the weights with the best validation performance retained.
During pretraining, the BrainBERT representations are passed to a spectrogram prediction head which consists of a two feed forward linear layers. The first layer, hidden layer, has dimension
15

---

Published as a conference paper at ICLR 2023
b
5.%
E714
3X0%
peroria
2.7
Intrinsi Dimension
Figure 8: Intrinsic dimension averaged by region We find the intrinsic dimension (ID) for all held-out electrodes, and consider the electrodes in the top 10-th percentile. Among these electrodes, the percentage which fall in each region is shown in (a). However, the electrodes are not distributed uniformly across the brain, so to get normalized view of the regions, we take the mean across electrodes in each region (b). Error bars show 95% confidence interval
= 768 and GeLU activation. The second layer, the output layer, has dimension = 40 This matches the height of the spectrogram, which is determined by the number of frequencies of interest; see appendix B.
As part of our ablation test, also include BrainBERT with randomly initialized weights. For this, weights are initialized according to the uniform Xavier scheme (Glorot & Bengio, 2010).
H FINE-TUNING TRAINING PARAMETERS
The classifier a fully connected linear layer with dnn = 768, dons = and sigmoid activation. The model is trained using binary cross entropy loss. When fine-tuning, we use the Adam W optimizer, with 1e - 3 for the classification head and |r = le - for the BrainBERT weights. When training with frozen BrainBERT weights, use the AdamW optimizer with ir 1e 3.
All models, both this classifier and the baseline models, are trained for 1,000 updates. Validation performance is computed every 100 updates, and test accuracy is reported using the weights with the best validation performance.
I INTRINSIC DIMENSION
We use a standard method for determining intrinsic dimension (Fan al., 2010). For given thresh- old B, the intrinsic dimension is the d E N such that the ratio of explained variance for d dimensions of N dimensional PCA above 3:
Σd E var(y:)
ΣS var(y;) > 3
(11)
We use PCA of N 200 and threshold of 0.95. For each session in the held out data, we segment the neural recordings into 5s intervals, for which the spectrogram is a matrix Y e paxm. We produce the BrainBERT embeddings E E Raxm for each spectrogram, and average across the time-dimension to obtain single n-dimensional vector representation of the interval. For each electrode, we find the intrinsic dimension of the manifold on which these BrainBERT embeddings lie. We find the regions with the highest intrinsic dimension, by first taking the electrodes in the top 10-th percentile. Among these electrodes, most of them lie in the precentral gyrus (13.5%), the lateral orbitofrontal cortex (10.1%), and the pars opereularis (9.0%). However, it should be noted that the electrodes are not distributed evenly across the cortex, so to get a sense of which regions have the highest ID, should normalize across the number of electrodes in region. We find that
16

---

Published as a conference paper at ICLR 2023
the regions with the highest mean ID are the supramarginal gyrus, the lateral orbitofrontal cortex, and the amygdala.
1.1 TASKS
Sentence onset
Once the movie transcript has been aligned to the brain activity, the brain activity which corresponds to the sentence onsets can be collected, each embedded 5s of context. These intervals form the set of positive examples.
Speech non-speech For this task, the positive examples are formed from the of context surrounding any word. For both this task and the sentence onset task, the negative examples formed by finding intervals which do not overlap with any intervals of speech audio. Each example is embedded in 5s of context.
Volume For each word in the transcript, the volume is measured as the root-mean-square for a 500ms interval starting from the word onset. Those words which lic one standard deviation above the mean volume are labeled "high-volume". Those words which lie one standard deviation below the mean volume are labeled as "low-volume".
Pitch For each word in the transcript, the pitch is extracted using librosa's piptrace function over a Mel-spectrogram (sampling rate 48,000 Hz, FFT window length of 2048, hop length of 512, and 128 mel filters). Those words which lie one standard deviation above the mean pitch are labeled "high-pitch". Those words which lie one standard deviation below the mean pitch labeled as "low-vpitch".
17

---

Published as a conference paper at ICLR 2023
J DATASET STATISTICS
Sentence onset Non-speech
Speech/Non-speech
Volume
Pitch
Subject
Onset
Speech
Non-speech
High
Low
High
Low
subject-1
1,587
1,587
6,333
6,333
1,135
542
1,723
1,724
subject-2
1,071
1,071
6,413
6,413
662
137
1,042
1,066
subject-3
2,057
2,057
2,581
2,581
1,350
980
1,591
1,592
subject-4
542
542
2,500
2,500
295
229
378
364
subject-5 1,059
1,059
3,183
3,183
769
521
1,009
980
subject-6 1,668
1,668
2,367
2,367
1,092
815
1,311
1,309
subject-10 1,971
1,971
1,971
1,971
1,350
980
1,591
1,592
Table 3: Annotated data statistics The number of examples per task and per subject for the held-out sessions are shown here. For the sentence onset task and speech vs. non-speech task, the number of examples is explicitly balanced between classes. Non-speech examples correspond with Is intervals which do not overlap with any word-audio in the movie.
Subj.
Age (yrs.)
# Elec- trodes
Movie
Recording time (hrs)
Held- out
19
91
Thor: Ragnarok
1.83
1
Fantastic Mr. Fox
1.75
The Martian
0.5
X
12
100
Venom
2.42
Spider-Man: Homecoming
2.42
Guardians of the Galaxy Guardians of the Galaxy
2.5
2
3
Avengers: Infinity War Black Panther
4.33
1.75
Aquaman
3.42
18
91
Cars 2
1.92
3
Lord of the Rings
2.67
Lord of the Rings 2 (extended edition)
3.92
4
9
135
Megamind Toy Story
2.58
1.33
Coraline
1.83
11
205
Cars 2
1.75
5
Megamind
1.77
12
152
Incredibles
1.15
Shrek
1.68
Megamind
2.43
7
6
109
Fantastic Mr. Fox
1.5
8
4.5
102
Ant Man
2.28
9
16
72
Sesame Street Episode
1.28
10
12
173
Cars 2
1.58
X
Spider-Man: Far from Home
2.17
Table 4: Subject statistics Subjects used in BrainBERT training, and held-out downstream evalu- ation. The number of uncorrupted, electrodes that can be Laplacian re-referenced are shown in the second column The average amount of recording data per subject is 4.3 (hrs).
18

---

Published as a conference paper at ICLR 2023
K ELECTRODE VISUALIZATION
For each subject, pre-operative TI MRI scans without contrast were processed with FreeSurfer (Fis- chl et al., 2004), which performed skull stripping, white matter segmentation, and surface genera- tion. iELVis (Groppe et al., 2017) was used to co-register a post-operative fluoroscopy scan to the preoperative MRI. Electrodes were manually identified using BiolmageSuite (Joshi et al., 2011). and then assigned to one of 94 regions (according to the Desikan-Killiany atlas (Desikan et al., 2006)) using FreeSurfer's automatic parcellation. The alignment to the atlas manually verified for each subject.
Depth electrodes in the white matter, if they within 1.5 of the gray-white matter boundary, were projected to the nearest point on that boundary, and were labeled as coming from that region (for the purposes of region analyses). This procedure very similar the post brain-shift correction methods used for electroeorticoograpy electrodes (Yang et al., 2012). For solely visualization pur- poses, all electrodes identified to lie in the gray matter first projected to the pial surface (using nearest neighbors), and then mapped to an average brain (using Freesurfer's 's fsaverage atlas) for the visualizations shown fig. and fig.
the gray-white matter boundary were
L PRETRAINING PERFORMANCE
BrainBERT (superint)
b
Masked superiet
Predicted
031009110 105
Updllte
c
BrainBERT (STFT)
Masked STFT
Predicted
V0151 1006
Figure Pretraining performance (a) During pretraining, BrainBERT with superlet inputs achieves 0.81 content-aware reconstruction loss. The L1 reconstruction component (not pictured) of this loss is 0.41. (c) BrainBERT with STFT inputs achieves 0.42 content aware reconstruction loss, of which the L1 component accounts for 0.20. (b) and (d) show sample reconstructions produced by BrainBERT after 500k updates.
19

---

Published as a conference paper at ICLR 2023
M
SUPPLEMENTARY FIGURES
Intrinsic Dim.
a
b
433
333
130
16
18
29
25
Figure 10: Intrinsic dimension of STFT features In fig. 6, we show plot the intrinsic dimension for each electrode based the BrainBERT representation of that electrode's activity. For comparison, a similar plot (b) can be made using the raw short-time Fourier transform (STFT) features. Unlike for the BrainBERT representations, the distribution of dimensions tightly grouped around a few values (a).
Sentence onset
Speech non-speech
Volume
Pitch
Figure 11: Location of test electrodes The results in section 4 are given according to fixed subset of electrodes per task. For each task, this subset is determined by training a linear classifier over all electrodes and taking the top 10 electrodes with the best performance. The location of these electrodes is shown here.
20

---

Published as a conference paper at ICLR 2023
N
SUPPLEMENTARY RESULTS
Sent.
Speech/Non- Pitch speech
Volume
onset
Linear (58, time domain)
.63+-.04
.58 + +.06
.58 + .07
.56 + +.19
Top 10
BrainBERT (STFT)
.82+-007
.93 + .03
.75 + .03
.83 + .09
BrainBERT (superlet)
.78+.008
.86 + .06
.62 +.5
.70 1010
Linear (5s, time domain)
.63..04
.5 .04
.58 + .06
.61 +
Top 20
CortexBERT (STFT) .82..08
.91 #
.74 .06
Cor (superlet) .77=.10
.81
.68 + .06
.76 #66
Table 5: Performance evaluated per task and per electrode. In order keep computational costs reasonable, since there are many baselines and ablations to compare against, we only evaluate the performance for subset of all electrodes. For given task, this subset is selected by first training linear classifier over all electrodes. In section 4, we find the top 10 electrodes with the highest linear classifier performance. This subset is then held fixed over all comparisons. The variance reported is calculated with respect to this set of electrodes. In this table, we report the results for the top 20 electrodes as well for comparison.
046
0.4
Figure 12:
Layerwise analysis We compute the decoding performance of the frozen BrainBERT features layer of the transformer encoder stack. We see that for all tasks, performance increases with depth in the stack, peaking the second to last layer. Note that in this work, used the features taken from the last layer. And for the purposes of showing the advantages of BrainBERT's pre-training, the performance we obtain sufficient.
0.20
Datta ROCAUUC
0.00
-020
Figure 13: Generalization vs. age. The ability of BrainBERT to transfer to new subjects does not depend on age (Pearson's r -0.27, with p-value 0.451). The y-axis shows the delta between the AUC-ROC, averaged across all tasks, between BrainBERT pretrained with and without particular subject. The x-axis shows the age of the subject. See table 6 for complete breakdown per task.
21

---

Published as a conference paper at ICLR 2023
Sentence onset
Speech/Non- speech
Pitch
Volume
Average
pretrain w/ subject
.74 09
.85 + .06
.79 = .05
.92 = .03
.82 = .09
pretrain w/o subject
.73
.85 + .05
.77 = .10
.92 = .03
.82 = .10
Linear (5s)
.62 .04
.56 + .04
.52 + .02
.62 + .06
.58 + .06
pretrain w/ subject
.83 + .09
.74 :4
.73 = .10
.73 .11
pretrain w/o subject
.622.008
+
.77 +
.77 .12
Linear (5s)
.59 00
.53 + .03
152.000
.61 #.11
.56 .08
pretrain w/ subject
.85
110
.65 .03
.86 110
.79 112
pretrain w/o subject
.44.05
.78
.61 -07
.88 = .03
.78 .13
Linear (5s)
.64=.02
.57
.54
.56 = 003
.58 .05
pretrain w/ subject
.57=
.04
.66 + .08
.69 .06
.72 = .15
pretrain w/o subject 4
.60=.08
.11
.66 = .6
.75 .09
.73 =
Linear (5s)
.58
.54
.61 + .04
.64 = 005
.59 .05
pretrain w/ subject
.63+.11
.76
.69 - 03
.70 .06
.70 = .10
pretrain w/o subject 6
.64=.11
.16
.76 + 03
.78 = 004
.73 .11
Linear (5s)
.56+.04
.34
.53 + 06
.58 = 06
.55 006
pretrain w/ subject
.74
.91 .04
.68 + .0
.83 - .08
.79 110
pretrain w/o subject
.75=.00
.86
.66 + .66
.85 + .02
.78 .11
Linear (5s)
.63#.04
.05
.58 = .0
.53 = 005
.58 = .05
pretrain w/ subject 5
.71
.75 .22
.65 + .03
.84 + .04
.74 + .09
pretrain w/o subject
.74
.81
.65 = .07
.89 = .03
777
Linear (5s)
.56=.05
.54
.52 = 04
.55 = .03
.54 5.4
pretrain w/ subject
.68+.10
.66 + 05
.76 + .04
.74 + .12
pretrain w/o subject 8
.70=.06
.12
.72 - .6
.75 .08
.76 .10
Linear (5s)
.61=1.04
.54
.59 + .04
.57 .05
.58 .05
pretrain w/ subject
.75=.11
.09
.75 + .04
.59 + .11
.72 #
pretrain w/o subject 9
.78+1.06
.80 110
.76 + .03
.56 + .16
.72 .114
Linear (5s)
.55
.53
.53 = .02
.80 = .08
.60 + .12
pretrain w/ subject 10
.79+.05
.11
.68 + .07
.90 + .05
.80 1111
pretrain w/o subject 10
.76+.06
.84
770
+ 03
.86 + 08
.79 + .09
Linear (5s)
.622.04
.53 .03
.53
+ .04
.56
= .05
.56 = .06
Table 6: Held-one-out analysis. For each subject, report the performance of versions of BrainBERT with superlet input. One version is trained without that particular subject, and one ver- sion is trained using all subjects. Fine-tuning performance is reported on subset of electrodes be- longing to the subject in question. As before, this subset is determined by running a linear classifier over electrodes and taking the top 10 electrodes with the highest performance. The performance of that linear classifier is also shown for comparison. In general, we see a close match between the two versions of BrainBERT.. performs better. This can be attributed to the fact that one subject's data may have negative contri- bution when it comes to learning useful representations for a particular task.
some instances, the version without a particular subject even
22


Reviews:
[{"review_id": "gP30UsZw2G", "reviewer": "Reviewer_98Er", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes BrainBERT , a self-supervised representation learning approach for intracranial recordings inspired by success in NLP and speech recognition. \n\nFirst, BrainBERT is trained on intracranial recordings while the subjects were watching videos without driven by any particular decoding task. Then, pretrained model is either finetuned or used as a feature for linear decoding of different low-level and high-level tasks.\n\nThe experiments show that features learned by BrainBERT\n1. Perform better than linear decoders directly trained on recordings\n2. Requires less data to reach good decoding performance \n3. Generalizes to new subject and electrodes", "summary_of_the_review": "This paper presented exciting new results by application BERT type training on intracranial recordings. The approach presented show promise and has potential to be applied on several other type of brain recordings to gain more interesting insights about the brain.\n\nTherefore, I recommend acceptance.", "strength_and_weaknesses": "Strengths:\n1. The paper is clearly written and easy to follow.\n2. The approach is validated on multiple tasks from determining volume level (low-level) to speech onset (high level) , compared with relevant baselines with relevant ablations thus making the evaluation of the proposed approach quite solid.\n3. The results are promising given the brain datasets are generally smaller, this approach shows a new direction of feature learning directly using brain recordings and then applying on decoding tasks.\n\nWeaknesses:\n1. The only weakness of this work I can see is that it is build on existing approach BERT and does not propose something entirely novel. However, I do not believe it is a major weakness as it is applied to a different problem and modified to take into account different modality.\n2. It is not clear to me how 10 best electrodes were selected. I recommend clarifying it and perhaps adding some results with best 50 or best 20 just to give an idea how much impact selection have on the decoding performance. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "V29sdub6vV", "reviewer": "Reviewer_TZqs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a transformer-based architecture with a masked modeling task to operate on intra-cranial recordings directly. Some of the important components of this model include a content-aware loss to capture the burst-like nature of neural activity, a superlet representation of the raw neural data and a masking procedure based on removing frequency/time bands. The authors show that using this pretrained architecture leads to better downstream identification performance on a suite of tasks like detecting speech onsets, distinguishing between speech and non-speech sounds etc..that cannot be attributed to the complex architecture of the model alone. They also look at the impact of using frozen vs. trainable encoding layers on downstream accuracy, generalizing to new subjects and accuracy as a function of training examples. Lastly, they apply PCA to the hidden states of each electrode input to compute the electrode’s intrinsic dimensionality.", "summary_of_the_review": "Overall, I think this paper is a promising methodological advancement over linearized encoding models. However, I thought the paper was a bit confusing to read and found some of the empirical results to be weak (effective dimensionality).", "strength_and_weaknesses": "Strengths: This paper presents a new and interesting computational framework for identification tasks using neural recordings. I think this will be an important research direction going forward and commend the authors for pursuing a novel line of work.\n\nWeaknesses: While the paper was interesting, several of the implementation details were confusing or missing altogether. I highlight a few examples below.\n\n- How does the model handle different electrodes? Is training shared across them? I was also confused by “First, the Linear (5s time domain input) network is trained once per electrode, for all held-out electrodes.” on page 6 under baselines.\n    - I am particularly interested in the trade-offs of using fewer electrodes and if the authors found better performance on downstream task based on the location of the electrodes, as would be expected.\n- Is there any possibility for the model to encode information in the population code?\n- What is the context window during training? I presume 5 seconds from details in the introduction of section 3.\n- What do the training curves look like? What is the pretraining performance? How hard was hyper-parameter tuning?\n- Can the authors comment on the fact that the decoding accuracy is above the deep NN baseline only when the model is finetuned?\n- Nit: I would decrease the curvature contrast in Fig. 5B so the electrode values are more visible.\n- I am not sure what to make of the intrinsic dimensionality analysis as the links to working memory and a sensory lobe seem arbitrary. Furthermore, the distribution across cortex doesn’t seem to follow any prior findings on gradients or functional distinctions during speech processing.\n- How many hours of data per subject on average?\n- What does the prediction head look like?\n- Is there any relationship between SNR and subject-transfer abilities (there must be!)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IlB_YhYso2k", "reviewer": "Reviewer_1MuJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I'm increasing my score from 5 to 6 as the authors have addressed several of my concerns.\nHowever, I am still not convinced about the utility of this architecture or pretrained weights beyond the specific experimental setting that is considered by the authors because the task someone is performing has a big effect on the brain activity. I would expect a paper that claims to deliver the BERT of brain recordings to show this. So I really hope that the authors indeed include a disclaimer about this in their manuscript.\n\n^^^^^^^^^^^ POST-REBUTTAL ^^^^^^^^^^^^^^^^^^\n\nThis work proposes an approach to pretrain a transformer from scratch, akin to BERT (though with fewer layers), to predict masked intracranial recordings of people viewing movies. The pretrained model can generalize to heldout subjects and electrode positions. The claim /hope is that this pretrained model can be used, similarly to pretrained models in NLP, as a multipurpose tool that can be easily fine-tuned for \"downstream\" neuroscience tasks. The representations of intracranial recordings that are constructed by the pretrained BrainBERT lead to higher decoding accuracy in several \"tasks\" (e.g predicting the volume of the stimulus audio, predicting whether the stimulus is speech vs non-speech, etc), than other baseline representation methods (e.g. the raw recordings, a Fourier transform of the raw recordings, a 5 layer feed forward neural network).\n\n\n", "summary_of_the_review": "In theory, this work should represent an advance for the computational neuroscience community, but it is not clear how general-purpose the representations that this model produces will be to brain recordings from different types of stimuli and different types of experimental tasks. I would change my assessment of the work, if the authors are able to address the three major concerns that I described above.", "strength_and_weaknesses": "Strengths:\n- clearly written for the most part\n- potentially impactful (if major concerns are addressed, see below)\n- timely\n\nWeaknesses:\nMajor:\n1. the 10 subjects used to pretrain the model are very young (ages 4-19, the distribution of ages among the subjects is not clear) and it's not clear how well data from these subjects would generalize to a more general population. The work shows some results from holding out one of these subjects, but it's not clear whether these results are averaged over holding out each of the subjects, or they are for only one randomly selected subject. It would be helpful if the authors can clarify this and show whether there are any differences in results when generalizing to subjects of different ages. The authors should also discuss this possible limitation of their pretrained model.\n2. a truly general-purpose pretrained model would be able to generate brain activity affected by different stimuli and tasks, but there is no validation of this ability of the model in the current work. How do the authors envision this model be used -- with intracranial recordings collected under any stimulus and any task, or only for naturalistic movie stimuli with passive viewing? It's important that this is discussed in the work.\n3. the tested \"tasks\" for which the pretrained embeddings are useful are claimed to vary from low-level to high-level, but to me all of these tasks are indeed low-level because they should not require any processing beyond early auditory cortex. It would be more convincing if a higher-level task, such as word-level decoding, is enabled by the pretrained model\n\nMinor:\n- the name BrainBERT would be more fitting for a model that incorporates multiple types of neural recordings. A name that more accurately reflects the actual recordings used would make it easier to remember and distinguish from any follow-up work that incorporates other types of recordings.\n\nClarifications / Questions:\n1. some of the baseline models are said to have \"randomized weights\" -- does this actually mean that the weights were randomized or that they were randomly initialized? Randomly initialized weights would be more convincing as a baseline, since it's not clear what the weight distribution is and how effective randomizing them would be for reducing the effect of training\n2. how was the stopping point for training determined for all models? \n3. are the intrinsic dimensionality results only possible because of the pretrained model? Can these analyses be done directly with the raw data or the Fourier transform, and if so, would the results be different?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7xWOmNf3-Y", "reviewer": "Reviewer_x1NZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "EDIT: I have upgraded my overall score (6 to 8), as well as my correctness score (2 to 4).\n\nThis paper proposes the use of the bidirectional transformer BERT to model intracranial EEG (iEEG) recordings collected from human patients while watching movie stimuli. They transform the iEEG data into spectrograms and use an approach from speech audio modeling to train the model to reconstruct masked parts of the spectrogram (akin to a masked language modeling objective). They call this proposed model BrainBERT. They then show (1) that the representations learned by BrainBERT can be used for classification tasks, such as speech vs. non-speech, (2) that the method generalizes to a held-out subject with different electrode locations, and (3) a variety of controls and ablation studies providing insight into what parts of the proposal are useful. They assert that this model provides useful embeddings for intracranial recordings in a reusable, subject- and electrode-agnostic way. They also assert that the improved performance on neural decoding tasks is an important contribution to neuroscience, where \"many core results in neuroscience hinge on whether a linear decoder can perform a certain task...also critical to building the next generation of brain-machine interfaces.\"", "summary_of_the_review": "Overall, I found the proposal and results interesting, and it is certainly a novel application of language modeling approaches that could be quite useful to neuroscience. However, the paper makes many strong claims which are often not supported by the data presented in the paper. I would mainly like the authors to pull back on the strength of some of their claims, and have suggested additional analyses that would bolster some of their current ones. I also encourage the authors to either isolate their claims to BrainBERT's potential use in BCI, or add more content elaborating how a neuroscientist might use the model to better understand the brain. I am rating this a marginal accept because I believe there is some merit to this approach, but the paper needs to be improved. Discussion with the other reviewers and with the authors may change this rating in either direction.", "strength_and_weaknesses": "Strengths\n- Method is novel and generalizes to a held-out subject with different electrode positions\n- Authors test a suite of 4 decoding tasks, ranging from low-level (volume/pitch) to high-level feature learning\n- Several analyses are provided to understand the contributions of different pipeline choices -- e.g. STFT vs superlet-generated spectrogram, linear decoders with different inputs, a deep NN decoder, fine-tuning the whole network vs. the classification head only. This provides key information for future experimenters that may wish to tune the pipeline for specific tasks, or create a variant of the current proposal.\n- Authors promise to make the data and trained models available, which should enable their vision of reusability for neuroscientists\n- Method has potential to simplify the development of brain-computer interfaces (BCI)\n\nWeaknesses\n- The generalizability claim is quite bold, and ought to be backed up with more data. The authors only show one held-out subject, but then claim that the method generalizes across *subjects* (plural). I would appreciate a complete leave-one-out analysis, leaving out each subject in turn and quantifying the change in classification accuracy with & without subject N.\n    - This claim is also made when using the top 10 electrodes across all subjects, and the top 10 electrodes for the held-out subject. To support the generalizability claim across electrodes, it would be useful to see where these electrodes are, or at least see a metric of how far apart they are in cortex. Does the model still generalize if the electrodes are in quite different locations?\n- The \"few-shot learning\" claim is also too strong -- the model does require fewer training samples, and may be within the range of other few-shot learning claims in LM papers (which are typically some fraction of the original training set and represent many labeled samples, rather than computer vision papers which are just 2-5 samples per class). However, it definitely does not fall into the range the authors suggest is crucial for many clinical and BCI studies where, in the authors' words, \"subjects may participate in only a few dozen trials\"\n- The intrinsic dimensionality (ID) analysis is of questionable value as it is presented, and not well-supported by the data as far as I can tell.\n    - The authors suggest that there is a clustering of high ID electrodes in motor areas. This does not appear to be true to me -- no more than, say, left lateral PFC, which is where I see a cluster -- perhaps the authors can circle the area they are referring to in Fig. 5.\n    - The authors report two areas with the highest ID in the text, with no interpretation other than it is \"a novel view of functional regions in a task agnostic way.\" First of all, it's not clear what the novelty is -- typically neuroscientists report areas with lower and lower ID, and draw conclusions from this. What is the interpretation for a high ID area, and what is unexpected about these two regions having a high ID? Second, this is not actually task agnostic. All the data used to train their model is from movie viewing, and they may have seen extremely different ID numbers for other tasks (e.g. motion tasks that are not just passive perception). I grant that BrainBERT has the potential to be task-agnostic, but it isn't currently.\n- The authors suggest that \"embeddings of the neural data provide a new means by which to investigate the brain\", but they fall short of illustrating how a neuroscientist would do this. What is a conclusion one could draw from the type of data presented here? One might claim that pitch is processed in lower areas of the language processing stream, e.g. auditory cortex. This claim would need to be supported by showing differences in BrainBERT performance when using electrodes from this area vs. electrodes from another area. Even so, a neuroscientist might ask why one should use BrainBERT embeddings instead of training decoders directly on neural data. I encourage the authors to think about what additional utility / inferential power BrainBERT would provide that other methods cannot. Other authors have claimed that using NN models of the brain provide the ability to perform causal interventions (on the model, instead of the brain), which cannot be done with noninvasive neuroimaging methods like fMRI. However, causal interventions can actually be done with iEEG experiments -- so what else might BrainBERT provide?\n- Some clarity issues -- see below.\n\nFurther questions/suggestions\n- It seems the authors are only using the last layer output of the frozen BrainBERT to train the decoders. Is this correct? If so, I'm curious how different layer outputs would change the performance across tasks. It is possible that tasks requiring higher-level features do better when using late layer embeddings, while tasks requiring lower-level features (e.g. pitch/volume) do better when using early layer embeddings.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster