Title: Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals
Venue: NeurIPS 2024
Accepted: yes
Keywords: neuroscience

sEEG

speech decoding

self-supervision

neuro-AI
Abstract: Invasive brain-computer interfaces with Electrocorticography (ECoG) have shown promise for high-performance speech decoding in medical applications, but less damaging methods like intracranial stereo-electroencephalography (sEEG) remain underexplored. With rapid advances in representation learning, leveraging abundant recordings to enhance speech decoding is increasingly attractive. However, popular methods often pre-train temporal models based on brain-level tokens, overlooking that brain activities in different regions are highly desynchronized during tasks. Alternatively, they pre-train spatial-temporal models based on channel-level tokens but fail to evaluate them on challenging tasks like speech decoding, which requires intricate processing in specific language-related areas. To address this issue, we collected a well-annotated Chinese word-reading sEEG dataset targeting language-related brain networks from 12 subjects. Using this benchmark, we developed the Du-IN model, which extracts contextual embeddings based on region-level tokens through discrete codex-guided mask modeling. Our model achieves state-of-the-art performance on the 61-word classification task, surpassing all baselines. Model comparisons and ablation studies reveal that our design choices, including (\romannumeral1) temporal modeling based on region-level tokens by utilizing 1D depthwise convolution to fuse channels in the ventral sensorimotor cortex (vSMC) and superior temporal gyrus (STG) and (\romannumeral2) self-supervision through discrete codex-guided mask modeling, significantly contribute to this performance. Overall, our approach -- inspired by neuroscience findings and capitalizing on region-level representations from specific brain regions -- is suitable for invasive brain modeling and represents a promising neuro-inspired AI approach in brain-computer interfaces. Code and dataset are available at https://github.com/liulab-repository/Du-IN.

Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals

Abstract

Invasive brain-computer interfaces with Electrocorticography (ECoG) have shown promise for high-performance speech decoding in medical applications, but less damaging methods like intracranial stereo-electroencephalography (sEEG) remain underexplored. With rapid advances in representation learning, leveraging abundant recordings to enhance speech decoding is increasingly attractive. However, popular methods often pre-train temporal models based on brain-level tokens, overlooking that brain activities in different regions are highly desynchronized during tasks. Alternatively, they pre-train spatial-temporal models based on channel-level tokens but fail to evaluate them on challenging tasks like speech decoding, which requires intricate processing in specific language-related areas. To address this issue, we collected a well-annotated Chinese word-reading sEEG dataset targeting languagerelated brain networks from 12 subjects. Using this benchmark, we developed the Du-IN 1 model, which extracts contextual embeddings based on region-level tokens through discrete codex-guided mask modeling. Our model achieves stateof-the-art performance on the 61-word classification task, surpassing all baselines. Model comparisons and ablation studies reveal that our design choices, including (i) temporal modeling based on region-level tokens by utilizing 1D depthwise convolution to fuse channels in the ventral sensorimotor cortex (vSMC) and superior temporal gyrus (STG) and (ii) self-supervision through discrete codex-guided mask modeling, significantly contribute to this performance. Overall, our approach inspired by neuroscience findings and capitalizing on region-level representations from specific brain regions - is suitable for invasive brain modeling and represents a promising neuro-inspired AI approach in brain-computer interfaces. Code and dataset are available at https://github.com/liulab-repository/Du-IN.

90

橙汁

0.05

电脑

0.1

苹果

Du-IN

苹果

0.8

⾳乐

0.01

61-word classification

sEEG recordings

subj-01

subj-02

subj-03

subj-04

subj-05

subj-08

subj-09

subj-10

subj-11

subj-07

subj-06

subj-12

1 Introduction

Brain signals refer to the biometric information collected from the brain. Their patterns provide valuable insights toward understanding the physiological functions of the brain and the mechanism of related diseases, leading to various applications, including speech decoding [13, 19, 37], sleep cognition research [35, 55], neurological disorders detection [28, 54], and so on. Due to the high signal-noise ratio, invasive recording methods (e.g., stereoElectroEncephaloGraphy (sEEG), ElectroCorticoGraphy (ECoG)) usually reveal these underlying mechanisms better than non-invasive recording methods. Many previous works [29, 19] have shown that decoding speech from EEG signals is difficult, and the performance is limited. Compared with ECoG, sEEG imposes less trauma on patients and provides more stereotactic information from specific brain regions. Although some studies [37, 36] have recently shown promise for building high-performance speech decoders based on ECoG, there are few attempts made to explore the potential of sEEG-based speech decoding.

Modeling intracranial neural signals, especially sEEG, has gained significant attention, but several issues remain unresolved. Current research on modeling neural signals is divided into two lines based on the basic modeling units (e.g., channel-level tokens or group-level tokens 2 ). Some studies [54, 28] utilize shared embedding blocks to embed single channels into channel-level tokens, neglecting the specificity of brain computation [8]; then they adopt spatial-temporal integration to model spatial relationships among them, attempting to regain the precise state of the brain. However, these methods mainly focus on channel-level classification tasks, e.g., seizure detection, yet fail to validate them on more challenging group-level classification tasks, e.g., speech decoding. Other studies [19, 21] fuse all channels (across the brain) to build brain-level tokens, overlooking the brain's desynchronization nature [7]; then they adopt temporal modeling to capture the rapid process of brain dynamics. Besides, labeling data at scale in medical experiments is often impractical or costly, emphasizing the need to maximize label efficiency. Hence, developing an efficient pre-training framework that draws on prior neuroscience findings is highly appealing, as it can make the most of abundant unlabeled data.

The primary challenge in modeling intracranial neural signals lies in extracting meaningful tokens, requiring careful consideration of two key factors. (1) Temporal scale. Since intracranial neural signals have high temporal resolution and signal-noise ratio, these tokens must capture rapid dynamic changes in brain activity. (2) Spatial scale. Considering the brain's desynchronization nature, these tokens should correctly capture the information of each brain region for further integration and, if needed, decouple different parts of brain dynamics within each brain region. To better assess how well different models capture the intricate processing within each brain region, we can evaluate these methods on tasks mainly involving a few brain regions.

Since speech mainly involves specific brain regions related to vocal production, as demonstrated in Section 2.1, we utilize speech decoding tasks to evaluate which model can effectively extract information from specific brain regions. Since there are too few open source sEEG language datasets [1, 49], we collected a well-annotated Chinese word-reading sEEG dataset (vocal production), including 12 subjects, which makes up for the problem of missing sEEG recordings in language tasks. Inspired by neuroscientific findings, we systematically demonstrate the locality and specificity of brain computation and propose the Du-IN model to solve the abovementioned issues. Compared to other existing methods for modeling brain signals, Du-IN achieves SOTA performance on the 61-word classification task, demonstrating the effectiveness of our model in extracting meaningful tokens that can capture both the rapid changes and the precise state of specific brain regions. It marks a promising neuro-inspired AI approach [42, 41] in BCI.

To sum up, the main contributions of our work comprise:

Awell-annotated Chinese word-reading sEEG dataset, addressing the lack of sEEG language dataset. The dataset will be publicly available.

Demonstration of brain-specific computation - achieving the best decoding performance only requires about one electrode in specific brain regions (i.e., vSMC, STG).

Anovel framework for sEEG speech decoding - Du-IN, which learns region-level contextual embeddings through discrete codex-guided mask modeling.

SOTA performance on the sEEG speech decoding task - Du-IN achieves 62.70% top-1 accuracy on the 61-word classification task, surpassing all other baselines.

2 Related Works

2.1 Neural Basis of Language Function

Neuroscientific research [5, 17, 43] in the past has extensively explored brain regions supporting language functionality. In neuroscience, the investigation into language functionality related to speech has been categorized into two main streams: one dedicated to semantic processing and the other to vocal production. Previous studies [4, 43] have shown that brain regions associated with semantic processing primarily include left inferior frontal gyrus (IFG), left anterior temporal lobe (ATL), and bilateral middle temporal gyrus (MTG).

As for vocal production, which is also the focus of our work, it is predominantly governed by motor information related to language articulation, primarily involving ventral sensorimotor cortex (vSMC), bilateral superior temporal gyrus (STG), and bilateral dorsal laryngeal motor cortex (dLMC) [5, 17, 9]. Our analysis results based on our collected word-reading sEEG dataset also confirm this point, as illustrated in Figure 4.

2.2 Language Decoding in BCI

The keys to decoding natural language from brain signals are (1) high-quality recordings, and (2) well-designed models with good representations. Compared to non-invasive recordings (e.g., EEG), invasive recordings manifest advantages in providing detailed information about specific brain regions with a high signal-noise ratio. Since speech mainly involves some specific brain regions, obtaining detailed recordings of these brain regions will significantly enhance the decoding performance. Existing works [13, 37, 21] have shown the great potential of building a high-performance decoder based on invasive recordings.

The other key is well-designed models with good representations. Existing work for brain-to-language representations can be classified into two categories: self-supervision or alignment with representation models pre-trained on other modalities (e.g., text, audio). BrainBERT [49] learns general embeddings through self-supervised mask modeling. DeWave [19] introduces discrete codex encoding and aligns neural representations with text embeddings from BART [32], thus enhancing the extraction of semantic processing-related information from EEG recordings. Metzger et al. [36] align neural representations with acoustic embeddings to improve the extraction of vocal production-related information from ECoG recordings.

2.3 Self-supervised Learning in BCI

In recent years, self-supervised pre-training has made significant progress in natural language processing [16, 39, 6] and computer vision [3, 25, 11]. However, its potential in BCI is far from being explored. BrainBERT (for sEEG) [49] embeds single channels into channel-level tokens and utilizes mask modeling to learn general representations. Brant (for sEEG) [54, 53], PopT (for sEEG) [10] and some works (for EEG) [28, 23] further adopt spatial-temporal integration to model spatial relationships among them. Some works (for EEG) [31, 20, 50, 22] take the other way - fusing all channels (across the whole brain) to build brain-level tokens, and it uses self-supervised learning to learn contextual representations. Considering the difference among brain regions, MMM (for EEG) [52] further splits channels into different groups to build region-level tokens.

All existing pre-training methods for sEEG primarily pre-train spatial-temporal models based on channel-level tokens yet only evaluate them on channel-level classification tasks, e.g., seizure detection. However, unlike EEG pre-training methods, their effectiveness over more challenging group-level classification tasks, e.g., speech decoding. Besides, there is no standard channel configuration for sEEG recordings, unlike EEG recordings, which makes modeling spatial relationships in sEEG more challenging.

3 Method

The overall architecture of Du-IN is illustrated in Figure 2, where the raw sEEG signals are fused across channels to build region-level tokens and further encoded for downstream tasks.

3.1 Task Definition

Due to the lack of open-source sEEG datasets related to language tasks, we follow the experimental design outlined by Moses et al. [37] to collect a well-annotated Chinese word-reading sEEG dataset (vocal production). During the experiment, each subject speaks aloud 61 pre-determined Chinese words 50 times; see Appendix A for more details. We formulate the multi-channel sEEG signals as X ∈ R C × T , where C is the number of sEEG channels and T is the total timestamps. The associated word label is denoted as y ∈ Y , where Y represents the set of 61 pre-determined words. In summary, this dataset comprises paired sEEG-word data ( ⟨X , y ⟩ ), and the model aims to decode the corresponding word y from a sequence of raw sEEG signals X .

3.2 Model Architecture

Linear

Conv

X3

Spatial Encoder

Temporal

Embedding

K

Add & Norm

FeedForward

Add & Norm

Attention

LN

V

Q

LN

Transformer Encoder

Output

Embedding

time

channel

patch

Input sEEG Signals

TI'4

TI'5

TI'6

F'9

e i

e

t

i

e

p

i

We introduce the Du-IN Encoder, a general architecture for sEEG speech decoding tasks that can deal with any input sEEG signals with arbitrary time length, as shown in Figure 2. The key operation for archiving this is segmenting the sEEG signals into patches, inspired by patch embeddings in images [18]. For each sample X , we use a W -length window without overlap to segment it into patches, obtaining X = { x i ∈ R C × W | i = 1 , ..., N } , where N = ⌊ T W ⌋ is the number of patches.

Spatial Encoder. As each sEEG patch has multiple channels, it is vital to fuse different channels to extract meaningful features before patch-wise interaction by self-attention. We employ a spatial encoder, which consists of a linear projection and several convolution blocks, to encode each sEEG patch into a patch embedding. The linear projection transforms the raw sEEG signals into the hidden neural space, and its weights are utilized for subsequent analysis. The convolution block is composed of a 1D depthwise convolution layer and a batch normalization layer [27]. We denote the output patch embeddings from the spatial encoder as

where d is the dimension of the embeddings.

Temporal Embedding. In order to enable the model to be aware of the temporal information of patch embeddings, we utilize the parameter-free position embeddings introduced in [48], i.e., E t = { e t 1 , ..., e t t max } . Note that t max is the hyperparameter determining the maximum number of time patches and t max ≥ N . Given one arbitrary patch embedding e i in Equation 1 from the spatial encoder, we add the corresponding temporal embedding to it:

which forms the input embeddings E init for the Transformer Encoder.

Transformer Encoder. Finally, the sequence of embeddings will be directly fed into the Transformer encoder [48] to get the final encoded E = { e i ∈ R d | i = 1 , ..., N } . To make the training of the Transformer more stable and efficient, we incorporate some modifications [14] inspired by LaBraM [28]. We add layer normalization to the queries and keys before the dot-product attention mechanism, which avoids over-large values in attention logits:

where d head is the dimension of attention head and LN denotes layer normalization [2]. For downstream classification tasks, we flatten the output embeddings followed by a classification head.

3.3 Du-IN VQ-VAE Training

Prior to pre-training Du-IN through mask modeling, we need to tokenize the sEEG patches into discrete tokens. We introduce vector-quantized neural signal regression, which is trained by reconstructing the original sEEG signals, as shown in Figure 3. The key components are the Du-IN Encoder, which encodes the raw sEEG samples into embeddings, and the Du-IN Regressor, which reconstructs the original sEEG signals. The idea is basically inspired by VQ-VAE [47], which encodes images into discrete latent embeddings.

c

Input sEEG Signals

TI'4

TI'5

TI'6

F'9

Du-IN Encoder

Neural Codex

Lookup

Replace

Du-IN Regressor

Reconstructed

sEEG Signals

TI'4

TI'5

TI'6

F'9

Du-IN VQ-VAE Training

Mask

TI'4

TI'5

TI'6

F'9

Symmetric Mask

TI'4

TI'5

TI'6

F'9

Du-IN Encoder

729

69

42

660

6

534

Token Prediction Head

69

660

6

729

42

534

M

M

M

M

M

M

(a)

(b)

l2-norm

l2-norm

e i

j

z i

z q

(e i

)

Wv

MW

Du-IN MAE Training

Du-IN Encoder. We define a neural codex C = { c j | j = 1 , ..., N codex } ∈ R N codex × d codex , where N codex is the number of the discrete neural embeddings and d codex is the dimension of each embedding. Given a sEEG sample X , the Du-IN Encoder, illustrated in Figure 2, first encodes it to embeddings E = { e i ∈ R d | i = 1 , ..., N } . After that, we utilize a linear projection z c to get the mapped embeddings z c ( E ) = { z c ( e i ) ∈ R d codex | i = 1 , ..., N } in the codex space. Then, the codex looks up the nearest neighbor of each embedding z c ( e i ) in the neural codex C . This procedure can be formulated as

where ℓ 2 represents ℓ 2 normalization and z q ( e i ) is the quantized vector after the quantizer. This is equivalent to finding the closest neural embedding by cosine similarity and such ℓ 2 normalization improves the codex utilization [38].

Du-IN Regressor. The Du-IN Regressor consists of a Transformer decoder and a stack of transposed convolution layers. Given a sequence of the vector-quantized embeddings Z = { z i | i = 1 , ..., N } , the Du-IN Regressor convert these discrete embeddings back into raw sEEG signals ˜ X = { ˜ x i | i = 1 , ..., N } . The mean squared error (MSE) loss is utilized to guide the regression. The total loss for training the Du-IN VQ-VAE is defined as:

where sg represents the stop-gradient operation, which is an identity at the forward pass and has zero gradients. To stabilize the codex update, we use the exponential moving average strategy [47].

3.4 Pre-training Du-IN

Masked sEEG Modeling. To enforce Du-IN learning contextual representations, we propose masked sEEG modeling. The whole procedure is presented in Figure 3. As illustrated in Figure 2, given a sEEG sample X , the spatial encoder first transforms it to patch embeddings E p = { e p i | i = 1 , ..., N } . Given these patch embeddings E p , around 50% of patch embeddings are patch-wisely chosen and masked. The masked position is termed as M . Then, a shared learnable embedding e [ M ] ∈ R d is used to replace the original patch embeddings:

where δ ( · ) is the indicator function and m i = δ ( i ∈ M ) . After that, the masked embeddings E m will be added by temporal embeddings, and then fed into the Transformer encoder. The output embeddings E will be used to predict the indices of the corresponding codes from the codex in the Du-IN VQ-VAE through a linear classifier:

The training loss of mask modeling is defined as:

Symmetric Masking. Inspired by LaBraM [28], we further introduce a symmetric masking strategy to improve training efficiency. We calculate the inverse of the generated mask M , obtaining ˆ M . Similarly, we use the new mask ˆ M to perform the mask modeling, obtaining the mask modeling loss L sym M . The total loss for pre-training the Du-IN model (i.e., Du-IN MAE model) is defined as:

4 Experiments

4.1 Dataset

Due to the lack of open-source sEEG datasets related to language tasks, we follow the experimental design outlined by Moses et al. [37] to collect a well-annotated Chinese word-reading sEEG dataset (vocal production), including 12 subjects. The subjects undergo a surgical procedure to implant 7 to 13 invasive sEEG electrodes, each with 72 to 158 channels, in their brain. For each subject, the dataset contains 15 hours of 2000Hz recordings, 3 hours of which are task recordings.

Pre-training dataset. For each subject, the pre-training dataset contains all sEEG recordings (with about 54 million timestamps) of that subject. To stabilize computing resource usage, the time length of sEEG sample X is set to 4 seconds.

Downstream dataset. For each subject, 3 hours of the sEEG recordings are task recordings. The sEEG signals are segmented into about 3000 3-second samples, each of which is paired with the corresponding word label (from 61 pre-determined words).

4.2 Implementation Details

Preprocess. We first filter the sEEG signals between 0.5Hz and 200Hz to remove low-frequency noise. Then, a notch filter of 50Hz is applied to avoid power-line interference. After that, all sEEG signals are resampled to 1000Hz and bi-polar re-referenced [33]. Finally, we perform z-score normalization on each channel to guarantee normalized data scales across all channels.

Model Configurations. The length of the sEEG patch is 100ms, resulting in 40 patches per sample in the pre-training dataset and 30 patches per sample in the downstream dataset. The "Spatial Encoder" contains one linear projection and three 1D convolution layers, transforming the original sEEG patches into patch embeddings with d = 160 . The following "Transformer Encoder" contains an 8-layer Transformer encoder with model dimension d = 160 , inner dimension (FFN) d ff = 320 , and 8 attention heads. See Appendix C for more details.

Pre-training. During the pre-training, we use either all sEEG recordings (15 hours) or the sEEG recordings without task recordings (12 hours) to train the Du-IN VQ-VAE and Du-IN MAE models. To enhance the robustness of the learned codex and representations, we further use data augmentation described in Appendix D. For each subject, the model is pre-trained on a Linux system with 2 CPUs (Intel Xeon Gold 6230 40-Core Processor) and 1 GPU (NVIDIA Tesla V100 32GB) for ∼ 1.2 days.

Fine-tuning. During the downstream evaluation, we split the task recordings into training, validation, and testing splits with a size roughly proportional to 80%, 10%, and 10%. All experiments are conducted on the same machine with the same set of random seeds. The train/validation/test splits are the same across different models. We also use data augmentation, as described in Appendix D, to make the most of the gathered dataset. We employ cross-entropy loss (multi-class classification) as the training loss. Our experiments are conducted on one V100 GPU by Python 3.11.7 and PyTorch 2.1.2 + CUDA 12.3. The best models are trained based on the training set, selected from the validation set according to accuracy, and finally evaluated on the test set. For model comparison, we report the average and standard error values (of all subjects) on six different random seeds to obtain comparable results. For the results of the subject-wise evaluation, we report the average and standard deviation values (of each subject) in Appendix K.

4.3 Channel Contribution and Selection

As demonstrated in Section 2.1, previous neuroscience studies reveal that vocal production predominantly engages specific brain regions. Given the sparse distribution of implanted sEEG electrodes (each containing 8-16 channels), it's vital to exclude redundant electrodes unrelated to vocal production, thus improving decoding performance. We retain electrodes implanted in relevant brain regions and evaluate the performance based on the remaining electrodes. Table 1 demonstrates that excluding approximately 85% electrodes even leads to a dramatic increase in decoding performance.

To further understand the detailed contribution of each channel, we analyze the weights of linear projection in the spatial encoder. In detail, we calculate the contribution scores of channels per subject and organize them accordingly, as described in Appendix H. Figure 4 demonstrates that (1) the brain regions effective for speech decoding align with findings from previous neuroscience research, and (2) our model achieves optimal decoding performance with approximately 10 channels, 80% of which originate from the same electrode. To streamline, we utilize these top 10 channels (selected according to train-set) for both pre-training and downstream evaluation.

4.4 Comparasion with Other Models

Table 2 presents the results of our Du-IN model and the advanced baselines that are designed for either brain signals or general time series. See Appendix B and Appendix C.3 for detailed descriptions of models. The results demonstrate that our Du-IN model outperforms all baselines. It's worth noting that the models (i.e., the foundation models designed for brain signals) that adopt spatial-temporal integration to model spatial relationships among channel-level tokens perform worse than the models that adopt temporal modeling based on region-level tokens, challenging the generalizability of current strategies to model spatial relationships among channels with Transformer.

right

left

(a)

0.75

1

0.5

0.25

0

(b)

1 PT: Whether the model is pre-trained before evaluation.

As BrainBERT [49] doesn't consider the spatial relationships among channels, we mainly focus on understanding why Brant [54], LaBraM [28] and LaBraM-Popt [28, 10] fail to effectively capture the discriminative features on the speech decoding task. These models typically build channel-level tokens by segmenting non-overlapping patches with large receptive fields (e.g., 1 second) from single channels. However, this approach makes it challenging to capture the rapid process of brain dynamics. Moreover, while these models further utilize Transformer to capture the spatial relationships among these tokens, they do not encourage region-level embeddings, either through their architecture [52] or their pre-training objective [10]. Therefore, the effectiveness of building brain foundation models based on these spatial-temporal backbones is still under exploration, especially for cognitive tasks (e.g., speech decoding), which are of great value in the field of neuroscience.

Besides, unlike LaBraM [28], Brant doesn't introduce spatial embeddings to identify the spatial location of each channel. Since the electrodes are sparsely distributed in the brain and the raw sEEG signals on the same electrode are highly correlated, it's fairly easy to identify their spatial relationships through their values. As demonstrated in iTransformer [34], this modeling approach is well suited for detecting time-delay events, e.g., seizure detection. For speech decoding tasks,

sEEG often requires bi-polar re-reference (or Laplacian re-reference) to remove the high correlations among channels, thus avoiding model overfitting [49]. Once the correlations among channels have been removed, Brant will lose the ability to model spatial relationships among channels.

For other baselines that use temporal modeling based on region-level tokens, we provide a detailed explanation of their performance differences as follows. TS-TCC [20] tokenizes raw sEEG signals into region-level tokens with a stack of 1D depthwise convolution blocks, but it lacks a temporal Transformer for further integration over time. CNN-BiGRU [37] introduces a stack of GRU layers on top of these tokens to perform temporal integration. EEG-Conformer [44] introduces a temporal Transformer to better integrate global temporal information, which makes it outperform CNNBiGRU. However, EEG-Conformer tokenizes raw sEEG signals with the temporal-spatial convolution, applying the same convolutional kernel across different channels, which overlooks the specificity of brain computation [8]. This also raises a challenge for the effectiveness of current sEEG foundation models, which rely on shared convolution blocks across individual channels. Neuro-BERT [50] further introduces mask modeling to learn contextual embeddings, which makes it outperform EEGConformer. DeWave [19] utilizes the Conformer model [24] for tokenization, which involves more parameters but is less effective than 1D depthwise convolution.

4.5 Ablation Study

Self-Supervision Initialization. As illustrated in Figure 3, the Du-IN model entails a two-stage pre-training process, wherein both the Du-IN VQ-VAE model and the Du-IN MAE model are trained. Previous studies utilize different strategies [19, 12, 28] to leverage these pre-trained models to enhance the performance of downstream tasks. Here, we evaluate these different strategies for comparison; see Appendix C.3 for detailed definitions. Table 2 shows that initializing weights from the Du-IN MAE model captures contextual embeddings effectively, resulting in the highest decoding performance.

Pre-training with/without Downstream Datasets. During the pre-training stage, we hope that the Du-IN VQ-VAE model can extract general tokens of that brain region, thus guiding the Du-IN MAE model to learn general representations that are not specific to any particular task. Although no label data is used during the pre-training stage, to eliminate the influence of the pre-training data on downstream tasks, we compare the results with or without incorporating the downstream task dataset into the pre-training stage. Table 3 shows a slight performance drop when excluding downstream datasets. However, the decoding performance is still higher than the baseline performance without pre-training, which means that the degradation is mainly due to the decrease of the pre-training dataset. We hope that, with more pure recordings, our model can achieve better decoding performance.

Discrete Codex. During the Du-IN VQ-VAE training stage, the Du-IN VQ-VAE model encodes sEEG patches into discrete codes and then reconstructs the original signal from these codes. We evaluate performance against varying codex sizes (512 to 8192) to ascertain if codex size affects the quality of the learned codex. As illustrated in Figure 5, while extremely small codex size lacks representation diversity, extremely large codex size often leads to codex collapse. We suspect that our existing training data might not be adequate for larger codex sizes. Furthermore, our experiments suggest that the model performs optimally when the codex dimension, denoted as d codex = 64 , is slightly less than the model dimension, d = 160 , yielding a more effective regularization effect.

Perception Time Window. We also conduct the ablation study on the model structure for the spatial encoder described in Section 3.2. As the spatial encoder transforms the sEEG signals within a given patch to a patch embedding, it compresses the sEEG signals for perception. As described in Section 4.2, the model utilizes a receptive field of 100ms. We conduct an ablation study of different receptive fields and report it in Figure 5. The model performance notably drops with a receptive field smaller than 60ms and gradually declines as the receptive field exceeds 160ms. The model reaches a

small peak around 100ms to 140ms. We think this phenomenon is rational since sEEG is known for its ability to capture the rapid dynamics of specific brain regions precisely.

(b)

(a)

(c)

5 Limitations

Despite Du-IN's enhancements in speech decoding via discrete codex-guided mask modeling, it is still restricted to close-set speech decoding tasks (i.e., the word set only includes 61 pre-determined words). However, a parallel to our work, Feng et al. [21], which follows previous works [26, 45], build an acoustic-inspired model that can decode arbitrary Chinese words by predicting syllable components (initials, finals, tones). Although their method requires a large amount of labeled data, their experimental design mirrors ours closely. The difference lies in the requirement for the subject to repeat syllable components, instead of entire words. Therefore, with slight modifications, our model can support open-set speech decoding tasks.

Additionally, the experiments in this paper are restricted to the vocal production part of language decoding, i.e., speech decoding. A more interesting but difficult task is to decode language from the semantic level, in which large language models have been wildly used to improve the model performance [46, 19]. However, due to the locality of sEEG recordings, it is still under exploration whether sEEG recordings can fully capture semantic-related information across brain regions.

6 Conclusion

This paper proposes Du-IN, a framework for speech decoding, which learns contextual embeddings through discrete codex-guided mask modeling on specific brain regions. To evaluate our model, we collect a well-annotated Chinese word-reading sEEG dataset to address the lack of sEEG language dataset. Inspired by neuroscientific findings, we analyze the effective brain regions for speech decoding and achieve the best decoding performance with about one electrode in specific brain regions, which dovetails with the past neuroscientific research on language. Comprehensive experiments demonstrate that our model outperforms both supervised and sEEG-based self-supervised baselines, effectively capturing the intricate processing within specific brain regions. It marks a promising neuro-inspired AI approach in BCI. In the end, we hope our work can have implications for future developments in sEEG-based self-supervised models with more consideration over how to build the basic representation units so that the model can maximally benefit from the pre-training stage.

7 Broader Impacts

Our method advances the feasibility of invasive BCI technology by being the first to demonstrate speech decoding using a single sEEG electrode, which holds significant potential for clinical applications. For patients who have lost their ability to communicate or perform daily tasks due to neurological conditions like locked-in syndrome or amyotrophic lateral sclerosis (ALS), our approach offers a less invasive alternative to technologies like ECoG or microelectrode arrays, thereby reducing the risk of brain damage.

Ethics Statement

Experiments that contribute to this work were approved by IRB. All subjects consent to participate. All electrode locations are exclusively dictated by clinical considerations.

Our informed consent signing process is as follows:

If the experimental participants are adults and have full civil capacity, we will ask them to sign a written informed consent after the participants have fully informed consent;

If the experimental participants are minors or do not have full civil capacity, we will ask the participant's legal guardian to sign a written informed consent after the participants and their legal guardians have fully informed consent.

Our informed consent form includes the following points:

Contact information of research institutions and researchers;

Research direction and purpose;

Risks involved in the research;

Personal information, data and usage methods to be used in the research;

Privacy protection statement (all personal identification information (PII) will not be disclosed);

Data storage statement (retained after deleting all personal identification information (PII));

Voluntary statement of participants;

Statement that participants can withdraw unconditionally at any time.

Our data storage and protection procedures include the following processes:

Our data collection, transfer, and analysis tasks are only completed by researchers who have signed relevant confidentiality agreements;

The collected raw data will be copied twice as soon as possible, one copy to a storage computer that is not connected to the Internet and encrypted, and the other copy to a mobile hard disk and encrypted and stored offline;

The use of the data is only authorized to the research leader and the main researchers (less than 5 people), among which the main researchers can only access data that does not contain personal identification information (PII);

After the study is completed, all personal identification information (PII) on both nodes (storage computer, mobile hard disk) will be deleted immediately.

To prevent unauthorized access or possible data leakage, we use double encryption on the storage computer, that is, a static password and a dynamic password (received by mobile phone or email); physical isolation is used on the mobile hard disk, that is, it is locked in a filing cabinet, and the key is only kept by the research leader and the main researchers.

References

Miguel Angrick, Maarten C Ottenhoff, Lorenz Diener, Darius Ivucic, Gabriel Ivucic, Sophocles Goulis, Jeremy Saal, Albert J Colon, Louis Wagner, Dean J Krusienski, et al. Real-time

synthesis of imagined speech processes from minimally invasive recordings of neural activity. Communications biology , 4(1):1055, 2021.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Jeffrey R Binder, Julie A Frost, Thomas A Hammeke, Robert W Cox, Stephen M Rao, and Thomas Prieto. Human brain language areas identified by functional magnetic resonance imaging. Journal of Neuroscience , 17(1):353-362, 1997.

Kristofer E Bouchard, Nima Mesgarani, Keith Johnson, and Edward F Chang. Functional organization of human sensorimotor cortex for speech articulation. Nature , 495(7441):327-332, 2013.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Gyorgy Buzsaki. Rhythms of the Brain . Oxford university press, 2006.

Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature human behaviour , 7(3):430-441, 2023.

Josh Chartier, Gopala K Anumanchipalli, Keith Johnson, and Edward F Chang. Encoding of articulatory kinematic trajectories in human speech sensorimotor cortex. Neuron , 98(5):10421054, 2018.

Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning populationlevel representations of intracranial activity. ArXiv , 2024.

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning , pages 1691-1703. PMLR, 2020.

Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang, Yifan Wang, Dongsheng Li, and Lili Qiu. Eegformer: Towards transferable and interpretable large-scale eeg foundation model. arXiv preprint arXiv:2401.10278 , 2024.

Cheol Jun Cho, Edward Chang, and Gopala Anumanchipalli. Neural latent aligner: cross-trial alignment for learning representations of complex, naturalistic neural data. In International Conference on Machine Learning , pages 5661-5676. PMLR, 2023.

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning , pages 7480-7512. PMLR, 2023.

Rahul S Desikan, Florent Ségonne, Bruce Fischl, Brian T Quinn, Bradford C Dickerson, Deborah Blacker, Randy L Buckner, Anders M Dale, R Paul Maguire, Bradley T Hyman, et al. An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage , 31(3):968-980, 2006.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Benjamin K Dichter, Jonathan D Breshears, Matthew K Leonard, and Edward F Chang. The control of vocal pitch in human laryngeal motor cortex. Cell , 174(1):21-31, 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin. Dewave: Discrete eeg waves encoding for brain dynamics to text translation. arXiv preprint arXiv:2309.14030 , 2023.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112 , 2021.

Chen Feng, Lu Cao, Di Wu, En Zhang, Ting Wang, Xiaowei Jiang, Heng Ding, Chenhao Zhou, Jinbo Chen, Hui Wu, et al. A high-performance brain-sentence communication designed for logosyllabic language. bioRxiv , pages 2023-11, 2023.

Navid Mohammadi Foumani, Geoffrey Mackellar, Soheila Ghane, Saad Irtza, Nam Nguyen, and Mahsa Salehi. Eeg2rep: enhancing self-supervised eeg representation through informative masked inputs. arXiv preprint arXiv:2402.17772 , 2024.

Pierre Guetschel, Thomas Moreau, and Michael Tangermann. S-jepa: towards seamless crossdataset transfer through dynamic spatial attention. arXiv preprint arXiv:2403.11772 , 2024.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100 , 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Christian Herff, Dominic Heger, Adriana De Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk, and Tanja Schultz. Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers in neuroscience , 9:217, 2015.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Weibang Jiang, Liming Zhao, and Bao liang Lu. Large brain model for learning generic representations with tremendous EEG data in BCI. In The Twelfth International Conference on Learning Representations , 2024.

Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Are eeg-to-text models working? arXiv preprint arXiv:2405.06459 , 2024.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Demetres Kostas, Stephane Aroca-Ouellette, and Frank Rudzicz. Bendr: using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience , 15:653659, 2021.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Guangye Li, Shize Jiang, Sivylla E Paraskevopoulou, Meng Wang, Yang Xu, Zehan Wu, Liang Chen, Dingguo Zhang, and Gerwin Schalk. Optimal referencing for stereoelectroencephalographic (seeg) recordings. NeuroImage , 183:327-335, 2018.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

Yuchen Liu and Ziyu Jia. Bstt: A bayesian spatial-temporal transformer for sleep staging. In The Eleventh International Conference on Learning Representations , 2022.

Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian E Dougherty, Jessie R Liu, Peter Wu, Michael A Berger, et al. A highperformance neuroprosthesis for speech decoding and avatar control. Nature , 620(7976):10371046, 2023.

David A Moses, Sean L Metzger, Jessie R Liu, Gopala K Anumanchipalli, Joseph G Makin, Pengfei F Sun, Josh Chartier, Maximilian E Dougherty, Patricia M Liu, Gary M Abrams, et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine , 385(3):217-227, 2021.

Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-quantized visual tokenizers. 2022.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019.

Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience , 22(11):1761-1770, 2019.

Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Neuroscience , 22(1):55-67, 2021.

Jingwei Sheng, Li Zheng, Bingjiang Lyu, Zhehang Cen, Lang Qin, Li Hai Tan, Ming-Xiong Huang, Nai Ding, and Jia-Hong Gao. The cortical maps of hierarchical linguistic structures during speech perception. Cerebral cortex , 29(8):3232-3240, 2019.

Yonghao Song, Qingqing Zheng, Bingchuan Liu, and Xiaorong Gao. Eeg conformer: Convolutional transformer for eeg decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 31:710-719, 2022.

Pedram Z Soroush, Christian Herff, Stephanie K Ries, Jerry J Shih, Tanja Schultz, and Dean J Krusienski. The nested hierarchy of overt, mouthed, and imagined speech activity evident in intracranial recordings. NeuroImage , 269:119913, 2023.

Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience , 26(5):858-866, 2023.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Brainbert: Self-supervised representation learning for intracranial recordings. arXiv preprint arXiv:2302.14367 , 2023.

Di Wu, Siyuan Li, Jie Yang, and Mohamad Sawan. Neuro-bert: Rethinking masked autoencoding for self-supervised neurological pretraining. IEEE Journal of Biomedical and Health Informatics , 2024.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations , 2022.

Ke Yi, Yansen Wang, Kan Ren, and Dongsheng Li. Learning topology-agnostic eeg representations with geometry-aware modeling. Advances in Neural Information Processing Systems , 36, 2024.

Zhizhang Yuan, Daoze Zhang, Junru Chen, Geifei Gu, and Yang Yang. Brant-2: Foundation model for brain signals. arXiv preprint arXiv:2402.10251 , 2024.

Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foundation model for intracranial neural signal. Advances in Neural Information Processing Systems , 36, 2024.

Hui Zheng, Zhongtao Chen, Haiteng Wang, Jianyang Zhou, Lin Zheng, and Yunzhe Liu. Universal sleep decoder: Aligning awake and sleep neural representation across subjects. arXiv preprint arXiv:2309.16457 , 2023.

A Experiment Design

Time

Next Trial

Prepare

Read

Rest

0.5s

2s

0.5s

+

苹果

苹果

Due to the lack of open-source sEEG datasets related to language tasks, we follow the experimental design outlined by Moses et al. [37] to collect a well-annotated Chinese word-reading sEEG dataset, including 12 subjects (9 male, 3 female; aged 15-53, µ 27.8, σ 10.4) with pharmacologically intractable epilepsy.

In the word-reading task, the subject speaks aloud individual words from a 61-word set while we simultaneously record his brain activity (measured by sEEG) and voice. The word set is chosen based on the following criteria:

The versatility of the words in generating a range of sentences.

The simplicity of using the words to express fundamental caregiving requirements.

The diversity of word pronunciations to cover as many Chinese pronunciation combinations as possible.

A list of the words contained in this 61-word set is provided in Table 4.

All data are collected as a series of "blocks" (25 blocks in total), with each block lasting about 10 minutes and consisting of multiple trials. During each block of this task, all words (from the 61-word set) are presented individually twice, leading to a total of 122 trials.

Each trial in a block of this task starts with one word shown on the screen in white text. After 0.5 seconds, the text will turn green and remain on the screen for 2 seconds. This color transition from white to green represents the go cue for each trial, and the subject is instructed to speak the word aloud as soon as the text turns green. Afterward, the text will be replaced with a blank screen with a centered cross. After 0.5 seconds, the task continues to the next trial. The word presentation order is randomized within each task block.

Besides, we also collected non-task recordings of subjects in their daily life. Apart from sleep periods, there are roughly 12 hours of non-task recordings during wakefulness. In summary, for each subject, we collect about 15 hours of sEEG recordings, of which 3 hours are task recordings.

B Details of Baselines

In experiments, we compare our model to the existing supervised or self-supervised methods on brain signals. The details of these baseline models are given here:

TS-TCC [20]: A self-supervised model that consists only of a CNN module to capture local features. This model learns robust temporal and discriminative representations from time series by designing a tough cross-view prediction task and a contextual contrasting module. Since sEEG is a unique type of time series, this model is suitable to serve as a baseline for comparison.

CNN-BiGRU [37]: A supervised model that consists of both CNN module and Bi-GRU module, to capture contextual features from EEG signals. This model is mainly designed for ECoG-based vocal production tasks, similar to ours. Since ECoG and sEEG are both intracranial neural signals of the brain, this model is suitable to serve as a baseline for comparison.

EEG-Conformer [44]: A supervised model that consists of both CNN module and Transformer module, to encapsulate local and global features in a unified EEG classification framework. EEG-Conformer is mainly designed for EEG-based motor imagination tasks. Since the data modes of EEG and sEEG are similar, and the signals primarily pertain to vocal production, this model is suitable to serve as a baseline for comparison.

Neuro-BERT [50]: A self-supervised model that consists of both CNN module and Transformer module, to encapsulate local and global features. This model learns robust contextual representations from EEG by introducing mask modeling. Since the data modes of EEG and sEEG are similar, this model is suitable to serve as a baseline for comparison.

DeWave [19]: A supervised model that consists of both Conformer module [24] and Transformer module, to encapsulate local and global features for language decoding. We adopt its encoder, which consists of a 6-layer Conformer and a 6-layer Transformer. Then, we add a classification head, which is also used in our model, for downstream word classification. Since DeWave is also designed for language decoding, this model is suitable to serve as a baseline for comparison.

BrainBERT [49]: A self-supervised model for sEEG recordings that bridges modern representation learning approaches to neuroscience. BrainBERT builds universal representation based on the superlet spectrograms of one single sEEG channel without modeling the spatial relationships among channels. Since the downstream tasks for BrainBERT are also related to language decoding (e.g., sentence-onset detection, speech vs. non-speech detection, etc.), this model is suitable to serve as a baseline for comparison.

Brant [54]: A self-supervised model for sEEG recordings that can capture both long-term temporal dependency and spatial correlation from neural signals. Brant is mainly designed for medicine, serving as a sEEG foundation model. Although Brant mainly evaluates its performance on the low-level modeling tasks [51] (e.g., neural signal forecasting, imputation, etc.), Brant achieves SOTA performance on some high-level modeling tasks (e.g., seizure detection). As a foundation model in sEEG pre-training field, this model is suitable to serve as a baseline for comparison.

LaBraM [28]: A self-supervised model for EEG recordings that learns generic representations with tremendous EEG data. LaBraM serves as an EEG foundation model, achieving SOTA performance on various downstream EEG tasks. Since the spatial embeddings are pre-defined according to the EEG caps, LaBraM can only be trained within one subject under the sEEG setting. Since the data modes of EEG and sEEG are similar, this model is suitable to serve as a baseline for comparison.

LaBraM+PopT [28, 10]: A self-supervised model based on LaBraM, simply replacing the learnable spatial embeddings with hard-coded spatial embeddings from PopT [10] to enable multi-subject pre-training under the sEEG setting.

The detailed implementations of these baseline models are given here:

For the TS-TCC method [20], the hyper-parameters are optimized for better performance, as they also have different hyper-parameter settings for different datasets in their original

implementation. The data samples are resampled to 400Hz. The sizes of convolution kernels are changed to {25, 8, 8} (other attempts include {8, 8, 8}, {15, 8, 8}, {20, 8, 8}, and {30, 8, 8}); the sizes of pooling kernels are changed to {10, 2, 2} (other attempts include {2, 2, 2}, {5, 2, 2}, and {20, 2, 2}); the numbers of pooling strides are changed to {10, 2, 2} (other attempts include {2, 2, 2}, {5, 2, 2}, and {20, 2, 2}). All other hyper-parameters are the same as the original implementation.

For the CNN-BiGRU method [37], the hyper-parameters are the same as the original implementation. The data samples are resampled to the specified sampling rate.

For the EEG-Conformer method [44], the hyper-parameters are the same as the original implementation. The data samples are resampled to the specified sampling rate.

For the Neuro-BERT method [50], the hyper-parameters are optimized for better performance, as they also have different hyper-parameter settings for different datasets in their original implementation. The data samples are sampled to 400Hz. The sizes of convolution kernels are changed to {40,} (other attempts include {20,} and {80,}); the numbers of convolution strides are changed to {40,} (other attempts include {20,} and {80,}).

For the DeWave method [19], the hyper-parameters are the same as the original implementation. The data samples are resampled to the specified sampling rate.

For the BrainBERT method [49], the hyper-parameters are optimized for better performance. We change the "nperseg" and "noverlap" arguments of "scipy.signal.stft" function from {400, 350} to {1600, 1400} (other attempts include {200, 175}, {800, 700} and {3200, 2800}).

For the Brant method [54], the hyper-parameters are optimized based on the Brant-Tiny model for better performance. We change the length of the patch segment from 6 seconds to 1 second. Besides, we change the linear embedding layer to the convolution embedding layer, which is also used in LaBraM [28]. The numbers of convolution filters are {96, 96, 96} (other attempts include {192, 192, 192}); the sizes of convolution kernels are {9, 9, 3} (other attempts include {19, 9, 3} and {9, 9, 3}); the numbers of convolution strides are {5, 5, 1} (other attempts include {10, 5, 1}) and {5, 5, 2}).

For the LaBraM method [28], the hyper-parameters are the same as the original implementation of the LaBraM-Base model. The data samples are resampled to the specified sampling rate.

For the LaBraM-PopT method [28, 10], the hyper-parameters are the same as the original implementation of the LaBraM-Base model. The data samples are resampled to the specified sampling rate.

When evaluating the decoding performance of these baseline models, we follow the same experiment setup of the Du-IN CLS model:

For one subject, we split the downstream dataset into training, validation, and testing splits with a size roughly proportional to 80%, 10%, and 10%.

The data samples are 3 seconds with the specified sampling rate corresponding to each model.

The samples in the train-set are augmented following the pipeline defined in Appendix D.

For the self-supervised methods, the pre-training setup follows the original setup of each model:

For the TS-TCC model, we use all sEGG recordings for each subject to pre-train it. The data samples are 4 seconds.

For the Neuro-BERT model, we use all sEGG recordings for each subject to pre-train it. The data samples are 4 seconds.

For the BrainBERT model, we use around 180 hours of sEEG recordings from either each subject or 12 subjects for pre-training. This pre-training dataset is larger than the one (approximately 45 hours) used in the original paper. The data samples are 4 seconds.

For the Brant model, we also use all sEEG recordings from either each subject or 12 subjects to pre-train it. While the total pre-training dataset is smaller than the one (around 2700 hours) used in the original paper, the number of subjects (i.e., the number of sEEG location configurations) is greater than in the original paper. The data samples are 4 seconds.

For the LaBraM model, we use all sEGG recordings for each subject to pre-train it. The data samples are 4 seconds.

For the LaBraM-PopT model, we use all sEEG recordings from 12 subjects to pre-train it. The data samples are 4 seconds.

C Model Details

C.1 Du-IN VQ-VAE

The architecture of the Du-IN VQ-VAE model contains three parts: (1) Du-IN Encoder, (2) Vector Quantizer, and (3) Du-IN Regressor. The overall architecture of "Du-IN Encoder" is shown in Figure 2. The "Vector Quantizer" is implemented similarly in LaBraM[28]. The "Du-IN Regressor" contains:

Transformer Decoder: A stack of Transformer layers.

Time Regression Head: A stack of 1D Transposed Convolution layers and one linear projection layer.

The hyperparameters for Du-IN VQ-VAE training are shown in Table 5.

C.2 Du-IN MAE

The architecture of the Du-IN MAE model contains two parts: (1) Du-IN Encoder, and (2) Token Prediction Head. The overall architecture of the "Du-IN Encoder" is shown in Figure 2. The hyperparameters of "Du-IN Encoder" are the same as those in Du-IN VQ-VAE. It's worth noting that when training Du-IN MAE, the weights of the "Du-IN Encoder" are randomly initialized, instead of loaded from the pre-trained Du-IN VQ-VAE model. The hyperparameters for Du-IN MAE training are shown in Table 6.

C.3 Du-IN CLS

The architecture of the Du-IN CLS model contains two parts: (1) Du-IN Encoder, and (2) Label Prediction Head. The overall architecture of the "Du-IN Encoder" is shown in Figure 2. The hyperparameters of "Du-IN Encoder" are the same as those in Du-IN VQ-VAE. It's worth noting that the "Du-IN Encoder" weights in Du-IN CLS can be loaded from either the pre-trained Du-IN MAE or the pre-trained Du-IN VQ-VAE. In the ablation experiments shown in Table 2, our models have different suffixes:

Du-IN: The original Du-IN CLS model. All weights of this model are randomly initialized.

Du-IN (vqvae+vq): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the pre-trained Du-IN VQ-VAE. When fine-tuning it on the downstream task, the "Vector Quantizer" in the pre-trained Du-IN VQ-VAE is inserted between "Du-IN Encoder" and "Label Prediction Head". This is the same operation in DeWave[19].

Du-IN (vqvae): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the pre-trained Du-IN VQ-VAE. This is the same operation in EEGFormer [12].

Du-IN (mae): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the pre-trained Du-IN MAE. This is the same operation in LaBraM [28].

Du-IN (poms): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the Du-IN MAE, which is pre-trained on multiple subjects. The modification of the Du-IN VQ-VAE and the Du-IN MAE to support multi-subject pre-training includes (1) initializing different spatial encoders for different subjects and (2) sharing the same transformer encoder and neural codex.

The "Label Prediction Head" is an MLP with one hidden layer, flattens the output embedding sequence from upstream, and maps this feature embedding to the final prediction through MLP. The hyperparameters for Du-IN CLS training are shown in Table 7.

D Data Augmentation

To enhance the robustness of learned representations during both the pre-training and fine-tuning stages, we apply data augmentation in both datasets.

Pre-training Dataset. In our implementation, we segment sEEG recordings into 8-second samples with a 4-second overlap. When fetching a sample, we randomly select a starting point between 0 and 4 seconds, then extract a 4-second sample beginning from that point.

Downstream Dataset. Since a trial lasts for 3 seconds, employing the jittering mentioned above leads to the blending of information from other trials. In our implementation, we segment sEEG recordings into 3-second samples. When fetching a sample, we randomly choose a shift step between 0 and 0.3 seconds, then shift the sample either to the left or right, padding it with zeros.

E Du-IN Pre-training Analysis

The pre-training of Du-IN can be interpreted as the training of a variational autoencoder [30, 3]. Let x denote the original sEEG signal, ˜ x the corrupted sEEG by mask, and z the neural tokens. Considering the evidence lower bound (ELBO) of the log-likelihood p ( x | ˜ x ) , i.e., recovering the original sEEG signal from its corrupted version:

where (1) q ϕ ( z | x ) denotes the Du-IN Encoder in the Du-IN VQ-VAE that obtains neural tokens; (2) p ψ ( x | z ) decodes the original sEEG signal given input neural tokens; (3) p θ ( z | ˜ x ) recovers the neural tokens based on the masked sEEG signal, which is our Du-IN pre-training task.

The whole framework is optimized through a two-stage procedure as [47, 40]. For the first stage, we train the Du-IN Encoder in the Du-IN VQ-VAE as a discrete variational autoencoder by minimizing the reconstruction loss -E z i ∼ q ϕ ( z | x i ) log p ψ (˜ x i | z i ) with a uniform prior. For the second stage, we set q ϕ as well as p ψ fixed and learn the prior p θ by minimizing the loss D KL . For simplicity, q ϕ ( z | x i ) is defined as a one-point distribution with the most likely neural tokens ˆ z i = arg max q ϕ ( z | x i ) .

z

Consequently, we can rewrite Equation 10 as

where the first term is the objective for vector-quantized neural signal regression in the first stage (i.e., the Du-IN VQ-VAE model), and the second term is the objective for Du-IN pre-training in the second stage (i.e., the Du-IN MAE model).

F Visualization of Vector-Quantized sEEG Regression

We further visualize how the sEEG signals are reconstructed. As depicted in Figure 7, although some details are missing, the overall trend of the signals is reconstructed well. Meanwhile, there is a stable decrease in the reconstruction loss during training, which indicates the discrete codex does learn high-level information from sEEG signals.

G Visualization of Mask sEEG Modeling

Figure 8 demonstrates the convergence curves of the total pre-training loss and masked sEEG modeling accuracy of the Du-IN MAE model. We observe that there is a stable decrease in the mask modeling loss, and the mask modeling accuracy achieves about 20%, which is similar to [28].

Original sEEG signals

Reconstructed sEEG signals

(a)

(b)

(a)

(b)

H Channel Contribution Analysis

For each subject, after training the Du-IN model (with random initialization) on the downstream dataset, we utilize the weights W ∈ R C × D of linear projection in the spatial encoder to calculate the contribution scores S of channels:

where C is the number of channels, D is the dimension of projected embedding and | · | gets the absolute value. Then, we normalize S using its maximum value to ensure it falls within the [0,1] range. Finally, given the variability in model performance across subjects, we further adjust the channel contribution scores based on the decoding performance of that subject, i.e., S = { s i · p | i = 1 , ..., C } , where p represents the decoding performance of that subject.

After calculating the channel contribution scores of all subjects, we project them to the standard brain template according to the MNI (Montreal Neurological Institute) locations of channels, using Nilearn 0.9.2. Since the electrodes are sparsely distributed within the brain, we use Scipy 1.8.1 to interpolate and smooth the channel contribution matrix and use NiLearn to plot the channel contribution map demonstrated in Figure 4 (a).

With the sorted channels within each subject, we evaluate the effect of the number of channels on the decoding performance. For each subject, we evaluate the Du-IN model with { 5 , 10 , 15 , 20 , 30 , 60 } channels (sorted by channel contribution scores), and the averaged performance (across subjects) is demonstrated in Figure 4 (b).

I Effectiveness of Region-Specific Channel Selection

DeWave [19] successfully reconstructs 128-channel EEG signals with the same setting of vectorquantizer. However, this is not the case under the sEEG setting, which is shown in Table 8. It's worth noting that sEEG signals are fundamentally different from EEG signals due to (1) the high information density and (2) the high specificity of different regions. Due to the desynchronization nature [7] of the brain during awake tasks, only specific brain regions are related to tasks. Therefore, only after region-specific channel selection, the Du-IN VQ-V AE model can successfully reconstruct the original signals, thus identifying the fine-grained state of brain regions.

J Additional Group-Wise Evaluation

The cross-entropy loss of different methods from each subject is provided in Table 9, with the best in bold and the second underlined. For model comparison, we report the average and standard deviation values (within each subject) on six different random seeds to obtain comparable results. "Std" means standard deviation.

K Subject-Wise Evaluation

The detailed performance of different methods from each subject is provided in Table 10, Table 11, and Table 12, with the best in bold and the second underlined. For model comparison, we report the average and standard deviation values (within each subject) on six different random seeds to obtain comparable results. "Std" means standard deviation.

1 PT: Whether the model is pre-trained before evaluation.

L Effectiveness of Vector-Quantized Neural Signal Prediction

To verify the effectiveness of vector-quantized neural signal prediction, we elaborate on two types of experimental settings as illustrated in Table 13. The comparison between Du-IN and Setting 1 demonstrates that the codex is effective for masked sEEG modeling. The comparison between Du-IN and Setting 2 demonstrates that introducing the codex can prevent the model from focusing too much on reconstructing details, thus enabling the Du-IN MAE to learn better contextual embeddings.

M Ablation on Mask Ratio

In this experiment, we conduct different mask ratio settings to explore its impact. It is noted that we introduce the symmetric masking strategy, so we only need to validate half of the mask ratios. As the mask ratio is set to r , the symmetric masking will mask 1 -r proportion of sEEG patches. The ablation results are provided in Table 14. It can be induced that the best mask ratio is 0.5 (0.5) for our dataset.

N Ablation on Pre-training Epochs

The impact of the number of pre-training epochs (of the Du-IN VQ-VAE model) is demonstrated in Table 15. We use the checkpoints according to the specified epochs to pre-train the Du-IN MAE model for 400 epochs. Once the reconstruction loss of the Du-IN VQ-VAE model converges, the Du-IN VQ-VAE model can extract the state of the brain region well, thus leading to better performance.

The impact of the number of pre-training epochs (of the Du-IN MAE model) is demonstrated in Table 16. We use the checkpoints according to the specified epochs for downstream classification. Once the mask modeling loss of the Du-IN MAE model converges, the Du-IN MAE model learns robust contextual embeddings, thus leading to better performance.

O Subject-Wise Electrode Locations

We provide detailed information on the locations of the implanted sEEG electrodes for each subject. Red channels are the top 10 channels (selected through channel contribution analysis) for both pre-training and downstream evaluation, as described in Section 4.3. As the majority of subjects have sEEG electrodes implanted on only one side of their brains to locate the source of epilepsy, we provide side views of either the left or right brain areas here.

subj-04

L

R

subj-01

L

R

L

subj-02

R

subj-03

L

R

subj-05

L

R

L

subj-06

R

subj-07

L

R

subj-08

L

R

subj-09

L

R

subj-10

L

R

L

subj-11

R

L

subj-12

R

...........

P Subject-Wise Selected Channels

The MNI coordinates, and brain region labels (according to Harvard-Oxford cortical and subcortical structural atlases [15]) for selected channels are listed below. The channels for each subject are arranged in descending order based on their contribution scores.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Three of the four contributions mentioned at the end of the "Introduction" section are explicitly included. The contribution related to neuroscience-inspired analysis is simplified as "inspired by neuroscience findings" at the end of the "Abstraction" section.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a separate "Limitations" section; see Section 5 for more details.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed information related to our model and baselines in Appendix C and Appendix B, respectively. Besides, we provide code and dataset in https://github.com/liulab-repository/Du-IN.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code and dataset in https://github.com/liulab-repository/Du-IN. Due to the lack of open-source sEEG datasets related to language, we collected a wellannotated Chinese word-reading sEEG dataset, and evaluated our model on this dataset. However, respecting the efforts of the data collectors, we only provide the dataset of some subjects to reproduce the experimental results in the main text. The whole dataset will be publicly available to ensure the reproducibility of this work.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed information related to our model and baselines in Appendix C and Appendix B, respectively.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For the main results, we report the average and standard error values (of all subjects) on six random seeds. For detailed subject-wise evaluation, we report the average and standard deviation values (of each subject) on six random seeds.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Detailed information related to the training process is provided in Section 4.2

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines .

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This work aims to explore the feasibility of intracranial neural signals to decode speech, which mainly has positive impacts on society.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: The ethics statements are provided in Section 7.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: The ethics statements are provided in Section 7.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed Du-IN, a speech decoding framework that incorporates a self-supervised pre-training stage and a classification stage. Experiments are conducted on 12 subjects with a task of word classification. Comparisons are conducted to demonstrate effectiveness.\n\nSTRENGTHS:\n* (a) The paper is clearly written and easy to follow.\n\n* (b) The task of speech decoding using intracranial recordings is interesting.\n\nWEAKNESSES:\n* (a) Although the task of decoding speech from intracranial recordings is an interesting topic, the algorithmic novelty of this paper is limited. The pretraining framework the authors proposed is very similar to BEIT other than the underlying data modality being images and sEEG. Simply applying an existing model BEIT to sEEG data with minor modifications to the backbone architecture does not meet the high standards required for acceptance at a top conference like NeurIPS. \n\n* (b) The comparisons conducted are insufficient. In particular, besides BrainBERT and Brant, the authors compared with, there are also many pre-training approaches for neurological signals.  I list some more up-to-date approaches from a contrastive learning perspective and a masked autoencoding perspective. For contrastive-based approaches,  there are TS-TCC [1] and [2]. For masked autoencoding based approaches, there are Biot [3] and Neuro-BERT [4]. As a matter of fact, the authors could at least follow the comparison setup in Brant for a fair comparison. \n\n* (c) Reproduction details are missing. The reported results on BrainBERT and Brant are ~7.5% and ~12.5%, as compared to the reported performance of ~63% of the proposed  Du-IN. It is astonishing that the proposed Du-IN outperforms BrainBERT and Brant by almost 50%. Both BrainBERT and Brant are masked autoregressive based pertaining approaches. Although BrainBERT and Brant vary from Du-IN, it does not make sense that pretraining could lead to such a tremendous difference. The authors should report the hyperparameters used to reproduce Brant and BrainBERT. Otherwise, It is highly possible that the authors did not tune  BrainBERT and Brant properly. \n\n* (d) Although the decoding of speech is an interesting question, it seems rather well-established in open vocabulary scenarios [5][6], decoding words of limited vocabulary traces back to at least ten years ago. \n\n\n\n\n**References**\n\n[1] Woo, Gerald, et al. \"Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting.\" arXiv preprint arXiv:2202.01575 (2022).\n\n[2] Baevski, Alexei, et al. \"Data2vec: A general framework for self-supervised learning in speech, vision, and language.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Yang, Chaoqi, M. Westover, and Jimeng Sun. \"Biot: Biosignal transformer for cross-data learning in the wild.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Wu, Di, et al. \"Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining.\" IEEE Journal of Biomedical and Health Informatics (2024). \n\n[5] Willett, Francis R., et al. \"A high-performance speech neuroprosthesis.\" Nature 620.7976 (2023): 1031-1036.\n\n[6] Metzger, Sean L., et al. \"A high-performance neuroprosthesis for speech decoding and avatar control.\" Nature 620.7976 (2023): 1037-1046.\n\nQUESTIONS:\n* Are decoding words the same as decoding pronunciations in [1][2]? Is it possible that the sentiment is associated with words that the decoding task is not speech but sentiment?\n\n**References**\n\n[1] Willett, Francis R., et al. \"A high-performance speech neuroprosthesis.\" Nature 620.7976 (2023): 1031-1036.\n\n[2] Metzger, Sean L., et al. \"A high-performance neuroprosthesis for speech decoding and avatar control.\" Nature 620.7976 (2023): 1037-1046.\n\nLIMITATIONS:\nThe authors discussed the drawbacks of the proposed framework which is essentially close-set prediction rather than open-set. Potential negative societal impact was not mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSETTING: Decoding spoken speech from invasive neural recordings, to wit, sEEG.  In particular, the MS aims to classify 61 independently produced (Chinese) words.\n\nAPPROACH: The focus of this study is on unsupervised pretraining via autoencoding, with either (a) vector-quantization or (b) masking.  Masks are 100-ms long and cover all channels.  For both VQ and masking, the core model is a \"spatial encoder\" (linear + conv) followed by a (temporal) transformer.  There are ~15 hours of data per subject.  After pre-training, this model is trained end-to-end (on ~3 hours of labeled data) on the classification task.\n\nRESULTS: The architecture outperforms other architectures proposed for this task, and masking does appear to improve performance.\n\nSTRENGTHS:\nThis is a nice study and the results are compelling.  Temporal masking looks promising as a generic technique for sEEG data.  The dataset is extremely useful and is likely to become the basis of benchmarks for future research in this area.\n\nWEAKNESSES:\nThe paper does not address some important questions (admittedly there is not much space), which I list below.\n\nIn addition to these, I list here some minor issues:\n\n--For comparing with other results in the decoding literature, it would be very useful to report cross entropy in addition to (or instead of) accuracy, since the latter can only be interpreted in light of the fact that there are 61 classes.\n\n--The reader shouldn't have to go to Appendix C.3 just to find out what the names in Table 2 mean.  Also, what does \"CLS\" stand for?\n\n--The (vqvae+vq) and (vavae) rows in Table 2 should be marked as \"pre-trained.\"\n\n--The MS calls the token's temporal width the \"perception field.\"  Is this a standard term?  Perhaps the authors mean \"receptive field\"?\n\n--The authors use \"codex\" where they mean to use \"code.\"\n\nQUESTIONS:\n(1) The spatial encoder:  The name suggests that its convolutions are across channels, but this is inappropriate across sEEG electrodes (and within electrodes should take into account the spatial layout of channels).  Can the authors clarify this?  (I see in Appendix H that the *linear layer* combines channels.)\n\n(2) The authors give some reasons for the superiority of their model over BrainBERT and Brant, but it would be helpful to have some insight on the performance relative to other models as well.  E.g.,\n        \n(a) The MS focuses on pretraining, but in fact it appears that the biggest gain over the baselines comes from the architecture itself (Du-IN trained fully supervised)---an improvement of ~25 percentage points over the CNN-BiGRU!  Can the authors shed light on this difference? \n\n(b) The EEG-conformer likewise uses a \"temporal transformer,\" but performance is ~10 percentage points lower than the supervised-trained Du-IN.  What do the authors see as the key difference here?\n\n(c) The authors mention LaBraM but don't compare Du-IN to it.  It would be nice to see this comparison as well....\n\n(3) Intracranial EEG data is generally (but not always!) filtered into the range of ~75-150, because otherwise the low frequencies, which have higher power, tend to dominate.  (The low frequencies can be provided as separate channels.)  This is a problem because low frequenceies generally contain much less information about speech.  In this study, frequencies below 70-Hz are not filtered out.  Indeed, it appears that reconstructions under vector quantizations retain mostly low-frequency information (Fig. 7).  Can the authors explain these results?\n\n(4) Channel selection:\n \n(a) \"To streamline, we utilize these top 10 channels for both pre-training and downstream evaluation.\"  Does that mean that, for each subject, the best 10 electrodes (for that subject) were used?\n\n(b) If the choice to use just 10 channels was made on the basis of decoding results on the test set, then the results are biased upwards.  Can the authors confirm that this choice was made based on validation data?  (Is Fig. 4b on validation data?)\n\n(c) How does channel selection interact with model choice?  Were all models in Table 2 evaluated with just 10 channels?  Does their performance as a function of #channels also resemble Fig. 4b?\n\n(5) It seems quite possible from Fig. 4a that the authors are decoding speech perception as much or more than speech production, since many of the most highly contributing electrodes are below the Sylvian fissure.  This would limit the use of this decoder as a speech prosthesis for people who cannot speak.  Can the authors comment on this?\n\nOn this head, although Fig. 4 is a nice summary, but it would be very useful to have a list/table of the electrodes used for decoding for each subject, labeled by cortical area.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Du-IN, a novel decoding technique for speech production the leverages sEEG data. To achieve their goals, the authors gathers a novel dataset. The Du-IN model learns multi-variate (across channel) representations from sEEG signal using a discrete cookbook and a masked modeling based self-supervised training. The proposed algorithm achieves SoTA performances on the speech 61-words classification task. Furthermore, analysis of optimal model configuration and ablation studies reveal that best performances are achieved with electrodes in brain regions known to be involved in speech production.\n\nSTRENGTHS:\nThe paper presents strong results of high significance for the field of speech decoding. In particular:\n- The authors have collected a novel sEEG dataset, which will be made publicly available, hence fostering further progress in the field of brain signal decoding.\n- The authors propose a novel framework for sEEG signal decoding - the Du-IN algorithm, which achieves SoTA performances, thus significantly advancing the field of brain decoding.\n- Authors provide additional ablations studies to validate design choices and recover known results from Neuroscience, strengthening the overall soundness of their technique.\n- The paper is well written and clearly organized, with a detailed Appendix providing further details.\n\nWEAKNESSES:\n- In Table 2 both the Du-IN (vqvae+vq) and Du-IN (vqvae) are marked as not pre-trained. However in the Appendix C.3 it is stated that their Neural Encoder weights are loaded from the pre-trained Du-IN VQ-VAE model.\n- Minor spelling: Line 156 DuIR $\\to$ DuIN\n\nQUESTIONS:\n- There seems to exist substantial variation of decoding accuracy across subjects. Is this difference due to different electrode locations, or number of electrodes (even if only 10 channels seem to be optimal) or are other concerns with data quality relevant?\n- Could the manual selection of channel be partially subsumed by a sparsity regularization term on the channel-to-embedding projection matrix?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation of their work with a dedicated section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a neural network architecture, a self-supervised learning (SSL) method for pre-training models for sEEG decoding and a novel dataset.\n- The architecture starts with segmenting the sEEG signal along the time dimension, followed by a linear module combining information across the different sEEG channels into embedding vectors and ending with a transformer with attention along the time dimension.\n- The SSL method has two stages: \n    - 1. The architecture followed by an additional decoder network is trained to reconstruct the original signals (autoencoder). However, the output of the transformer is not directly passed to the decoder, instead, each token is replaced with the closest embedding vector from a trainable codebook (vector quantisation).\n    - 2. Time windows of the input are masked and the architecture is trained to predict which codebook embedding represents best each masked window.\n- The dataset introduced contains sEEG recordings of 12 subjects while they were speaking aloud Chinese words from a vocabulary of size 61. \n\nIn the experiments, the models were pre-trained using the SSL tasks using unlabelled data from a single subject and fine-tuned on labelled data from the same subject for classifying the 61 words. \nThe authors conducted different ablation studies regarding pre-training (no pre-training, and only step 1 of pre-training using quantized features for fine-tuning or not), dataset, codex size and dimensionality, and length of the time windows.\nThey also compare with pre-existing methods, two including a pre-trining phase and two without. The two with pre-training are pre-trained on all 12 subjects.\n\nSTRENGTHS:\n*Significance*: As pointed out by the authors, sEEG speech datasets are rare. Their promise of publishing the dataset they introduce here is good news for the community as it will lower the entry threshold for future research. Additionally, they demonstrate how SSL-based pre-training  allows improving performance compared to only supervised training.\n\n*Clarity*: The text has a good structure and is well-written. The figures also help in understanding the method.\n\n*Originality*: SSL techniques are still rather novel to the BCI community. This work sheds light on some of them.\n\nWEAKNESSES:\n**Major**\n1. The authors pre-train their method on the test subject (Lines 201-202) whereas the baseline methods with pre-training (BrainBERT and Brant) were pre-trained on all 12 subjects (lines 511-518). This does not allow for a fair comparison. Additionally, this choice is not properly justified.\n2. The authors mention in section 4.5 Ablation study, that they found optimal hyperparameters for their architecture (codex of size 2048 and dimension 64, and perceptual field of 100ms). They use these hyper parameters to compare to the other methods (see Table 5). However, the manuscript does not mention the use of a holdout set for conducting this ablation study. This is problematic for two reasons:\n    1. This suggests the method could have a low capacity to generalise be overfitting to their dataset\n    2. This is not fair for the other methods as no hyperparameter optimisation is mentioned for them.\n3. The authors mention they resampled their signals to 1000Hz (line 210). However, some architecture compared have been introduced with different sampling rates (Brant used 250Hz and EEG-Conformer used 200 and 250Hz). The text does not mention any adaptation of the architectures compared or of the sampling rate. This is problematic as a convolution kernel supposed to cover 1s of signal at 250Hz will only cover 0.25s at 1000Hz. \n4. Some aspects of the method are undefined or unclear. Please see the Questions section below. These aspects need to be clarified in the manuscript.\n\n\n**Minor**\n1. Because of the flaws in experimental design described above, the statements about the superiority of their method compared to others (lines 14, 71, 248 and 318) are unjustified.\n2. All figures should be in a vectorized format.\n3. The text in the figures should be approximately constant and the same as the article’s text. It is too small in figures 5 and 7. It is on the limit in figures 1,2 and 3.\n4. The term “fusing channels” is used lines 16, 39, 108 and 117 but is rather unclear. It is only explained line 134.\n5. Line 29: provide examples of non-invasive recording methods (EEG, MEG…)\n6. Section 2.3, additional publication the authors should be aware:\n    1. A preprint of Brant-2 is now available https://arxiv.org/abs/2402.10251\n    2. SSL methods trained on EEG with attention mechanism for combining information across channels: EEG2Rep (https://arxiv.org/abs/2402.17772) and S-JEPA (https://arxiv.org/abs/2403.11772).\n7. Line 54: Indeed, there is a lack of open sEEG recordings but some still exist. In reference [1], they mention they use a public dataset, in [16] they mention their data is available upon request and Brant-2 (not cited here), they use a 26-subject public dataset. Existing sEEG datasets should be cited.\n8. Line 98: typo “Metzger1” \n9. As I understand, the terms “Du-IN Encoder”, “neural transformer”, “neural encoder”, and “neural tokeniser” all refer to the architecture in figure 2. If this is indeed the case, it would improve readability to chose one of them and stick to it throughout the manuscript.\n10. Line 166: missing “to” in “$z_c$ to get”\n11. Figures 2 and 3: It would improve readability to place the different variables ($e^p_i$, $e_i$, $e^t_i$, $c_j$,…) on these figures.\n12.  Line 162: for consistency, you could use $j$ as index for the codex instead of $i$.\n13. Lines 180-193: the objective function of the masked training is unclear (see questions).\n14. Line 247: Please mention that the baseline descriptions can be found in appendix C.3.\n15. Line 301: The research line of Feng et al. was already explored by Christian Herff in 2015 (https://doi.org/10.3389/fnins.2015.00217)  and is still today (https://www.sciencedirect.com/science/article/pii/S1053811923000617). It could be interesting to draw a parallel.\n\n\n\n**Overall comment**\nThe introduction of a novel sEEG dataset is valuable but the evaluation of the machine learning algorithms that are introduced is poor. Therefore,  I believe NeurIPS might not be the best fit for this publication.\n\nQUESTIONS:\n1. Lines 134-140: The description of the spatial encoder is ambiguous:\n    1. Does “linear” mean fully-connected layer? Appendix C.1 mentions “Linear Projection: 10->16” which would mean that an operation is applied to each channel independently and that the layer is actually not a fully-connected one.\n    2. Convolutions are mentioned, but over which dimensions? Over time? Channels? Both?\n2. Figure 3: What are the numbers below “l2-norm” and bellow “neural regressor” corresponding to?\n3. Why did you use a different signal length for pre-training and for downstream classification?\n4. Line 210: Why did you use a sampling rate of 1000Hz? With a low-pass filtering at 200Hz, a sampling rate of 400Hz would be sufficient\n5. Line 211: Did you apply the normalisation on each channel independently (i.e., mean and std computed for one channel only) or using all channels together?\n6. Line 222: How is the training time split between VQ-VAE and MAE training?\n7. Line 228, “selected from the validation set”: does this mean early stopping? Was it performed using the validation loss, validation accuracy or another metric?\n8. Table 2, “Du-IN (mae)”: this row corresponds to the scores obtained after a VQ-VAE training followed by a MAE training. Is the first step exactly “vqvae” or “vqvae+vq”? To lift the ambiguity, you could rename the row to “Du-IN (mae+vqvae)” or “Du-IN (mae+vqvae+vq)”.\n9. Why refer to your method as a “VAE”? As I understand, for a given input, the “reconstructed sEEG signals” are obtained in a deterministic manner (please, correct me if I’m wrong am wrong). The term “variational” in VAE suggests there is a stochastic operation involved.\n10. Lines 180-193: the objective function of the masked training is unclear:\n    1.  Line 188: “neural tokens” is ambiguous. What is being predicted?\n    2. Equation (7) and (8): how are the $z_i$ obtained? \n    3. Equation (8): $m_i$ is undefined. What is the meaning of $m_i$?\n\nLIMITATIONS:\nThe authors included a Limitations section in their manuscript in which they already mention other approaches to speech decoding, at the phonemes level and at a semantic level. \n\nIt could be relevant to remind the reader that sEEG recordings are invasive. Therefore, they are only done with patients who would require them anyway for medical reasons (epilepsy monitoring, etc…). And this presents a major obstacle to making these methods widely accessible and usable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDue to the locality and specificity of brain computation, existing models that learn representations of a single channel yield unsatisfactory performances on more difficult tasks like speech decoding. This paper tries to build a stereoElectroEncephaloGraphy (sEEG) foundation model with Vector Quantization (VQ) pre-training. We hypothesize that building multi-variate representations within certain brain regions can better capture the specific neural processing. To this end, we collect a well-annotated Chinese word-reading sEEG dataset, targeting language-related brain networks over 12 subjects, and we developed the Du-IN model that can extract contextual embeddings from specific brain regions through discrete codebook-guided mask modeling. Extensive comparison and ablation experiments verify that our model achieves SOTA performance on the downstream 61-word classification task.\n\nSTRENGTHS:\n**(S1)** This paper provides an interesting solution for the sEEG learning framework that models multiple sEEG channels by a spatial encoder and quantizes sEEG signals from the temporal aspect. To evaluate the proposed Du-IN framework, the authors collect and contribute a Chinese word-reading sEEG dataset for speech decoding (as word classification).\n\n**(S2)** The manuscript is clearly structured and easy to follow. Empirical analysis and ablations of proposed modules provide comprehensive understandings.\n\nWEAKNESSES:\n**(W1) The employed techniques in the proposed framework lack novelty to some extent.** Firstly, except for the spatiotemporal modeling architecture for sEEG, the VQVAE tokenization and masked code modeling pre-training framework is adopted from popular BeiT frameworks [1, 2], which has been adopted to various scenarios like [3, 4, 5]. Some specially designed modules are necessary for introducing the BeiT-like framework, and the current version is not effective and well-designed from my aspect (view W2 for concerns). Secondly, the authors regard the speech decoding tasks as word classification, which might be too coarse to decode the actual meanings of human speeches. Therefore, the conducted decoding task might not be sufficient to verify the effectiveness of the Du-IN framework.\n\n**(W2) Weak comparison experiments.** Firstly, the authors only provide a few supervised learning and self-supervised pre-training baselines in Table 2 with a single-word classification task (as speech decoding). More relevant contrastive learning [6] and masked modeling pre-training methods [7] for sEEG should be considered, which should also be referred to in the background section. Meanwhile, some supervised baselines with more parameters can be added for a fair comparison as Du-IN models. Secondly, the results of some baseline methods are very bad (e.g., BrainBERT and Brant). It’s hard to believe that pre-training models will yield worse results than random initialized supervised learning models. I suggest the authors conduct a fair and comprehensive comparison experiment.\n\n**(W3) Not explicitly modeling multiple subjects.** Since the invasive sEEG signals for different subjects have different channels, I couldn’t find an explicit modeling strategy in the proposed encoders and pre-training framework that considers fusing or selecting information from multiple subjects. There might be some challenges, like representation conflicts and missing channels of different subjects, which will decrease the overall cross-subject performances. I suggest the authors conduct such analysis of multiple subjects, e.g., cross-validation of several subjects and transfer learning of pre-trained representation across datasets of different subjects.\n\n### Reference\n\n[1] MAGE: Masked Generative Encoder to Unify Representation Learning and Image Synthesis. CVPR, 2023.\n\n[2] BEIT: BERT Pre-Training of Image Transformers. ICLR, 2021.\n\n[3] Human Pose as Compositional Tokens. CVPR, 2023.\n\n[4] MAPE-PPI_Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding. ICLR, 2024.\n\n[5] VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling. ICML, 2024.\n\n[6] Time-series representation learning via temporal and contextual contrasting. IJCAI, 2021.\n\n[7] Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining. IEEE Journal of Biomedical and Health Informatics, 2024.\n\n==================== Post-rebuttal Feedback ====================\n\nAfter considering the authors' rebuttal and other reviewers' comments, I decided to keep my score because some of my concerns were not solved directly (e.g., W1 and W3). Meanwhile, the authors have provided some useful results (e.g., more comparison experiments) to support the effectiveness of the proposed methods. Since the revision could not be uploaded, I encourage the authors to improve the manuscript further for submission to the next conference or journal.\n\nQUESTIONS:\n**(Q1)** For different subjects with different channels, does the spatial encoder need different parameters for each subject? And does the output embedding of each subject in the same shape?\n\n**(Q2)** Does the authors conduct ablation studies of masked code modeling, e.g., the masking ratio and pre-training epochs? Since the proposed Du-IN is a framework with several networks and the pre-traning & fine-tuning pipeline, it’s better to analyze and clearify each module for better understanding and practical usage.\n\nLIMITATIONS:\nAlthough the authors discussed some limitations in Sec. 5, potential negative societal impact of the proposed dataset and the Du-IN framework were not clearly discussed. More details of the implementations and the collected dataset can be provided for the sake of the neuroscience community.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}