Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals
Authors: Hui Zheng
Haiteng Wang
Weibang Jiang
Zhongtao Chen
Li He
Peiyang Lin
Penghu Wei
Guoguang Zhao
Yunzhe Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: neuroscience
sEEG
speech decoding
self-supervision
neuro-AI
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose  the Du-IN model, a framework for sEEG-to-Word translation task, learning contextual embeddings through discrete codebook-guided pre-training on specific brain regions.

Abstract: Invasive brain-computer interfaces with Electrocorticography (ECoG) have shown promise for high-performance speech decoding in medical applications, but less damaging methods like intracranial stereo-electroencephalography (sEEG) remain underexplored. With rapid advances in representation learning, leveraging abundant recordings to enhance speech decoding is increasingly attractive. However, popular methods often pre-train temporal models based on brain-level tokens, overlooking that brain activities in different regions are highly desynchronized during tasks. Alternatively, they pre-train spatial-temporal models based on channel-level tokens but fail to evaluate them on challenging tasks like speech decoding, which requires intricate processing in specific language-related areas. To address this issue, we collected a well-annotated Chinese word-reading sEEG dataset targeting language-related brain networks from 12 subjects. Using this benchmark, we developed the Du-IN model, which extracts contextual embeddings based on region-level tokens through discrete codex-guided mask modeling. Our model achieves state-of-the-art performance on the 61-word classification task, surpassing all baselines. Model comparisons and ablation studies reveal that our design choices, including (\romannumeral1) temporal modeling based on region-level tokens by utilizing 1D depthwise convolution to fuse channels in the ventral sensorimotor cortex (vSMC) and superior temporal gyrus (STG) and (\romannumeral2) self-supervision through discrete codex-guided mask modeling, significantly contribute to this performance. Overall, our approach -- inspired by neuroscience findings and capitalizing on region-level representations from specific brain regions -- is suitable for invasive brain modeling and represents a promising neuro-inspired AI approach in brain-computer interfaces. Code and dataset are available at https://github.com/liulab-repository/Du-IN.

Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals
Hui Zheng"24, Hai-Teng Wang" Wei-Bang Jiang', Zhong-Tat Chen', Li He', Pei-Yang Lin', Peng-Hu Wei Guo-Guang Zhao², Yun-Zhe Liu
Beijing Normal University, PPeking University, SShanghai Jiao Tong University, *Chinese Institute for Brain Research, CCapitll Medical University, Xuanwu Hospital, Beijing Equal contribution, tyunzhe liu@bnu. edu.
Abstract
Invasive brain- computer interfaces with Electrocorticography (ECoG) have shown promise for high- perfformromcc speech decoding in medical applications, but less damaging methods like intracranial aecro--lectroennephallggaap (sEEG) remain underexplored. With rapid advances in representation learning, leveraging abundant recordings enhance speech decoding is increasingly attractive. However, popular methods often pre-train temporal models based on brain-level tokens, overlooking that brain activities different regions highly desynchronized during tasks. Alternatiely, they pre-train spatial-temporal models based on channel-level tokens but fail to evaluate them on challenging tasks like speech decoding, which requires intricate processing in specific language-relate areas. To address this issue, we collected a well-annotated Chinese word-reading sEEG dataset targeting language- related brain networks from 12 subjects. Using this benchmark, we developed the Du-INN model, which extracts contextual embeddings based on region-lever tokens through discrete codex- sedex-guided mask modeling. Our model achieves state- of-the-art performance on the 61-word classification task, surpassing all baselines. Model comparisons and ablation studies reveal that our design choices, including (i) temporal modeling based on region-level tokens by utilizing ID depthwise con- volution to fuse channels in the ventral sensorimotor cortex (vSMC) and superior temporal gyrus (STG) and (ii) self-supervision through discrete codex-guided mask modeling, significantly contribute to this performance. Overall, approach inspired by neuroscience findings and capitalizing on region-level representations from specific brain regions is suitable for invasive brain modeling and represents a promising neuro-inspired AI approach in brain-computer interfaces. Code and dataset available at https///tthhhoooommmmmaagaaoooo/
GEEG movdiigs
6 reed clmuifratiin
R asi
Tu-hi
Du-IN
402/12
ChNi BITBU
BexaREKT
02y-11
Texe
Figure 1: Overall illustration of sEEG decoding setup and comparison with SOTA baselines.
DuuINN refers the phonetic transcription of "Mth 22 (i.e., promunciation) in Chinese.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Brain signals refer to the biometric information collected from the brain. Their patterns provide valuable insights toward understanding the physiological functions of the brain and the mechanism of related diseases, leading to various applications, including speech decoding L3 19 37,, sleep
cognition research B5 551, neurological disorders detection [28 54], and so on. Due to the high signal-noise ratio, invasive recording methods (e.g.. Miereeeleettonnnepallorraaa
(sEEG), Elec- troCorticGGraphy (ECoG)) usually reveal these underlying mechanisms better than non- non-innasivv recording methods. Many previous works (29 19] have shown that decoding speech from EEG signals is difficult, and the performance is limited. Compared with ECoG, sEEG imposes less trauma on patients and provides more stereotactic information from specific brain regions. Although some studies 337 36) have recently shown promise for building high-performance speech decoders based on ECoG, there are few attempts made to explore the potential of sEEG-based speech decoding.
Modeling intracranial neural signals, especially SEEG, has gained significant attention, but several issues remain unresolved. Current research on modeling neural signals is divided into two lines based on the basic modeling units (e.g., channel-level tokens or group-level tokens". Some studies [54] [2 utilize shared embedding blocks to embed single channels into channel-level tokens, neglecting the specificity of brain computation [8] then they adopt spatial-tempora integration to model spatial relationships among them, attempting to regain the precise state of the brain. However, these methods mainly focus on channel-level classification tasks, e.g., seizure detection, yet fail to validate them on more challenging group-level classification tasks, e.g., speech decoding. Other studies [19 21 fuse all channels (across the brain) to build brain-level tokens, overlooking the brain's 's desynchronization nature 77: then they adopt temporal modeling to capture the rapid process of brain dynamics. Besides, labeling data at scale in medical experiments is often impractical or costly, emphasizing the need to maximize label efficiency. Hence, developing an efficient pre-traing vre-training framework that draws on prior neuroscience findings is highly appealing, it can make the most of abundant unlabeled data.
The primary challenge in modeling intracranial neural signals lies in extracting meaningful tokens, requiring careful consideration of two key factors. (1) Temporal scale. Since intracranial neural signals have high temporal resolution and signal-noise ratio, these tokens must capture rapid dynamic changes in brain activity. (2) Spatial scale. Considering the brain's 's desynchronizatioon nature, these tokens should correctly capture the information of each brain region for further integration and, needed, decouple different parts of brain dynamics within each brain region. To better assess how well different models capture the intricate processing within each brain region, we can evaluate these methods on tasks mainly involving a few brain regions.
Since speech mainly involves specific brain regions related to vocal production, as demonstrated in Section 2.1 we utilize speech decoding tasks to evaluate which model can effectively extract information from specific brain regions. Since there are too few open source sEEG language datasets 11 49), we collected well-annotated Chinese word-reading sEEG dataset (vocal production), including 12 subjects, which makes up for the problem of missing sEEG recordings in language tasks. Inspired by neuroscientific findings, we systematically demonstrate the locality and specificity of brain computation and propose the Du-IN model to solve the abovementioned issues. Compared to other existing methods for modeling brain signals, Du-IN achieves SOTA performance on the 61-word classification task, demonstrating the effectiveness of our model in extracting meaningful tokens that can capture both the rapid changes and the precise state of specific brain regions. It marks a promising neuro-ing ceuro-inspired Al approach [42 41 in BCI.
To sum up, the main contributions of our work comprise:
1. A well-annotated Chinese word-reading sEEG dataset, addressing the lack of sEEG language dataset. The dataset will be publicly available.
2. Demonstration of brain- seain-specific computation - achieving the best decoding performance only requires about one electrode in specific brain regions (i.e., vSMC, STG).
3. A novel framework for sEEG speech decoding - Du-IN, which learns reggion-level contextual embeddings through discrete codex- -adex- mask modeling.
4. SOTA performance on the SEEG speech decoding task Du-IN achieves 62.70% top-1 accuracy on the 61-word classification task, surpassing all other baselines.
'The term "group-level" includes "bvain-level" and "region-level." and distinct from "channel-level."

---

2 Related Works
2.1 Neural Basis of Language Function
Neuroscientific research [ 43 in the past has extensively explored brain regions supporting language functionality. In neuroscience, the investigation into language functionality related to speech has been categorized into two main streams: one dedicated to semantic processing and the other to vocal production. Previous studies 4 [ have shown that brain regions associated with semantic processing primarily include left inferior frontal gyrus (IFG), left anterior temporal lobe (ATI.), and bilateral middle temporal gyrus (MTG).
As for vocal production, which is also the focus of our work, it is predominantly governed by motor information related to language articulation, primarily involving ventral sensorimotor cortex (vSMC), bilateral superior temporal gyrus (STG), and bilateral dorsal laryngeal motor cortex (dLMC) 17 9 Our analysis results based on our collected word-reading sEEG dataset also confirm this point, as illustrated in Figure
2.2
Language Decoding in BCI
The keys to decoding natural language from brain signals are (1) high-quality recordings, and (2) well-designed models with good representations. Compared to non-invasive recordings (c.g., EEG), invasive recordings manifest advantages in providing detailed information about specific brain regions with a high signal-noise ratio. Since speech mainly involves some specific brain regions, obtaining detailed recordings of these brain regions will significantly enhance the decoding performance. Existing works 11 37 21 have shown the great potential of building a high- gegg-performance decoder based on invasive recordings.
The other key well-designed models with good representations. Existing work for brain-to-language representations can be classified into two categories: self-supervision or alignment with representation models pre-trained on other modalities (e.g., text, audio). BrainBERT [4 learns general embeddings through self-supervised mask modeling. DeWave [19] introduces discrete codex encoding and aligns neural representations with text embeddings from BART (32), thus enhancing the extraction of semantic processing-relatee information from EEG recordings. Metzger et al. (36) align neural representations with acoustic embeddings to improve the extraction vocal production-relatt information from ECoG recordings.
2.3 Self-supervised Learning in BCI
In recent years, self-supervised pre-training has made significant progress natural language processing [16] 33 6 and computer vision ( 25 [1]. However, its potential in BCI is far from being explored. BrainBERT (for sEEG) [49 embeds single channels into channel-level tokens and utilizes mask modeling to learn general representations. Brant (for sEEG) [54] (53. PopT (for sEEG)
further adopt spatial- temporal integration to model spatial relationships among them. Some works (for HEG) 31
10 and some works (for EEG) 28 23
50 22 take the other way fusing all channels (across the whole brain) to build brain-level tokens, and it uses self-supervised learning to leam contextual representations. Considering the difference among brain regions, MMM (for EEG) 52 further splits channels into different groups to build region-level tokens.
20
All existing pre-training methods for sEEG primarily pre-train spatial-temporal models based on channel-level tokens yet only evaluate them on channel-level classification tasks, e.g., seizure de- tection. However, unlike EEG pre-training methods, their effectiveness group-level classification tasks, e.g., speech decoding. Besides, there is no standard channel configu- ration for sEEG recordings, unlike EEG recordings, which makes modeling spatial relationships in sEEG more challenging.
more challenging
3 Method
The overall architecture of Du-IN illustrated in Figure where the raw sEEG signals are fused across channels to build region-level tokens and further encoded for downstream tasks.
3

---

3.1
Task Definition
Due to the lack of open- sEEG datasets related to language tasks, we follow the experimental design outlined by Moses et al. [37 to collect a -nnooattt Chinese word word-reading sEEG dataset (vocal production). During the experiment, each subject speaks aloud 61 pre-determined Chinese words 50 times; see Appendix for more details. We formulate the multi-channel sEEG signals as X E CCXT where C is the number of sEEG channels and is the total timestamps. The associated word label is denoted as y E V/ where y represents the set of 61 pre-determined words. In summary, this dataset comprises paired sEEG-word data ((K2 y)), and the model aims to decode the corresponding word y from sequence of raw SEEG signals X.
3.2 Model Architecture
Temporal
Trandarmer Encaler
Ioput xEEG Signals
Sputial Encuder
Output Embedding
Banch Noven
R Norm
119
patch
Figure 2: The overall architecture of Du-IN Encoder. Du-IN Encoder is used as an encoder in all Du-IN models (i.e., Du-IN VQ- VQ-VAE Du-IN MAE, Du-IN CLS (classification)) Appendix Cf more details.
We introduce the Du-IN Encoder, a general architecture for sEEG speech decoding tasks that can deal with any input SEEG signals with arbitrary time length, shown in Figure 2 The key operation for archiving this is segmenting the sEEG signals into patches, inspired by patch embeddings in images [18]. For each sample X. we use W W-length window without overlap to segment it into patches, obtaining X' = {xx E RCx xxC W = I,.....N. N}. where N = W
T is the number of patches.
Spatial Encoder. As each SEEG patch has multiple channels, it is vital to fuse different channels to extract meaningful features before patch-wiss wise interaction by self-attention. We employ spatial encoder, which consists of a linear projection and several convolution blocks, to encode each sEEG patch into patch embedding. The linear projection transforms the raw SEEG signals into the hidden neural space, and its weights are utilized for subsequent analysis. The convolution block is composed of a 1D depthwise convolution layer and a batch normalization layer (27. We denote the output patch embeddings from the spatial encoder as
Ep = (e E RR i where d is the dimension of the embeddings.
= I,.,,N
Temporal Embedding. In order enable the model to be aware of the temporal information of patch embeddings, we utilize the parameter-free free position embeddings introduced in [48). i.c.. E (ei,.... ..., e. ). Note that tmax the hyperparameter determining the maximum number of time patches and tmaz 2 N. Given one arbitrary patch embedding e, in Equation| I from the spatial encoder, we add the corresponding temporal embedding it:
Einit = {e"" + e/ji = N).
which forms the input embeddings Emitt for the Transformer Encoder.
Transformer Encoder. Finally, the sequence of embeddings will be directly fed into the Trans- former encoder [48] to get the final encoded È = {ee E Rdli NY. To make the training of the Transformer more stable and efficient, we incorporate some modifications [14] inspired by LaBraM (28). We add layer normalization to the queries and keys before the dot-product attention mechanism, which avoids over-large values attention logits:
Attention( Q, K, V) softmax LN(Q)LN(K) V
Vihead
where dhead is the dimension of attention head and LN denotes layer normalization 2 For down- stream classification tasks, flatten the output embeddings followed by a classification head.

---

3.3 Du-IN VQ-VAE Training
Prior to pre-training Du-IN through mask modeling. we need tokenize the sEEG patches into discrete tokens. We introduce quctor-quantied neural signal regression, which is trained by recon- structing the original sEEG signals, as shown in Figure The key components are the Du-IN Encoder, which encodes the raw sEEG samples into embeddings, and the Du-IN Regressor, which reconstructs the original sHEG signals. The idea is basically inspired by VQ-VAE which encodes images into discrete latent embeddings.
(a) Du-IN VQ-VAE Training
Reconstracted
Imput sEEG Signals
3
SEGG Signals
Du-IN Rogressor
Da-IN Eucoder
(b)
Mask
Token Prediction Head
650
T71
Du-IN Encoder
Symmetric Mask
T14
T15
T14
F9
Do-IN
Figure 3: Overview of Du-IN VQ-VAB training and Du-IN MAE training. (a). We train the Du-IN Encoder in the Du-IN VQ- VAE discretize sEEG signals into discrete neural tokens by reconstructing the original sEEG signals. (b). During the training of Du-IN MAE, part of sEEG patches are masked while the objective predict masked tokens from visible patches.
Du-IN Encoder. We define neural codex C - (c,li - 1, 1,..., Ncodew E IR Nendex Xdendes where Neodez the number of the discrete neural embeddings and deodez the dimension of each em- bedding. Given a sEEG sample X, the Du-IN Encoder, illustrated in Figure 2 first encodes it to embeddings & {e, E Rd 1,.., N). After that, we utilize linear projection Ze to get the mapped embeddings 2. (E) (z. Reddde I,....N N) in the codex space. Then, the codex looks up the nearest neighbor of each embedding (e;) in the neural codex C. This procedure can be formulated
2y(E) (e)))=== N). Zg(e:) = C..
2, = arg [l()(z.(e))) (2(c,)
where E2 represents & normalization and z, (ee) is the quantized vector after the quantizer. This is equivalent to finding the closest neural embedding by cosine similarity and such &g normalization improves the codex utilization [38].
Du-IN Regressor. The Du-IN Regressor consists a Transformer decoder and a stack of transposed convolution layers. Given a sequence of the vector-quantized embeddings 2 {z = 1..., N). the Du-IN Regressor convert these discrete embeddings back into sEEG signals X (ã.li ..., N}. The mean squared error (MSE) loss utilized to guide the regression. The total loss for training the Du-IN VQ- VAE is defined as:
Luquae
Σ 1=1
||..
xx 112 [|sg(z.(e:)] 22)]) (e:e)) 3 (e.))) - sg(z,
(5)
5

---

where represents the stop-gradne graddindtnnt operation, which is an identity at the forward pass and has zero gradients. To stabilize the codex update, we use the exponential moving average strategy (47.
3.4 Pre-training Du-IN
Masked sEEG Modeling. To enforce Du-IN learning contextual representations, we propose masked sEEG modeling. The whole procedure presented in Figure As illustrated in Figure 2 given a sEEG sample X, the spatial encoder first transforms it to patch embeddings &p = { 1. ...., NY. Given these patch embeddings Ep. around 50% of patch embeddings are patch wisely chosen and masked. The masked position is termed as M. Then, shared learnable embedding eIM] Rd is used to replace the original patch embeddings:
mm = li = NY. e.ºh m, 3 e(M] + m1)m,) O
(6)
where 6()) is the indicator function and my 8(i M). After that, the masked embeddings & will be added by temporal embeddings, and then fed into the Transformer encoder. The output
embeddings & will be used predict the indices of the corresponding codes from the codex in the Du-IN VQ- -A-E through a linear classifier:
p==(e:)) softmax
The training loss of mask modeling is defined as:
LM LM= -- Σ m, log p(z(le))
(8)
SEM
Symmetric Masking. Inspired by LaBraM (28], we further introduce a symmetric masking strategy to improve training efficiency. We calculate the inverse of the generated mask M, obtaining M. Similarly, we use the new mask M to perform the mask modeling, obtaining the mask modeling loss Caym. The total loss for pre-training the Du-IN model (i.e., Du-IN MAE model) is defined as:
Lmac LM + LM
Experiments
4.1 Dataset
Due to the lack of open-source sEEG datasets related to language tasks, we follow the experimental design outlined by Moses et al. (37] to collect a well-annotated Chinese word-reading sEEG dataset (vocal production), including subjects. The subjects undergo a surgical procedure to implant to 13 invasive sEEG electrodes, each with 72 to 158 channels, in their brain. For each subject, the dataset contains hours of 2000Hz recordings, 3 hours of which are task recordings.
Pre-training dataset. For each subject, the pre-training dataset contains all sEEG recordings (with about 54 million timestamps) of that subject. To stabilize computing resource usage, the time length of sEEG sample X' is set to 4 seconds.
Downstream dataset. For each subject, 3 hours of the sEEG recordings are task recordings. The sEEG signals are segmented into about 3000 3-second samples, each of which paired with the corresponding word label (from 61 pre-determined words).
4.2 Implementation Details
Preprocess. We first filter the sEEG signals between 0.5Hz and 200Hz to remove low-frequency noise. Then, a notch filter of 50Hz is applied to avoid power-line interference. After that, all sEEG signals are resampled 1000Hz and bi-polar re-referenced [33]. Finally, we perform z-score normalization each channel to guarantee normalized data scales across all channels.

---

Model Configurationss The length of the sEEG patch is 100ms, resulting in 40 patches per sample in the pre-training dataset and 30 patches per sample in the downstream dataset. The "Spatial Encoder" contains one linear projection and three ID convolution layers, transforming the original sEEG patches into patch embeddings with = 160. The following "Transformer Encoder" contains an 8-layer Transformer encoder with model dimension d = 160, inner dimension (FFN) d5r = 320, and 8 attention heads. See Appendix C for more details.
Pre-training. During the pre-training, we use either all sEEG recordings (15 hours) or the sEEG recordings without task recordings (12 hours) to train the Du-IN VQ-VAE and Du-IN MAE models. To enhance the robustness of the learned codex and representations, we further use data augmentation described in Appendix [ For each subject, the model is pre-trained Linux system with 2 CPUs (Intel Xeon Gold 6230 40-Core Processor) and GPU (NVIDIA Tesla V100 32GB) for ~ 1.2 days.
Fine-tuning. During the downstream evaluation, we split the task recordings into training, valida- tion, and testing splits with a size roughly proportional to 80%, 10%, and 10%. All experiments are conducted on the same machine with the same set of random seeds. The train/ train/validation/tes splits are the same across different models. We also use data augmentation, as described in Appendix [ to make the most of the gathered dataset. We employ cross-entropy loss (multi-class classification) as the training loss. Our experiments are conducted on one V100 GPU by Python 3.11.7 and Py PyTorch 2.1.2 CUDA 12.3. The best models are trained based on the training set, selected from the validation set according to accuracy, and finally evaluated on the test set. For model comparison, we report the average and standard eπrоr values (of all subjects) six different random seeds to obtain comparable results. For the results of the subject-wise evaluation, we report the average and standard deviation values (of each subject) in Appendix K
4.3 Channel Contribution and Selection
As demonstrated in Section 2.1 previous neuroscience studies reveal that vocal production predomi- nantly engages specific brain regions. Given the sparse distribution of implanted sEEG electrodes (each containing 8-16 channels), it's vital exclude redundant electrodes unrelated vocal produc- tion, thus improving decoding performance. We retain electrodes implanted in relevant brain regions and evaluate the performance based on the remaining electrodes. Table demonstrates that excluding approximately 85% electrodes even leads to dramatic increase in decoding performance.
Table 1: The performance of Du-IN with or without electrode selection.
Methods
of Channels (Averaged) Accuracy (%) + Ste (%)
Du-IN (w/o electrode selection) Du-IN (w/ electrode selection)
109.75
30. 12+5-66.
12.25
55.92:14.96
To further understand the detailed contribution of each channel, we analyze the weights of linear projection in the spatial encoder. In detail, we calculate the contribution scores of channels per subject and organize them accordingly, as described in Appendix]] H.] Figure demonstrates that (1) the brain regions effective for speech decoding align with findings from previous neuroscience research, and (2) our model achieves optimal decoding performance with approximately 10 channels, 80% of which originate from the same electrode. To streamline, we utilize these top 10 channels (selected according to train-set) for both pre-training and downstream evaluation.
4.4 Comparasion with Other Models
Table presents the results of our Du-IN model and the advanced baselines that are designed for either brain signals or general time series. See Appendix B and Appendix C33 for detailed descriptions of models. The results demonstrate that our Du-IN model outperforms all baselines. It's worth noting that the models (i.e., the foundation models designed for brain signals) that adopt spatial-temporal integration to model spatial relationships among channel-level tokens perform worse than the models that adopt temporal modeling based on region-level tokens, challenging the generalizability current strategies to model spatial relationships among channels with Transformer.
7

---

a
(b)
Ablations on of Channels
left
right
60
53
0.75
Aceuracy (%)
50
0..
45
0.25
40
35
0
# of Channels
Figure 4: The channel contribution analysis. (a). The channel contribution map. (b). The effect of the number of channels (sorted according to channel contribution scores) on decoding performance.
Table 2: The performance of different methods (with the best bold and the second underlined).
Config
Methods
Token Level
Model Size
Accuracy (%) + Ste (%)
PT
MS²
TS-TCC 20
Region
x
0.32M
24.85
CNN-BiGRU 37
Region
X
0.54M
32.04:15.45
BEG-Conformer
Region
X
2.34M
45.82:14.66
Neuro-Brel BerooBBRRT5 50
Region
x
2.14M
49.51
DeWave 19
Region
X
5,70M
32.4314.48
BrainBERT 49
Channel
X
43.58M
6.72=1.59
BrainBERT 49
Channel
43.58M
7.50=1.76
Brant 54
Channel
X
69.35M
11.1613.55
Brant 54
Channel
X
69.35M
12.42+4.110
LaBraM 28
Channel
6.85M
11.53++2.6
LaBraM-PopT 28
10
Channel
6.85M
11.78+2..70
Du-IN
Region
x
4.38M
56.29+55.20
Du-IN (vqvae+vq)
Region
X
4.38M
44.17
Du-IN (vqvae)
Region
x
4.38M
58.2444.83
Du-IN (mae)
Region
4.38M
62.70 4.69
Du-IN (poms)
Region
5.18M
59.18+4..63
PT: Whether the model is pre-trained before evaluation.
MS: Whether the model is pre-trained across multiple subjects.
As BrainBERT [49 doesn't 't consider the spatial relationships among channels, we mainly focus on understanding why Brant [54], LaBraM (28 and LaBraM-Popt 28 I0 fail to effectively capture the discriminative features on the speech decoding task. These models typically build channel-level tokens by segmenting non-overlapping patches with large receptive fields (e.g., second) from single channels. However, this approach makes it challenging capture the rapid process of brain dynamics. Moreover, while these models further utilize Transformer to capture the spatial relationships among these tokens, they do not encourage region-level embeddings, either through their architecture [52) their pre-training objective [TO]. Therefore, the effectiveness of building brain foundation models based these spatial-tempora backbones is still under exploration, especially for cognitive tasks (e.g., speech decoding), which are great value in the field neuroscience.
Besides, unlike LaBraM (28), Brant doesn't introduce spatial embeddings to identify the spatial location each channel. Since electrodes are sparsely distributed in the brain and the raw sEEG signals on the same electrode are highly correlated, it's fairly easy to identify their spatial relationships through their values. As demonstrated iTransformer (34], this modeling approach is well suited for detecting time-delay events, c.g., seizure detection. For speech decoding tasks,

---

sEEG often requires bi bolpa re-reference (or Laplacian re-reference) to remove the high correlations among channels, thus avoiding model overfitting 49 Once the correlations among channels have been removed, Brant will lose the ability to model spatial relationships among channels.
For other baselines that use temporal modeling based on region-level tokens, provide detailed explanation of their performance differences as follows. TS-TCC [20] tokenizes raw sEEG signals into region-level tokens with a stack of ID depthwise convolution blocks, but it lacks a temporal Transformer for further integration over time. CNN-BiGRU (37 introduces stack of GRU layers on top of these tokens to perform temporal integration. EEG-Conformer 44 introduces temporal Transformer to better integrate global temporal information, which makes it outperform CNN- BiGRU. However, EEG-Conformer tokenizes raw sEEG signals with the temporal-spatial convolution, applying the same convolutional kernel across different channels, which overlooks the specificity of brain computation 8 This also raises challenge for the effectiveness of current sEEG foundation models, which rely on shared convolution blocks across individual channels. Neuro-BERT [50] further introduces mask modeling to learn contextual embeddings, which makes it outperform EEG- Conformer. Wave [19] utilizes the Conformer model (24] for tokenization, which involves more parameters but is less effective than ID depthwise convolution.
4.5 Ablation Study
Superviipo Initialization. As illustrated in Figure 3 the Du-IN model entails a two-stage pre-training process, wherein both the Du-IN VQ-VAE model and the Du-IN MAE model are trained. Previous studies utilize different strategies 19 12) 28 to leverage these pre-trained models to enhance the performance of downstream tasks. Here, we evaluate these different strategies for comparison; see Appendix C.3 for detailed definitions. Table shows that initializing weights from the Du-IN MAE model captures contextual embeddings effectively, resulting in the highest decoding performance.
Pre-training with/without Downstream Datasets. During the pre-training stage, hope that the Du-IN VQ-VAE model can extract general tokens of that brain region, thus guiding the Du-IN MAE model learn general representations that are not specific to any particular task. Although no label data is used during the pre-training stage, to eliminate the influence of the pre-training data on downstream tasks, compare the results with or without incorporating the downstream task dataset into the pre-training stage. Table 3 shows slight performance drop when excluding downstream datasets. However, the decoding performance is still higher than the baseline performance without pre- pre-training, which means that the degradation is mainly due to the decrease of the pre- pre-raining dataset. We hope that, with more pure recordings, our model can achieve better decoding performance.
Table 3: Ablation study on whether pre-training with the downstream dataset (DD) or not, Accuracy ( (%) = Ste (%)
Methods
Pre-training Dataset Size
Du-IN (mae w/o DD)
12 hours per subject
60.02+4..4
Du-IN (mae w/ DD)
15 hours per subject
62.70-L4.69
Discrete Codex. During the Du-IN VQ-VAE training stage, the Du-IN VQ-VAE model encodes SEEG patches into discrete codes and then reconstructs the original signal from these codes. We evaluate performance against varying codex sizes (512 to 8192) to ascertain if codex size affects the quality of the learned codex. As illustrated in Figure 5 while extremely small codex size lacks representation diversity, extremely large codex size often leads codex collapse. We suspect that our existing training data might not be adequate for larger codex sizes. Furthermore, our experiments suggest that the model performs optimally when the codex dimension, denoted as deodez = 64, is slightly less than the model dimension, d = 160, yielding a more effective regularization effect.
Perception Time Window. We also conduct the ablation study on the model structure for the spatial encoder described in Section 3.2 As the spatial encoder transforms the sEEG signals within a given patch to patch embedding, it compresses the sEEG signals for perception. As described in Section 4.2 the model utilizes a receptive field of 100ms. We conduct an ablation study of different receptive fields and report it in F The model performance notably drops with a receptive field smaller than 60ms and gradually declines the receptive field exceeds 160ms. The model reaches

---

small peak around 100ms to 140ms. We think this phenomenon is rational since sEEG is known for its ability capture the rapid dynamics of specific brain regions precisely.
(a)
65.00
Ablations on Codox Sizes
65.00
Ablations on Codex Dimension
(c)
65.00
Ablations on Recentive Field
Acorncy (99)
62.25
Accosta (%
62.25
62.25
Accuncy
60.00
60.00
57.75
57.75
57.75
55.00
1824
6144
8192
55,00
52
128
55,00
140
180
220
Codex Sizes
Codex Dimension
Receptive Field (ms)
Figure 5: Ablation study on different codex sizes, codex dimensions, and receptive fields.
5
Limitations
Despite Du-IN's enhancements in speech decoding via discrete codex-guided mask modeling, is still restricted to close-set speech decoding tasks (i.e., the word set only includes 61 pre-determined words). However, a parallel to our work, Feng et al. (21). which follows previous works [26] (4). build an acoustic-inspired model that can decode arbitrary Chinese words by predicting syllable components (initials, finals, tones). Although their method requires large amount of labeled data, their experimental design mirrors ours closely. The difference lies in the requirement for the subject to repeat syllable components, instead of entire words. Therefore, with slight modifications, our model can support open-set speech decoding tasks.
Additionally, the experiments this paper are restricted to the vocal production part of language decoding, i.e., speech decoding. A more interesting but difficult task to decode language from the semantic level, in which large language models have been wildly used to improve the model performance 46 T9. However, due the locality of sEEG recordings, it is still under exploration whether sEEG recordings can fully capture semantic-related information across brain regions.
Conclusion
This paper proposes Du-IN, a framework for speech decoding, which learns contextual embeddings through discrete codex-guided mask modeling specific brain regions. To evaluate our model, we collect well-annotated Chinese word-reading sEEG dataset to address the lack of sEEG language dataset. Inspired by neuroscientific findings, we analyze the effective brain regions for speech decoding and achieve the best decoding performance with about one electrode specific brain regions, which dovetails with the past neuroscientific research on language. Comprehensive experiments demonstrate that our model outperforms both supervised and BEEG-based self-supervised baselines, effectively capturing the intricate processing within specific brain regions. It marks promising neuro-insping AI approach in BCI. In the end, we hope our work can have implications for future developments in sEEG-based self-supervised models with more consideration over how to build the basic representation units so that the model can maximally benefit from the pre-training stage.
7
Broader Impacts
Our method advances the feasibility of invasive BCI technology by being the first to demonstrate speech decoding using a single sEEG electrode, which holds significant potential for clinical ap- plications. For patients who have lost their ability to communicate or perform daily tasks due to neurological conditions like locked-in syndrome amyotrophic lateral sclerosis (ALS), our approach offers a less invasive alternative to technologies like ECoG or microelectrode arrays, thereby reducing the risk of brain damage.
10

---

Acknowledgements
This study was supported by the National Science and Technology Innovation 2030 Major Program (2022ZD0205500), the National Natural Science Foundation of China (32271093), the Beijing Natural Science Foundation (Z230010, L222033), and the Fundamental Research Funds for the Central Universities. We would like to extend our sincere appreciation to Dr. Zhi-Feng Yue for his coordination and support in securing the computing resources essential for this study. Besides, we also sincerely appreciate the LaBram 28 team for their valuable discussion on the visual design of Figure Banreaandd and Figure
Ethics Statement
Experiments that contribute to this work were approved by IRB. All subjects consent to participate. All electrode locations are exclusively dictated by clinical considerations.
Our informed consent signing process is as follows:
1. If the experimental participants are adults and have full civil capacity, we will ask them to sign a written informed consent after the participants have fully informed consent;
2. If the experimental participants are minors or do not have full civil capacity, we will ask the participant's legal guardian sign written informed consent after the participants and their legal guardians have fully informed consent.
Our informed consent form includes the following points:
1. Contact information of research institutions and researchers;
2. Research direction and purpose;
3. Risks involved in the research;
4. Personal information, data and usage methods be used in the research;
5. Privacy protection statement (all personal identification information (PII) will not be dis- closed);
6. Data storage statement (retained after deleting all personal identification information (PII));
7. Voluntary statement of participants;
8. Statement that participants can withdraw unconditionally at any time.
Our data storage and protection procedures include the following processes:
1. Our data collection, transfer, and analysis tasks are only completed by researchers who have signed relevant confidentiality agreements;
2. The collected raw data will be copied twice soon possible, one copy to storage computer that is not connected to the Internet and encrypted, and the other copy to a mobile hard disk and encrypted and stored offline;
3. The use of the data is only authorized the research leader and the main researchers (less than 5 people), among which the main researchers can only access data that does not contain personal identification information (PII);
4. After the study is completed, all personal identification information (PII) on both nodes (storage computer, mobile hard disk) will deleted immediately.
To prevent unauthorized access possible data leakage, computer, that is, static password and a dynamic password (received by mobile phone or email); physical isolation is used on the mobile hard disk, that is, it is locked in filing cabinet, and the key only kept by the research leader and the main researchers.
double encryption the storage
References
[1] Miguel Angrick, Maarten Ottenhoff, Lorenz Diener, Darius Ivucic, Gabriel Ivucic, Sophoeles Goulis, Jeremy Saal, Albert J Colon, Louis Wagner, Dean J Krusienski, al. Real-time

---

synthesis of imagined speech processes from minimally invasive recordings of neural activity. Communications biology, 4(1):1055, 2021.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprini arXiv:2106.082555 2021.
[4] Jeffrey R Binder, Julie A Frost, Thomas A Hammeke, Robert Cox, Stephen M Rao, and Thomas Prieto. Human brain language areas identified by functional magnetic resonance imaging. Journal of Neuroscience, 17(1):353-365 1997.
[5] Kristofer E Bouchard, Nima Mesgarani, Keith Johnson, and Edward F Chang. Functional organization of human sensorimotor cortex for speech articulation. Nature, 495(7441):327-332, 2013.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
[7] Gyorgy Buzsaki. Rhythms of the Brain. Oxford university press, 2006.
[8] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3):430-44)) 2023.
[9] Josh Chartier, Gopala K Anumanchipalli, Keith Johnson, and Edward F Chang. Encoding of articulatory kinematic trajectories in human speech sensorimotor cortex. Neuron, 98(5):0042- 1054, 2018.
[10] Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedar- madji. Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning population- level representations of intracranial activity. ArXiv, 2024.
[11] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference machine learning, pages 1691-1703. PMLR, 2020.
[12] Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang. Yifan Wang, Dongsheng Li, and Lili Qiu. Eegformer: Towards transferable and interpretable large-scale eeg foundation model. arXiv preprint arXiv:2401.10277 2024.
[13] Cheol Jun Cho, Edward Chang, and Gopala Anumanchipalli. Neural latent aligner: cross-trial alignment for learning representations of complex, naturalistic neural data. In International Conference Machine Learning, pages 5661-5676 PMLR, 2023.
[14] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[15] Rahul Desikan, Florent Ségonne, Bruce Fischl, Brian Quinn, Bradford C Dickerson, Deborah Blacker, Randy L Buckner, Anders M Dale, R Paul Maguire, Bradley T Hyman, et al. An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage, (13):66--9800 2006.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180.004000 2018.
[17] Benjamin K Dichter, Jonathan D Breshears, Matthew K Leonard, and Edward F Chang. The control of vocal pitch in human laryngeal motor cortex. Cell, 174(1):21-31, 2018.
12

---

[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[19] Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin. Dewave: Discrete eeg waves encoding for brain dynamics to text translation, arXiv preprint arXiv:2309. 14030, 2023.
[20] Emadeldeen Hldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2 106. 141 12, 2021.
[21] Chen Feng, Lu Cao, Di Wu, En Zhang, Ting Wang, Xiaowei Jiang, Heng Ding, Chenhao Zhou, Jinbo Chen, Hui Wu, et perf-pperormance brain- sentence communication designed for logosyllabic language. bioRxiv, pages 2023-11, 2023.
[22] Navid Mohammadi Foumani, Geoffrey Mackellar, Soheila Ghane, Saad Irtza, Nam Nguyen, and Mahsa Salehi. Eeg2rep: enhancing self--uper eeg representation through informative masked inputs. arXiv preprint 17X7v 2024.
[23] Pierre Guetschel, Thomas Moreau, and Michael Tangermann. S-jepa: towards seamless cross- dataset transfer through dynamic spatial attention. arXiv preprint arXiv:2403. 2024.
[24] Anmol Gulati, James Qin, Chung-Chery Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang. Yonghui Wu, et al. Conformer: Convolution-augmennte transformer for speech recognition. arXiv preprint arXiv:2005.08 2020.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[26] Christian Herff, Dominic Heger, Adriana De Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk, and Tanja Schultz. Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers in neuroscience, 9:217, 2015.
[27] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456, pmlr, 2015.
[28] Weibang Jiang, Liming Zhao, and Bao liang Lu. Large brain model for learning generic representations with tremendous EEG data in BCI. The Twelfih International Conference on Learning Representations, 2024.
[29] Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Are eeg-to-text models working? arXiv preprini urXi::2405.06455 2024.
[30] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013.
[31] Demetres Kostas, Stephane Kroca-Ouelleds and Frank Rudzicz. Bendr: using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers Human Neuroscience, 15:653659, 2021.
[32] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. arXiv preprint a91iv:1910. 13461, 2019.
[33] Guangye Li, Shize Jiang, Sivylla E Paraskevopoulou, Meng Wang, Yang Xu, Zehan Wu, Liang Chen, Dingguo Zhang, and Gerwin Schalk. Optimal referencing for stereo- elettreencephalographic (seeg) recordings. Neurolmage, 183:327-335, 2018.
13

---

[34] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.0662.. 2023.
[35] Yuchen Liu and Ziyu Jia. Bstt: A bayesian spatial-temporal transformer for sleep staging. In The Eleventh International Conference on Learning Representations, 2022.
[36] Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian E Dougherty, Jessie Liu, Peter Wu, Michael Berger, et al. A high- performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976):1037- 1046, 2023.
[37] David A Moses, Sean L Metzger, Jessie R Liu, Gopala K Anumanchipalli, Joseph G Makin, Pengfei F Sun, Josh Chartier, Maximilian Dougherty, Patricia M Liu, Gary M Abrams, et al. Neuroprosthesis for decoding speech in paralyzed person with anarthria. New England Journal of Medicine, 385(3):177-277, 2021.
[38] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-quantized visual tokenizers. 2022.
[39] Alec Radford, Karthík Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[40] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[41] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al.
A deep learning framework for neuroscience. Nature neuroscience, 22 (111:1776111770, 2019.
[42] Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Neuroscience, 22(1):55 67, 2021.
[43] Jingwei Sheng. Li Zheng, Bingjiang Zhehang Cen, Lang Qin, Li Hai Tan, Ming-Xiong Huang, Nai Ding, and Jia-Hong Gao. The cortical maps of hierarchical linguistic structures during speech perception. Cerebral cortex, 3233-33333 2019.
[44] Yonghao Song, Qingqing Zheng, Bingchuan Liu, and Xiaorong Gao. Eeg conformer: Convolu- tional transformer for eeg decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:710-719, 2022.
[45] Pedram Z Soroush, Christian Herff, Stephanie K Ries, Jerry J Shih, Tanja Schultz, and Dean J Krusienski. The nested hierarchy of overt, mouthed, and imagined speech activity evident in intracranial recordings. Neurolmage, 269:1 19913, 2023.
[46] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):8588866 2023.
[47] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[49] Christopher Wang, Vighnesh Subramanian, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Brainbert: Self-supervised representation learning for intracranial recordings. arXiv preprint arXiv: /4367, 2023.
[50] Di Wu, Siyuan Li, Jie Yang, and Mohamad Sawan. Neuro-bert: Rethinking masked autoen- coding for supevvsee neurological pretraining. IEFE Journal of Biomedical and Health Informatics, 2024.
14

---

[51] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. The eleventh international conference on learning representations, 2022.
[52] Ke Yi, Yansen Wang, Kan Ren, and Dongsheng Li. Learning tagology eeg representa- tions with geometry- aware modeling. Advances in Neural Information Processing Systems, 36, 2024.
[53] Zhizhang Yuan, Daoze Zhang, Junru Chen, Geifei Gu, and Yang Yang. Brant-2: Foundation model for brain signals. arXiv preprini 2402:22002,1251,, 2024.
[54] Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foundation model for intracranial neural signal. Advances in Neural Information Processing Systems, 36, 2024.
[55] Hui Zheng. Zhongtao Chen, Haiteng Wang, Jianyang Zhou, Lin Zheng, and Yunzhe Liu. Universal sleep decoder: Aligning awake and sleep neural representation across subjects. arXiv preprint arXiv: 16457, 2023.
15

---

A Experiment Design
0.5s
2s
0.5s
Time
Next Trial
Prepare
Read
Rest
Figure 6: The experiment design of our sEEG word-reading task.
Due to the lack of open-source sEEG datasets related to language tasks, we follow the experimental design outlined by Moses et al. [37 to collect well-annotated Chinese word-reading sEEG dataset, including 12 subjects (9 male, 3 female; aged 15-53, 27.8, 10.4) with pharmacologically intractable epilepsy.
In the word-reading task, the subject speaks aloud individual words from 61-word set while we simultaneously record his brain activity (measured by sEEG) and voice. The word is chosen based on the following criteria:
The versatility of the words generating a range of sentences.
The simplicity of using the words express fundamental caregiving requirements.
The diversity of word pronunciations to many Chinese pronunciation combinations as possible.
list of the words contained in this 61-word set is provided in Table
data are collected series of "blocks" (25 blocks in total), with each block lasting about 10 minutes and consisting of multiple trials. During each block of this task, all words (from the 61-word set) are presented individually twice, leading to a total 122 trials.
Each trial in a block of this task starts with one word shown on the screen in white text. After 0.5 seconds, the text will turn green and remain on the screen for seconds. This color transition from white to green represents the go cue for each trial, and the subject is instructed to speak the word aloud as soon as the text turns green. Afterward, the text will be replaced with a blank screen with centered cross. After 0.5 seconds, the task continues to the next trial. The word presentation order is randomized within each task block.
Besides, we also collected non-task recordings of subjects in their daily life. Apart from sleep periods, there are roughly 12 hours of non-task recordings during wakefulness. In summary, for each subject, we collect about 15 hours of sEEG recordings, of which 3 hours are task recordings.
16

---

Table 4: The Chinese words and their corresponding English translations in the 61-word set,
Words
Translations
Words
Translations
Words
Translations
mouth
pineapple
#b
help
1
get
Н
friend
washbasin
T
calm
1 A pretty
clothes
tofu
X
rice
HOZE
put on
noodle
E It
towel
*[1
close the door
computer
X T
stool
1)7]
knife
4
headache
gummies
A
vinegar
#
vegetables
toilet
72
chopped green onion
F.L
cell phone
basketball
piano
C
mood
JT.
loofah
garlic paste
how
Hig
sausage
need
you
hold
orange juice
#k
look for
f
[A]
pork
cat
T
wear
be
TR A
family
18 #
hot water
3 +
nurse
AA gij
change dressing
15
drink
thirsty
E
look
10
bowl
f Ht
steak
T
feel
give
3
play
[n k
problem
takeouts
have
5R
music
T0
reserve
sweet dumpling
willing
R
17

---

B Details of Baselines
In experiments, we compare our model the existing supervised or self-supervised methods on brain signals. The details of these baseline models are given here:
TS-TCC20)) A self-supervised model that consists only of CNN module to capture local features. This model learns robust temporal and discriminative representations from time series by designing a tough cross-view prediction task and a contextual contrasting module. Since sEEG is a unique type of time series, this model is suitable to serve as a baseline for comparison.
CNN-BiGRU A supervised model that consists of both CNN module and Bi-GRU module, to capture contextual features from EEG signals. This model is mainly designed for ECoG-based vocal production tasks, similar to ours. Since BCoG and sEEG are both intracranial neural signals of the brain, this model is suitable to serve as a baseline for comparison.
HEEG-Conformer//e A supervised model that consists of both CNN module and Trans- former module, to encapsulate local and global features in a unified EEG classification framework. EEG-Conforme is mainly designed for EEG-based motor imagination tasks, Since the data modes of EEG and sEEG are similar, and the signals primarily pertain to vocal production, this model is suitable serve baseline for comparison.
Neuro-BERT(Sin A self-supervised model that consists of both CNN module and Trans- former module, to encapsulate local and global features. This model learns robust contextual representations from EEG by introducing mask modeling. Since the data modes of EEG and sEEG are similar, this model is suitable to serve as a baseline for comparison.
We : A supervised model that consists of both Conformer module (24] and Trans- former module, to encapsulate local and global features for language decoding. We adopt its encoder, which consists of a 6-layer Conformer and a 6-layer Transformer. Then, we add a classification head, which is also used in our model, for downstream word classification. Since Wave is also designed for language decoding, this model is suitable to serve as a baseline for comparison.
BrainBERT [9] A self-supervised model for SEEG recordings that bridges modern repre- sentation learning approaches neuroscience. BrainBERT builds universal representation based on the superlet spectrograms of one single sEEG channel without modeling the spatial relationships among channels. Since the downstream tasks for BrainBERT are also related to language decoding (e.g., sentence-onsel detection, speech vs. non-speech detection, etc.), this model is suitable serve as a baseline for comparison.
Brant A self-supervised model for sEEG recordings that capture both long-term temporal dependency and spatial correlation from neural signals. Brant is mainly designed for medicine, serving sEEG foundation model. Although Brant mainly evaluates its performance on the low-level modeling tasks [51 (e.g.. neural signal forecasting, imputation, etc.), Brant achieves SOTA performance some high-level modeling tasks (e.g., seizure detection). As foundation model in SEEG pre-training field, this model is suitable to serve baseline for comparison.
LaBraM (2 A self-supervised model for EEG recordings that learns generic representa- tions with tremendous EBG data. LaBraM serves as an EEG foundation model, achieving SOTA performance on various downstream EEG tasks. Since the spatial embeddings are pre-defined according to the EEG caps, LaBraM can only be trained within one subject under the sEEG setting. Since the data modes of EEG and sEEG similar, this model is suitable to serve as a haseline for comparison.
self-supervised model based on LaBraM, simply replacing the learnable spatial embeddings with hard-coded spatial embeddings from PopTT [10 to enable multi-subject pre-training under the sEEG setting.
LaBraM+PopTT 28 10
The detailed implementations of these baseline models are given here:
For the TS-TCC method 20), the hyper-parameter are optimized for better performance, they also have different hyper-parameter settings for different datasets in their original
18

---

implementation. The data samples are resampled to 400Hz. The sizes of convolution kernels
are changed to (25, 8) (other attempts include {8, 8. 8). [15, 8, 8). (20. 8, 8), and [30, 8)); the sizes of pooling kernels are changed to (10, 2, 2] (other attempts include [2, 2, 2},
(5, 2, 2), and (20, 2. 2)); the numbers of pooling strides are changed to (10, 2, 2) (other
attempts include (2, 2, 2), (5, 2, 2), and (20, 2, 2)). All other hyper-parameters are the same as the original implementation.
For the CNN-BiGRU method [77, the syper-parameters the same as the original implementation. The data samples are resampled to the specified sampling rate.
For the EFG-Conformer method implementation. The data samples are resampled to the specified sampling rate.
the hyper-parameters are the same as the original
are optimized for better perfor- mance, they also have different hyper-parameter settings for different datasets in their original implementation. The data samples are sampled to 400Hz. The sizes of convolution kernels changed to (40.) (other attempts include (20.) and (80,)) 1); the numbers of
For the Neuro- BERT method [50,, the
convolution strides are changed to (40.) (other attempts include (20.) and (80,)).
For the Weveve method 19. the hyppr-parameters are the same as the original implementa- tion. The data samples are resampled to the specified sampling rate.
are optimized for better performance. We change the "nperseg" and "noverlap" arguments of secipy.signal.ste function from (400, 350) to 11600, 1400) (other attempts include {200, 175). (800, 700) and (3200, 2800)):
For the BRainBERT method [49]. the
For the Brant method [54], the hyper-parametees are optimized based on the Brant- Brant-Tiny model for better performance. We change the length of the patch segment from 6 seconds second. Besides, we change the linear embedding layer to the convolution embedding layer, which is also used in LaBraM (28). The numbers of convolution filters are (96, 96, 96} (other attempts include (192, 192, 192]); the sizes of convolution kernels are {9, 9, 3) (other attempts include (19, 3) and (9. 9, 3)); the numbers of convolution strides are (5, 5, 1} (other attempts include (10, 5, 13) and (5, 2))..
For the LaBraM method tion of the LaBraM-Baa model. The data samples are resampled to the specified sampling rate.
(28). the
the same the original implementa-
For the LaBraM-Popl method 28 10 the paree-paraeeters are the same as the original implementation of the LaBraM-Base model. The data samples are resampled to the specified sampling rate.
When evaluating the decoding performance of these baseline models, we follow the same experiment setup of the Du-IN CLS model:
For subject, split the downstream dataset into training, validation, and testing splits with a size roughly proportional to 80%, 10%, and 10%.
The data samples are 3 seconds with the specified sampling rate corresponding to each model.
The samples in the train-set are augmented following the pipeline defined in Appendix D
For the self-supervised methods, the pre-training setup follows the original setup of each model:
For the TS-TCC model,
all sEGG recordings for each subject to pre-train it. The data samples are 4 seconds.
For the Neuro-BERT model, we use all sEGG recordings for each subject to pre-train The data samples are 4 seconds.
For the BrainBERT model, we use around 180 hours of sEEG recordings from either each subject or 12 subjects for pre-training. This pre-training dataset is larger than the one (approximately 45 hours) used in the original paper. The data samples are seconds.
For the Brant model, we also use all sEEG recordings from either each subject or 12 subjects to pre-train it. While the total pre-training dataset is smaller than the one (around 2700 hours) used in the original paper, the number of subjects (i.e., the number of sEEG location configurations) is greater than in the original paper. The data samples are 4 seconds.
19

---

For the LaBraM model, we use all sEGG recordings for each subject to pre-train it. The data samples are 4 seconds.
For the LaBraM-PopT model, all sEEG recordings from 12 subjects pre-train it. The data samples are seconds.
20

---

C Model Details
C.1 Du-IN VQ-VAE
The architecture of the Du-IN VQ-VAE model contains three parts: (1) Du-IN Encoder, (2) Vector Quantizer, and (3) Du-IN Regressor. The overall architecture of "Du-IN Encoder" is shown in Figure
The "Vector Quantizer" is implemented similarly in LaBraM288) The "Du-IN Regressor" contains:
Transformer Decoder: A stack of Transformer layers.
Time Regression Head: A stack of 1D Transposed Convolution layers and linear
projection layer.
The hyperparameters for Du-IN VQ-VAE training shown in Table 5
Table 5: The hyperparameters for Du-IN VQ- VAE training.
Module
Sub-Module
Name
Value
Linear Projection
16
of Input Channels
[16, 116,128,1288
of Output Channels
[128,128,16,
Spatial Encoder
Kernel Size
(19,3,3)
Stride
(10,1,1]
Padding
(9,1,1)
of Transformer Layers
Du-IN Encoder
Hidden Size
160
MLP Size
320
Transformer Encoder
MLP Dropout Ratio
[0.2.0.)
of Attention Heads
Attention Head Size
64
Attention Dropout Ratio
0.2
Codex Size
2048 x 64 160())an)) + 64
Vector Quantizer
Embeddiin-tto-ouue Projection
160
Projection
64 > 160
# of Transformer Layers
Hidden Size
160
MLP Size
320
Transformer Decoder
MLP Dropout Ratio
(0.2,0.)
# of Attention Heads
Attention Head Size
64
Attention Dropout Ratio
0.2
Du-IN Regressor
of Input Channels
(160,128, 128, 128)
of Output Channels
Kernel Size
(3.3,10,9,19]
Time Regression Head
Stride
(1.1.10,1.10)
Padding
Output Padding
Linear Projection
16 + 10
Batch Size
64
Maximum Learning Rate
3e-4
Minimum Learning Rate
5e-5
Learning Rate Scheduler
Cosine
Optimizer
Optimizer Type
Adam W
Adam 8
(0.9, 0.99)
Weight Decay
0.01
Total Epochs
400
Warm-up Epochs
40
21

---

C.2 Du-IN MAE
The architecttre of the Du-IN MAE model contains two parts: (1) Du-IN Encoder, and (2) Token Prediction Head. The overall architecture of the "Du-IN Encoder" is shown in Figure 2 The hyperparameters of "Du-IN Encoder" are the same those in Du-IN VQ-VAE. It's worth noting that when training Du-IN MAE, the weights of the "Du-IN Encoder" are randomly initialized, instead of loaded from the pre-trained Du-IN VQ-VA VAE model. The hyperparameters for Du-IN MAE training are shown in @
Table The hyperparameters for Du-IN MAE training.
Module
Sub-Module
Name
Value
Token Prediction Head
Linear Projection
160 - 2048
Batch Size
64
Maximum Learning Rate
3e-4
Minimum Learning Rate
5e-5
Learning Rate Scheduler
Cosine
Optimizer
Optimizer Type
AdamW
Adam 8
(0.9, 0.99)
Weight Decay
0.05
Total Epochs
400
Warm-up Epochs
40
C.3 Du-IN CLS
The architecture of the Du-IN CLS model contains two parts: (1) Du-IN Encoder, and (2) Label Prediction Head. The overall architecture of the "Du-IN Encoder" is shown in Figure 2 The hyperparameters of "Du-IN Encoder" are the same those in Du-IN VQ-VAE. It's worth noting that the "Du-IN Encoder" weights Du-IN CLS can be loaded from either the pre-trained Du-IN MAE or the praited Du-IN VQ- VAE. In the ablation experiments shown in Table our models have different suffixes:
Du-IN: The original Du-IN CLS model. All weights of this model are randomly initialized.
Du-IN (vqvac+vq): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the pre-trained Du-IN VQ-VAE. When fine-tuning on downstream task, the "Vector Quantizer" in the pre-trained Du-IN VAE is inserted between "Du-IN Encoder" and "Label Prediction Head". This is the same operation DeWave
Du-IN (vqvae): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the pre-trained Du-IN VQE. This is the same operation in BRGFormer [12].
Du-IN (mac): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the pre-trained Du-IN MAE This is the same operation LaBraM 28
Du-IN (poms): The weights of the "Du-IN Encoder" in the Du-IN CLS model are loaded from the Du-IN MAE, which is pre-trained on multiple subjects. The modification of the Du-IN VQ-VAE and the Du-IN MAE to support multi-subject pre-training includes (1) initializing different spatial encoders for different subjects and (2) sharing the same transformer encoder and neural codex.
The "Label Prediction Head" is an MLP with one hidden layer, flattens the output embedding sequence from upstream, and maps this feature embedding to the final prediction through MLP. The hyperparameters for Du-IN CLS training are shown in Table
22

---

Table 7:
The hyperparameters for Du-IN CLS training.
Module
Sub-Module
Name
Value
Flatten
Label Prediction Head
Linear Projection
30 x 160 + 128(R)LU) + 61
Batch Size
32
Maximum Learning Rate
2e-4
Minimum Learning Rate
5e-6
Learning Rate Scheduler
Cosine
Optimizer
Optimizer Type
Adam (0.9, 0.99)
Adam 8
Weight Decay
0.05
Total Epochs
200
Warm-up Epochs
20
23

---

D Data Augmentation
To enhance the robustness of learned representations during both the pre-training and fine-tuning stages, we apply data augmentation in both datasets.
Pre-training Dataset. In our implementation, we segment sEEG recordings into 8-second samples with 4-second overlap. When fetching sample, randomly select a starting point between 0 and 4 seconds, then extract a 4-second sample beginning from that point.
Downstream Dataset. Since trial lasts for 3 seconds, employing the jittering mentioned above leads to the blending of information from other trials. In our implementation, we segment sEEG recordings into 3-second samples. When fetching sample, randomly choose shift step between O) and 0.3 seconds, then shift the sample either to the left or right, padding it with zeros.
E Du-IN Pre-training Analysis
The pre-training of Du-IN can be interpreted as the training of a variational autoencoder 30 3 Let x denote the original sEEG signal, à the corrupted sEEG by mask, and the neural tokens. Considering the evidence lower bound (ELBO) of the log- ligllike p(z[ū), i.e., recovering the original sEEG signal from its corrupted version:
log p(x,)(,)) 2
Σ (EDD
Ber (log Pv (2//=.)) - - DDKL (xl./uoll/ po) (10)
(ED
Neural Token Reconstruction
where (1) (=(=))) denotes the Du-IN Encoder in the Du-IN VQ-VAE that obtains neural tokens; (2) (z(2) decodes the original sEEG signal given input neural tokens; (3) (=(2(ā) recovers the neural
tokens based on the masked sEEG signal, which is our Du-IN pre-training task.
The whole framework is optimized through two-stage procedure as 47 40 For the first stage,
train the Du-IN Encoder the Du-IN VQ-VAE as discrete variational autoencoder by minimizing the reconstruction loss - 2, (z/z,) log P (=: with uniform prior. For the second stage, set 4s well pe fixed and learn the prior pe by minimizing the loss DKL For simplicity, g0 (z(x)) is defined as a one-point distribution with the most likely neural tokens 23 = arg max do (Z r.).
Consequently,
rewrite Equation 10 as
log p(r.(e))
Σ (DD
Ez [log Pap (f.12;)) +
log po TTp
(11)
(ED
Neural Token Reconstruction
Masked sEEG Modeling
where the first term is the objective for vector-quantized neural signal regression in the first stage (i.e., the Du-IN VQ-VAE model), and the second term is the objective for Du-IN pre-training in the second stage (i.e., the Du-IN MAE model).
F Visualization of Vector-Quantized sEEG Regression
We further visualize how the sEEG signals are reconstructed. depicted in Figure 7 although some details are missing, the overall trend of the signals is reconstructed well. Mcanwhile, there is stable decrease in the reconstruction loss during training, which indicates the discrete codex does learn high-level information from sEEG signals.
G Visualization of Mask sEEG Modeling
Figure g demonstrates the convergence curves of the total pre-training loss and masked sEEG modeling accuracy of the Du-IN MAE model. We observe that there stable decrease in the mask modeling loss, and the mask modeling accuracy achieves about 20%, which is similar to (28).

---

(a)
VQ sEBO Reconstraction Loss
(b)
Original SEEG signals
Recoustructed SEEG signals
1.0
wmwl/Mmnn Ma
(o0g) Wog M M0 HooSA
0.8
0.6
100
200
300
400
Epochs
Figure The visualization of Vector-Quantien sEEG Regression. (a). The reconstruction loss curve during the training process of the Du-IN VQ-VAE model. (b). The visualization of reconstructed sEEG signals.
(a)
Masked sEEG Modeling Loss
(b)
Masked sEEG Modeling Accuracy
25
20
Accuracy (%)
15
Looss
10
5
0
100
200
300
400
0
100
200
300
400
Epochs
Epochs
Figure 8: The loss curve and accuracy curve during the training process of the Du-IN MAE model.
H
Channel Contribution Analysis
For each subject, after training the Du-IN model (with random initialization) on the downstream dataset, we utilize the weights W E RCxD of linear projection in the spatial encoder to calculate the contribution scores S of channels:
D
S = 1 1..., CJ.
1
Σ W.yl
(12)
D
where is the number of channels, D the dimension projected embedding and | gets the absolute value. Then, we normalize S using its maximum value to ensure it falls within the [0. range. Finally, given the variability in model performance across subjects, we further adjust the channel contribution scores based on the decoding performance of that subject, i... S = {s; pli 1,.., C}. where p represents the decoding performance of that subject.
After calculating the channel contribution scores of all subjects, project them to the standard brain template according the MNI (Montreal Neurological Institute) locations of channels, using Nileam 0.9.2. Since the electrodes are sparsely distributed within the brain, we use Scipy 1.8.1 to interpolate and smooth channel contribution matrix and use NiLeam to plot the channel contribution map demonstrated in Figure (a).
With the sorted channels within each subject, we evaluate the effect of the number of channels on the decoding performance. For each subject, we evaluate the Du-IN model with {5, 10, 15, 20, 30, 60) channels (sorted by channel contribution scores), and the averaged performance (across subjects) is demonstrated in Figure((b)
25

---

I Effectiveness of Region-Specifie Channel Selection
DeWave successfully reconstructs 128-channel EEG signals with the same setting of vector- quantizer. However, this is not the case under the sEEG setting, which is shown in Table X It's worth noting that sEEG signals are fundamentally different from EEG signals due to (1) the high information density and (2) the high specificity of different regions. Due to the desynchronization nature 7 of the brain during awake tasks, only specific brain regions are related to tasks. Therefore, only after region-specific channel selection, the Du-IN VQ-VAT model can successfully reconstruct the original signals, thus identifying the fine-grained state of brain regions.
Table 8: Ablations to validate the effectiveness of region-specific channel selection.
Settings
Setting
Setting
Setting 3
MSE
0.2969
0.5211+0.0492
0.9673+0.0148
Setting 1:
Select top-10 channels relevant to speech decoding for neural signal reconstruction.
Setting 2: Randomly select 10 channels for neural signal reconstruction.
Setting 3: Use all channels (109.75 channels average) for neural signal reconstruction.
J
Additional Group-Wir Wise Evaluation
The cross-entropy loss of different methods from each subject is provided in Table with the best in bold and the second underlined. For model comparison, we report the average and standard deviation values (within each subject) on six different random seeds to obtain comparable results. "Std" means standard deviation.
Table 9: The cross-entropy loss of different methods (with the best in bold and the second underlined).
Config
Methods
Token Level
Model Size
Cross-Entropy + Ste
PT
MS
TS-TCC
Region
X
0.32M
3.8871:0.3072
CNN-BiGRU 37
Region
X
0.54M
4.0294=:076211
EEG-Conformer
Region
X
2.34M
3.8165+0.3356
Neuro-BERY 50
Region
X
2.14M
3.641620.4360
DeWave
Region
5.70M
4.1891
BrainBERT 40
Channel
X
43.58M
4.625410.1984
BrainBERT 49
Channel
43.58M
4.6190+0.2132
Brant 54
Channel
69.35M
4.7962+0.7082
Brant 54
Channel
X
69.35M
5.0294 1.0621
LaBraM 28
Channel
6.85M
4.8591+0..2223
LaBraM-Pop'T
28
10
Channel
6.85M
4.6564
Du-IN
Region
X
4.38M
3.5083+0.3003
Du-IN (vqvae+vq)
Region
X
4.38M
3.7244+0.3104
Du-IN (vqvae)
Region
4.38M
L..2781
x
Du-IN (mac)
Region
4.38M
3.3707
Du-IN (poms)
Region
5.18M
3.4429
PT: Whether the model is pre-trained before evaluation.
MS: Whether the model is pre-trained across multiple subjects.
26

---

K Subject-Wise Evaluation
The detailed performance of different methods from each subject provided in Table I0 Table and Table with the best in bold and the second underlined. For model comparison, we report the average standard deviation values (within each subject) on six different random seeds obtain comparable results. "Std" means standard deviation.
Table 10: The performance of different methods from subjects (01-04).
Config
Accuracy (%) + Std (%)
Methods
PT
MS²
subj-01
subj-02
subj-03
subj-04
TS-TCC 20
X
26.90:11.72
61.57=1.21
6.65+1.09
20.66:10.79
CNN-BiGRU 37
XX
46.46+t4.03
68.06+1.56
4.35=0044
17.68
BEG-Conformer 44
X
58.41
69.82+11.22
19.50+11.71
49.65++2.38
Neuro-BERT
V
x
60.441:2.23
72.97=1.47
28.38
52.76=+3.41
Wawa 19
X
43.31+4.80
57.1243.001
3.70++5.35
33.52-+3.98
BrainBERT
5.30
23.49:1.29
2.74+0.40
5.18=0.38
BrainBERT 49
6.76
25.64+1.23
2.97+0.66
5.09+00.4
Brant 54
8.64
47.97
0.06
2.74
Brant 54
7.47
54.26111.63
3.34
1.35
LaBraM 28
12.55#1.117
39.20=11.53
3.54
13.30=1.19
LaBraM-PopT 28 10
14.14
39.50=
3,28+0.55
12.77=11772
Du-IN
71.25+1.444
77.99
23.0444.76
59.91
Du-IN (vqvae+vq)
X
50.1543.80
62.79114.67
20.72:2.15
48.24:2.65
Du-IN (vqvae)
12.36:11555
79.16:11112
29.21
13.83+1183
Du-IN (mae)
X
78.60-10.79
83.61+0.38
38.80
70.98
Du-IN (poms)
73.2340.67
1.01
28.89:
64.24 1.29
PT: Whether the model is pre-trained before evaluation.
MS: Whether the model is pre-trained across multiple subjects.
27

---

Table 11: The performance of different methods from subjects (05-08).
Config
Accuracy (%) + Std (%)
Methods
PT
MS²
subj-05
subj-06
subj-07
subj-08
TS-TCC
X
34.53-H1.34
9.73
24.83++1.15
20.08++:.60
CNN-BiGRU 37
X
51.26::44.3
31.52:
47.75+#1.12
24.6411.44
EEG-Conformer 44
X
65.44
31.06:2.58
1..89+1.86
42.12:22.08
Neuro-BERT 50
V
x
71.61=1.97
36.63:2.15
50.23+2.63
1..24=1.36
DeWave 19
X
45.20:1.008
26.88
38.2442.15
29.15
BrainBERT
9.59
2.67=0.31
4.7940.64
5.10=0.59
BrainBERT 49
11.28
3.00=0.47
5.31
5.22-0.76
Brant [4
24.10:2.11
5.09=1.27
7.74=1.46
8.66=
Brant 54
28.83++1.93
5.28+1148
9.70+1.85
8.93+
LaBraM 28
15.52+ 1.48
6.33
12.55+0.50
7.41 +0.86
LaBraM-PopT 10
17.7341.11
1.61
1..90
6.41
Du-IN
X
77.60++.20
41.91+1.80
52.35 59.63+2220
Du-IN (vqvae+vQ)
x
63.46:42.28
34.8411.98
45.20:12.44
40.14
x
Du-IN (vqvae)
78.56+1.24
13.29
62.29+11.49
54.10
x
Du-IN (mae)
81.56
46.90.11.02
65.45:11.74
59.09-10.98
Du-IN (poms)
76.91+11.38
47.01=11.76
61.68
55.17=1.37
PT: Whether the model is pre- pre-teained before evaluation.
MS: Whether the model is pre-trained across multiple subjects.
Table 12: The performance of different methods from subjects (09-12).
Config
Accuracy (%) + Std (%)
Methods
PT
MS
subj-09
subj-10
subj-11
subj-12
TS-TCC 20
X
37.75+11.22
5.71=
35.72:+0.95
14.12:10.68
CNN-BiGRU
X
44.03-45.885
7.11
28.44:1..42
13.17:33.41
BEG-Conformer 44
X
56.51 1.98
22.22:11.07
57.1042..03
29.87=1144
Neuro-BERY 50
V
x
54.12:4.111
24.6611.288
62.9910.93
36.07=2.336
De DaWave 19
X
41.98.14.60
6.22+0.94
44.60+3.35
19,22-2299
BrainBERT 49
1.82
2.55+0.49
8.73
3.71
BrainBERT 49
7.20
2.49-0.43
10.60+11.22
4.41=0.88
Brant 54
6.82+1.44
2.8410.26
8.76+1.55
7.53
Brant 54
6.46+1.66
3.00=0.31
9.82+1.71
7.82+1.66
LaBraM 28
8.97
3.50+0.30
7.92+0.61
7.19+0.54
LaBraM-PopTT 28
10
9.35 1.09
3.91+00.1
7.84+0.92
7.74+1.24
Du-IN
X
66.39+4.47
27.07+2.24
73.56+1.09
44.76+3.74
Du-IN (vqvae+vq)
60.06+1.61
22.05+1..76
50.31+4.69
32.06
x
Du-IN (vqvae)
67.18+11.22
31.06+1.59
72.41
45.38+2226
Du-IN (mae)
69.18+1.99
34.23+1.117
7..5
0..54
Du-IN (poms)
36.90+1.34 7..71+1.48
72.80+
43.53+3..0
PT: Whether the model is pre-traine me-ttrinee before evaluation.
MS: Whether the model is pre-trained across multiple subjects.
28

---

L Effectiveness of Vector-Quantized Neural Signal Prediction
To verify the effectiivnness of vector-quantized neural signal prediction, we elaborate on two types of experimental settings as illustrated in Table [3] The comparison between Du-IN and Setting demonstrates that the codex effective for masked sEEG modeling. The comparison between Du-IN and Setting 2 demonstrates that introducing the codex can prevent the model from focusing too much on reconstructing details, thus enabling the Du-IN MAE to learn better contextual embeddings.
Table 13: Ablations to validate the effectiveness of vector-quantized neural signal prediction.
Model
Du-IN (mae)
Setting 1
Setting 2
Acc. (%)
Ste (%)
62.70 4.69
60.92++4.35
58.72+-5.02
We directly predict output embeddings of the Du-IN Encoder in the Du-IN VQ-VAB by maximizing cosine similarity instead of predicting the discrete neural tokens from the codex. Setting 2: We discard the Du-IN Encoder in the Du-IN VQ-VAE and directly reconstruct raw EEG patches by minimizing MSE loss.
Setting I:
M Ablation on Mask Ratio
In this experiment, we conduct different mask ratio settings to explore its impact. It is noted that we introduce the symmetric masking strategy, so we only need validate half of the mask ratios. As the mask ratio is set to the symmetric masking will mask - r proportion of sEEG patches. The ablation results are provided in Table [4] It can be induced that the best mask ratio is 0.5 (0.5) for our dataset.
Table 14: Ablations to explore the impact of mask ratios.
Mask Ratio
0.5 (0.5)
0.4 (0.6)
0.3 (0.7)
0.2 (0.8)
0.1 (0.9)
Acc. (%) + Ste (%)]
62.7011.66
60.58
59.58
58.9214.07
58.5543.94
N Ablation on Pre-training Epochs
The impact of the number of pre-training epochs (of the Du-IN VQ- VQ-VAE model) is demonstrated in [5] We use the checkpoints according to the specified epochs to pre-train the Du-IN MAE model for 400 epochs. Once the reconstruction loss of the Du-IN VQ-VAE model converges, the Du-IN VQ- VAE model can extract the state of the brain region well, thus leading to better performance.
The impact of the number of pre-training epochs (of the Du-IN MAE model) demonstrated in Table We use the checkpoints according to the specified epochs for downstream classification. Once mask modeling loss of the Du-IN MAE model converges, the Du-IN MAE model learns robust contextual embeddings, thus leading to better performance.
16
Table 15: Ablations explore the impact of the pre-training epochs (of the Du-IN VQ-VAE model).
# of Epochs
5
10
50
100
400
Acc. (%) + Ste ( (%)
50.02.14.91
52.29455.09
61.09
62.59.14.32
62.70-14.69
Table 16: Ablations explore the impact of the pre-training epochs (of the Du-IN MAE model).
# of Epochs
5
10
50
100
400
Acc. (%) + Ste (%)
57.87
58. 12.494
61.8914.62
62.47
62.70:14.69
29

---

O Subject-Wiid Wise Electrode Locations
We provide detailed information on the locations of the implanted sEEG electrodes for each subject. Red channels are the top 10 channels (selected through channel contribution analysis) for both pre-training and downstream evaluation, as described in Section 4.3 As the majority of subjects have sEEG electrodes implanted only one side of their brains to locate the source of epilepsy, we provide side views of either the left or right brain areas here.
subj-01
L
R
subj-02
L
R
subj-03
L
subj-04
L
R
Figure 9: Electrode locations from subjects (01-04).
30

---

subj-05
L
R
subj-06
subj-07
L
R
subj-08
L
R
Figure 10: Electrode locations from subjects (05-08).
31

---

subj-09
L
R
subj-10
L
R
subj-11
L
R
subj-12
Figure 11: Electrode locations from subjects (09-12) 12).
32

---

P Subject-Wise Selected Channels
The MNI coordinates, and brain region labels (according to Harvard- cortical and subcortical structural atlases [15] for selected channels are listed below. The channels for each subject are arranged in descending order based on their contribution scores.
Table 17: The MNI coordinates and brain region labels of selected channels from subjects (01-04).
MNI coordinate
Subjects
Brain Region
Du-IN Accuracy (%)
x
-57
-16
19
Central Opercular Cortex L
-61
-16
20
Postcentral Gyrus L
-54
-16
17
Central Opercular Cortex L
-67
-15
23
Postcentral Gyrus L
-25
-30
49
White L
subj-01
71.25
-64
-15
21
Postcentral Gyrus L
-51
-17
16
Central Opercular Cortex
22
-31
49
White L
48
-17
15
Central Opercular Cortex
-18
-32
49
White L
33
-27
7
White R
37
228
Heschls Gyrus R
30
-27
6
White R
48
299
9
Planum temporale R
51
-29
10
Planum temporale R
subj-02
77.99
41
288
8
White R
46
-3
-9
Insula R
55
29
11
Planum temporale R
49
-3
-7
Planum temporale R
43
-3
-10
Insula R
-38
-30
6
White I.
-58
-31
Superior Temporal Gyrus
-31
-30
7
White L
-35
-30
White
49
-11
1
Heschls Gyrus R
subj-03
23.04
48
-36
26
Parietal Operculum Cortex
42
-11
-1
Insula R
-55
-31
5
Superior Temporal Gyrus L.
41
-30
6
Planum temporale I.
45
-11
0
Heschls Gyrus R
44
-10
32
Precentral Gyrus L
455
-11
35
Precentral Gyrus L
-53
-6
-1
Planum temporale L
-44
28
Precentral Gyrus L
46
-12
39
Precentral Gyrus L
subj-04
59.91
-52
Planum temporale L.
-43
-8
24
White L
-38
-3
6
Insula L
-53
-7
-5
Superior Temporal Gyrus L
-16
43
25
White
33

---

Table 18: The MNI coordinates and brain region labels of selected channels from subjects (05-08).
MNI coordinate
Subjects
Brain Region
Du-IN Accuracy (%)
y
Z
-48
-16
33
Postcentral Gyrus L
46
15
29
Postcentral Gyrus L
-43
-14
22
White L
-51
-17
39
Postcentral Gyrus L
-44
-15
26
White L
subj-05
77.60
-53
-17
43
Postcentral Gyrus L
-50
-16
36
Postcentral Gyrus L
41
-14
19
Insula L
-55
-18
46
Postcentral Gyrus L
-49
-37
9
White L
56
-1
10
Central Opercular Cortex R
58
-4
4
Planum temporale R
52
4
21
Precentral Gyrus R
53
3
17
Precentral Gyrus R
57
-2
7
Central Opereular Cortex
subj-06
41.91
54
1
14
Precentral Gyrus R
51
24
Precentral Gyrus R
64
-25
-4
Middle Temporal Gyrus R
21
-1
11
White R
49
7
28
Precentral Gyrus R
-38
-18
2
Insula L
-44
-23
Heschls Gyrus
-41
-21
Heschls Gyrus L
-35
-16
2
Insula
-50
-28
0
Superior Temporal Gyrus L
subj-07
59.63
-47
-66
Planum temporale L
-52
-30
0
Superior Temporal Gyrus L
-66
-16
40
White L
-22
0
-8
Insula L
-39
-22
41
Postcentral Gyrus L
-40
20
4
Heschls Gyrus L.
-43
-21
4
Heschls Gyrus
-37
-20
4
Insula L
-50
-22
4
Heschls Gyrus
-47
-21
4
Heschls Gyrus L
subj-08
52.35
-55
3
24
Precentral Gyrus L
-64
-24
3
Superior Temporal Gyrus L Superior Temporal Gyrus
-61
-23
3
-54
-22
3
Planum temporale I.
-52
2
22
Planum temporale I.
34

---

Table 19: The MNI coordinates and brain region labels of selected channels from subjects (09-12).
MNI coordinate
Subjects
Brain Region
Du-IN Accuracy (%)
-58
-13
21
Postcentral Gyrus L
-55
-12
19
Central Opercular Cortex
-53
-11
17
Central Opercular Cortex I.
-50
-10
15
Central Opercular Cortex L
-61
-15
23
Postcentral Gyrus L
subj-09
66.39
-63
-66
25
Postcentral Gyrus L
-45
Central Opercular Cortex
-47
-9
12
Central Opercular Cortex L
-42
Insula L
52
-2
26
Precentral Gyrus R
-34
-47
41
Supramarginal Gyrus L
-42
-55
41
Angular Gyrus I.
-39
-53
41
Angular Gyrus I.
-37
-50
41
Supramarginal Gyrus I.
-25
-37
42
White L
subj-10
27.07
-44
-58
41
Angular Gyrus L
-27
-39
42
White L
-18
-41
48
Postcentral Gyrus L
-34
-34
-23
Temporal Fusiform Cortex
-13
-40
43
Precuneous Cortex L
39
-22
3
Heschls Gyrus R
36
-23
Insula R
47
-22
2
White R
32
-23
White R
43
-22
Planum temporale R
subj-11
73.56
53
-9
16
Central Opercular Cortex
57
Postcentral Gyrus R
50
-10
16
Central Opercular Cortex
61
-21
Superior Temporal Gyrus R
39
-14
13
Insula R
45
-0
11
Heschls Gyrus R
38
-20
Insula R
42
-20
11
Heschls Gyrus R
49
-20
Heschls Gyrus R
53
-19
10
Heschls Gyrus R
12
44.76
60
-19
Planum temporale R
56
-19
Planum temporale R
60
-58
Middle Temporal Gyrus R
56
-57
Middle Temporal Gyrus R
35
-21
12
Insula R
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Three of the four contributions mentioned at the end of the "Introduction" section explicitly included. The contribution related neuroscience-insee inspired analysis is simplified as "inspired by neuroscience findings" at the end the "Abstraction" section.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide separate "Limitations" section; see Section for more details.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information related to model and baselines in Appendix C and Appendix [ respectively. Besides, we provide code and dataset in http://gghuub.oom//lluaaa-pppppe
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide code and dataset in https://github.. wttp///uaabrppoossoorroooooaaaaa IN Due to the lack of open-source sEEG datasets related to language, we collected well- annotated Chinese word-reading sEEG dataset, and evaluated our model on this dataset. However, respecting the efforts of the data collectors, we only provide the dataset some subjects to reproduce the experimental results in the main text. The whole dataset will be publicly available to ensure the reproducibility of this work.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed information related model and baselines in Appendix
C and Appendix B respectively.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For the main results, report the average and standard error values (of all subjects) on six random seeds. For detailed subject- wise evaluation, we report the average and standard deviation values (of each subject) six random seeds.
36

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Detailed information related the training process provided in Section 4.2
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httphh/ttsss//.er........ppp//la
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics http://p///eeaaep...op/p/iiiiiaa
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work aims explore the feasibility of intracranial neural signals to decode speech, which mainly has positive impacts on society.
11. Safeguards
Question:
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: The paper poses such risks.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Justification:
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The ethics statements are provided in Section
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
37

---

Answer: [Yes]
Justification: The ethics statements are provided in Section 7
38


Reviews:
[{"review_id": "UbDFMClI4q", "reviewer": "Reviewer_eGF6", "summary": "Due to the locality and specificity of brain computation, existing models that learn representations of a single channel yield unsatisfactory performances on more difficult tasks like speech decoding. This paper tries to build a stereoElectroEncephaloGraphy (sEEG) foundation model with Vector Quantization (VQ) pre-training. We hypothesize that building multi-variate representations within certain brain regions can better capture the specific neural processing. To this end, we collect a well-annotated Chinese word-reading sEEG dataset, targeting language-related brain networks over 12 subjects, and we developed the Du-IN model that can extract contextual embeddings from specific brain regions through discrete codebook-guided mask modeling. Extensive comparison and ablation experiments verify that our model achieves SOTA performance on the downstream 61-word classification task.", "questions": "**(Q1)** For different subjects with different channels, does the spatial encoder need different parameters for each subject? And does the output embedding of each subject in the same shape?\n\n**(Q2)** Does the authors conduct ablation studies of masked code modeling, e.g., the masking ratio and pre-training epochs? Since the proposed Du-IN is a framework with several networks and the pre-traning & fine-tuning pipeline, it’s better to analyze and clearify each module for better understanding and practical usage.", "limitations": "Although the authors discussed some limitations in Sec. 5, potential negative societal impact of the proposed dataset and the Du-IN framework were not clearly discussed. More details of the implementations and the collected dataset can be provided for the sake of the neuroscience community.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**(S1)** This paper provides an interesting solution for the sEEG learning framework that models multiple sEEG channels by a spatial encoder and quantizes sEEG signals from the temporal aspect. To evaluate the proposed Du-IN framework, the authors collect and contribute a Chinese word-reading sEEG dataset for speech decoding (as word classification).\n\n**(S2)** The manuscript is clearly structured and easy to follow. Empirical analysis and ablations of proposed modules provide comprehensive understandings.", "weaknesses": "**(W1) The employed techniques in the proposed framework lack novelty to some extent.** Firstly, except for the spatiotemporal modeling architecture for sEEG, the VQVAE tokenization and masked code modeling pre-training framework is adopted from popular BeiT frameworks [1, 2], which has been adopted to various scenarios like [3, 4, 5]. Some specially designed modules are necessary for introducing the BeiT-like framework, and the current version is not effective and well-designed from my aspect (view W2 for concerns). Secondly, the authors regard the speech decoding tasks as word classification, which might be too coarse to decode the actual meanings of human speeches. Therefore, the conducted decoding task might not be sufficient to verify the effectiveness of the Du-IN framework.\n\n**(W2) Weak comparison experiments.** Firstly, the authors only provide a few supervised learning and self-supervised pre-training baselines in Table 2 with a single-word classification task (as speech decoding). More relevant contrastive learning [6] and masked modeling pre-training methods [7] for sEEG should be considered, which should also be referred to in the background section. Meanwhile, some supervised baselines with more parameters can be added for a fair comparison as Du-IN models. Secondly, the results of some baseline methods are very bad (e.g., BrainBERT and Brant). It’s hard to believe that pre-training models will yield worse results than random initialized supervised learning models. I suggest the authors conduct a fair and comprehensive comparison experiment.\n\n**(W3) Not explicitly modeling multiple subjects.** Since the invasive sEEG signals for different subjects have different channels, I couldn’t find an explicit modeling strategy in the proposed encoders and pre-training framework that considers fusing or selecting information from multiple subjects. There might be some challenges, like representation conflicts and missing channels of different subjects, which will decrease the overall cross-subject performances. I suggest the authors conduct such analysis of multiple subjects, e.g., cross-validation of several subjects and transfer learning of pre-trained representation across datasets of different subjects.\n\n### Reference\n\n[1] MAGE: Masked Generative Encoder to Unify Representation Learning and Image Synthesis. CVPR, 2023.\n\n[2] BEIT: BERT Pre-Training of Image Transformers. ICLR, 2021.\n\n[3] Human Pose as Compositional Tokens. CVPR, 2023.\n\n[4] MAPE-PPI_Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding. ICLR, 2024.\n\n[5] VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling. ICML, 2024.\n\n[6] Time-series representation learning via temporal and contextual contrasting. IJCAI, 2021.\n\n[7] Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining. IEEE Journal of Biomedical and Health Informatics, 2024.\n\n==================== Post-rebuttal Feedback ====================\n\nAfter considering the authors' rebuttal and other reviewers' comments, I decided to keep my score because some of my concerns were not solved directly (e.g., W1 and W3). Meanwhile, the authors have provided some useful results (e.g., more comparison experiments) to support the effectiveness of the proposed methods. Since the revision could not be uploaded, I encourage the authors to improve the manuscript further for submission to the next conference or journal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.\n\n**Answer to W1**:\n\n1. It’s true that the BEiT framework is popular; it has been used widely in recent years. The core idea is that it introduces discrete tokenization for mask modeling, which helps extract semantic information from data, thus enhancing model performance. For the same reason, we adopt this idea for sEEG decoding. However, we think the novelty of our study is that we are the first to find a way to adopt this idea to sEEG data for speech decoding properly and demonstrate why and how this idea works with a solid neuroscience foundation. DeWave[R1] indeed has attempted to use VQ-VAE (but not including MAE) to enhance language decoding based on EEG. However, the performance of directly applying their method to sEEG data was relatively low; see the performance of Du-IN (vqvae+vq)(44%). Besides, due to the brain's desynchronization nature[R2] during awake tasks, only specific brain regions are related to tasks. Therefore, only after region-specific channel selection can the Du-IN VQ-VAE model successfully reconstruct the original signals, thus identifying the fine-grained state of brain regions for mask modeling. Please see Table 2 in the general rebuttal pdf for more details. Moreover, we achieve the best decoding performance, only requiring about one electrode. It is really important because it reduces the medical burden and potential risks for patients.\n\n2. It’s true that word classification may not exactly decode semantic meanings of speech. However, we want to emphasize that this study aims to develop a system that can decode speech (vocal production) from limited intracranial data, which will help patients in the future.\n\n**Answer to W2:**\n\n1. Thank you for the helpful suggestion. TS-TCC(24%) and Neuro-BERT(49%) are added for comparison; please refer to Table 1 in the general rebuttal pdf for details. TS-TCC uses 1D depthwise convolutions as ours to capture local features. TS-TCC performs worse than CNN-BiGRU due to the lack of GRU for temporal integration. Besides, there may be some misunderstanding about Neuro-BERT. It is developed for EEG and evaluated only on single-channel EEG tasks, e.g., seizure detection. We note Neuro-BERT is also evaluated on non-EEG tasks (e.g., Hand Gesture Recognition), which use depthwise convolution to integrate multiple channels. Neuro-BERT outperforms EEG-Conformer since it introduces MAE to learn contextual embeddings. DeWave-CLS[R1] (32%) is added, which replaces our depthwise convolutions with Conformer and has more parameters. We have cited these works in our revised paper, which is available in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n2. BrainBERT and Brant indeed outperform their randomly initialized models. Brant is a great work that serves as the medical field's foundation model and achieves SOTA on many medically valuable tasks, e.g., seizure detection. But their architecture may not be suitable for some cognition tasks, e.g., speech decoding. For the reason of their relatively low performances, please refer to the general rebuttal for more details.\n\n**Answer to W3:**\n\nThank you for the helpful suggestion. We add Du-IN (poms)(59%), which is pre-trained over 12 subjects. The representation conflicts do exist, which performs slightly worse than Du-IN (mae)(62%), which is pre-trained on that subject. But we should note that this is not the core aim of our study. We aim to find a way to leverage redundant pure recordings of one subject to build the codebook of speech-related brain regions, which can identify fine-grained region states to improve the efficiency of mask modeling, thus improving speech decoding performance.\n\n**Answer to Q1:**\n\nThank you for the valuable feedback. We initialize different spatial encoders for different subjects; all these spatial encoders have the same hyperparameters. The output embeddings are of the same shape. The following transformer encoder and vector quantizer are shared. We have clarified these in our revised paper, which is available in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n**Answer to Q2:**\n\nThank you for the helpful suggestion. We add these ablation studies in Table 3,5,6 in the general rebuttal pdf. For ablations on pre-training epochs:\n\n1. We use the checkpoints of Du-IN VQ-VAE according to the specified epochs to train the Du-IN MAE model for 400 epochs. Once the reconstruction loss of the Du-IN VQ-VAE model converges, the Du-IN VQ-VAE model can extract the state of the brain region well, thus leading to better performance.\n\n2. We use the checkpoints of Du-IN MAE according to the specified epochs for downstream classification. Once the mask modeling loss of the Du-IN MAE model converges, the Du-IN MAE model learns robust contextual embeddings, thus leading to better performance.\n\n**References:**\n\n[R1]Duan Y, Zhou J, Wang Z, et al. Dewave: Discrete eeg waves encoding for brain dynamics to text translation[J]. arXiv preprint arXiv:2309.14030, 2023.\n\n[R2]Buzsaki G. Rhythms of the Brain[M]. Oxford university press, 2006.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ZNaXu9uz8", "reviewer": "Reviewer_bbQK", "summary": "The authors introduce a neural network architecture, a self-supervised learning (SSL) method for pre-training models for sEEG decoding and a novel dataset.\n- The architecture starts with segmenting the sEEG signal along the time dimension, followed by a linear module combining information across the different sEEG channels into embedding vectors and ending with a transformer with attention along the time dimension.\n- The SSL method has two stages: \n    - 1. The architecture followed by an additional decoder network is trained to reconstruct the original signals (autoencoder). However, the output of the transformer is not directly passed to the decoder, instead, each token is replaced with the closest embedding vector from a trainable codebook (vector quantisation).\n    - 2. Time windows of the input are masked and the architecture is trained to predict which codebook embedding represents best each masked window.\n- The dataset introduced contains sEEG recordings of 12 subjects while they were speaking aloud Chinese words from a vocabulary of size 61. \n\nIn the experiments, the models were pre-trained using the SSL tasks using unlabelled data from a single subject and fine-tuned on labelled data from the same subject for classifying the 61 words. \nThe authors conducted different ablation studies regarding pre-training (no pre-training, and only step 1 of pre-training using quantized features for fine-tuning or not), dataset, codex size and dimensionality, and length of the time windows.\nThey also compare with pre-existing methods, two including a pre-trining phase and two without. The two with pre-training are pre-trained on all 12 subjects.", "questions": "1. Lines 134-140: The description of the spatial encoder is ambiguous:\n    1. Does “linear” mean fully-connected layer? Appendix C.1 mentions “Linear Projection: 10->16” which would mean that an operation is applied to each channel independently and that the layer is actually not a fully-connected one.\n    2. Convolutions are mentioned, but over which dimensions? Over time? Channels? Both?\n2. Figure 3: What are the numbers below “l2-norm” and bellow “neural regressor” corresponding to?\n3. Why did you use a different signal length for pre-training and for downstream classification?\n4. Line 210: Why did you use a sampling rate of 1000Hz? With a low-pass filtering at 200Hz, a sampling rate of 400Hz would be sufficient\n5. Line 211: Did you apply the normalisation on each channel independently (i.e., mean and std computed for one channel only) or using all channels together?\n6. Line 222: How is the training time split between VQ-VAE and MAE training?\n7. Line 228, “selected from the validation set”: does this mean early stopping? Was it performed using the validation loss, validation accuracy or another metric?\n8. Table 2, “Du-IN (mae)”: this row corresponds to the scores obtained after a VQ-VAE training followed by a MAE training. Is the first step exactly “vqvae” or “vqvae+vq”? To lift the ambiguity, you could rename the row to “Du-IN (mae+vqvae)” or “Du-IN (mae+vqvae+vq)”.\n9. Why refer to your method as a “VAE”? As I understand, for a given input, the “reconstructed sEEG signals” are obtained in a deterministic manner (please, correct me if I’m wrong am wrong). The term “variational” in VAE suggests there is a stochastic operation involved.\n10. Lines 180-193: the objective function of the masked training is unclear:\n    1.  Line 188: “neural tokens” is ambiguous. What is being predicted?\n    2. Equation (7) and (8): how are the $z_i$ obtained? \n    3. Equation (8): $m_i$ is undefined. What is the meaning of $m_i$?", "limitations": "The authors included a Limitations section in their manuscript in which they already mention other approaches to speech decoding, at the phonemes level and at a semantic level. \n\nIt could be relevant to remind the reader that sEEG recordings are invasive. Therefore, they are only done with patients who would require them anyway for medical reasons (epilepsy monitoring, etc…). And this presents a major obstacle to making these methods widely accessible and usable.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "*Significance*: As pointed out by the authors, sEEG speech datasets are rare. Their promise of publishing the dataset they introduce here is good news for the community as it will lower the entry threshold for future research. Additionally, they demonstrate how SSL-based pre-training  allows improving performance compared to only supervised training.\n\n*Clarity*: The text has a good structure and is well-written. The figures also help in understanding the method.\n\n*Originality*: SSL techniques are still rather novel to the BCI community. This work sheds light on some of them.", "weaknesses": "**Major**\n1. The authors pre-train their method on the test subject (Lines 201-202) whereas the baseline methods with pre-training (BrainBERT and Brant) were pre-trained on all 12 subjects (lines 511-518). This does not allow for a fair comparison. Additionally, this choice is not properly justified.\n2. The authors mention in section 4.5 Ablation study, that they found optimal hyperparameters for their architecture (codex of size 2048 and dimension 64, and perceptual field of 100ms). They use these hyper parameters to compare to the other methods (see Table 5). However, the manuscript does not mention the use of a holdout set for conducting this ablation study. This is problematic for two reasons:\n    1. This suggests the method could have a low capacity to generalise be overfitting to their dataset\n    2. This is not fair for the other methods as no hyperparameter optimisation is mentioned for them.\n3. The authors mention they resampled their signals to 1000Hz (line 210). However, some architecture compared have been introduced with different sampling rates (Brant used 250Hz and EEG-Conformer used 200 and 250Hz). The text does not mention any adaptation of the architectures compared or of the sampling rate. This is problematic as a convolution kernel supposed to cover 1s of signal at 250Hz will only cover 0.25s at 1000Hz. \n4. Some aspects of the method are undefined or unclear. Please see the Questions section below. These aspects need to be clarified in the manuscript.\n\n\n**Minor**\n1. Because of the flaws in experimental design described above, the statements about the superiority of their method compared to others (lines 14, 71, 248 and 318) are unjustified.\n2. All figures should be in a vectorized format.\n3. The text in the figures should be approximately constant and the same as the article’s text. It is too small in figures 5 and 7. It is on the limit in figures 1,2 and 3.\n4. The term “fusing channels” is used lines 16, 39, 108 and 117 but is rather unclear. It is only explained line 134.\n5. Line 29: provide examples of non-invasive recording methods (EEG, MEG…)\n6. Section 2.3, additional publication the authors should be aware:\n    1. A preprint of Brant-2 is now available https://arxiv.org/abs/2402.10251\n    2. SSL methods trained on EEG with attention mechanism for combining information across channels: EEG2Rep (https://arxiv.org/abs/2402.17772) and S-JEPA (https://arxiv.org/abs/2403.11772).\n7. Line 54: Indeed, there is a lack of open sEEG recordings but some still exist. In reference [1], they mention they use a public dataset, in [16] they mention their data is available upon request and Brant-2 (not cited here), they use a 26-subject public dataset. Existing sEEG datasets should be cited.\n8. Line 98: typo “Metzger1” \n9. As I understand, the terms “Du-IN Encoder”, “neural transformer”, “neural encoder”, and “neural tokeniser” all refer to the architecture in figure 2. If this is indeed the case, it would improve readability to chose one of them and stick to it throughout the manuscript.\n10. Line 166: missing “to” in “$z_c$ to get”\n11. Figures 2 and 3: It would improve readability to place the different variables ($e^p_i$, $e_i$, $e^t_i$, $c_j$,…) on these figures.\n12.  Line 162: for consistency, you could use $j$ as index for the codex instead of $i$.\n13. Lines 180-193: the objective function of the masked training is unclear (see questions).\n14. Line 247: Please mention that the baseline descriptions can be found in appendix C.3.\n15. Line 301: The research line of Feng et al. was already explored by Christian Herff in 2015 (https://doi.org/10.3389/fnins.2015.00217)  and is still today (https://www.sciencedirect.com/science/article/pii/S1053811923000617). It could be interesting to draw a parallel.\n\n\n\n**Overall comment**\nThe introduction of a novel sEEG dataset is valuable but the evaluation of the machine learning algorithms that are introduced is poor. Therefore,  I believe NeurIPS might not be the best fit for this publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.\n\n**Answer to Major W1:** Thank you for the helpful suggestion. We have added DuIN(poms)(59.18%), which is pre-trained on all subjects for a fair comparison. Since BrainBERT and Brant support pre-training on multi-subjects, this operation can maximize the downstream performance according to their reported scaling law.\n\n**Answer to Major W2:** Thank you for the valuable feedback.\n\n1. All experiments are conducted on the same machine with the same set of random seeds; the train/valid/test splits are the same across different models.\n\n2. In fact, the performances of BrainBERT(7.5%) and Brant(12.4%) we report are the results after model optimization, and their performance with the original hyperparameter settings is even lower(5.6%&8.7%). Please see the general rebuttal for more details. We have clarified these in our revised paper, which is available in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n**Answer to Major W3:** Actually, we resampled the signals to the required frequency of each model. Please refer to line 508 in the Appendix.\n\n**Answer to Minor W2&3&4&5&8&9&10&11&12&14:** Thank you for the valuable feedback. We have corrected all formatting, spelling, references, and wording issues. We have updated the paper in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n**Answer to Minor W6&7&15:** Thank you for the helpful suggestion. We have cited these works and updated the paper in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n**Answer to Q1:** Thank you for the valuable feedback.\n\n1. Yes, we use “linear” to refer to the fully connected layer, which projects 10 channels to 16 dimensions.\n\n2. We use `torch.nn.Conv1d` with `groups=1`, i.e., depthwise convolution over time.\n\n**Answer to Q2:** It corresponds to the index of discrete codes in the codex.\n\n**Answer to Q3:** Thank you for the valuable feedback. We followed LaBraM's settings and used 4s signals for pre-training. The signal length of downstream tasks is usually shorter than the pre-training length.\n\n**Answer to Q4:** Thank you for the valuable feedback. We tried the effects of different sampling rates and finally found that there was not much difference between 400 and 1000Hz, but 1000Hz is slightly better. At the same time, we use a sampling rate of 1000Hz to minimize the difference with the BrainBERT model because they also perform sEEG language-related tasks and use 2000Hz.\n\n**Answer to Q5:** We did it on each channel independently; please refer to line 211 in Section 4.2.\n\n**Answer to Q6:** The training time splits are the same.\n\n**Answer to Q7:** Thank you for the valuable feedback. Yes, this means early stopping, and we performed using the validation accuracy.\n\n**Answer to Q8:** Du-IN (mae) is obtained after a VQ-VAE training followed by a MAE training, i.e., loading weights from the Du-IN MAE. “+vq” means we use quantized embeddings after the vector-quantizer for the downstream task, which is a specific operation when loading weights from the Du-IN VQ-VAE.\n\n**Answer to Q9:** Thank you for the valuable feedback. It's true that our method is not variational; we followed the name “VQ-VAE” proposed in [R1].\n\n**Answer to Q10:** Thank you for the valuable feedback.\n\n1. The “neural tokens” means the code index in the codex of the Du-IN VQ-VAE model.\n\n2. It is obtained from Equation (4).\n\n3. Sorry for the confusion, it is the $\\delta(i\\in\\mathcal{M})$ from Equation (6).\n\n**References:**\n\n[R1]Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in 434 neural information processing systems, 30, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t01xMtQPfc", "reviewer": "Reviewer_1qa8", "summary": "The authors introduce Du-IN, a novel decoding technique for speech production the leverages sEEG data. To achieve their goals, the authors gathers a novel dataset. The Du-IN model learns multi-variate (across channel) representations from sEEG signal using a discrete cookbook and a masked modeling based self-supervised training. The proposed algorithm achieves SoTA performances on the speech 61-words classification task. Furthermore, analysis of optimal model configuration and ablation studies reveal that best performances are achieved with electrodes in brain regions known to be involved in speech production.", "questions": "- There seems to exist substantial variation of decoding accuracy across subjects. Is this difference due to different electrode locations, or number of electrodes (even if only 10 channels seem to be optimal) or are other concerns with data quality relevant?\n- Could the manual selection of channel be partially subsumed by a sparsity regularization term on the channel-to-embedding projection matrix?", "limitations": "The authors have adequately addressed the limitation of their work with a dedicated section in the paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper presents strong results of high significance for the field of speech decoding. In particular:\n- The authors have collected a novel sEEG dataset, which will be made publicly available, hence fostering further progress in the field of brain signal decoding.\n- The authors propose a novel framework for sEEG signal decoding - the Du-IN algorithm, which achieves SoTA performances, thus significantly advancing the field of brain decoding.\n- Authors provide additional ablations studies to validate design choices and recover known results from Neuroscience, strengthening the overall soundness of their technique.\n- The paper is well written and clearly organized, with a detailed Appendix providing further details.", "weaknesses": "- In Table 2 both the Du-IN (vqvae+vq) and Du-IN (vqvae) are marked as not pre-trained. However in the Appendix C.3 it is stated that their Neural Encoder weights are loaded from the pre-trained Du-IN VQ-VAE model.\n- Minor spelling: Line 156 DuIR $\\to$ DuIN", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.\n\n**Answer to W1 & W2:** Thank you for the valuable feedback. These have been corrected in the revised paper, which can be accessed in the Anonymous Github Repository at line 328 in the initial manuscript.\n\n**Answer to Q1:** This difference is mainly due to different electrode locations, as we find that decoding vocal production is mainly related to specific brain regions (i.e., vSMC, STG), which is consistent with the previous neuroscience findings.\n\n**Answer to Q2:** Thank you for the helpful suggestion. Yes, this modification will make the channel selection more automatic and general.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQNJkKL0Nu", "reviewer": "Reviewer_gwJp", "summary": "SETTING: Decoding spoken speech from invasive neural recordings, to wit, sEEG.  In particular, the MS aims to classify 61 independently produced (Chinese) words.\n\nAPPROACH: The focus of this study is on unsupervised pretraining via autoencoding, with either (a) vector-quantization or (b) masking.  Masks are 100-ms long and cover all channels.  For both VQ and masking, the core model is a \"spatial encoder\" (linear + conv) followed by a (temporal) transformer.  There are ~15 hours of data per subject.  After pre-training, this model is trained end-to-end (on ~3 hours of labeled data) on the classification task.\n\nRESULTS: The architecture outperforms other architectures proposed for this task, and masking does appear to improve performance.", "questions": "(1) The spatial encoder:  The name suggests that its convolutions are across channels, but this is inappropriate across sEEG electrodes (and within electrodes should take into account the spatial layout of channels).  Can the authors clarify this?  (I see in Appendix H that the *linear layer* combines channels.)\n\n(2) The authors give some reasons for the superiority of their model over BrainBERT and Brant, but it would be helpful to have some insight on the performance relative to other models as well.  E.g.,\n        \n(a) The MS focuses on pretraining, but in fact it appears that the biggest gain over the baselines comes from the architecture itself (Du-IN trained fully supervised)---an improvement of ~25 percentage points over the CNN-BiGRU!  Can the authors shed light on this difference? \n\n(b) The EEG-conformer likewise uses a \"temporal transformer,\" but performance is ~10 percentage points lower than the supervised-trained Du-IN.  What do the authors see as the key difference here?\n\n(c) The authors mention LaBraM but don't compare Du-IN to it.  It would be nice to see this comparison as well....\n\n(3) Intracranial EEG data is generally (but not always!) filtered into the range of ~75-150, because otherwise the low frequencies, which have higher power, tend to dominate.  (The low frequencies can be provided as separate channels.)  This is a problem because low frequenceies generally contain much less information about speech.  In this study, frequencies below 70-Hz are not filtered out.  Indeed, it appears that reconstructions under vector quantizations retain mostly low-frequency information (Fig. 7).  Can the authors explain these results?\n\n(4) Channel selection:\n \n(a) \"To streamline, we utilize these top 10 channels for both pre-training and downstream evaluation.\"  Does that mean that, for each subject, the best 10 electrodes (for that subject) were used?\n\n(b) If the choice to use just 10 channels was made on the basis of decoding results on the test set, then the results are biased upwards.  Can the authors confirm that this choice was made based on validation data?  (Is Fig. 4b on validation data?)\n\n(c) How does channel selection interact with model choice?  Were all models in Table 2 evaluated with just 10 channels?  Does their performance as a function of #channels also resemble Fig. 4b?\n\n(5) It seems quite possible from Fig. 4a that the authors are decoding speech perception as much or more than speech production, since many of the most highly contributing electrodes are below the Sylvian fissure.  This would limit the use of this decoder as a speech prosthesis for people who cannot speak.  Can the authors comment on this?\n\nOn this head, although Fig. 4 is a nice summary, but it would be very useful to have a list/table of the electrodes used for decoding for each subject, labeled by cortical area.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a nice study and the results are compelling.  Temporal masking looks promising as a generic technique for sEEG data.  The dataset is extremely useful and is likely to become the basis of benchmarks for future research in this area.", "weaknesses": "The paper does not address some important questions (admittedly there is not much space), which I list below.\n\nIn addition to these, I list here some minor issues:\n\n--For comparing with other results in the decoding literature, it would be very useful to report cross entropy in addition to (or instead of) accuracy, since the latter can only be interpreted in light of the fact that there are 61 classes.\n\n--The reader shouldn't have to go to Appendix C.3 just to find out what the names in Table 2 mean.  Also, what does \"CLS\" stand for?\n\n--The (vqvae+vq) and (vavae) rows in Table 2 should be marked as \"pre-trained.\"\n\n--The MS calls the token's temporal width the \"perception field.\"  Is this a standard term?  Perhaps the authors mean \"receptive field\"?\n\n--The authors use \"codex\" where they mean to use \"code.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.\n\n**Answer to Q1:** Thank you for the helpful suggestion. We use 1D depthwise convolution, which convolves over time and integrates channels with learnable weights. Linear layer is fully-connected layer, which maps sEEG signals with 10 channels to 16 dimensions. We have clarified these in our revised paper, which is available in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n**Answer to Q2:** Thank you for the valuable feedback. Architecture itself indeed matters.\n\n1. EEG-Conformer(45%) outperforms CNN-BiGRU(32%) is because Transformer can better integrate global temporal information compared to RNN-based model.\n\n2. Du-IN(56%) outperforms EEG-Conformer(45%) is because:\n   - The difference between our depthwise convolution and their TS-convolution. Depthwise convolution applies different convolution kernels to different channels, whereas TS-convolution applies the same convolution kernel to different channels. Brain signals are different from other time series, e.g., traffic. The same pattern in different brain locations may have different meanings. Depthwise convolution can better spatially integrate brain channels to produce meaningful tokens.\n   - Our carefully designed convolution kernels for sEEG data. The first convolution layer covers about 20ms window on the brain signal with 1000Hz sampling rate, which can capture sufficiently fine-grained features to facilitate the following integration of low-frequency and high-frequency features for speech decoding.\n\n3. Thank you for the helpful suggestion. LaBraM(11.53%) and LaBraM-PopT(11.78%) are added for comparison; please refer to Table 1 in the general rebuttal pdf for details. For the reason for LaBraM’s relatively low performance, please refer to the general rebuttal for details.\n\n**Answer to Q3:** Thank you for the valuable feedback.\n\n1. This is indeed an important point. Specifically, although many previous studies believe that 75~150Hz (high gamma) band signals dominate speech information, this is because high gamma band signals are believed to mainly reflect the spike information of neurons[R1]. However, in recent years, more neuroscience studies have found that LFP can also decode neural information well[R2-R5], so we retained data from all frequency bands, giving us better performance.\n\n2. It's true that the reconstructed signal is mainly low-frequency information. Our method may not be the best to identify fine-grained states of specific brain regions since previous studies[R2-R5] found that all frequency bands contribute to speech decoding. In the future, it will be interesting to see whether introducing the STFT loss proposed in [R6], which further balances the reconstruction of low and high frequencies, leads to better performance.\n\n**Answer to Q4:** Thank you for the valuable feedback. (a) Yes, we selected the best 10 channels for each subject. (b) In the channel selection stage, we use the last training checkpoint for selection without referring to [valid,test]-set. So, these channels are selected purely based on train-set. We also tried to use the accuracy of valid-set to select channels, and the channels selected were the same. (c) Yes, all models are evaluated using the same 10 channels, and the performance of other models as a function of #channels is similar to Fig. 4b.\n\n**Answer to Q5:** Thank you for the helpful suggestion. Due to the difference in the number of electrodes on the left and right brain, we standardized and rendered them separately when drawing the map. We have changed to joint standardization and rendering. In fact, the important site is still the language motor area of the left brain. We have made this table and added it to the article, but due to the word limit of the rebuttal, we only show one subject here. Please see our revised paper for more details.\n\n|Subjs|MNI|coord-|-inate|Brain Region|ACC(%)|\n|:--|:--:|:--:|:--:|:--:|:--:|\n||__x__|__y__|__z__|||\n||-57|-16|19|Central Opercular Cortex L||\n||-61|-16|20|Postcentral Gyrus L||\n||-54|-16|17|Central Opercular Cortex L||\n||-67|-15|23|Postcentral Gyrus L||\n|01|-25|-30|49|White L|71.25|\n||-64|-15|21|Postcentral Gyrus L||\n||-51|-17|16|Central Opercular Cortex L||\n||-22|-31|49|White L||\n||-48|-17|15|Central Opercular Cortex L||\n||-18|-32|49|White L||\n\n**Answer to W1&2&3&4&5:** Thank you for the helpful suggestion. We have made these clear in our revised paper. Due to the limited space for comments and rebuttal pdf, we provide cross-entropy results in Appendix J of our revised paper.\n\n**Reference:**\n\n[R1]Buzsáki, G., & Wang, X. J. (2012). Mechanisms of gamma oscillations. Annual review of neuroscience, 35(1), 203-225.\n\n[R2]Proix, T., Delgado Saa, J., Christen, A., Martin, S., Pasley, B. N., Knight, R. T., ... & Giraud, A. L. (2022). Imagined speech can be decoded from low-and cross-frequency intracranial EEG features. Nature communications, 13(1), 48.\n\n[R3]Ahmadi, N., Constandinou, T. G., & Bouganis, C. S. (2021). Impact of referencing scheme on decoding performance of LFP-based brain-machine interface. Journal of Neural Engineering, 18(1), 016028.\n\n[R4]Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2015). A high performing brain–machine interface driven by low-frequency local field potentials alone and together with spikes. Journal of neural engineering, 12(3), 036009.\n\n[R5]Flint, R. D., Lindberg, E. W., Jordan, L. R., Miller, L. E., & Slutzky, M. W. (2012). Accurate decoding of reaching movements from field potentials in the absence of spikes. Journal of neural engineering, 9(4), 046006.\n\n[R6]Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BgNsEJEKH", "reviewer": "Reviewer_t1mV", "summary": "The authors proposed Du-IN, a speech decoding framework that incorporates a self-supervised pre-training stage and a classification stage. Experiments are conducted on 12 subjects with a task of word classification. Comparisons are conducted to demonstrate effectiveness.", "questions": "* Are decoding words the same as decoding pronunciations in [1][2]? Is it possible that the sentiment is associated with words that the decoding task is not speech but sentiment?\n\n**References**\n\n[1] Willett, Francis R., et al. \"A high-performance speech neuroprosthesis.\" Nature 620.7976 (2023): 1031-1036.\n\n[2] Metzger, Sean L., et al. \"A high-performance neuroprosthesis for speech decoding and avatar control.\" Nature 620.7976 (2023): 1037-1046.", "limitations": "The authors discussed the drawbacks of the proposed framework which is essentially close-set prediction rather than open-set. Potential negative societal impact was not mentioned.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* (a) The paper is clearly written and easy to follow.\n\n* (b) The task of speech decoding using intracranial recordings is interesting.", "weaknesses": "* (a) Although the task of decoding speech from intracranial recordings is an interesting topic, the algorithmic novelty of this paper is limited. The pretraining framework the authors proposed is very similar to BEIT other than the underlying data modality being images and sEEG. Simply applying an existing model BEIT to sEEG data with minor modifications to the backbone architecture does not meet the high standards required for acceptance at a top conference like NeurIPS. \n\n* (b) The comparisons conducted are insufficient. In particular, besides BrainBERT and Brant, the authors compared with, there are also many pre-training approaches for neurological signals.  I list some more up-to-date approaches from a contrastive learning perspective and a masked autoencoding perspective. For contrastive-based approaches,  there are TS-TCC [1] and [2]. For masked autoencoding based approaches, there are Biot [3] and Neuro-BERT [4]. As a matter of fact, the authors could at least follow the comparison setup in Brant for a fair comparison. \n\n* (c) Reproduction details are missing. The reported results on BrainBERT and Brant are ~7.5% and ~12.5%, as compared to the reported performance of ~63% of the proposed  Du-IN. It is astonishing that the proposed Du-IN outperforms BrainBERT and Brant by almost 50%. Both BrainBERT and Brant are masked autoregressive based pertaining approaches. Although BrainBERT and Brant vary from Du-IN, it does not make sense that pretraining could lead to such a tremendous difference. The authors should report the hyperparameters used to reproduce Brant and BrainBERT. Otherwise, It is highly possible that the authors did not tune  BrainBERT and Brant properly. \n\n* (d) Although the decoding of speech is an interesting question, it seems rather well-established in open vocabulary scenarios [5][6], decoding words of limited vocabulary traces back to at least ten years ago. \n\n\n\n\n**References**\n\n[1] Woo, Gerald, et al. \"Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting.\" arXiv preprint arXiv:2202.01575 (2022).\n\n[2] Baevski, Alexei, et al. \"Data2vec: A general framework for self-supervised learning in speech, vision, and language.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Yang, Chaoqi, M. Westover, and Jimeng Sun. \"Biot: Biosignal transformer for cross-data learning in the wild.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Wu, Di, et al. \"Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining.\" IEEE Journal of Biomedical and Health Informatics (2024). \n\n[5] Willett, Francis R., et al. \"A high-performance speech neuroprosthesis.\" Nature 620.7976 (2023): 1031-1036.\n\n[6] Metzger, Sean L., et al. \"A high-performance neuroprosthesis for speech decoding and avatar control.\" Nature 620.7976 (2023): 1037-1046.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.\n\n**Answer to W1:** It’s true that the BEiT framework is popular; it has been used widely in recent years. The core idea is that it introduces discrete tokenization for mask modeling, which helps extract semantic information from data, thus enhancing model performance. For the same reason, we adopt this idea for sEEG decoding. However, we think the novelty of our study is that we are the first to find a way to adopt this idea to sEEG data for speech decoding properly and demonstrate why and how this idea works with a solid neuroscience foundation. DeWave[R1] indeed has attempted to use VQ-VAE (but not including MAE) to enhance language decoding based on EEG. However, the performance of directly applying their method to sEEG data was relatively low; see the performance of Du-IN (vqvae+vq)(44%). Besides, due to the brain's desynchronization nature[R2] during awake tasks, only specific brain regions are related to tasks. Therefore, only after region-specific channel selection can the Du-IN VQ-VAE model successfully reconstruct the original signals, thus identifying the fine-grained state of brain regions for mask modeling. Please see Table 2 in the general rebuttal pdf for more details. Moreover, we achieve the best decoding performance, only requiring about one electrode. It is really important because it reduces the medical burden and potential risks for patients.\n\n**Answer to W2:** Thank you for the helpful suggestion. LaBraM(11%), TS-TCC(24%), and Neuro-BERT(49%) are added for comparison; please refer to Table 1 in the general rebuttal pdf for details. Since LaBraM outperforms BIOT on multiple EEG tasks, we report the performance of LaBraM instead of BIOT. We can see that BrainBERT(5.6%) & Brant(8.7%) & LaBraM(11%) all performs worse than other models. For the reason of (BrainBERT,Brant,LaBraM)'s relatively low performance, please refer to the general rebuttal for more details. TS-TCC uses 1D depthwise convolutions as ours to capture local features. TS-TCC performs worse than CNN-BiGRU due to the lack of GRU for temporal integration. Neuro-BERT outperforms EEG-Conformer since it introduces MAE to learn contextual embeddings. We have cited these works in our revised paper, which is available in the Anonymous Github Repo at line 328 in the initial manuscript.\n\n**Answer to W3:** Thank you for the helpful suggestion. We reproduce BrainBERT(5.6%) and Brant(8.7%) strictly following their original implementation.  We have uploaded our reproduction to the anonymous repository, which is available in the Anonymous Github Repo at line 328 in the initial manuscript. We also uploaded their original codes and checkpoints. You can directly use their checkpoints to check whether the reproduction is the same; please see README.md in the repository for more details. Brant will load the Brant-Large checkpoints provided by the original Brant paper. In our implementation, we use the Brant-Tiny model for pre-training and evaluation. The length of the patch segment is changed from 6 seconds to 1 second for a fair comparison with LaBraM.\n\n**Answer to W4:** Thank you for the helpful suggestion. There are indeed some studies on open vocabulary, but the problem is that all of these studies are based on ECoG or microelectrode arrays. These technologies may cause more brain damage than sEEG technology in clinical practice. More importantly, we found that only one electrode was needed for decoding, further reducing clinical risks. At the same time, unlike the other two technologies that can only obtain signals from the cortex, sEEG electrodes are implanted inside the brain. Due to the existence of cortical columns, sEEG can obtain more accurate and hierarchical information. The sEEG signal is fundamentally different from other technologies’, and there was lack of methodology to draw on in this case, thus decoding the limited vocabulary from sEEG signals was still a relatively new problem.\n\n**Answer to Q1:** Thank you for the valuable feedback. Yes, decoding words and decoding pronunciations are essentially the same in that both are decoding speech-motor signals. And we are not sure what you mean by “sentiment”. If you do mean sentiment(emotional information), then we think this is unlikely to affect decoding because almost all of the words we chose were neutral words and did not contain any sentimental information. But if you mean semantics, we cannot rule out the involvement of semantic information in decoding. Because although our task setting is to collect speech-motor information, it is very likely that humans also perform corresponding semantic processing when reading words. This is indeed an interesting question, but it requires more elaborate experimental settings and corresponding data to verify this.\n\n**Reference:**\n\n[R1]Duan Y, Zhou J, Wang Z, et al. Dewave: Discrete eeg waves encoding for brain dynamics to text translation[J]. arXiv preprint arXiv:2309.14030, 2023.\n\n[R2]Buzsaki G. Rhythms of the Brain[M]. Oxford university press, 2006.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response to AC and all reviewers
Thanks to all reviewers for careful reading and thoughtful feedback. We are grateful for acknowledging our method "looks promising as a general technique for sEEG data" and that our dataset “is likely to become the basis of benchmarks for future research in this area” (Reviewer gwJp). We are also excited that our work "presents strong results of high significance for the field of speech decoding" (Reviewer 1qa8). Further, we are delighted to hear our method "is rather novel and sheds light on the BCI community" (Reviewer bbQK).

In response to reviewers' comments, we performed additional experiments: 8 new model comparison experiments and 5 new ablation experiments, summarizing the additional studies. Please see the general rebuttal pdf for details. We will add these new experiments to our revised paper.

## About Model Comparison Experiment
To support the effectiveness of Du-IN for speech decoding, we added 1 supervised baseline (DeWave-CLS), 6 self-supervised baselines (TS-TCC, Neuro-BERT, original BrainBERT, original Brant, LaBraM, LaBraM-PopT) and 1 Du-IN variant (Du-IN (poms), pre-trained on multi-subjects), as a solution to similar concerns raised by some reviewers. These concerns include:
 - Insufficient comparison experiments (Reviewer t1MV, eGF6).
 - Not explicitly modeling multiple subjects (Reviewer eGF6, bbQK).
 - LaBraM comparison (Reviewer gwJp).

All the results can be found in Table 1 in the rebuttal pdf. All codes and checkpoints are available in the Anonymous Github Repo at line 328 in the initial manuscript.  First of all, we want to emphasize that **all baselines are reproduced strictly following to their original code**. Besides, all experiments are conducted on the same machine with the same set of random seeds, and **the train/valid/test splits are the same across different models**.
 - **Our model comparisons are indeed fair and comprehensive.**

For BrainBERT and Brant, we provide their original code and checkpoints, and you can directly load their weights using our model to check whether reproductions are the same. The reported performance of BrainBERT(7.5%) and Brant(12.42%) in the initial manuscript is after hyperparameter optimization. The performance of original BrainBERT(5.65%) and Brant(8.79%) is even lower. LaBraM(11.53%) and LaBraM-PopT(11.78%, which replace learnable spatial embeddings with hard-coded ones proposed by [R1] to enable pre-training on multiple subjects) perform similarly to them. Their relatively low performances are mainly because they don’t encourage region-level embedding either from architecture[R2] or loss[R1], which makes them hard to decode speech. MMM[R2] introduces an information bottleneck to encourage integrating channel-level embeddings into region-level embeddings. PopT[R1] does this due to their pre-training task. Our model directly uses depthwise convolution to get such region-level embeddings, that is why our model outperforms them by a 50% margin.

Additionally, we add TS-TCC(24.85%), Neuro-BERT(49.51%), and DeWave-CLS(32.43%) to further address the concerns about insufficient comparison experiments. Our Du-IN model (62.70%) still performs the highest among them.

We further provide Du-IN (poms) (59.18%), which is pre-trained on all 12 subjects and provides a fair comparison with BrainBERT and Brant, to address the concerns about modeling multi-subjects.

## About ablation experiment
To further demonstrate why our proposed model works for sEEG speech decoding, we conduct additional ablation experiments on our model, as a solution to similar concerns raised by some reviewers. All the results can be found in the rebuttal pdf:
 - Ablations on effectiveness of region-specific channel selection(Table 2).
 - Ablations on impact of mask ratios(Table 3).
 - Ablations on effectiveness of vector-quantized neural signal prediction(Table 4).
 - Ablations on impact of the pre-training epochs (of the Du-IN VQ-VAE and MAE)(Table 5,6).

The first ablations demonstrate that sEEG signals are fundamentally different from EEG signals due to the high information density and the high specificity of different regions. Other ablations demonstrate the robustness of our model.

## About specific responses
We have individually addressed all of your comments below, specifically addressing each reviewer’s concerns in the corresponding responses. Please note that in our responses, references in the format “[R1]” indicate citations that are newly added in the rebuttal, while references in the format "[1]" are citations from the original manuscript.

We have dedicated significant effort to improving our manuscript, and we sincerely hope that our responses will be informative and valuable. We would love to receive your further feedback.

**Reference:**

[R1]Chau G, Wang C, Talukder S, et al. Population Transformer: Learning Population-level Representations of Intracranial Activity[J]. ArXiv, 2024.

[R2]Yi K, Wang Y, Ren K, et al. Learning topology-agnostic eeg representations with geometry-aware modeling[J]. Advances in Neural Information Processing Systems, 2024, 36.

Meta-review: This work has received feedback from 5 reviewers two of which are championing the paper (revs 1qa8 and gwJp). Novelty of the contribution is challenged by certain reviewers that argue that the work has strong similarities with LaBraM and is strongly inspired by BeiT. Yet the present contribution is here applied to sEEG and not EEG or computer vision, and results suggest that Du-IN is a novel method offering a strong performance on a rich decoding task (yet not on open vocabulary). Besides code is provided at https://anonymous.4open.science/r/Du-IN-3B8F/README.md rather cleanly, and authors promise to release the data upon acceptance which should enable progress on this task by the ML community.

For these reason I recommend this work for acceptance.

Yet, I am asking the authors to clearly acknowledge what figures are paragraphs are taken or inspired from the Labram paper (https://arxiv.org/pdf/2405.18765) (cf fig1, fig2, "transformer encoder" paragraph) in the camera ready version.
Decision: Accept (poster)