Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordings
Authors: Di Wu
Siyuan Li
Chen Feng
Lu Cao
Yue Zhang
Jie Yang
Mohamad Sawan
Venue: ICLR 2025 Poster (2025)
Keywords: brain-computer interfaces; speech decoding; tonal language; Homogeneity-Heterogeneity Disentanglement
Primary area: applications to neuroscience & cognitive science
TL;DR: This paper introduces H2DiLR that disentangles and learns both the homogeneity and heterogeneity from intracranial recordings across multiple subjects, which achieves superior performances on our collected sEEG data from multiple participants.

Abstract: Recent advancements in brain-computer interfaces (BCIs) and deep learning have made decoding lexical tones from intracranial recordings possible, providing the potential to restore the communication ability of speech-impaired tonal language speakers. However, data heterogeneity induced by both physiological and instrumental factors poses a significant challenge for unified invasive brain tone decoding. Particularly, the existing heterogeneous decoding paradigm (training subject-specific models with individual data) suffers from the intrinsic limitation that fails to learn generalized neural representations and leverages data across subjects. To this end, we introduce Homogeneity-Heterogeneity Disentangled Learning for Neural Representations (H2DiLR), a framework that disentangles and learns the homogeneity and heterogeneity from intracranial recordings of multiple subjects. To verify the effectiveness of H2DiLR, we collected stereoelectroencephalography (sEEG) from multiple participants reading Mandarin materials containing 407 syllables (covering nearly all Mandarin characters). Extensive experiments demonstrate that H2DiLR, as a unified decoding paradigm, outperforms the naive heterogeneous decoding paradigm by a large margin. We also empirically show that H2DiLR indeed captures homogeneity and heterogeneity during neural representation learning.

Published as a conference paper at ICLR 2025
TOWARDS HOMOGENEOUS LEXICAL TONE DECODING FROM HETEROGENEOUS INTRACRANIAL RECORDINGS
Di Wu Siyuan Li Chen Feng²
Lu Cao Yue Zhang' Jie Yang Mohamad Sawan 3*
1 Westlake University, Hangzhou, China
2Zhejiagg University, Hangzhou, China
ABSTRACT
Recent advancements in brain-computer interfaces (BCIs) have enabled the decod- ing lexical tones from intracranial recordings, offering the potential to restore the communication abilities of speech-impaired tonal language speakers. How- ever, data heterogeneity induced by both physiological and instrumental factors poses significant challenge for unified invasive brain tone decoding. Traditional subject-specific models, which operate under a heterogeneous decoding paradigm, fail capture generalized neural representations and cannot effectively leverage data across subjects. To address these limitations, we introduce Homogeneity- Heterogeneity Disentangled Learning for neural Representations (H2DiLR), a novel framework that disentangles and learns both the homogeneity and hetero- geneity from intracranial recordings across multiple subjects. To evaluate H2DiLR, we collected serrooeectreeneeehallggaa
(sEEG) data from multiple participants reading Mandarin materials comprising 407 syllables, representing nearly all Man- darin characters. Extensive experiments demonstrate that H2DiLR, as a unified decoding paradigm, significantly outperforms the conventional heterogeneous de- coding approach. Furthermore, empirically confirm that H2DiLR effectively captures both homogeneity and heterogeneity during neural representation learning.
INTRODUCTION
The human language system, with its intricate and expansive syntactic structure, enables rich and complex communication. Decoding spoken language from within human brains has emerged significant topic of interest in neuroscience (Anumanchipalli et al., 2019; Willett et al., 2023; Feng et al., 2023; Lu et al., 2023; Liu et al., 2023). The decoding of vocal tone from brain measurements (Lu et al., 2023; Liu et al., 2023) is of particular research interest, due to the prominence tonal languages, which make up over 60% of the world's languages (Yip, 2002) and are spoken by approximately one-third of the global population (Dryer & Haspelmath, 2013). In these languages, tone plays a critical role in distinguishing lexical meaning at the syllable level.
Mandarin, for instance, is a widely spoken tonal language that has an extensive inventory of over 50,000 characters, with each associated with a syllable composed of initial, final, and tone (Du- anmu, 2007). Mandarin features four tones, each characterized by starting pitch height and contour. The same initial and final components can yield entirely different semantic meanings when uttered with different tones, as illustrated in Fig. 1. For instance, the syllable formed by the initial /b/ and the final /al can represent vastly different concepts depending on the tone: a high-level tone (Tone I) signifies 'eight' (/l), rising pitch contour (Tone 2) indicates 'pull' (#2). low falling-rising tone (Tone 3) means 'handle' (?C), and a high falling tone (Tone 4) translates to 'father' (.). Consequently, precise tone identification crucial for brain sentence decoding of tonal languages.
Recent studies have shown the feasibility of decoding tones using non-invasive seorophysiologiaal signals such electroencephalogramm (EEG) (Yang et al., 2021; Li et al., 2021a) and more promisingly, intracranial recordings such as electrocorticography (ECoG) (Liu et al., 2023). While EEG provides a non-invasive method, BCoG offers superior spatiotemporal resolution and reduced signal attenuation, leading to better decoding performance and interpretability. Nonetheless, the heterogeneity evoked by
Coonsponding Authors yangj @eettt aake edu. and sawan hawangwest lake. en.

---

Published as a conference paper at ICLR 2025
both physiological and instrumental factors is major challenge for invasive brain neural decoding. In particular, physiological variations among subjects and discrepancies in electrode configuration due to diverse electrode implantation conditions prevent unified decoding actoss subjects. Conversely, developing heterogeneous models for each subject leads to poor generalization ability due to data seareity, which is primarily due the cumbersome process of intracranial data acquisition and the inability of participants to wear electrodes for long time periods. Thus, efficiently integrating heterogeneous intracranial recordings from multiple subjects enable consistent decoding remains critical challenge.
To tackle this, we propose two- stage neural representation learn- ing framework called Homogeneity- Heterogeneity Disentangled Learning for neural Representations (H2DiLR) that captures both homogeneous and heterogeneous information from in- tracranial recordings across multiple subjects, enabling unified neural de- coding. The intuition behind proposed H2DiLR R is straightforward: although physiological and instru- mental differences exist among dif'- ferent subjects (heterogeneity), the same brain regions in different in- dividuals exhibit similar functions during the tone production process (homogeneity). Figure provides an overview of H2DiLR in the context of lexical tone decoding. Concretely, in disen- tanglement (H2D) stage (first stage), perform neural feature extraction from the novel perspective of vector quantization (Van Den Oord et al., 2017), where the learned latent neural embedding (neural code) captures dis- criminative pattern-aware information. We further maintain online-optimizable codebooks to store these learned neural codes as matching templates for disentangled neural representation learning: one shared codebook for all subjects to capture homogeneity, while private codebooks for each subject capture heterogeneity. The disentangled H2D neural representations are then used for tone decoding in the second stage.
SEEG
Semantics in Speech
Piich Centours of Manduria Toncs
(/ Eight)
High
4 Mid-lligh
bá (t Pull)
bă (E Handle)
2 Mid-low
bà (8 Father)
Sitgect
Disaangaa cebedding
Puhote Codetwok of
Trander
Scroettic Spece
Stared Codkbok e
Quantiootion
Ereoder Mal
Decoding Heed
Privetc Colchast Ci
Figure 1: Illustration of H2DiLR for unified lexical tone decoding with sEEG from multiple participants. In the homo- heterogeneity disentanglement (H2D) stage, the continuous latent representations from the encoders are disentangled into H2D representations, which are constructed by discretized code embeddings in a shared codebook (homogeneous tone articulation neural codes) and private codebooks (heteroge- neous personalized neural codes). The learned H2D repre- sentations are utilized for tone decoding in the second stage.
the womooenniity-heerrgeee
We comprehensively evaluate the effectiveness of H2DiLR using steroeeletreencchaabggppyy (sEEG) data collected from multiple participants reading Mandarin materials containing 407 sylla- bles. Compared to previous tone decoding studies that focused on smaller subsets of syllables. our study has greater practical relevance. Extensive experiments demonstrate that H2DiLR outperforms traditional subject-specific decoding paradigms by a significant margin. Moreover, we empirically show that H2DiLR successfully disentangles homogencous and heterogeneous components in the learned neural representations, as evidenced by visualizations of neural codes and subject categoriza- tion tests. H2DiLR, neural representation learning, can be applied to various neural decoding applications. Overall, our contributions are as follows:
-urrpose framework for somogeneity-heterggeeee
disentangled
(1) We introduce a neural representation learning framework, H2DiLR, which disentangles ho- mogeneity and heterogeneity, enabling unified neural decoding across multiple subjects with heterogeneous recordings caused by physiological and instrumental factors;
(2) To the best of our knowledge, we the first to perform unified tone decoding across subjects on a comprehensive set of 407 Mandarin syllables, covering nearly all Mandarin characters, providing results of greater practical significance.
2

---

Published as a conference paper at ICLR 2025
(3) Extensive experiments demonstrate the effectiveness of H2DiLR, with average top-1 tone decoding accuracy improvement of 12.2% over conventional subject-specific models. Moreover, empirically verify that H2DiLR successfully captures both homogeneous and heterogeneous neural representations.
2 RELATED WORK
Brain Language Decoding. With the mature application of biomedical circuits and the advance- ment of deep learning, the in-depth exploration of the perceptual and processing mechanisms of the human brain in response to language and speech has attracted increasing research attention in neuro- science. Recent studies have demonstrated the feasibility of decoding language and speech intentions from both non-invasive (Défossez et al., 2023; Tang et al., 2023; Sereshkeh et al., 2018; Si et al., 2021; Deng et al., 2010; DaSalla et al., 2009; Chi et al., 201 1) and invasive neural recordings (Moses et al., 2019; Makin et al., 2020; Moses et al., 2021; Angrick et al., 2019). Early studies primarily focused on the binary classification of language components such as syllables, phonemes, and words from non- invasive brain signals like functional magnetic resonance imaging (IMRI, functional near-infrared spectroscopy (fNIRS), and electroencephallggaphy (DEG))aa et al., 2009; Deng al., 2010; Chi et al., 1). Similar to language decoding, recent work has also demonstrated the feasibility of non-invasively decoding perceived music(Denk et al., 2023). In contrast non-invasive approaches, intracranial electreencephalography such as electrocorticography (ECoG), offers superior spatial resolution and signal-to-nn -onnose ratios, leading to more robust decoding performance (Willett et al.. 2023; Metzger et al., 2023). Anumanchipalli et al. (2019) pioncered sentence-level English decoding using recurrent neural networks (RNNs) to predict Mel-frequency cepstral coefficients (MFCC) from ECoG signals, which were then converted into speech via vocoder (Anumanchipalli et al., 2019). Beyond non-tonal languages, accurate recognition of lexical tones is crucial for decoding tonal languages from brain signals due to their distinctive pitch articulation and their critical role in differentiating lexical meaning(Yan)) (Yang et al., 2021; Li et al., 2021a; Liu et al., 2023; Lu et al., 2023; Li et al., 2021b). For instance, Liu et al. (2023) employed long short-term memory (LSTM) networks to predict Mel spectrograms from ECoG signals, successfully generating sound waves of syllables /mi/ and /ma/ along with their corresponding four tones using the Griffin-Lim algorithm. However, these studies are often limited by small datasets, as data is typically recorded from patients undergoing neurosurgery, resulting in restricted data availability. Guo & Chen (2022) performed multi-class classification of four tones across vowels /al, fi/, /o/, and /u/ using manually extracted features from fNIRS. Despite these advancements, existing studies are mostly confined tone decoding on limited syllables, using small subject-specifie datasets. In this work, we expand upon prior research by performing sull-spectrum tone decoding across all possible sy syllables in Mandarin Chinese using stereoelectroencephallogogg decoding framework that integrates neural recordings from multiple subjects.
(sEEG). Additionally, we propose a unified brain tone
Heterogeneity in Neural Representation Learning. Benefiting from large-scale training corpora, recent breakthroughs in natural language processing (NLP) have demonstrated the exceptional capabilities of large language models as general-purposs task solvers (Brown et al., 2020). Similarly, in computer vision (CV), generative models have shown remarkable performance when trained on extensive datasets (Ho et al., 2020). However, these advances often rely on the assumption that data is independent and identically distributed (IID), which is rarely applicable to neural representation learning due to the heterogeneity inherent in neurological data acquisition. Heterogeneity in neural data manifests in various ways, including physiological and neural differences across individuals, variations in electrode configurations during data collection, and fluctuations in neural activity across recording sessions (Wu et al.. 2023). To address these challenges. several researchers have attempted to train universal models praiisily spatiotemporal encoders by combining data from multiple subjects or sessions to overcome the heterogeneity caused by physiological and neural variability (Zhang et al., 2024; Cai et al., 2023; Jiang et al., 2024; Ye et al., 2024; Fazli et al., 2009; Autthasan et al., 2022; Liu et al., 2021; Ng & Guan, 2024). However, most existing approaches assume homogeneous experimental setups, where the number and placement of electrodes are consistent across subjects. In addressing heterogeneity due to different electrode configurations, LaBraM (Jiang et al., 2024) and MMM (Yi et al., 2023) introduced pre-training strategies based on the standardized 10-20 and 10-10 EEG acquisition systems to mitigate channel compatibility issues during model training. However, these methods are restricted to non-invasive EEG systems,
3

---

Published as a conference paper at ICLR 2025
Domnstream Tasks
Downstream Tasks
VQ Reconstrction
Dowastream Tasks
Downstream Tasks
Decoder
Neuna Decoder
Nounl Dacoder
MLP
MLP
Shured
Sharrd
Clicchook
Cooebook
Entooding
Quantoatin
Quantiotion
Quatiattinn
Quantiction
Privuc Cldcbook
Privaee Colkbook
Encoder
Encoder In
VQ Encoder
VQ Encoder m
VQ Encoder
VQ Eocoder in
M Subject
M W Subject
MM Subject m
Subject
Subject
Subject Slage-2: Neural Decoding
(4) Heterogemzous Training
(b) Stage=1: Hlemm-bbeerogeeeity Disentanglementt
Figure 2: Overview of the proposed H2DiLR learning paradigm compared to the heterogeneous learning paradigm. The VQ encoders, decoders, a shared codebook, and private codebooks are learnable and trained in a self-supervised manner during the H2D stage. It is worth noticing that the VQ decoders are discarded after stage one. In the neural decoding stage, all encoders and codebooks frozen for H2D representation generation, which is used for further decoding with transformers. The red lines and marks denote loss propagation.
limiting their applicability to broader neural decoding tasks. BIOT (Yang et al., 2023) tackled data heterogeneity by introducing handcrafted embeddings to align neural representations across subjects. However, BIOT primarily focuses on developing encoder architectures for multi-dataset training and does not sufficiently address the nuances of homogeneity and heterogeneity across subjects or datasets. In contrast, this work proposes novel learning paradigm that disentangles and models both homogeneity and heterogeneity from heterogeneous neural data, enabling unified neural decoding across subjects.
3 HOMOGENIIY-HETTERGEENTT DISENTANGLED LEARNING FOR NEURAL REPRESENTATION
We first present the overall learning paradigm of H2DiLR comparison to other existing learning paradigms in Sec. 3.1. We then propose unified pattern-aware neural tokenization (UPaNT) in Sec. 3.2, the prerequisite for homogeneity--heerogeneee
disentanglement (H2D). The details of H21) claborated in Sec. 3.3, along with the corresponding model architectures described in Sec. 3.4.
3.1 LEARNING PARADIGM H2DILR
Managing heterogeneity caused by physiological or instrumental factors remains a fundamental challenge in neural representation learning. As illustrated in Fig. 2-(a), existing approaches typically adopt a purely heterogeneous training paradigm, wherein subject-specific models are trained indepen- dently for each individual. This learning paradigm effectively handles heterogeneity with apparent drawbacks: the lack of unified neural representation learning capability across individuals and poor generalization of learned representations, particularly in invasive scenarios with limited data.
To overcome these limitations, we propose novel learning paradigm named H2DiLR, which contains an H2D stage and a neural decoding (ND) stage, as shown in Fig. 2. In the H2D stage, we learn neural representations that capture both homogencous and heterogencous features by leveraging data from all subjects in an unsupervised, task-agnostic manner through vector- quatti-qua (VQ) style reconstruction( Van Den Oord al., 2017). We will elaborate on VQ in Sec. 3.2. At high level, H2D stores homogeneous information in a shared, trainable codebook accessible to all subjects, while subject-specific private codebooks capture heterogeneous information. In the ND stage, the parameters of encoders and cookbooks are frozen for H2D representation extraction. A lightweight transformer adopted for specific downstream neural decoding tasks in supervised manner.
3.2 UNIFIED PATTERN-AWARE NEURAL TOKENIZATION
Prerequisites for maanogeneiit-hhetrrggeeeee Disentanglement. To successfully disentangle ho- mogeneity and heterogeneity in neural recordings across multiple subjects, neural representation learning must meet two key requirements: (i) Unify the latent representation space of hetero-
4

---

Published as a conference paper at ICLR 2025
Cudebook of
Codebook of
Private Codebook o2
Shared Codebook e2
Selecting vi. sharad
(L-iil, private sodes
L
L
LXD
1- al..kaiaa oodas
Gork pinas
Encoder E1
Encoder Es
Encoder Bm
Decoder D;
Decoder Dg
Decoder Di
M
M
m Tx
T'
T
2x 0s2
Subject
Subject!
Subject
Reconstraction
Reconstruction
Reconstruction
Figure 3: Illustration of the proposed Homogeneity subjects, which contains encoders {E,}- decoders {D.) and a shared codebook CS and private codebooks {C," m for quantization. For each sample, 2L tokens are selected from its embedding and discretized with the shared codebook, while the rest of (1 - L tokens are quantized by the corresponding private codebook The red lines and marks denote training loss propagation.
hettrgggnncty-hettteeeentt heterogeneity Disentanglement (H2D) for m
geneous neural recordings for neural decoding. This property allows the neural representation learning algorithm to handle data heterogeneity and build a single decoding model; (ii) Extract features with explicit semantic patterns for further H2D. Take speech decoding for example. The articulation of speech involves the intricate coordination of oral organs, including the tongue, larynx, vocal tract, and others. During vocalization, these organs exhibit explicit muscle movements associated with specific states under neural control (Jürgens, 2009). For instance, the muscles in the larynx bring the vocal cords closer to realize pitch voicing. Consequently, extracting neural representations associated with the opening and closing of the vocal cords is critical for lexical tone decoding. Likewise, capturing additional critical articulation patterns, such as manner of articulation (MOA) and place of articulation (POA), will benefit neural representation learning.
Driven by the above design principles, we propose unified pattern-an aware neural tokenization (UPaNT) to characterize neural patterns (pitch articulation in case) in unified manner during neural representation learning based on vector-quantized (VQ) autoencoding (Van Den Oord et al., 2017)). It's important to note that UPaNT, without H2D, could be considered homogeneous training paradigm since manages to learn neural representations from multiple subjects in a unified manner. The concept of vector quantization (VQ) was initially introduced for learning discrete representations in the context of natural images. Holistically, VQ discretizes the continuous latent representations generated by the encoder by substituting them with the nearest quantized embeddings from an online optimizable codebook and further reconstructs the original input with quantized representations. The discretized embeddings in the codebook demonstrate explicit semantic information (Zhou al., 2022). In vision representation learning, for example, these discretized embeddings often correspond to interpretable patterns like color and texture.
Consider we collect intracranial recordings from m subjects, {S given the same neural decoding task. We assume substantial differences exist among the m sets recordings [S. 1 1 due to electrode configuration variations among subjects. X, XX,xTXCC denotes the of recordings collected from subject i with different number of data samples N,, differentt number of channels C., and the same segment length T. The neural recordings X' are first mapped into latent feature, Eş(x, 0,) RLxD in the continuous latent space by a set of encoders (E;(: {.)) parameterized by network parameters {0,)m² A finite VQ codebook of K key-value pairs, = {(k, e(k))) where each code owns its learnable code embedding e(k) E DR. can discretize each token in 2 by quantization function 2
M, Q(2;:C) argmin [e]k]lz; e(k))l2.
where 2, E IR L denotes the 5th token of z with 1 S < L, M, E [K] T. is code mapping indices.
With assigned M, the latent feature 2 can be indexed and quantized to the VQ embedding by the closest I-of-K embedding vectors in the codebook C as &, = e(M,). Then, a set of decoders (De(; (+)))) th parameterized by { m maps the VQ embedding 2 back to the input space to
5

---

Published as a conference paper at ICLR 2025
reconstruct the original neural recordings a:
② (:) Di(e(M) vt),
(2)
Since differentiation through the quantization in Eq. (1) is ill-posed during gradient backward, the thigghtt-hhouh-sstiiae (STE) (Bengio et al., 2013) is employed as the approximation. i.e. (e(M,) + 23) zj. Overall, the learning objective of VQVAE on X' includes Lree for reconstruction of autoencoders, Leade for the codebook learning, and Ccommit for quantization:
Lvq =
Σ 1 &111 + || Z 1
free
Geode
+
2 sg. 1.
Ccommit
where sg[-] denotes the stop gradient operation and / > 0 is paper-parameter set to 0.25 by default.
3.3 HOMOGINEIT--IITERGGENNNN DISENTANGLEMENT
heterogeneity disentanglement (H2D) based on UPaNT. To capture the homogeneous and heterogeneous neural representations, we first define a shared codebook C = {(k, e [E[KK]] to maintain the common neural embeddings with high-level semantic patterns extracted from all subjects under the same task. We also keep private codebooks (C.) == to encode the unique patterns of m different subjects, where C {(k: ef (k)) )E(KPP) Given the th neural recording sample Tn, from subject we first rank all tokens in the continuous encoded feature An,i with the similarity between the tokens and the corresponding selected nearest code embeddings in the shared codebook C R = rank Zn. - e (M....) )l)/ii)) where rank(-) denotes the ascending ranking. Based on the ranking result, we split the tokens into the homogeneous group and the heterogeneous group, where the top-v tokens (the most similar S tokens for CC) are quantized with the shared codebook, while the rest I tokens quantized using the corresponding private codebook subject The partition factor v E [0. 1] to balance homogeneous and heterogeneous representations, The H2D quantization can be formulated as:
We formalize homogeneity nomogeneitt-hhteroggenigg
2n,as (M Bm,(()) S vL
(M Rn, > vL
The shared codebook C and the private codebooks CP¹² are then updated with two different strategies. The shared codebook is updated by:
2n,1, (1 a-) + a2n.s Rn, SL,
(5)
where a is the momentum coefficient for the exponential moving average (EMA). Note that the EMA update of the codebook in Eq. (5) reduces the training instability caused by updating conflicts the certain code from latent tokens of different subjects (Razavi et al., 2019). The private codebooks are updated as follows:
Ceri Σ
|s (n.i,)] - Eni. Rm,(() > vL.
(6)
Note that CPri is applied only to the rest vL tokens. We use the commitment loss align latent embeddings to the relevant codes as the same design as in VQVAE:
ccom L
| - 2g(2n..,
Meanwhile, our proposed H2D adopts unsupervised autoencoding as the pretext task during neural representation learning using the reconstruction loss from Eq. (3). Overall, the learning objective of H2D is defined as:
m
CHDD
Ch + 3C C
(8)

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with heterogeneous (supervised and self-super methods), homogeneous (UPaNT), and heterogeneity- homooeetttt disentanglement approach (H2DiLR) for brain tone decod-
ing. CL denotes contrastive learning, and MM denotes masked modeling on sEEG. Top-1 accuracy
(%) for fine-tuning evaluations are reported. Bold and underline denote the best and second best.
Paradigm
Method
Pre-training
Backbone
Subject
Subject
3
Subject
Avg.
Sea
CNN
34,94=2.17 36.7345,88
27.1043.22
27.10=1.43
314713,16
FFCL
CNN+LSTM
37,47+2.40
37.71+5,42
26.61+3.188
29.47=1669
32.82+33117
TS-TCC
CL
CNN
39.59
41.47
+2778
28.82
32.33
35.55
Heterogeneous
CoST
CL
CNN
43,95 -1.48 40.41
+4.77
31.02
+1.00 34,53
+2.39
37.47 =2,41
ST-Trassformrr
Transformer
39.02 22.39 37.22
+4668
27.51
+2.11
29.63
+2.8
33.35 +3.04
NeuroBERT
MM
Transformer
=1.80 43.10
+2.76
29.80
1198 32.65
+2.89 36.94
-2.37
H2DiLR (ours)
0
Transforner
43,61 #2.12 42.15
1.63
+1.51
35.92
1143
38.98
1167
Homogeneous
BIOT
MM
Transformer
42.45 =6.99 40.90
+.5.7
33.55
+2,95
33,88
1.89
37.47
22,26
H2DILR (ours)
UPaNT only
Transformer 45.47 =3.04 44.65
1.884 35.59
+2.37
35.76
40.37
+2.11
Disentanglement
H2DiLR (ours) UPaNT+H2D
Transformer 49,06 #2.15 47.84 1181
39.18 11.6 38.61
1,49
43.67
1178
For simplicity, set the homogeneous neural representation component to have the same number of tokens as the heterogeneous component, i.e., V/ = 0.5. The size for the shared codebook K 8 is set to be x KPP resulting in total code size of K 2mK from all codebooks.
3.4 MODEL ARCHITECTURE
The architectures of VQ encoders, VQ decoders, and neural decoders can take on any arbitrary design, provided that they effectively accomplish the reconstruction and disentanglement tasks in the H2D stage, as well as the decoding task in the ND stage. In our work, we discover that very lightweight architectures achieve promising results in lexical tone decoding. The VQ encoders consist of five 1-D convolution layers with a kernel size of four and a stride of two. Due to varying input channel numbers across different subjects, the channel dimension is first mapped by the stem layer and then progressively increased to 512 before being reduced back to 256. The VQ-decoder adopts a symmetrical design to the VQ-encoder, wherein 1-D convolution layers are substituted with transpose convolution layers. For the ND stage, we adopt lightweight transformer as the neural decoder with the multi-head self-attention (MSA) structure with pre-normalization and residual connection as in ViT (Dosovitskiy et al., 2020). The patch embedding is performed by 1-D convolution stem layer with a kernel size of five and stride five to ensure non-overlapping patch embedding. The output dimension of the stem layer is 128, and the hidden dimension of the Feed Forward Network (FFN) is set to 512. A fully connected layer is added to map the output to desired decoding output formats. A detailed description of network architectures is provided in Tab. A2.
uniform count of 64
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Data acquisition for tone decoding. We recruited four participants undergoing epilepsy monitor- ing with stereo electroencephalographh (sEEG) electrodes implanted in the Second Affiliated Hospital of Zhejiang University School of Medicine to participate in this study. The distribution of electrodes for all four participants shown in Fig. 4. The experimental protocol was approved by the review board of the Second Affiliated Hospital of Zhejiang University School of Medicine. All participants gave their written, informed consent prior to testing. For each participant, we selected contacts related to speech and excluded those located in the visual cortex and white matter. All participants are asked to read 407 monosyllabic Mandarin characters, each with a unique tone, three times, covering all common pronunciations of Mandarin characters. To make the pronunciation process of the participants as similar as possible to normal speech, carrier words are added before and after each character to form sentence. Thus, in each trial, participants are required to read a complete sentence containing the target syllable. The collected sEEG signals are downsampled to 1000 Hz with power line interference removed. See Appendix for details.
Evaluation Protocols. We assess decoding performance using the top-1 accuracy (Acc). Data for each participant is divided into an 80% training set and 20% testing set, with 20% of the training data further allocated for validation. We conducted the experiment five times using different random seeds and reported the mean and standard deviation. Refer Appendix B for implementation details.
7

---

Published as a conference paper at ICLR 2025
Participant
Participant
Participant
Participant
Dorsal
Posterior
Right
Anterior
Posterior
Right
Left
Posterior
Anterior
Ventral
Antorior
Left
Figure 4: The anatomy of four participants mapped onto the standard Montreal Neurological Institute template brain, with directions indicated. All chosen contacts for Participant 1 are situated in the left hemisphere, while those for Participants 2 and 3 are in the right hemisphere. Participant Four's selected contacts are distributed across both hemispheres. The brain structures housing these chosen contacts cover most regions associated with speech, including the Superior Temporal Gyrus (STG), Middle Temporal Gyrus (MTG), ventral Sensorimotor Cortex (vSMC), Inferior Frontal Gyrus (IFG), Precentral Gyrus, and Postcentral Gyrus. Additionally, signals from several subcortical structures are recorded, such as the Thalamus, Hippocampus, Insula, and Amygdala.
4.2 DECODING PERFORMANCE COMPARISON
Baseline. We select representative approaches of both the heterogeneous learning and homoge- neous learning paradigms baselines in comparison to proposed homogeneity homogeneity-hhterogeeei -eterogeeeetty disentangled learning paradigm (H2DiLR). For the heterogeneous learning paradigm, we consider three supervised approaches featuring diverse backbone designs (Jing et al., 2023; Li et al., 2022; Song et al., 2021). Additionally, we include three methods utilizing contrastive and masked modeling pre-training (Woo et al., 2022; Eldele et al., 2021; Wu et al., 2022) for a fair comparison, the H2I) stage of our H2DiLR can be regarded as pre-training stage. Furthermore, we consider H2DiLR with O, which indicates no shared codebook, as another variant within the heterogeneous learning paradigm. For the homogeneous learning paradigm, we examine BIOT (Yang et al., 2023) and the UPaNT component of H2DiLR.
Results comparison. We compare the performance of H2DiLR with baselines in Tab. 1. It is observed that baselines of the heterogeneous training paradigm with pre-training outperform those with no pre-training. Heterogeneous approaches generally suffer from the scarcity of data from indi- vidual subjects, where self-supervised pre-training methods prove more effective in capturing neural representations and subsequently improving decoding performance. Although BIOT can integrate data from multiple individuals for homogeneous decoding, its performance does not significantly outperform heterogeneous approaches and even yields worse results compared to heterogeneous decoding methods with pre-training. This is due to the fact that BIOT eliminates data heterogeneity in terms of channel count, sampling rate, and data length but fails to explore and utilize the inherent heterogeneity and heterogeneity embedded in the brain recordings from multiple subjects. Compared to BIOT and heterogeneous baselines, our proposed UPaNT demonstrates a better decoding perfor- mance due to its pattern-awant feature extraction capabilities. With H2D, our proposed H2DiLR further improves the decoding performance on top of UPaNT by disentangling the homogeneous and heterogeneous neural representations, leading significant gain over existing approaches.
4.3 PRELIMINARY VERIFICATION OF H2D
Homogeneous representations correlate with tonal decoding. We first evaluate whether the homogeneous representations learned by our proposed H2D capture unified neural features across subjects in the context of tone decoding. Specifically, visualize the neural codes learned in the
8

---

Published as a conference paper at ICLR 2025
Neural Shared Codabook IG0
Viausbatton Neura Codes Codebzok UPaNT
Tome Blok
Tome
Tome
Figure 6: Reconstructed sEEG from the Amyg- dala (Amyg), Superior Temporal Gyrus (STG), and Comparison of UMAP visualization of Middle Temporal Gyrus (MTG) in the H2D stage, neural codes learned by H2D and UPaNT w.r.t showing that H2D achieves disentangled quantiza- different tone classes.
Figure 5:
tion while preserving heterogeneous information.
shared codebook of H2D in comparison to the codes learned by UPaNT using UMAP (MeInnes et al., 2018). Each neural code is assigned the tone class to which it is most frequently mapped during the quantization process. As shown in Fig. 5, The neural codes learned by UPaNT w.r.t. each tone class are scattered with no clear pattern, while the distribution of neural codes from the shared codebook of H2D demonstrates a clear separation between the Tone and other tones. Since the H2D training stage is unsupervised training, we hypothesize that such clustering effect might correlate with the two key functions of the larynx: pitch voicing and modulation (lowering and rising pitch) (Lu et al., 2023). This is consistent with the fact that Tone 2,3 and 4 involve pitch change but not Tone 1. This finding and superior decoding performance, as seen Tab. I, prove that H2D better captures tone-related neural features by disentangling homogeneity from heterogeneity during neural representation learning.
Heterogeneous representations capture subject-specific information. To verify whether the heterogeneous representations learned by our proposed H2D capture the heterogeneity. we propose a subject classification task to determine whether the learned heterogeneous representation extracts sufficiently discriminative information to classify sEEG signals from different subjects. In particular, we adopt the same network architecture utilized for tone classification this test and follow the same experimental setup described in Sec. 4. As shown in Tab. 2, using heterogeneous representations yields a much higher subject classification performance but an inferior tone decoding performance than the homogeneous representation, suggesting that the heterogeneous representation extracts more subject-specific information rather than tone-related neural features. We also show in Fig. 6 that heterogeneous representations capture instrumental heterogeneity reconstruct sEEG from different brain regions across subjects.
4.4 ABLATION STUDY
This section ablates three key designs and the scaling effect of the network parameters and subject count. The average top-1 accuracy for tone decoding all subjects is reported, and subject classi- fication tasks are designed using the same experimental setup as Sec. 4.1. Furthermore, additional experiments diverse neural decoding tasks demonstrate the effectiveness of the proposed H2DiLR as a general neural representation learning framework, detailed Sec. C.I.
Ablation on v. We first study how the heterogeneous and homogeneous representation partition ratio y influences the learned representations terms of tone decoding and subject classification tasks. By default, H2D sets v 0.5, and a higher value of μ indicates more homogencous information captured during representation learning. With a v value of H2D degenerates to the UPaNT. It is observed in Tab. 2 that smaller value of y leads to better subject classification performance with heterogeneous representation, indicating more heterogeneity captured. Also, a smaller value of leads to inferior tone decoding performances with homogeneous representation. Results show that 0.5 strikes good balance for H2D.
Ablation on codebook size and dimension. By default, H2DiLR utilizes a codebook size and dimension of 256 and a fixed value of 0.5. We study how value changes in size and dimension affect the reconstruction performance in the H2D stage and the tone decoding performance the ND stage. We the mean squared error (MSH) the reconstruction performance metric and report results in

---

Published as a conference paper at ICLR 2025
Table 2: Ablation H2D codebook partition Table 3: Ablation study on the codebook dimension ratio v. Note that 'Homo-' and 'Hetero-" " de- (dim.) ) and the total codebook size in the H2D stage note using homogeneous and heterogeneous of H2DiLR.
representation, respectively.
Method
UPaNT
UPaNT+H2D
Subject
Tone
Value
Code dim.
Code size
Code dim.
Cude size
Homo-
Hetero-
Homo-
Hetero-
Helro
MSE
Acc.
MSE Acc.
MSE Acc.
MSE Acc.
0.25
71.7
85.3
35.2
36.9
42.78
64
0.096 38.86 0.103 38.37 0.092 39.76 0.094
39.27
0.5
71.9
82.6
38.0
37.9
43.67
128
0.081 39.76 0.079 39. 0.079 42.78 0.077 42.37
0.75
73.1
80.7
38.5
35.4
41.31
256
0.079
40.37 0.079 40.37 0.076 43.67 0.076 43.67
1.0
73.5
40.4
40.37
512
0.078
40.08 0.071 40.41 0.073 43.35 0.071 43.59
Table 4: Scaling-up up network parameters (embedding dimension and layers) for better performances.
Method
Encoder
Transformer
Subject
Subject 2
Subject
Subject 4
Avg.
UPaNT+H2D
ConvNet-4-64
Transformer--128
49.06 +2.15
47.84
=181 39.18
1168 38.61 11.9 43.67 1177
UPaNT-H2D
ConvNet-4-128
Transfomer-4-128
50.38 + 1.81
48.57 + 1.69 40.04 + 1.44 39.57 + 1.68 44.64 + 1.72
UPaNT+H2D
ConvNet-4-64
Transformer-82255
52.12 + 1.55
49.62
+ 1.54 41.23 + 1.73 40.36 + 1.25 45.82
+ 1.63
UPaNT+H2D
ConvNet-4-128 Transformer-8-256
52.27 2.02
49.56
1.87 41.39 + 160 40.58 1.54 45.95 + 1.65
Table 5: Scaling-uy up participants count to verify the learned representations across participants.
Participant Count
Participant 1
Participant 2
Participant 3
Participant 4
H2DiLR (m = 2)
46.25 1,89
45.73 11.83
H2DiLR (m = 3)
48.16 + 2.04
46.25 + 1.61
38.22 2.05
H2DiLR (m = 4)
49.06 +2.15
47.84 +1.81
39.18 =
38.61 +1.49
Tab. with the default settings greyed out. We observe that codebook sizes and dimensions larger than 128 lead to quite similar reconstruction performances, while a size and dimension of 64 yield much worse reconstruction performance due to the limited expressive capacity. It is worth noticing that better reconstruction does not necessarily mean a better decoding performance. We hypothesize that neural codes that reconstruct signal the best might not be optimal for neural decoding.
Ablation on scaling-up network parameters. To verify the scaling-up effects of network parame- ters, we double the embedding dimension and the depth of network blocks. ConvNet-N-D represents ConvNet encoder with N layers and D-dimensional input embedding, with ConvNet-4-64 being the baseline setup as in Tab. 1. For the Transformer, Transformer-N indicates a model with N layers and D-dimensional embedding, with Transformer-4- 128 being the baseline. As demonstrated in Tab. 5, enlarging the Transformer model by both deepening and widening it (as in the Transformer-8-255 configuration) leads more significant performance improvements compared merely widening the encoder. Moreover, the combined scaling of both the encoder and the Transformer models to their larger configurations yields the highest performance enhancements across all participants tested.
Ablation on scaling-up subject count. As demonstrated in Tab. 1, we presented the decoding performance improvement achieved by H2DiLR using data from all four participants (m = 4). To provide better understanding of how varying subject numbers affect model performance, we conduct an additional ablation study focusing Without loss of generality, we analyzed decoding performance with participants and 2 for m = 2. and participants 1, 2, and 3 for = 3. The preliminary results, as seen from Tab. 5, indicate consistent trend: an increase in the number of participants leads to improved decoding performance.
varying the number of participants (ranging from 2 to 4).
5 CONCLUSION AND LIMITATION
This paper presents manoggnniity-hetrrogneeeee disentangled learning for neural representations (H2DiLR), which disentangles and models the homogeneity and heterogeneity from intracranial recordings of multiple subjects for neural decoding. Extensive tone decoding experiments on collected sEEG of multiple participants reading Mandarin suggest that H2DiLR enables unified tone decoding across subjects with superior performance compared to existing methods. We list three potential limitations of this work: (1) The generalization of the trained H2DiLR on unseen new subjects, which is of great practical value, remains to be explored. (2) Additional interpretability of the learned neural codes is required. Establishing mapping between learned neural codes with functionalities of different brain regions for better interpretability remains promising future research direction. (3) Due to the complexity of intracranial recording data acquisition, the current constraints prevent us from expanding our subject pool further.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by ST12030-Major Projects (20222D0208805)) the National Natural Science Foundation of China (Grant No. 623B2085), and the "Pioneer" and "Leading Goose" R&D Program of Zhejiang (2024C03002).
REFERENCES
Miguel Angrick, Christian Herff, Emily Mugler, Matthew C Tate, Marc W Slutzky, Dean J Krusienski, and Tanja Schultz. Speech synthesis from ecog using densely connected 3d convolutional neural networks. Journal of neural engineering, 16(3):036019, 2019. 3
Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang. Speech synthesis from neural decoding of spoken sentences. Nature, 568(7753) 498, 2019. 1,3
Phairot Autthasan, Rattanaphon Chaisaen, Thapanun Sudhawiyangkul, Phurin Rangpong, Suktipol Kiatthaveephong. Nat Dilokthanakul, Gun Bhakdisongkhram, Huy Phan, Cuntai Guan, and Theer- awit Wilaiprasitporn. Min2net: End-to-end multi-task learning for subject-independen motor imagery eeg classification. IEEE Transactions Biomedical Engineering, 696)):105-22188, 2022. doi: 10. 20.1000/
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. International
Conference Learning Representations (ICLR), 2022. 16
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through
stochastic neurons for conditional computation. arXiv preprint arXiv: /30..:338333404 3432, 2013. 6
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:18 3
Donghong Cai, Junru Chen, Yang Yang, Teng Liu, and Yafeng Li. Mbrain: A multi-channel self- supervised learning framework for brain signals. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1:30-141,5 2023.
Xuemin Chi, John B Hagedorn, Daniel Schoonover, and Michael D' Beg-based discrimination of imagined speech phonemes. International Journal of Bioelectromannttssm, 1344:2001-206 2011.
Charles S DaSalla, Hiroyuki Kambara, Makoto Sato, and Yasuharu Koike. Single-trial classification of vowel speech imagery using common spatial patterns. Neural networks, 2334411334 2009. 3
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decod- ing speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10): 1097-1107, 2023.
Siyi Deng, Ramesh Srinivasan, Tom Lappas, and Michael D'Zmura. Beg classification of imagined syllable rhythm using hilbert spectrum methods. Journal of neural engineering, 7(4):046006, 2010.
Timo Denk, Yu Takagi, Takuya Matsuyama, Andrea Agostinelli, Tomoya Nakai, Christian Frank, and Shinji Nishimoto. Brain2music: Reconstructing music from human brain activity. arXiv preprint arXiv:2307. 1078, 2023. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020. 7
Matthew Dryer and Martin Haspelmath. The world atlas of language structures online. 2013. 1
San Duanmu. The phonology of standard Chinese. OUP Oxford, 2007.
11

---

Published as a conference paper at ICLR 2025
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-serie representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2352-2359, 2021. 8, 16, 17
Siamac Fazli, Cristian Grozea, Marton Danoczy, Benjamin Blankertz, Florin Popescu, and Klaus- Robert Müller. Subject independent eeg-based bei decoding. In Y. Bengio, D. Schuurmans, Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 22, Curran Associates, Inc., 2009. 3
Chen Feng. Lu Cao, Di Wu, En Zhang, Ting Wang, Xiaowei Jiang, Heng Ding, Chenhao Zhou, Jinbo Chen, Hui Wu, al. A high-performance brain-sentence communication designed for logosyllabic language. bioRxiv, pp. 2023-11, 2023.
Zengzhi Guo and Fei Chen. Decoding lexical tones and vowels in imagined tonal monosyllables using fiirs signals. Journal of Neural Engineering, 19(6):66007,, 2022. 3
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 6851, 2020.
Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large brain model for learning generic represen- tations with tremendous eeg data in bei. arXiv preprint 1.Xi::24005.8765. 2024.
Jin Jing, Wendong Ge, Shenda Hong, Marta Bento Fernandes, Zhen Lin, Chaoqi Yang. Sungtae An, Aaron F Struck, Aline Herlopian, loannis Karakis, et Development of expert-level classification of seizures and rhythmic and periodic patterns during eeg interpretation. Neurology, 100(17): c1750-e1762, 2023. 8, 16, 17
Uwe Jürgens. The neural control of vocalization in mammals: a review. Journal of Voice, 23(1):1-10. 2009. 5
Hongli Li, Man Ding, Ronghua Zhang, and Chunbo Xiu. Motor imagery eeg classification algorithm based cnn-lstm feature fusion network. Biomedical signal processing and control, 72:103342 2022. 8, 16, 17
Mingtao Li, Sio Hang Pun, and Fei Chen. Mandarin tone classification in spoken speech with eeg signals. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 1163-1166. IEEE, 2021a. 3,3
Siyuan Li, Di Wu, Fang Wu, Zelin Zang, and Stan.Z.Li. Architecture-agnostic masked image modeling- from vit back to cnn. In Proceedings of the International Conference on Machine Learning (ICML), 2023. 16
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng, Yufei Huang, and Stan Z. Li. Vqdna: Unleashing the power of vector quantization for multi-species genomic sequence modeling. In Proceedings of the International Conference on Machine Learning (ICML), 2024. 16
Yuanning Li, Claire Tang, Junfeng Lu, Jinsong Wu, and Edward F Chang. Human cortical encoding of pitch in tonal and non-tonal languages. Nature communications, 12(1):1 161, 2021b. 3
Shuaiqi Liu, Xu Wang, Ling Zhao, Jie Zhao, Qi Xin, and Shui-Hua Wang. Subject-independen emotion recognition of eeg signals based on dynamic empirical convolutional neural network. CEEE/ACM Transactions on Computational Biology and Bioinformatics, 1700-177,) 2021. doi: 10. 10.1100/TCBBB.220003018
Yan Liu, Zehao Zhao, Minpeng Xu, Haiqing Yu, Yanming Zhu, Jie Zhang, Linghao Bu, Xiaoluo Zhang, Junfeng Lu, Yuanning Li, et al. Decoding and synthesizing tonal language speech from brain activity. Science Advances, 9(23)). 2023. 1,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations (ICLR), 2019. 16
12

---

Published as a conference paper at ICLR 2025
Junfeng Lu, Yuanning Li, Zehao Zhao, Yan Liu, Yanming Zhu, Ying Mao, Jinsong Wu, and Edward F Chang. Neural control of lexical tone production in human laryngeal motor cortex. Nature Communications, 14(1):6917, 2023. 3,9
Joseph G Makin, David A Moses, and Edward Chang. Machine translation of cortical activity to text with an encoder-decoder framework. Nature neuroscience, 23(4):575-582, 2020. 3
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 180.v:480.003426 2018. 9
Sean L. Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian Dougherty, Jessie R Liu, Peter Wu, Michael A Berger, et al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976): 1037-1046, 2023.
David A Moses, Matthew K Leonard, Joseph G Makin, and Edward F Chang. Real-time decoding of question- and-answer speech dialogue using human cortical activity. Nature communications, 10 (1):3096, 2019.
David A Moses, Sean L Metzger, Jessie R Liu, Gopala K Anumanchipalli, Joseph G Makin, Pengfei F Sun, Josh Chartier, Maximilian E Dougherty, Patricia M Liu, Gary M Abrams, et al. Neuroprosthe- sis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine, 385(3):217-227, 2021. 3
Han Wei Ng and Cuntai Guan. Subject-independent meta-learning framework towards optimal training of eeg- classifiers. Neural Networks, pp. 106108, 2024. 3
Ali Razavi, Aäron Van Den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 6
Alborz Rezazadeh Sereshkeh, Rozhin Yousefi, Andrew T Wong. and Tom Chau. Online classifica- tion of imagined speech using functional near-infrared spectroscopy signals. Journal of neural engineering, 16( (6(1):060005 2018. 3
Ali Shoeb and John Guttag. Application of machine learning to epileptic seizure detection. In ICML 2010 Proceedings, 27th International Conference on Machine Learning, pp. 975-982, 08 2010. 16
Xiaopeng Si, Sicheng Li, Shaoxin Xiang, Jiayue Yu, and Dong Ming. Imagined speech increases the hemodynamic response and functional connectivity of the dorsal motor cortex. Journal of Neural Engineering, 18(5)0056448, 2021.
Yonghao Song, Xueyu Jia, Lie Yang, and Longhan Xic. Transformer-base learning for eeg decoding. arXiv preprint 1rXii:2106.111700 2021. 8, 16, 17
ppatial-tempore feature
Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858-866, 2023. 3
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6309-6318, Red Hook, NY, USA, 2017, Curran Associates Inc. ISBN 9781510860966 2, 4,
Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew Glasser, Leigh R Hochberg, Shaul Druckmann, et al. A high- performance speech neuroprosthesis. Nature, 62077976) 2023. 1,3
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal- -rend representations for time series forecasting. In International Conference on Learning Representations (ICLR), 2022. 8, 16, 17
Di Wu. Siyuan Li, Jie Yang, and Mohamad Sawan. neuro2vec: Masked fourier spectrum prediction for neurophysiological representation learning. ArXiv, abs/2204. 12440, 2022. 8, 16, 17
13

---

Published as a conference paper at ICLR 2025
Di Wu, Jie Yang. and Mohamad Sawan. Transfer learning electromyographhy (emg) tasks: approaches and beyond. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023. 3
Chaoqi Yang, M Brandon Westover, and Jimeng Sun. Biot: Biosignal transformer for cross-data learning in the wild. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: https://openzeviie nwt. Fot.://oooo..dec2222yleeee
8. 16, 17
Shiau-Ru Yang, Tzyy-Ping Jung. Chin-Teng Lin, Kuan-Chih Huang, Chun-Shu Wei, Herming Chiueh, Yue-Loong Hsin, Guan-Ting Liou, and Li-Chun Wang. Recognizing tonal and nontonal mandarin sentences for eeg-based brain-computer interface. IEEE Transactions on Cognitive and Developmental Systems, 14(4):1666-16) 1677, 2021. 1,3
Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer 2: multi-context pretraining for neural spiking activity. Advances in Neural Information Processing Systems, 36, 2024. 3
Ke Yi, Yansen Wang, Kan Ren, and Dongsheng Li. Learning appology-annostin eeg representations with geometry-awant modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Moira Jean Winsland Yip. Tone. Cambridge University Press, 2002.
Daoze Zhang, Zhizhang Yuan, Yang Yang, Junru Chen, Jingjing Wang, and Yafeng Li. Brant: Foundation model for intracranial neural signal. Advances in Neural Information Processing Systems, 36, 2024.3
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URL https:/ / oppnrevii ew, net/ fo rum? d=ydopy-e60g.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A TONE DECODING DATA AQUISITION
This work proposes to use stereotactic electroencephalograply (sEEG) as means of collecting intracranial neurophysiological data for tone decoding. sEEG is a novel international technique that has emerged in recent years as a localization method for epileptic foci. This technique simultaneously records the brain electrical activity of epilepsy patients in different cranial structures, involving many brain networks associated with advanced cognitive functions, such the hippocampus, frontal lobe, amygdala, cingulate gyrus, parietal lobe, and precuneus. Patients can undergo assessments and tests of advanced cognitive functions during the interictal period (when there are no symptoms of epilepsy, and the patient's 's behavior is indistinguishable from that of a normal, healthy person). Therefore, sEEG is currently recognized an invasive method for studying advanced cognitive functions in the human brain and does not pose additional risks, such as cranial trauma for patients during the research process.
Table A1: Participant characteristics, including sex, age, education level, and handedness
Subject Number
Subject 1 male
Subject 2
Subject 3
Subject 4 female
sex age education
male
female
20 Bachelor's right handed
25
19 Bachelor's right handed
19
High school
High school right handed
handedness
right handed
In this study, we recruited four participants undergoing epilepsy monitoring with stereo electroen- cephalograph (sEEG) electrodes implanted in the Second Affiliated Hospital Zhejiang University School of Medicine participate in this study. The distribution of electrodes for all four participants is shown in Fig. 4. We also provide basic information on participants, including sex, age, education level, and handedness in Tab. A1. The experimental protocol was approved by the institutional review board of the Second Affiliated Hospital Zhejiang University School of Medicine. All participants gave their written, informed consent before testing. For each participant, we selected channels related to speech and excluded those located in the visual cortex and white matter. All participants are asked to read 407 monosyllabic Mandarin characters, each with unique tone, three times, covering all common pronunciations of Mandarin characters. A comprehensive list of characters and their corresponding syllables is provided in Tab. A5 through Tab. A8. To make the pronunciation process of the participants as similar as possible to normal speech, carrier words are added before and after each character to form a sentence. Therefore, in each trial, participants must read a complete sentence containing the target syllable. For example, if the target syllable is 'ài', the sentence presented and to be read by the participant is "R E (I read love three times). Our reading material is carefully designed by Mandarin linguists to cover many pronunciation phenomenons as possible to enable a brain decoding algorithm with generality. The reading material contains syllables of four tones subject to uniform distribution. has no specific pitch contour and is typically used on less emphasized syllables where the preceding syllable primarily influences its pitch. It is not considered fifth tone in addition to the four tones but rather special tonal variation of Tone 4, which physically manifests itself as a shortening of the length of the tone and a weakening of the strength of the tone. Consequently, we do not treat the neutral tone as an additional fifth class.
worth noticing that there is an additional neutral tone, which
Neural signals were recorded using a multi-channel electrophysiologccal recording device, specifically the Neurofax EEG-1200 produced by Nihon Kohden Corporation. Japan, and were recorded at sampling rate of 2000Hz. Each channel was subjected to visual and quantitative inspection for artifacts or excessive noise. We also record the audio signals of participants to provide time stamps for slicing the targeted syllable from the sentence being read. The neural signals were low-pass filtered at 300 Hz and notch filtered at 50 Hz, 100 Hz, 150 Hz, and 200 Hz to remove power line interference. The signals were then downsampled to 1000 Hz. Signal fragments are padded to the maximum length of 1000.
15

---

Published as a conference paper at ICLR 2025
Table A2: Detailed architecture specifications for models utilized in the H2D stage and ND stage in this work. AvgPool stands for average pooling in ConvNet and rel. pos. denotes relative positional encoding in Transformers.
H2D Stage ConvNet 4x 64, stride 2
ND Stage Transformer
Stem
5x 1, 128, stride 5
MSA, 128, rel. pos.
1x1, 128 AvgPool, stride 2
Block
1 1, 512
1x1, 128
MSA. 128
4x1, 256
Block
AvgPool, stride 2
x
1x1, 512
x
1x1, 128
MSA, 128
1x 512 AvgPool, stride 2
Block
x
1x 512
x
1x1, 128
MSA, 128
Block
14x1, 256]
1x1, 512
1
1x1. 128
# params.
1.55 x 10"
0.958 x 10"
B IMPLEMENTATION DETAILS
All our experiments are implemented by PyTorch and conducted on workstations with NVIDIA A100 GPUs. For all baselines (Woo et al., 2022; Eldele et al., 2021; Wu et al., 2022: Yang et al., 2023) with open-source code, we reproduce results using the official code and setups provided by the authors. We use the implementation provided by BOIT (Yang et al., 2023) for baselines with no publicly available codes (Jing et al., 2023; Li et al., 2022; Song et al., 2021). I). For baselines with pre-training involved, we train each model with fixed epoch number of 40. We AdamW as the optimizer with a base learning rate of 3e-4. Cosine annealing decay is adopted for learning rate scheduling. We momentum factors B1 B2 0.9, 0.999 with weight decay of 0.01. The batch size is set to 32. A dropout ratio of 0. is adopted. For the pre-training stage of baselines with pre-training, we follow the experimental setup provided by the authors. Next, we give the pre-training details for our proposed UPaNT and H2D. Following masked modeling methods (Wu et al., 2022: Li al., 2023) with VQ tokenizers (Bao al., 2022; Li al., 2024), we use Adam W (Loshchilov Hutter, 2019) optimizer with a base learning rate of 5e-5, 31, B2 0.9, 0.999, and the weight decay of 0.01. A cosine learning rate schedule is also adopted. We train fixed number of 1000 epochs during pre-training with batch size 32. For UPaNT and H2D, we use 3 0.25 and 0.5 with total codebook size of 256, i.e., K = 256 for UPaNT only and K = 32 for H2D. For the fine-tuning stage of all approaches with pre-training, for baselines with no pre-training, with the only difference being a learning rate of 5e-5.
adopt the same training setup previously described
C ADDITIONAL ABLATION STUDY
C.1 VERIFICATION OF H2DILR ON SEIZURE PREDICTION
Our proposed Homogeneity- Heceettgnttttttreeneett Disentangled Learning for Neural Representations (H2DiLR) is primarily inspired by the challenges of heterogeneous neural decoding from intracranial recordings, particularly in tasks involving the decoding of lexical tones where electrode placement and count may vary across subjects. To showcase its versatility, we extend the application of H2DiLR beyond its initial focus. As a proof of concept, we demonstrate its effectiveness representation learning framework in a different neural decoding task task-epilepsy seizure prediction. This demonstrates H2DiLR's adaptability and potential applicability across range of complex neural decoding challenges.
general neural
Dataset and Pre- Pre-processing. The publicly available CHB-MIT (Shoeb & Guttag, 2010) dataset includes 637 recordings from 23 epileptic patients. Signals were collected from 22 bi-polar electrodes placed according to the international 10-20 system of 256 Hz sampling frequency and 16-bit resolution. Clinical experts annotated the start and end times of seizures through visual inspection. Only patients
16

---

Published as a conference paper at ICLR 2025
Table A3: Comparison with heterogeneous (supervised and self-supervised methods), homogeneous (UPaNT), and saccrrocneity-homoggee disentanglement approach (H2DiLR) for epileptic seizure prediction. CL denotes contrastive learning, and MM denotes masked modeling with sEEG signals.
Top-1 accuracy (%) for fine-tuning evaluations are reported. Bold and underline denote the best and second best results.
Paradigm
Method
Pre-traiming
Backhone
Acc(%6]
Sen(5) 1
FPRC/1 1
SPaRCNet (Jing el al., 2023)
CNN
84.33413:26
86.74.12.94
0.18
FTCL (Li et al., 2022)
CNN+LSTM
85.2112.588
89.37:15.42
0.16:10.02
Heterogenoous
ST-Transformer (Song et al., 2021)
Transformer
85.74 1:239
91.84
43.26
0.22 10.05
TS-TCC (Eldele al., 2021)
CL
CNN
86.16
90.42
+2.78
0.20 +00.4
CoST (Woo et al., 2022)
CL
CNN
18,45
92.19
14.77 0.18 10.04
NeuroBERT (Wu et al., 2022)
MM
Transformer
89.36 #2.12 92.54
12.98 0.15 10.06
BIOT (Yang ot al., 2023)
MM
Transformer
87.22 #2.41 90.53
11.8
0.17 =0.04
Homogeneous
H2DiLR (Ours)
UPaNT only
Transformer
+2.34
+1.95
0.13
Disentanglement
H2DiLR (Ours)
UPaNT+HED
Transtormer
93.97 +2.16 94.82 +1,27 0.10 00002
whose channel configurations remained consistent throughout data collection were included in our study. For seizure prediction, 'lead seizure' is defined as one occurring after seizure-free interval of at least four hours. We set the seizure prediction horizon (SPH) and preictal interval length (PIL) at 5 and 30 minutes, respectively. Interictal samples are generated using 20-second non-overlapping sliding window, and preictal samples with a 20% overlapping 20-second window to address sample imbalance. All samples were normalized channel-wis by subtracting the mean and dividing by the standard deviation to standardize the input data for further analysis.
We follow the same experimental setup described in Sec. B and report the averaged accuracy (Acc), sensitivity (Sen), and false positive rate (FPR) across all subjects over five random seeds in Tab. A3. The prediction performance observed for the baseline approaches mirrors the patterns seen in the lexical tone decoding task, where heterogeneous baselines with pre-training outperform those without, demonstrating enhanced feature extraction capabilities. BIOT does not significantly outperform the heterogeneous approaches and. in some cases, yields inferior results compared to pre-trained heterogeneous decoding methods. This can be attributed to BIOT's primary focus on addressing data heterogeneity in terms of channel count, sampling rate, and data length without adequately tackling the inherent heterogeneity insight: unlike in computer vision and natural language processing, merely aggregating neurological data into a larger dataset does not necessarily enhance performance. In contrast, our proposed UPaNT showcases superior prediction performance due to its pattern-aware feature extraction capabilities. Building on this, H2DiLR, through its ability to disentangle homogeneous and heterogeneous neural representations, significantly enhances prediction performance.
brain recordings across different subjects. This underscores a key
Table A4: Brain anatomical area contribution analysis in tone decoding for both cortical and subcorti- cal issues. The contribution score normalized to the range between zero and one.
Brain region
Brain structure
Contribution
Thalamus
Subcortical
0.92
Hippocampus
Subcortical
0.85
Amygdala
Subcortical
0.83
Precentral Gyrus
Cortical
0.97
Insula
Cortical
0.85
Postcentral Gyrus
Cortical
0.94
Inferior Frontal Gyrus
Cortical
0.90
Superior Temporal Gyrus
Cortical
0.85
Supramarginal Gyrus
Cortical
0.60
Angular Gyrus
Cortical
0.66
D ANATOMICAL REGION CONTRIBUTION
We study the contribution of different anatomical brain regions in the task of tone decoding. Due to the dtal-stage training paradigm of our H2DiLR framework and the challenges of differentiating through quantization during backpropagation, we cannot directly analyze the contributions of specific regions during neural decoding. To address this, conducted separate analysis by combining
17

---

Published as a conference paper at ICLR 2025
the nomngeneity-heeerogeeeitt disentanglement (H2D) pre-training and neural decoding stages into a single-stage model. We applied cross-entropy loss with a Transformer decoder to quantized representations, using straight-through estimator (STE) to approximate gradients. Please note, however, that this setup may introduce training instability and slight performance degradation; is used solely analyze region contributions. We calculated channel saliency scores through class activation maps, normalized between 0 and I, and aggregated them across brain regions linked to speech processing (e.g., Superior Temporal Gyrus, Precentral Gyrus, Postcentral Gyrus), as well as reference regions like the Angular Gyrus. Our findings are summarized in Tab. A4. Our analysis indicates that the Precentral and Postcentral Gyrus are key contributors among cortical regions, with subcortical regions like the Thalamus also playing significant role in tone decoding.
E BROADER IMPACTS
The aaanogeneity-hetrrggeeee disentangled learning for neural representations (H2DiLR) pre- sented in this paper aims to advance domains of both fundamental machine learning algorithms and neuroscience.
From the standpoint of algorithmic progress, H2DiLR marks eepp forward in the field of neural representation learning. By addressing the inherent data heterogeneity in neural recordings from multiple subjects, H2DiLR offers a novel approach to disentangling homogeneous and heterogeneous neural representations. This breakthrough has implications beyond the specific application domain of tonal language decoding. as it lays the groundwork for developing more generalized and adaptable neural representation learning algorithms. H2DiLR's s ability to capture both homogeneous and heterogeneous neural patterns during representation learning provides researchers with a powerful tool for uncovering underlying neural structures and dynamics that may be obscured by data heterogeneity, Moreover, the framework's 's capacity to learn generalized neural representations across diverse subjects holds promise for realizing large models similar LLM facilitate wide range of applications, including cognitive neuroscience, clinical diagnosis, and brain-computer interface design.
In terms practical applications, H2DiLR extends the field of neural prosthesis, particularly in the context of restoring communication abilities for speech-ine impaiied individuals who speak tonal languages. The ability to decode lexical tones from intracranial recordings using deep learning algorithms has profound implications for speech prosthesis systems tailored to tonal language speakers. By leveraging H2DiLR's unified decoding paradigm, researchers and developers can design more robust and adaptive speech prosthesis systems that accommodate individual variations in neural activity while also facilitating more natural and efficient communication for users.
From the perspective of neuroscience, the proposed H2DiLR has the potential to enhance the interpretability and transferability of neural models, thereby contributing to the understanding of brain function and dynamics. The disentanglement of homogeneous and heterogeneous neural representations facilitated by H2DiLR provides researchers with a clearer insight into the underlying neural processes involved in various cognitive tasks and behaviors. By mapping learned neural codes and electrodes, H2DiLR helps elucidate how different brain regions encode information and interact with each other, shedding light the complex mechanisms underlying cognitive functions such as language processing and motor control.
One concern with H2DiLR, as with any technology involving intracranial recordings, is the risk of privacy invasion. Decoding thoughts or speech could potentially access highly personal information without proper consent or awareness. If such data were misused or accessed by unauthorized parties, it could lead to significant privacy breaches.
18

---

Published as a conference paper at ICLR 2025
Table A5: The first set of 407 Mandarin Chinese characters used as reading material, with corre- sponding syllables organized by initial phonetic order. For infrequently used characters, participants were presented directly with the syllable pronounce.
Characters
Syllables
Characters
Syllables
Characters
Syllables
chè
HB
dião
X
ài
X chèn
dié
32
302 chéng
[ ding
50
áigg
chi
diā
ão
SR
chóng
A döng
3
bá
chóu
dou
bái
[
chü
13t
dú
15
băn
chuä
chuä
duăn
bāng
1000
chuäi
x
dui
bão
chuän
dün
K6
bēi
chuáng
+
duó
bēn
chui
36
01
böng
chún
8
en
4
bí
chuō ēng
ēng
biän
X
ci
ér
1
bião
X
cōng
11
fá
bië
A còu
făn
bin
cu
fäng
771
bing
cuàn
AE
féi
18
bó
cui
8)
fön
#/
bü
7F
cún
[X]
fēng
#
cā
cuŌ
fú
M
cái
#]
då
75
föu
7
cán
dài
f+J
fù
cāng
dăn
1/t
gå
cáo
9 däng
gåi
10
cè
T] dào
ii
gän
cän
dé
gäng
céng
15 děi #:
gâo
chá
dèn dèn
$9]
gē
chái
dēng
4
gëi
chăn J di
gēn
chàng dia
gēng
chão F dian
#
góng
19

---

Published as a conference paper at ICLR 2025
Table A6: The second set 407 Mandarin Chinese characters used as reading material, with corresponding syllables organized by initial phonetic order.
Characters
Syllables
Characters
Syllables
Characters
Syllables
góu Fi
jīn
X
lêng
fli
gū
jing
33
лT gua
jiông
1145
liä
f guài Y
jiù
lián
guan
246
jü
F
liáng
guāng
juān
Nop
lião
gui
jué
liè
gün
192
jün
**
lín
guó
R
kä
līng
4
käi
80
liü
1%
hái
T7
kän
lóng
X
hán
käng
138
lóu
07
háng
#
kão
lù
#6
hão #$
kē
55
luăn
kè
#C
lún
hei H
kěn
luó
11
hèn 3/t
kēng
Y
lü
héng
FL
köng
lüè
hōng
kōu
mä
höu
kù
mäi
hä zt
kuä
102
màn
huà BAL kuài
It:
máng
huai T
kuän -B
máo
#6
huàn XE
kuáng
mēi
huäng
kui
["]
mén
16
hui
kün
měng
+
hün
[
kuò
*
mi
huò
lä
[H]
miàn
*
lái
miáo
40
jià
län
K
miè
jiān
lang
mín
jiäng
lão
ming
2
jião
lëi
136
miù
jië
16i
mó
20

---

Published as a conference paper at ICLR 2025
Table A7: The third 407 Mandarin Chinese characters used as reading material, with corre- sponding syllables organized by initial phonetic order. For infrequently used characters, participants were presented directly with the syllable pronounce.
Characters
Syllables
Characters
Syllables
Characters
Syllables
1i
móu
pén
#
ru
* mù
Till
pêng ruá ruá
pí
EX ruăn
näi
110 piän
01
rui
9
nán
piào
YE
rün
$
náng
piä
553
ruò
nào
pin
să
1/3
ping
sâi
[X]
nèi X
pô
<0=
sän
nèn
T1
pou
säng
néng a6
pü
são
28
ní
50
qí
X2
sè
4
nián #
qiā
sën
A niáng
qián
sēng
nião f
qiäng
shä
1
nië
qiáo
Off
shäi
15
nín 97
qiè
[A]
shän
ning
qín
[9j
shäng
4
niú
qing
shão
æ
nóng
93
qióng
[
she
nou
qiú
idf
shuí
1x
nú
HX
qũ
74:
shén
nuän
quàn
shéng
3
nuó
li
que
Zi
shí
qún
F
shöu
nüè
rán
shō
i.::
rång
ki
shuä
öu
ráo
jip
shuài
17
pá
rè
##
shuän
[4]
pāi
rēn
shuäng
pán
#7
rēng
shuï
39
páng
H
ri
100
shùn
30
pão
róng
100
shuè
péi
[A
rôu
3F
sī
21

---

Published as a conference paper at ICLR 2025
Table A8: The fourth of 407 Mandarin Chinese characters used reading material, with corresponding syllables organized by initial phonetic order.
Characters Syllables
Characters
Syllables
Characters
Syllables
söng 4
wü
4
zão
1 söu
DE
xi
zé
S
sü R
xiá
zéi
suän
xiän
.2%
zēn
[6]
xiàng
7ēng
161
sün
xiào
zhä
101
suő
xiě
föt
zhài
/
tă
C
xïn
zhán
19
tái
xing
zhäng
18
tán
xióng
zhão
80
täng
1%
xiū
zhē
tão
F
xů
i
zhè
tè
xuăn
++
zhēn
téng
xué
#
zhēng
xúń
zhí
tián
3
yá
B+
zhöng
tiáo
yän
/4
zhōu
tič
yäng
*
zhū
tíng
yào
BG
zhuä
töng
By
yē
zhuäi
160
tōu
X
yi
zhuän
tú
yín
11
zhuàng
tuán
188
yīng
i
zhuī
M8
tuī
189]
yö
zhūn
tün
*
yöng
KSE
zhuō
16
tuō
7111
you
577
zi
yu
Ad.
zöng
4
wåi
πC
yuán
x
zōu
7j
wån
F
yuè
zü
wång
yún
zuän
wéi
#
zá
19
zui
X
wén
#
zāi
A
zün
wēng
zån
zuō
wō
zång
22


Reviews:
[{"review_id": "bP4QcklMrK", "reviewer": "Reviewer_mtfq", "summary": "This paper introduces the Homogeneity-Heterogeneity Disentangled Learning for Neural Representations (H2DiLR), a framework designed to improve neural decoding by separating homogeneous and heterogeneous components from intracranial recordings across multiple subjects. The study demonstrates that H2DiLR significantly enhances tone decoding performance in Mandarin-speaking participants using stereoelectroencephalography (sEEG) data. The framework outperforms traditional methods by effectively capturing and leveraging both shared and subject-specific neural features.", "questions": "Can you provide the specific list of 407 Mandarin syllables and explain how they represent nearly all Mandarin characters?\n\nHow and why are the private and shared codebooks defined and utilized within your framework?\n\nHave you considered the computational cost implications of using separate encoders for each subject, and are there plans to optimize this aspect?\n\nHow does your work differentiate from the 2023 study \"A high-performance brain-to-sentence decoder for logosyllabic language,\" which also uses a dataset with 407 syllables?\n\nWhat is the total duration of the sEEG data in the dataset you provided?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The introduction of the Homogeneity-Heterogeneity Disentangled Learning for Neural Representations (H2DiLR) is a novel approach that effectively addresses the challenge of data heterogeneity in neural decoding. By disentangling shared and subject-specific neural features, the framework enhances the decoding accuracy across multiple subjects.\n\nThe study employs a robust experimental setup using stereoelectroencephalography (sEEG) recordings from multiple Mandarin-speaking participants.\n\nThe proposed method demonstrates a substantial improvement in tone decoding accuracy over traditional subject-specific models.", "weaknesses": "The statement that a \"comprehensive set of 407 Mandarin syllables covers nearly all Mandarin characters\" requires further clarification. You need to provide the specific vocabulary list used in the rebuttal materials to support this claim.\n\nThe concept of private and shared codebooks is introduced in the paper but lacks a detailed explanation. A more comprehensive rationale and description are needed to clarify how and why these codebooks are defined and utilized.\n\nSharing the data and code would greatly enhance the validation and reproducibility of the proposed method by the research community. This transparency would contribute significantly to the field and support further advancements.\n\nThe claim of being \"the first\" to work with 407 Mandarin syllables is inaccurate, as the 2023 study titled \"A high-performance brain-to-sentence decoder for logosyllabic language\" also mentions using a dataset with 407 syllables. Although this work is cited, the related work section lacks a detailed discussion of this study and how the current work differentiates itself.\n\nThe paper asserts that it disentangles homogeneity and heterogeneity, yet the evidence provided relies on feature visualization of specific samples. A statistically robust analysis is necessary to substantiate this claim. \n\nWhile the framework allows for joint training across multiple subjects, the use of separate encoders for each subject may significantly increase computational costs, which should be addressed and optimized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cN5cTIGmCk", "reviewer": "Reviewer_ScW5", "summary": "The paper discusses a method called Homogeneity-Heterogeneity Disentangled Learning for neural Representations (H2DiLR) for decoding tone from stereoelectroencephalography (sEEG) data.  H2DiLR disentangles shared (homogeneous) and individual-specific (heterogeneous) neural representations using a two-stage learning paradigm. The first stage involves unsupervised learning of neural features via vector quantization with shared and private codebooks, while the second stage applies these representations to decode tones using a transformer model​. The authors claim that H2DiLR outperforms previous baseline methods by 12% when evaluated on Top-1 accuracy.", "questions": "See weaknesses. What areas of the brain are most important to tone decoding? Does the exclusion of data from some electrodes have more of an impact than others? Is there any benefit of this method for neural encoding models that predict neural responses from feature input?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic of tone decoding is understudied, if somewhat narrow, and the idea to separate homogeneous and heterogeneous neural representations is interesting and could have use cases outside the specialized setting of tone decoding, if applied to semantic neural representations more generally. Overall, I think the idea is interesting enough to be in the \"borderline accept\" category of papers, mainly because I think the idea is general enough to have practicable value outside this particular application.", "weaknesses": "I feel that the authors limited themselves somewhat in specializing their approach to tone decoding only. I feel that the problem of disentangling subject specific effects from general ones is a wider problem with many downstream applications, most notably for semantic decoding, but also for problems of encoding as well. I would have liked to see a deeper exploration of this in the work. There is also disappointingly little exploration of the abundant neuroscience questions here - I would like to have seen a location-based ablation in addition to the parametric ablations of the method that showed which regions are most useful for tonal decoding. An analysis studying if there are region-specific differences to representational homogeneity (i.e. some regions tend to be more idiosyncratic to a particular subject) would also have been interesting, although I recognize that the chosen data setting, sEEG, has challenges in answering this question due to coverage differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VSMENZ6dsw", "reviewer": "Reviewer_AheB", "summary": "The authors introduce a neural network architecture, H2DiLR, a self-supervised learning (SSL) method for pre-training models for sEEG tone decoding (4-way classification task) and a novel dataset (407-syllable reading sEEG dataset).\n\n**Dataset**: The patients are asked to read 407 syllables aloud. To make their pronunciation as close to natural speech as possible, carrier words are added to form full sentences. The dataset includes `1221` trials (i.e., 3 trials per syllable). Simultaneously recorded audio signals are then used to precisely identify time stamps for extracting the target syllable from each trial. Each trial is truncated based on these time stamps, resulting in a neural recording of up to `1` second, labeled with one of `4` tones. According to the task paradigm demonstrated in [1], the total recording time for each subject is approximately `5` hours, with each trial lasting around 12 seconds. Therefore, the total amount of pre-training dataset is about `20` hours.\n\n**Model**: They introduce two different codebooks to disentangle the shared and private parts of multiple subjects. During the pre-training stage, they reconstruct the original sEEG signals. After pre-training, they use the frozen pre-trained VQ-Encoder to obtain quantized embeddings of the input sEEG signals. These embeddings are then passed through a temporal Transformer for further temporal integration, followed by a classification head for tone decoding.\n\n**Experiments**: Previous pre-training methods (including BIOT, NeuroBERT, etc.) are compared. Besides, the authors conducted different ablation studies regarding pre-training (partition ratio, # of subjects, etc.).\n\n---------------------------------\n\n**Summary**:\n\n**This study might benefit from further validation to enhance its robustness.**\n\n - The observed performance improvement might be attributed to the careful selection of hyperparameters in the codebook. See **Private Comment by Reviewer AheB (1/3)** and **Final Comment by Reviewer AheB (1/2)** for more details.\n - Given the current preprocessing approach, the analyzed neuroscience results might be biased. See **Private Comment by Reviewer AheB (2/3)** and **Final Comment by Reviewer AheB (2/2)** for more details.\n - The advantages of the VQ code over the original signal have not been clearly demonstrated through ablation studies. Besides, the author's goal of disentangling different subjects (i.e., the brain's desynchronization nature [3]) does not appear to align with the capabilities of the current model architecture. See **Private Comment by Reviewer AheB (3/3)** and **Final Comment by Reviewer AheB (1/2)** for more details.\n\nsEEG signals differ significantly from EEG signals, and cross-subject decoding remains a challenging task. Achieving notable improvements with such a small amount of subjects is uncommon in prior studies [1,2] related to sEEG-based speech decoding. While the authors' effort to tackle this difficult problem is commendable and encouraging, the effectiveness of the proposed method is ultimately more critical than the story itself. Therefore, I maintain my original **reject** vote. Thanks for the authors' contribution to the field of sEEG-based tone decoding.\n\n**Reference**:\n\n[1] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.\n\n[2] Chen J, Chen X, Wang R, et al. Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals[J]. bioRxiv, 2024.\n\n[3] Buzsaki G. Rhythms of the Brain[M]. Oxford university press, 2006.", "questions": "1. Table 1: What is the codebook size of Heterogeneous H2DiLR with ($\\nu=0$)? Is it `K=32`?\n\n2. Did you use a different signal length for pre-training and downstream classification? Are there any differences between the pre-training dataset and the downstream classification dataset?\n\n3. Line 879: Did you apply normalization to each channel independently (i.e., calculating the mean and standard deviation for each channel separately) or across all channels together?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Significance**: ~~**Open-source sEEG speech datasets are rare. Their promise of publishing the dataset upon acceptance (Line 780) is good news for the community as it will lower the entry threshold for future research.**~~ Additionally, they demonstrate how SSL-based pre-training allows improving performance compared to only supervised training (**doubtful**).\n\n**Clarity**: The text has a good structure and is well-written. The figures also help in understanding the method.", "weaknesses": "**Major**\n1. The authors don’t mention the introduction of data augmentation (e.g., temporal jittering, additive noise, etc.) to avoid overfitting, a common strategy used in sEEG-based speech decoding [2,3]. Since the number of available trials (i.e., $\\sim$1221 trials) within each subject is too small, I’m not sure whether the improvement of pre-training comes from this.\n\n2. The preprocess details are missing. Could you provide a detailed description of sEEG preprocessing (i.e., from the originally collected sEEG signals at a sample rate of 2kHz to the epoched trials at a sample rate of 1kHz)?\n\n3. In [1], their CNN baseline achieves about 40% on average, and their NAR model achieves 45.75% on average. All these methods are supervised (heterogeneous) baselines, and these results make it hard to assess the contribution of H2DiLR.\n\n4. No code and demo dataset is provided. Without the code and data, it's difficult to verify the claims.\n\n5. Some aspects of the method are undefined or unclear. Please see the Questions section below. These aspects need to be clarified in the manuscript.\n\n**Minor**\n1. Figure 1: It could mislead readers into thinking that H2DiLR uses Product Quantization. In Figure 3 and Section 3.3, the authors actually quantize each item in the embedding sequence using one of different codebooks. However, in Figure 1, the “disentangled embedding” suggests that different parts of each individual embedding are quantized with different codebooks.\n\n2. Line 155: typo “LaBraM Anonymous (2024)”\n\n3. Line 841: typo “the learning of 5e-5” should be “the learning rate of 5e-5”? Is the only difference between fine-tuning pre-trained models and training non-pretrained baselines the learning rate? Are all other settings the same?\n\n4. Use `\\citet` or `\\citep` instead of `\\cite` when referencing other works.\n\n5. Since you collected a 407-syllable reading sEEG dataset, how does H2DiLR perform on other types of syllable classification tasks, such as initial syllable classification? Including these additional results might further highlight H2DiLR's effectiveness.\n\n**Reference**\n\n[1] Feng C, Cao L, Wu D, et al. A high-performance brain-sentence communication designed for logosyllabic language[J]. bioRxiv, 2023: 2023.11. 05.562313.\n\n[2] Metzger S L, Littlejohn K T, Silva A B, et al. A high-performance neuroprosthesis for speech decoding and avatar control[J]. Nature, 2023, 620(7976): 1037-1046.\n\n[3] Willett F R, Kunz E M, Fan C, et al. A high-performance speech neuroprosthesis[J]. Nature, 2023, 620(7976): 1031-1036.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e8ze1NNzZK", "reviewer": "Reviewer_VGSo", "summary": "In this paper, the authors propose H2DiLR to capture both homogeneous and heterogeneous information from intracranial recordings across multiple subjects for lexical tone decoding. The H2DiLR framework first obtains discrete vectors through a reconstruction task (vq-autoencoding manner), then builds shared codebook and private codebook for the decoding of different subjects. Experiments on a newly acquired, not publicly released sEEG dataset show the H2DiLR method outperforms other baseline methods.", "questions": "1. The usage of non-invasive brain recordings is an emerging topic. What's the performance of H2DiLR in decoding EEG signal?\n2. Why the size of codebook and dimension of codebook are set as the same?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is clear. The authors seek to address a very important problem in brain decoding: dig homogeneity within the brain responses of different subjects, and hope the obtained homogeneity representation can help improve decoding performance. The presentation of this paper is also good.\n2. The proposed vq-autoencoding based method outperforms previous models in decoding performance. Additional ablation analysis further confirms the effectiveness of model design.\n3. The authors build a new sEEG dataset (promise to release if this paper gets accepted) which has potential to boost researches related to lexical tone decoding.", "weaknesses": "1. Since lexical tone decoding is not a very common task in the brain decoding area, I think the task / problem setting needs to be first introduced and defined in the methodology part.\n2. The paper lacks analysis of the generalization ability of captured common representation, i.e. if the shared codebook of three subjects well generalize to the fourth subject, and whether any three of the four subjects lead to similar shard codebook. The authors also mention this issue in limitations, but I think the dataset containing four subjects is enough to conduct such experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)