Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder (2024)

Abstract: Reconstructing natural language from noninvasive electroencephalography (EEG) holds great promise as a language decoding technology for brain-computer interfaces (BCIs). However, EEG-based language decoding is still in its nascent stages, facing several technical issues such as: 1) Absence of a hybrid strategy that can effectively integrate cross-modality (between EEG and text) self-learning with intramodality self-reconstruction of EEG features or textual sequences; 2) Under-utilization of large language models (LLMs) to enhance EEG-based language decoding. To address above issues, we propose the Contrastive EEGText Masked Autoencoder (CET-MAE), a novel model that orchestrates compound selfsupervised learning across and within EEG and text through a dedicated multi-stream encoder. Furthermore, we develop a framework called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable Representations), which leverages pre-trained modules alongside the EEG stream from CET-MAE and further enables an LLM (specifically BART) to decode text from EEG sequences. Comprehensive experiments conducted on the popular textevoked EEG database, ZuCo, demonstrate the superiority of E2T-PTR, which outperforms the baseline framework in ROUGE-1 F1 and BLEU-4 scores by 8.34% and 32.21%, respectively. Our proposed pre-trained EEG-Text model shows the potential to improve downstream tasks involving EEG and text. This opens up promising avenues for its application in inner speech BCI paradigms, meriting further investigation.

heading: 1 Introduction
text: Decoding natural language from non-invasive brain recordings with electroencephalography (EEG) is an emerging topic that holds promising benefits
*Corresponding author †Corresponding author
for patients suffering from cognitive impairments or language disorders. Thanks to the burgeoning development of pre-trained large language models (LLMs) (Zhao et al., 2023a), the potential of using an open vocabulary to decode human brain activity has been gradually unlocked. Specifically, through the commendable text understanding and generation capabilities of cutting-edge LLMs (Touvron et al., 2023; Ouyang et al., 2022), translating complex spatio-temporal EEG signals into nuanced textual representations, which is known as EEG-to-Text, is being achieved. Compared to conventional paradigms of brain-computer interfaces (BCIs), such as motor imagery (MI) (Al-Saegh et al., 2021), steady-state visual evoked potential (SSVEP) (Wang et al., 2016), and P300 (Cecotti and Graser, 2010), EEG-to-Text can convey much more intended commands from the human brain to computers, and thus presents a more extensive range of applications. Its potential as a novel and powerful BCI paradigm suggests it could contribute to advancements in the field of imagined or inner speech BCIs.
Several existing EEG studies (Li et al., 2022a; Yi et al., 2024) were focused on developing specialized pre-trained models for EEG only, aiming to extract universal semantic representations from the human brain. However, the pre-trained model
1 7278
bridging EEG and text has been ignored, which may be important to enhance the representation learning for inter-modality conversion (Bai et al., 2023). This motivates us to develop a hybrid model to orchestrate compound pre-trained representations across and within EEG and text. This endeavor faces the core challenge: How to bridge the semantic gap between EEG and text while establishing an implicit mapping in the latent representation space? Responding to this challenge, we focus on self-supervised learning (SSL), because of its great capability in multi-modal representation learning (Chen et al., 2024). Contrastive learning is one of the important SSL strategies, learning semantic-level representations across modalities (as CLIP does for language and image) (Radford et al., 2021). Masked modeling methods exhibit significant capability of intra-modality selfreconstruction, such as BERT (Devlin et al., 2019) in nature language processing and masked autoencoder (MAE) (He et al., 2022) in computer vision.
Inspired by the above prevailing SSL strategies, we propose a novel pre-trained model to align EEG and text, Contrastive EEG-Text Masked Autoencoder (CET-MAE), as shown in Figure2(a). CETMAE integrates contrastive learning and masked signal modeling through a dedicated multi-stream encoder. It effectively learns pre-trained representations of EEG and text by balancing the latent embeddings represented by self-reconstruction and the semantic-level aligned embeddings of text tokens and text-evoked EEG features. In terms of masked signal modeling, CET-MAE implements a high mask ratio (specifically, 75%) on both EEG and text data, presenting a meaningful challenge for the model to handle an increased amount of missing information during the reconstruction phase. This setting not only enhances the model’s understanding of individual modality but also facilitates cross-modal interactions and support.
Furthermore, to make the most of LLMs’ capability in language understanding and generation as well as to fully use pre-trained representations learned by CET-MAE, we introduce a new EEG-to-Text decoding framework, EEG-to-Text using Pre-trained Transferable Representations (E2TPTR). E2T-PTR utilizes pre-trained modules alongside the EEG stream from CET-MAE and further adopts the BART (Lewis et al., 2020) to decode language from EEG sequences. By transferring the pre-trained representations from CET-MAE, E2TPTR significantly enhances EEG-to-Text decoding,
surpassing both the baseline and state-of-the-art (SOTA) methods.
Our main contributions are summarised below:
• Introducing CET-MAE, the first pre-trained EEG-text model for EEG-based language decoding. CET-MAE integrates the reconstruction of text and EEG features with semantic alignment, forming a multi-stream SSL framework for both intra-modality and crossmodality representation learning.
• Developing a new EEG-to-Text framework via E2T-PTR. The new E2T-PTR framework can leverage CET-MAE’s pre-trained EEG representations and the capabilities of LLMs (BART) for text generation.
• Conducting extensive EEG-to-Text experiments on three, four, and five reading tasks in ZuCo. Results demonstrate that our framework outperforms previous works, offering valuable insights into leveraging pre-trained transferable representations to enhance EEGto-text decoding.

heading: 2 Related Works

heading: 2.1 Self-supervised Representations Learning
text: Multimodal self-supervised representation learning aims to explore the interactions between different modalities to produce semantically generalizable representations for downstream tasks.
In recent years, there have been substantial progresses across various modalities, such as visionlanguage pre-training (Zhao et al., 2023b; Lin et al., 2023). A range of existing methods rely on contrastive learning, which can effectively draw closer to the global representations of matched pairs in latent spaces with semantic-level self-supervised constraints. But contrastive learning sometimes tends to overlook the self-information of individual modalities, particularly at more granular levels. On the other hand, multimodal masked signal modeling integrates cross-modality self-learning with intra-modality self-reconstruction, focusing on reconstructing one modality from another. This approach may help the model learn the associations between modalities. However, it may lead to an excessive emphasis on fine-grained details, potentially weakening the overall cross-modality correlation and causing issues such as insensitivity to whether the inputs are matched pairs. A series of
2
recent works, such as CMAE (Huang et al., 2023), CAV-MAE (Gong et al., 2022) and SimVTP (Ma et al., 2022), have already successfully integrated both contrastive learning and masked signal modeling so that their complement advantages can be utilized.
Our work draws inspiration from the above SSL methods but with a novel strategy. In the proposed CET-MAE, the utilization of both text and EEG streams not only achieves an explicit contrastive learning objective to capture global coordination but also avoids erroneous learning processes. Meanwhile, the utilization of the joint stream can facilitate the information interaction between modal-specific embeddings to achieve masked signal modeling effectively. To the best of our knowledge, this is the first EEG-to-Text masked autoencoder that attempts to establish transferable representation learning between EEG and text.

heading: 2.2 Open Vocabulary EEG-to-Text Decoding
text: Previous works (Nieto et al., 2022; Kamble et al., 2023) on EEG-to-Text have been severely confined by a limited number of (several or tens of) words in terms of vocabulary size. These closed-vocabulary efforts primarily focused on recognizing low-level linguistic features, such as individual words or syllables. However, these works can hardly capture more complex, high-level semantic and contextual aspects of language.
The development of LLMs has significantly enhanced the field of EEG-based text decoding. The first work using LLM (Wang and Ji, 2022) integrates an additional EEG encoder to align the pre-trained BART for EEG-to-Text, providing important inspiration for subsequent works. CSCL (Feng et al., 2023) employs curriculum learning to effectively mitigate the discrepancy between subject-dependent and semantic-dependent EEG representations in EEG-to-Text translation. DeWave (Duan et al., 2024) uses a quantized variational encoder to convert continuous EEG signals into discrete sequences, alleviating the reliance on eye fixations. Despite advancements, prior efforts struggled to bridge the complex semantic gap between EEG and text on an open-vocabulary scale. Our proposed CET-MAE aims to tackle this challenge. Additionally, our E2T-PTR framework transfers CET-MAE’s representations and leverages the BART to achieve superior text generation outcomes.

heading: 3 Methods

heading: 3.1 Preliminary
text: ZuCo benchmark dataset. For our work, we use the ZuCo1.0 (Hollenstein et al., 2018) and ZuCo2.0 (Hollenstein et al., 2023) datasets, which contain the EEG and eye tracking data during five natural reading tasks. The corpus for sentiment reading (SR) task v1.0 comes from the movie reviews. The corpus for the remaining four tasks is sourced from Wikipedia and comprises two versions each of Natural Reading (NR) and TaskSpecific Reading (TSR), specifically NR v1.0, NR v2.0, TSR v1.0, and TSR v2.0. The word-level EEG was recorded and aligned by the eye-tracking fixations, and the sentence-level EEG was recorded during the entire reading procedure. We follow the preprocessing and dataset splits established by baseline work (Wang and Ji, 2022).
Natural masking ratios of EEG feature sequences. Our investigation reveals the word-level contextual EEG presentations in ZuCo datasets are severely corrupted due to missing eye-tracking fixations, leading to mismatches between EEG raw data and text, as shown in Figure1. This misalignment leads to fragmented word-level EEG feature sequences, which fails to capture the cohesive semantics of entire sentences and inevitably complicates the representations learning of EEG and text.
Different from previous works, we concatenate the word-level EEG features and the sentence-level EEG features as our EEG feature sequences E as
E = [Eword1, Eword2, .., EwordN , Esentence]. (1)
Incorporating sentence-level EEG features offers several benefits. First, it provides a holistic view of EEG sequences, enriching the interpretation of overall sentence semantics. Secondly, it acts as a form of data augmentation, which can mitigate the issue of data incompleteness, thereby alleviating semantic discrepancies caused by the misalignment between word-level EEG and text. To provide a clearer overview, we have presented the detailed statistics of the natural masking ratio (NMR) of EEG feature sequences under three categories of reading task combinations in Appendix A.
Definitions in EEG-to-Text Decoding. Given a sequence of EEG features E as the input to the model M, the aim is to decode the ground-truth word tokens W from open-vocabulary V via M. These corresponding EEG-Text pairs ⟨E,W⟩ are collected during natural readings.
3
During the testing phase, the model M operates with an implicit understanding of the ground-truth word tokens W. Its primary objective remains to decode the EEG feature sequences E to generate an output that closely matches tokens W. This involves the model generating the sequence of words with the highest probability within the probability distribution P of the V.

heading: 3.2 EEG-Text Masking
text: We perform random masking on the text tokens, followed by processing with BERT. For EEG masking, we adopted the following settings. Word-level EEG feature sequences are randomly masked, while
sentence-level EEG feature sequences are compulsorily masked. This aims to force the model to fully reconstruct the contextual semantics within the sentence-level EEG feature sequences.

heading: 3.3 CET-MAE Encoder
text: As illustrated in Figure 2(a), the CET-MAE model needs to extract the embeddings of text and EEG separately and then feed the embeddings into the multi-stream transformer encoder to learn the crossmodal representations.
Text encoder. We utilize the pre-trained encoder-decoder model BART as the text encoder. Due to the suitable capabilities in natural language
4
understanding and generation (Li et al., 2022b), we opt to freeze weights of the BART 1 encoder to maintain its high-level language comprehension from the last hidden states. Firstly, the text tokens are converted into high-quality text embeddings with positional encoding by BART. The learnable embeddings are then used to replace the masked word tokens.
EEG encoder. The EEG encoder is designed as a Multi-layer Transformer Encoder ) (Vaswani et al., 2017) to capture the temporal relationships from EEG sequences with spatial and frequency features in each token. A learnable linear projection layer is employed to transform the EEG embeddings from the EEG encoder, aligning their dimensions with those of the text embeddings.
Multi-stream Transformer encoder. The pivotal design of this module lies in the integration of EEG, text, and the joint streams. We implement the dual-modality streams for EEG-text contrastive learning, especially using a specialized head for each modality. It is equipped with the layer normalization (LN) and the feed-forward network (FFN) enabling the production of embeddings that preserve their unique properties(Gong et al., 2022). Notably, we control the learning process to ensure that learnable vectors at masked positions do not enter into the text stream, thereby preventing the inclusion of misleading contrastive feedback. Equally crucial for the two reconstruction tasks, the joint stream is utilized to facilitate the integration of the embeddings from both text and EEG modalities. This design aims to deepen the interaction and enhance the cooperation between EEG and text, fostering a more effective learning synergy.

heading: 3.4 CET-MAE Decoder
text: We apply a lightweight Transformer encoder as the EEG decoder. For EEG reconstruction tasks, EEG embeddings are first mapped to the original dimensions through a learnable linear projection layer. Subsequently, EEG embeddings with learnable masked tokens are inserted back into their original positions. The final EEG embeddings added to the positional embeddings are fed into the EEG decoder. Since the text encoder has already encoded the masked tokens and captured their positional information within the text, we employ a learnable linear projection layer as the text decoder to predict the masked text tokens.
1https://huggingface.co/facebook/bart-large

heading: 3.5 CET-MAE Training Objectives
text: CET-MAE is pre-trained by three objectives: (1) Masked Text Modeling (LT ): it aims to predict the masked text tokens by utilizing hybrid representations that integrate information from both textual and EEG embeddings. (2) Masked EEG Modeling (LE): it learns to reconstruct the original EEG feature sequences, especially predicting masked word- and sentence-level features based on hybrid representations, where the error is measured by mean square error (MSE). (3) EEG-Text Contrastive Learning (LCL): it involves a process where the corresponding EEG and text representations are computed by separate global average pooling layers. The objective is to bring the aligned pairs (matched EEG and text embeddings) closer together while pushing unpaired ones further apart. Our goal L is minimizing is the summation of these three learning objectives:
L = λT · LT + λE · LE + λCL · LCL (2)

heading: 3.6 E2T-PTR Framework
text: The proposed E2T-PTR is illustrated in Figure 2(b). It can be summarized into the following key points.
Word-sentence level input tokens. We add the sentence-level EEG features as our input tokens. As detailed in 3.1, concatenating the sentence-level EEG feature sequences as the last token can effectively alleviate the incoherent contextual semantics due to gaps in word-level EEG features.
Effective transfer capability. We investigate how to effectively transfer the cross-modality representations learned from the CET-MAE to downstream tasks such as EEG-to-Text decoding. The E2T-PTR employs a synergy of the following critical components: the EEG encoder, the linear projection layer, and the EEG-stream transformer encoder, all of which are integral components as outlined within the CET-MAE. For the LLM backbone, we also apply the BART which excels at natural language generation tasks.
Fine-tuning strategy. We fine-tune all parameters of E2T-PTR during the training phase. The weights of CET-MAE are first loaded into the EEG encoder, the linear projection layer, and the EEGstream transformer encoder. As the linguistic backbone of E2T-PTR, the BART is also fully fine-tuned to improve its ability to generate fine-grained text tokens from EEG embeddings.
5

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation
text: We pre-trained our CET-MAE models under three, four, and five reading tasks in ZuCo v1.0 and ZuCo v2.0. For fairness, we assessed the performance of E2T-PTR for the EEG-to-Text task under the identical dataset scale used during the pre-training phase. We adopt the BLEU and ROUGE-1 scores for evaluating the EEG-to-Text generation performance. More details are presented in Appendix B.

heading: 4.2 Implementation Details
text: The CET-MAE model features a robust EEG encoder with transformer encoder blocks (6 layers, 2048 hidden dimensions, and 8 attention heads). The EEG decoder is a lightweight transformer encoder of 1 layer with 8 heads. The multi-stream transformer encoder is designed with 1 layer, a 4096 hidden dimension, and 16 attention heads. The mask ratios for EEG feature sequences and textual tokens are set at 75% (which can achieve the best results based on trial-and-error). For the CET-
MAE pertaining objective L, we set λT =0.1, λE=1, λCL=0.01. This setting is refined through experiments to balance the gradients of each loss in the overall training objective, ensuring that the model learns effectively from each task. We pre-train the CET-MAE model from scratch for 100 epochs. Subsequently, we fine-tune the E2T-PTR model for EEG-to-Text tasks over 50 epochs, employing a batch size of 32 and utilizing the AdamW optimizer. More details are provided in Appendix B.
6

heading: 4.3 Main Results
text: Table 1 shows the performance of our E2T-PTR framework on the ZuCo benchmarks. In three reading tasks, E2T-PTR achieves BLEU-1 to BLEU-4 SOTA scores of 42.09%, 25.13%, 14.84%, and 8.99%, respectively. Moreover, it outperforms best in ROUGE-1 Precision, Recall, and F1 scores compared to recent works. Notably, without removing repetitive generated word tokens, E2T-PTR surpasses C-SCL in BLEU-1 and BLEU-2 scores across four reading tasks. Particularly under the five reading tasks with 18791 training samples, E2T-PTR scores 59.20%, 50.77%, 46.82%, and 44.63% in BLEU-1 to BLEU-4, significantly exceeding the baseline work EEG2Text.
Table2 presents a comparative analysis of the decoding results between our model and other models under three reading tasks. Our model E2T-PTR demonstrates an enhanced ability to generate more complete grammatical structures, which is evident from the reduced error rates and increased semantic coherence in the decoded sentences, exemplified by expressions such as “his father was” and “Bush was born in”. Our model also excels in decoding common and proper nouns, such as “office” and “University of the Sacred Heart”. It also adeptly produces semantically similar words, such as, “appointed” vs “elected”, and “Ernest Lundeen” vs “John Hemy”. Intriguingly, upon expanding our training samples to 1.75 times (10710 to 18791), we observe an obvious improvement in the translation quality of the model, especially concerning fine-grained recognition. More comprehensive results are included in the Appendix C.
Our investigation delved into the transfer performance of CET-MAE across varying EEG and text masking ratios under three reading tasks. Table 3 details the performance shifts under different combinations of masking ratios rising from 25% to 50%, and to 75%. We discovered that the CETMAE model excels at the higher masking ratios of 75%, starkly contrasting with the traditional 15% mask ratio suggested in BERT. This result is consistent with recent findings in multi-modal masked models (Ma et al., 2022; Geng et al., 2022), suggesting that inter-modal interactions may promote performance improvement. We further ponder this phenomenon and suggest that, in terms of CET-MAE structure, it appears to be suited for reconstructing masked EEG features and predicting masked word tokens. In terms of the masking strat-
egy, forcefully masking sentence-level EEG embeddings can better compel the model to learn global semantic information. Furthermore, we discuss the overall masking ratio for the EEG, the natural EEG masking ratio under three reading tasks is 32.51% as mentioned in Appendix A. Therefore, the total masking ratio for the EEG is 83.13% 2 (32.51% of natural + 50.62% of CET-MAE masked).
For a more rigorous validation, we further implemented the leave-one-subject-out validation strategy for both the CET-MAE model and the E2TPTR framework, detailed in Table 4. This validation approach proved extremely valuable in testing the generalization performance across different subjects within the EEG dataset. Given the inherent noise and individual variability in EEG data, it is crucial to evaluate how well a model performs under such conditions. The results obtained from the leave-one-subject-out validation not only exceeded our initial performance metrics presented in Table 1 but also underscored the strong generalizability of our models. These results affirm the ability of our models to effectively manage the inherent variability in EEG data, thereby demonstrating robust performance as each subject’s data was sequentially excluded from the training set.

heading: 4.4 Ablation Studies
text: Table 5 details the ablation experiments, affirming the effectiveness of each component in our approaches for EEG-to-Text generation quality. First, sentence-level EEG features positively impact BLEU scores, notably BLEU-1, underscoring their importance in capturing essential semantic information for improved text generation. Second, CET-MAE, focusing on masked signal modeling and contrastive learning between EEG and text, is fundamental. Integrating CET-MAE with the baseline framework (Wang and Ji, 2022) significantly boosts BLEU scores, especially BLEU-4. Third, combining E2T-PTR with CET-MAE enhances performance across metrics, particularly Precision, Recall, and F1 score of ROUGE-1, showcasing E2TPTR’s role in effectively transferring CET-MAE’s learned representations.

heading: 4.5 Transfer Performance of SSL Models
text: We further pre-train and compare the transfer performance of the following SSL models: 1) Contrastive EEG-Text (CET) learning model: The CET
2Overall Masking Ratio = NMR + (1 - NMR) × CET-MAE Masking Ratio.
7
Metrics (%) Our SSL Models
that has no reconstruction objective. For a fair comparison, we implement CET using the same encoder architecture (modal-specific encoders + multi-stream encoder) with CET-MAE but remove the reconstruction task (LE and LT ). We use this model to investigate the impact of contrastive learning. 2) EEG-text masked autoencoder (ET-MAE) model: The ET-MAE has the same architecture as CET-MAE but the contrastive loss (LCL) is set to 0. The masking strategy is the same as CET-MAE. We use this model to examine the effectiveness of masked signal modeling. 3) Our proposed CETMAE is detailed in Section 3.
To ensure fairness, CET and ET-MAE are pretrained with the same pipeline as CET-MAE. We assess their EEG-to-Text transfer performance using the E2T-PTR framework. Results in Table 6 demonstrate CET-MAE’s superiority over two other SSL models (CET and ET-MAE) across most evaluation metrics. Specifically, CET-MAE achieves
improvements of 0.32%, 0.45%, 0.51%, and 0.39% in BLEU-1 to BLEU-4, respectively, compared to CET. Against ET-MAE, CET-MAE records increases of 0.29%, 0.41%, 0.41%, and 0.46% for these metrics, respectively. The trend of enhancement is consistent in ROUGE-1 metrics as well.

heading: 5 Conclusion
text: This study contributes to the development of EEGbased language decoding by introducing an effective EEG-text pre-trained model, CET-MAE, and a highly capable and LLM-empowered EEG-to-Text decoding framework, E2T-PTR. CET-MAE uses a multi-stream architecture to incorporate both intraand cross-modality SSL within one unified system: 1) Intra-modality streams explore representative embeddings that reflect the intrinsic characteristics of EEG or text sequences, leveraging masked modeling with a mask ratio of up to 75%; 2) Intermodality stream provides dual-modal representations to enhance intra-modality reconstruction and constrains the encoder to maximize semantic consistency between text and its corresponding EEG sequences. E2T-PTR transfers pre-trained EEG representations and leverages BART’s capabilities for text generation from these consistent and representative features. Extensive experiments on the latest text-evoked EEG dataset, ZuCo, demonstrate the superiority of this work in both qualitative and quantitative assessments. The proposed CET-MAE model shows great potential for enhancing EEGbased language decoding tasks and could be utilized for other inner speech BCI datasets.
8
Limitation
The limitations of our study are summarized as follows:
Dataset Scale: The performance of both the CET-MAE model and the E2T-PTR framework is constrained by the scale of currently available datasets. We are in the process of developing our datasets to fully exploit the potential of our models and frameworks.
Teacher Forcing: While our results are pushing the open vocabulary EEG-to-Text decoding performances to a new SOTA, they still depend on the implicit use of teacher forcing, a common precondition in recent studiess (Wang and Ji, 2022; Duan et al., 2024; Feng et al., 2023; Xi et al., 2023). This reliance on teacher forcing could be constraining the full capabilities of the LLMs. Noted that recent work (Yang et al., 2024) has reported promising results with the autoregressive capabilities of large speech models like Whisper (Radford et al., 2023) on the MEG datasets (Schoffelen et al., 2019). This may offer potential solutions to the challenges of using teacher forcing in the EEG-to-Text field. Our future work will aim to verify the correctness of the aforementioned new methods and explore the autoregressive capabilities of LLMs to reduce reliance on teacher forcing.
Exploration of LLMs: We plan to explore more advanced LLMs to enhance our EEG-to-Text decoding capabilities. This will involve testing new models and techniques to improve performances and uncover deeper insights from EEG data.
Ethics Statement
In this work, we do not generate new EEG data, nor do we perform experiments on human subjects. We use the publicly available ZuCo v1.0 and ZuCo v2.0 datasets without any restrictions. We do not anticipate any harmful applications of our work.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (No. 62306089) and the China Post-doctoral Science Foundation (Nos. 2023M730873 and GZB20230960).

heading: A Natural Masking Ratio of Datasets
text: To provide a clear perspective, we present the detailed statistics of the NMR of EEG feature sequences for three categories of reading task combinations in Table7.

heading: B Datasets and Implementation Details
text: We utilize the combination of both ZuCo v1.0 and ZuCo v2.0 to form the final ZuCo benchmark. The EEG features are collected with a 128-channel system under the sampling rate of 500Hz. After the noise canceling process, only 105 channels are used. There are 8 frequency bands determined in the ZuCo dataset as follows: theta1 (4–6 Hz), theta2 (6.5–8 Hz) alpha1 (8.5–10 Hz), alpha2 (10.5–13 Hz), beta1 (13.5–18 Hz) beta2 (18.5–30 Hz) and gamma1 (30.5–40 Hz) and gamma2 (40–49.5 Hz). The Hilbert transform is applied in each of these time series. The final features of the EEG are formed by concatenating features from all 8 frequency bands, resulting in a vector with a dimension of 840. For three reading tasks, we pre-train and fine-tune the models on “SR v1.0 + NR v1.0 + NR v2.0”. For four reading tasks, we choose the combination of “SR v1.0 + NR v1.0 + NR v2.0 + TSR v1.0”. For five reading tasks, the models are pre-trained and fine-tuned on “SR v1.0 + NR v1.0 + NR v2.0 + TSR v1.0 + TSR v2.0”. During pre-training, the datasets were split into training and testing sets in a 90% to 10% ratio. During the EEG-to-Text fine-tuning phase, the datasets were further divided into training, validation, and testing sets with an 80%, 10%, and 10% split respectively. The test set samples remained consistent throughout the above two stages. The dataset statistics of EEG-to-Text decoding are detailed in Table 8. Our training hyper-parameters are listed in Table 9. To ensure a fair comparison, we conducted both pre-training and fine-tuning for the EEG-to-Text decoding task using datasets with the same combinations of reading tasks.

heading: C Generated Samples
text: We show more details in EEG-to-Text translation results generated on our models in Table 10, Table 11, and Table12. In our experiments, we aim to select the same sentences from the test sets of three, four, and five reading tasks where feasible. This enables us to directly observe and compare the generated results with the ground truth across different task conditions.

heading: D Subject-independent Performance
text: As reported in Table1, we present the average BLEU-N and ROUGE-1 scores for all 30 subjects. However, considering the individual variations of brain activities during semantic processing and cognitive operations within different subjects, we further provide individual BLEU-N and ROUGE-1 scores for each subject. We use radar charts shown in Figure3 and Figure4 to visually represent these differences, allowing for an intuitive comparison across subjects. For a detailed numeric breakdown of these variances, refer to Table13 and Table14.
E Impact of the Masking Strategy
The masking strategy is crucial in Masked Autoencoders. For the text, the BERT masking strategy has proven highly effective. For the EEG modality, we introduce a pivotal design that involves mandatory masking of sentence-level EEG feature sequences, as detailed in Section 3.2. We delve into the impact of this strategy on the EEG-to-Text decoding task. Comparative results between random and forced masking strategies are presented in Table 15. The forced masking strategy outperforms the random masking strategy in the EEG-to-Text decoding, highlighting the efficacy of our proposed strategy in compelling the model to reconstruct the contextual semantics within sentence-level EEG feature sequences comprehensively.
F Impact of the Multi-Stream Design
Our investigation, as detailed in Table 16, reveals the transfer performance of a multi-stream design in the CET-MAE and E2T-PTR frameworks. The multi-stream approach, which provides the specialized handling of text and EEG using separate streams, outperformed a single joint stream design. Notably, in the E2T-PTR framework, leveraging the EEG-specific stream for fine-tuning yielded a marked improvement in EEG-to-Text task performance over a joint modality stream. This modalityfocused approach appears to capitalize on the nuanced semantic information inherent in EEG embeddings, resulting in a more sophisticated and contextually relevant latent space. This is substantiated by the observed uptick in BLEU and ROUGE metrics. Our study underscores the criticality of finegrained, modality-specific processing approaches in the domain of EEG-Text representation learning.
11
12
13
14
15

heading: Future Work
text: Our future work will aim to verify the correctness of the aforementioned new methods and explore the autoregressive capabilities of LLMs to reduce reliance on teacher forcing. Exploration of LLMs: We plan to explore more advanced LLMs to enhance our EEG-to-Text decoding capabilities. This will involve testing new models and techniques to improve performances and uncover deeper insights from EEG data