Title: DeWave: Discrete Encoding of EEG Waves for EEG to Text Translation
Venue: NeurIPS 2023
Accepted: yes
Keywords: EEG; Neural Encoding; Brain Computer Interface
Abstract: The translation of brain dynamics into natural language is pivotal for brain-computer interfaces (BCIs), a field that has seen substantial growth in recent years. With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. These event markers may not be readily available or could be challenging to acquire during real-time inference, and the sequence of eye fixations may not align with the order of spoken words. To tackle these issues, we introduce a novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) it alleviates the order mismatch between eye fixations and spoken words by introducing text-EEG contrastive alignment training, and 2) it minimizes the interference caused by individual differences in EEG waves through an invariant discrete codex. Our model surpasses the previous baseline (40.1 and 31.7) by 3.06% and 6.34\%, respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. Furthermore, this work is the first to facilitate the translation of entire EEG signal periods without the need for word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset, respectively.

DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation

Abstract

The translation of brain dynamics into natural language is pivotal for braincomputer interfaces (BCIs). With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. To tackle these issues, we introduce a novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) it realizes translation on raw waves without marker by introducing text-EEG contrastive alignment training, and 2) it alleviates the interference caused by individual differences in EEG waves through an invariant discrete codex with or without markers. Our model surpasses the previous baseline (40.1 and 31.7) by 3 . 06% and 6 . 34% , respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. This work is the first to facilitate the translation of entire EEG signal periods without word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset.

Bob attended the University of Texas at Austin where he graduated, Phi Beta Kappa with a Bachelor's degree in Latin American Studies in 1973, taking only two and a half years to complete his work, and obtaining generally excel- lent grades.

Predict was the University of California at Austin in where he studied in Beta Kappa in a degree of degree in history American Studies in 1975. and a one classes a half years to complete the degree. and was a excellent grades.

1 Introduction

Decoding brain states into comprehensible representations has long been a focal point of research [20, 38, 8, 54]. Electroencephalogram (EEG) signals are particularly favored by researchers due to their non-invasive nature and ease of recording [31, 46]. Traditional EEG decoding techniques

largely focus on classifying brain states into restricted categories like Motor Imaginary (MI) [37, 7], Emotion [19, 47], Robotic Control [44, 55], and Gaming [28, 24]. However, these labels, bound to specific tasks, are insufficient for broad-based brain-computer communication. Consequently, there has been a surge of interest in brain-to-text (speech) translation in recent years. As the current trend leans towards large language models (LLM) [4, 12, 30] showcasing increasingly generalized intelligence capabilities, it becomes crucial to delve into ways of bridging the gap between brain signals and natural language representation. However, this area remains under-explored.

The early work in brain-to-text translation [14, 1, 26, 41] relied on external event markers like handwriting or eye-tracking fixations to segment whole brain signals into fragmentary features. This methodology treated the task as word-level classification on a small, closed vocabulary set, with each time step analyzed individually. Both invasive [14, 42] (ECoG) and non-invasive [33] (EEG) brain signals have been used in these approaches. Notably, utilizing handwriting as event markers on invasive signals has led researchers [49] to achieve state-of-the-art (SOTA) recognition accuracy on closed-set character-level recognition. Wang [48] expanded the vocabulary size substantially and demonstrated the feasibility of open-vocabulary brain-to-text translation by employing pre-trained language models with word-level EEG features. However, limitations persist. The order of event markers, particularly eye-tracking fixations used to segment EEG waves into word-level features, may not match the natural word order in language 3 . Moreover, current methods do not have the capacity for direct text translation.

In this paper, we present Discrete EEG Waves Encoding for Brain Dynamics to Text Translation (DeWave), a pioneering framework depicted in Figure 2. DeWave uses a vector quantized variational encoder to transform EEG waves into a discrete codex, linking EEG waves to tokens based on their proximity to codex book entries. This method offers two key advantages: it addresses significant distribution variances in EEG waves across individuals [29, 40, 9], and rectifies order mismatches between raw wave sequences and text without eye-tracking markers. Our raw waves encoder is guided by both self-reconstruction and a contrastive supervision alignment between text embeddings and vectorized raw waves. To navigate the challenges of training with limited parallel data, DeWave leverages large-scale pre-trained language models [6, 36, 3], specifically employing BART [21], which combines BERT's bidirectional context with GPT's left-to-right decoder. Notably, our discrete codex aligns more closely with actual language tokens than continuous EEG features, serving as an interpretable bridge between EEG input and the language model.

Experiments employ non-invasive EEG signals and data from the ZuCo dataset [16], a large-scale public resource that records eye-tracking and EEG during natural reading tasks. Notably, DeWave can be generalized for both word-level EEG features and raw EEG wave translation. With a robust codex representation, this work pioneers in translating the entire time period of EEG signals without the need for word-level order markers such as eye fixations. We assess the performance using standard translation metrics [34, 25]. With word-level EEG features, DeWave attains 42.8 BLEU-1 and 34.9 Rouge-1, surpassing the previous baseline (40.1 and 31.7) by 6 . 73% and 10 . 09% respectively on the ZuCo Dataset. For raw EEG waves without event markers, DeWave achieves 20.5 BLEU-1 and 29.5 Rouge-1. This work's contributions can be summarized in three main points.

This paper introduces discrete codex encoding to EEG waves and proposes a new framework, DeWave, for open vocabulary EEG-to-Text translation.

By utilizing discrete codex, DeWave is the first work to realize the raw EEG wave-to-text translation, where a self-supervised wave encoding model and contrastive learning-based EEG-to-text alignment are introduced to improve the coding ability.

Experimental results suggest the DeWave reaches SOTA performance on EEG translation, where it achieves 41.35 BLUE-1 and 33.71 Rouge-1, which outperforms the previous baselines by 3 . 06% and 6 . 34% respectively.

2 Related Works

The key to decoding natural language from EEG signals is good representations. Existing work for EEG-to-text representation can be categorized into hand-crafted representations and deep-learning

Eye Fixation

Sliced  Waves

Raw Waves

Band Filters

/gid335

Wave2Vec

Transformer

512

512

512

512

512

Vectorized EEG Featuers

Transformer

Encoders

1 9 6 5 8

Discrete Codex

Gradients

Pretrained BART

Decoder

Discrete Codex

I want to say hello and wave

/gid335

Indexing Codex

representations. Earlier works make use of traditional methods to obtain hand-crafted representations from a sequence of EEG signals. A continuous feature representation is extracted using methods such as statistical features [53], correlation coefficient matrix from fast Fourier transform (FFT) components [43, 51, 50], wavelet transform features [18, 32, 54], and Mel-frequency cepstral coefficients (MFCCs) [5]. These hand-crafted representations can be used to establish a mapping between EEG segments and words using distance-based methods such as linear discriminative analysis (LDA) or Support Vector Machine (SVM).

On the other hand, deep learning methods, such as [52], utilize a convolutional neural network (CNN) and a Long Short-Term Memory (LSTM) to learn deep features and perform classification of user's instructions. However, the direct representation is often insufficient to discriminate a larger number of imagined text categories. In [39], EEG signals are first classified into six phonological categories as the intermediate state using a deep convolutional autoencoder. Then latent features are used as input for another encoder to predict a total of 11 speech tokens. Recently, [48] proposed a framework that uses a multi-layer transformer encoder to project word-level EEG feature sequences into EEG embeddings. Then a pre-trained BART model is used to decode these embeddings into words.

3 Method

The overall process of DeWave is illustrated in Figure 2, where the word-level or raw EEG features are vectorized into sequences embedding and are fed into the discrete codex. The language model generates translation output based on discrete codex representation.

3.1 Task Definition

Given a sequence of word-level EEG features E , the aim is to decode the corresponding openvocabulary text tokens W . These EEG-Text pairs 〈 E , W 〉 are collected during natural reading, as defined in Section 4.1. We delve into two task settings: (1) Word-level EEG-to-Text Translation, where EEG feature sequences E are fragmented and re-ranked based on eye-fixation F aligned with each word token w in sequence W ; and (2) Raw EEG Waves to Text Translation, where EEG feature sequences E are directly vectorized into embedding sequences for translation without any event markers, a more challenging but practical real-time setting. DeWave is the pioneering work in this latter task.

3.2 Discrete Codex

Discrete representation is first proposed in VQ-VAE [45]. DeWave is the first work to introduce discrete encoding into EEG signal representation. The discrete representation could benefit both the word-level EEG features and the raw EEG wave translation. Introducing discrete encoding into brain waves could bring two aspects of advantages. 1) It is widely accepted that EEG features have a strong data distribution variance across different human subjects. Meanwhile, the datasets can only have samples from a few human subjects due to the expense of data collection. This severely weakened the generalized ability of EEG-based deep learning models By introducing discrete encoding, we

could alleviate the input variance to a large degree as the encoding is based on checking the nearest neighbor in the codex book. 2) The codex contains fewer time-wise properties which could alleviate the order mismatch between event markers (eye fixations) and language outputs. Meanwhile, because of this property of codex represents, DeWave is the first work that could realize the direct translation from raw EEG waves without any event marker to give the order.

Inference: Given the EEG waves E , it is firsted vectorized into embedding as introduced in Section 3.3 with ( X = Θ ( E , F ) ) or without ( X = Θ ( E ) ) eye fixations F , where X is the embedding sequence. A codex book { c i } ∈ R k × m is initialized with number k of latent embedding with size m . The vectorized feature X is encoded into z c ( X ) through a transformer encoder. The discrete representation is acquired by calculating the nearest embedding in the codex of input embedding x ∈ X as shown in Equation 1.

Different from the original VQ-VAE which decodes the original input, Dewave directly decodes the translation output given the representation z q ( X ) . Given a pre-trained language model, the decoder predicts text output with P ( W| z q ( X ) .

Learn: The codex is like a bridge connecting the vectorized EEG feature and the language model. Compared to learning direct EEG-to-text relation, DeWave learns a better discrete codex for the language model. It is easier to learn since We learn the discrete codex by the combination of the loss functions in three parts,

where the loss maximize the log-likelihood of language outputs log(P( W| z q ( X )) and minimize the distance between latent variable z and the codex value z . Here, the sg denotes the stop gradients. The learning is robust for β from 0.1-2.0, where we set it as 0.2 throughout the training process.

3.3 EEG Vectorization

Word-Level EEG Features With Event Markers: The EEG waves are first sliced into fragments according to the eye-tracking fixation of word sequences given in the annotation. Similar to [48], we calculate the statistical result of four frequency band filters, Theta band (5-7Hz), the Alpha band (8-13Hz), the Beta band (12-30Hz), and Gamma band (30Hz-) [27] to get the statistic frequency features of each fragment. It is noted that although different fragments may have different EEG window sizes, the statistical results are the same (embedding size 840). A multi-head transformer layer is applied to project the embedding into feature sequences with latent size 512.

Raw EEG Waves: Self-Guided Waves to Discrete Codex Our self-supervised EEG wave encoder transforms raw EEG signals into a sequence of embeddings [2, 56] as illustrated in Figure 3. It has two guiding principles: Self-Reconstruction, where the encoder is trained to transform and subsequently reconstruct the original waveforms from discrete codices; and Text Alignment, where the codices' encoding is semantically aligned with word vectors, fostering the development of text-aligned EEG signal representations.

For structure-wise, a conformer-based multi-layer encoder with specially designed hyperparameters is employed. The one-dimensional convolution layer processes the EEG waves to generate the embedding sequence 4 , fusing the EEG channels into a unique embedding for each period. We apply bi-directional transformer attention layers to the sequence to capture temporal relations.

For the reconstruction process, a decoder transformer and transpose convolution structure then convert these discrete embeddings back into raw waves. Given the reconstruction process as ˜ X = φ ( z q ( X ) , the self-supervised loss could be modified into:

where the model calculates the mean square loss between the reconstructed wave and the wave ground truth to perform self-supervised training.

To obtain a semantically coherent codex, we introduce a cross-modality contrastive learning approach distinct from prevalent methods. Unlike CLIP [35, 23, 22] that contrasts CLS embeddings between

/gid335

Transformer

Attention

Codex Encoders

Discret Codex

Codex

Decoder

Generate

Attention

/gid335

EEG2Vec

Text Embedding

Word2Vec

Source

Text

Wave Reconstruction

Encoder

ConFormer [13]

kernel (10,3,3,3,2) stride (3,2,2,2,2)

Codex Transformer

heads=8, dim=512, layer=6

Decoder

Codex Transformer

heads=8, dim=512, layer=6

CNNs

kernel (3,3,3), stride (2,2,3)

Transpose CNN

kernel (3) stride (2)

sample pairs within a mini-batch, our approach operates within a single EEG-text pair. We contrast the EEG-tokenized codex embeddings sequence z q with the text embeddings sequence z t . Assuming the raw wave feature extractors can produce a token sequence in an organized chronological order , we treat the diagonal EEG codex and text word2vec encoding pairs as positive pairs within the sequences. All other pairs are considered negative. The model is trained to minimize the distance between embeddings of positive pairs and maximize that of the negative pairs. For a given EEG-text pair ( i, j ) , the loss is defined in Equation 4

Here, τ is the temperature parameter, N is the total number of EEG-text pairs in a batch, and the sum in the denominator is over all N EEG-text pairs and N text-EEG pairs. The EEG embeddings are expected to correctly match with their corresponding text while distinguishing them from mismatched EEG-text pairs. The total loss then becomes a combination of the original Wave2Vec loss and the contrastive loss as L total = L wave + α L contrast .

By this means, the model not only learns to reconstruct the EEG signal but also learns a robust representation of the signal that aligns with the corresponding text embeddings. This cross-modal learning can potentially improve the translation system by bridging the gap between EEG signals and the semantic content of the text.

3.4 Language Model

We used large-scale text corpus pre-trained BART [21] as the generative language model for translation output. As the EEG-to-text translation data is quite limited, leveraging the BART model could introduce prior knowledge of text relations. In that case, the translation system only needs to learn a codex representation for the language model, which is easier to learn. The codex representations are fed into pre-trained 5 BART model and get the output hidden states. A fully connected layer is applied on the hidden states to generate English tokens from pre-trained BART vocabulary V .

3.5 Training Paradigm

DeWave is trained through a multi-stage process, where the training process is illustrated in Appendix C. In the first stage, we do not involve the language model in weight updates. The target of the first stage is to train a proper encoder projection θ codx and a discrete codex representation C for the language model. In the second stage, the gradient of all weights, including language model θ BART is opened to fine-tune the whole system.

4 Experiments

4.1 Dataset

DeWave utilize both ZuCo 1.0 [15] and 2.0 [17] for experiments. The dataset simultaneously recorded the text and EEG corpus during Normal Reading (NR) and Task-Specific Reading (TSR) tasks. The

EEG waves are collected with a 128-channel system under a sampling rate 500Hz through a frequency band filter from 0.1Hz to 100Hz. However, after the noise canceling process, only 105 channels [15] are used for translation. Similar to [48], we slice the EEG wave according to the eye fixation and calculate the frequency features. For raw EEG waves, the signal is normalized into a value range of 0 -1 for decoding. The reading task's data are divided into the train ( 80% ), development ( 10% ), and test ( 10% ) respectively by 10874, 1387, and 1387 unique sentences with no intersections. Please refer to Appendix A for a detailed description.

4.2 Implementation Details

For word-level EEG features, we use the 56 tokens each with an 840 embedding size. For raw EEG waves, we clip or pad the EEG waves up to sample point 5500 with a constant value of zero. A transformer layer with head number 8 and a 1 × 1 convolutional layer are combined to fuse multiple EEG channels into an embedding sequence with size 512. DeWave uses a codex with size 2048 where each codex latent is an embedding with size 512. The ablation study (Section 4.6) gives a discussion about the codex size. All models are trained on Nvidia V100 and A100 GPUs. For the self-supervised decoding for raw waves, we use a learning rate of 5e-4 and a VQ coefficient of 0.25 for training 35 epochs. For training the codex (stage 1), DeWave uses a learning rate of 5e-4 for 35 epochs. For finetuning the translation (stage 2), DeWave uses a learning rate of 5e-6 for 30 epochs. We use the SGD as the optimizer for training all the models. Due to limited space, refer to Appendix B for more details.

4.3 Evaluation Metrics

We evaluate translation performance using NLP metrics, BLEU and ROUGE, as shown in Table 1. For word-level EEG features, we compare our results to EEG-to-Text [48], maintaining a consistent language model for fairness. In the absence of methods for raw EEG waves, we establish a baseline EEG-to-Text † by segmenting the entire EEG waves into a sequence embedding using a 200ms time window with a 100ms overlap. We adapt Wave2Vec, originally developed for speech recognition, to brain waves and compare it with our approach, DeWave. Furthermore, we adapt unsupervised raw EEG waves classification methods BENDR [11] and SCL [10], using SSL pre-training and feature extraction for comparison, underscoring the impact of discrete encoding.

Word-Level EEG Features: For the word-level EEG feature, we observe that introducing the discrete brain representation could help DeWave reach BLEU-{ 1 , 2 , 3 , 4 } scores of 41 . 35 , 24 . 15 , 13 . 92 , and 8 . 22 , which respectively outperform the previous baseline by 1 . 23 (+3 . 06%) , 0 . 97 (+4 . 18%) , 1 . 31 (+10 . 38%) and 1 . 54 (+18 . 73%) . It is observed that the increasing ratio is more significant for larger grams evaluation. DeWave achieve ROUGE-1 score 28 . 84 (R), 31 . 69 (P), and 30 . 10 (F) which outperforms the previous baseline by -0 . 02 ( -0 . 06%) , 1 . 98 (+6 . 35%) , and 0 . 59 (+1 . 9%) . The increase of the matrix is higher for longer phrases (3-gram and 4-gram). This is rational since the single-word level EEG features are sliced by eye fixation during reading, which may naturally contain noises since the human subject may not really think about the corresponding words when looking at them.

Raw Waves: DeWave represents an unprecedented endeavor in translating raw EEG waves directly, obviating the need for event markers. This is achieved through the application of a learned Wave2Vec model, which converts waves into discrete codex tokens. When benchmarked against

the baseline method - which merely slices the wave according to a time window - DeWave exhibits marked improvement. It attains BLEU1 , 2 , 3 , 4 scores of 20 . 51 , 10 . 18 , 5 . 16 , and 2 . 56 , respectively, surpassing the baseline by margins of 7 . 44 (+56 . 92%) , 4 . 44 (+76 . 1%) , 2 . 61 (+102 . 35%) , and 1 . 42 (+129 . 09%) . This underscores the superiority of a learned discrete embedding representation over the rudimentary time-window baseline. In comparison with cutting-edge self-supervised learning (SSL) methods, DeWave consistently outperforms them, including contrastive learning (CL)-based methods such as SCL [10], the original Wave2Vec [2], and BENDR [11].

4.4 Cross-Subject Performance

Cross-subject performance is of vital importance for practical usage. To further illustrate the performance variance on different subjects, we train the model by only using the data from subject YAG and test the metrics on all other subjects. The results are illustrated in Figure 4, where the radar chart denotes the performance is stable across different subjects. Please refer to Appendix G for more detailed results.

SubjectPerfomance onBLEU-2

SubjectPerfomance onBLEU-4

SubjectPerfomanceonROUGE-R

SubjectPerfomance on ROUGE-F

YMDYFS

YMDYFS

YMDYFS

YMDYFS

YRH

YLS

YRH

YLS

YLS

YRH

YLS

YFR

YRP

YRP

YFR

YRP

YRP

YAC

YTL

22

24.

26

AG

YDC

YAC

YTL

7.5

8.0

8.5

9.0

9.5%

YDC

YAC

YTL

26

28

30

34YAG

YDC

YAC

LA

33

34

3AG

YDG

YSL

YIS

YSL

/YIS

YSL

/YIS

YSL

SIA

YAK

YRK

YAK

YRK

YAK

YRK

YAK

YRK

YMS

YDR

YMS

YDR

YMS

YDR

YMS

YDR

YSD

YSD

YSD

YSD

4.5 Generated Samples

In Table 2, we display visualized examples of text generated from unseen EEG signals. Despite the challenge of thought translation and limited prior research, our model yields meaningful results, aligning keywords and forming similar sentence structures, although it may not yet match traditional language translation tasks.

The model is more adept at matching verbs than nouns. For instance, implies ' vs. implies ', the author who wrote it ' vs. the man who wrote it ' both effectively convey the intended sentiment of the sentence. However, when it comes to nouns, we observe a tendency towards synonymous pairs rather than precise translations, such as the man' vs. the author', Burroughs' vs. Heroughs', edition' vs. version'. Our analysis suggests two potential causes for this. First, when the brain processes these words, semantically similar words might produce similar brain wave patterns. Given the inherent noise in brain waves, the codex might group these features under the same value. Second, the volume of EEG-to-Text pairs available for training is significantly smaller than that for traditional language translation. Hence, some degree of error in translating unseen nouns or sentences is to be expected.

Translation on raw EEG waves is naturally harder than word-level translation, as it lacks eye fixation to suggest the relationship between the period of waves and the word target. Table 2 meet our expectation that the results on raw EEG waves are not as good as those on word-level features, especially on the real semantic meaning of the sentence (sample (2) and sample (5) have the same target for comparison). However, the translation still could output the correct translation of certain words, such as ' much of ' vs. ' much of ', ' individual ' vs. ' individual ', and 'more complicated story' vs. 'more exciting thing'. Although EEG-to-text is a hard topic, DeWave suggests the feasibility of translation improvements.

4.6 Ablation Study

Discrete Codex DeWave encodes EEG waves into a discrete codex, aiming for a language modelfriendly representation. We evaluated performance against varying codex sizes (1024 to 8192) to ascertain if larger sizes yield better results. As depicted in Figure 5, we found no strong correlation between codex size and model performance. A codex size of 2048 yielded the highest BLEU score average, and while the ROUGE score slightly improved with larger sizes, there was no clear evidence that increasing codex size consistently enhanced performance.

Raw EEG wave performance fluctuates noticeably with codex size. Performance improves when increasing the codex size from 1024 to 2048, but any larger size reduces performance. This variation may result from the training formation; word-level EEG data, selected by eye fixations, contains less noise than raw waves. We hypothesize that our current training data may be insufficient for larger codex sizes. Additionally, our experiments indicate that the latent dimension of the codex doesn't significantly affect performance.

TranslationMetricsVariation

TranslationMetricsVariation

(onWordLevelEEGFeature)

(onRawWaves)

AblationMetricsonPerceptionField

30

30

40

25

35

25

（%)

Metrics (%)

20

(%)

20

BLEU-1

Metrics(

25m

BLEU-1

BLEU-1

Metrics

BLEU-2

BLEU-2

15

BLEU-2

15

BLEU-3

BLEU-3

BLEU-3

-×-

BLEU-4

X

BLEU-4

10

BLEU-4

10

ROUGE-R

15

ROUGE-R

ROUGE-R

ROUGE-P

10

ROUGE-P

5

ROUGE-P

5

ROUGE-F

ROUGE-F

ROUGE-F

40ms

80ms 120ms 160ms 200ms240ms 500ms1000ms

5

1024

2048

4096

6144

8192

1024

2048

4096

6144

8192

Perception Field

CodexSize

CodexSize

Perception Time Window We also conduct the ablation study on the model structure for the wave2vec model illustrated in Figure 5. As the model utilize a multi-layer CNN model to slide through the raw waves, the model compresses the waves for perception. The compress ratio decided how large the perception field is for each extracted embedding feature. As described in Section 3.3, the model utilized a perception field of 200ms with an overlap of 100ms. We conduct an ablation study of different perception fields and report it in Figure 5 Where it is observed that the model performance is significantly lower when the perception field is smaller than 80ms or larger than 240ms. The model could achieve similar results in the perception field 120ms to 240ms. The model reaches a small peak around 200ms to 240ms. We think this phenomenon is rational since the normal reading speed for humans is around 160-400 words per minute (reading speed may vary from different

material and human subjects). In other words, the reading period for each word is 150-375ms on average, which roughly meets our observation between 200ms-240ms.

Self-Supervision Initialization Theoretically, we could pre-train the codex by introducing a decoder and calculate the reconstruction loss with the original input for both word-level EEG and raw EEG waves. We prefix other parameters as our best setting with codex size 2048 and compare the impact in Table 3.

For word-level EEG features, the impact of pre-train is mostly on BLEU scores while the ROUGE score does not have much variance. Without pre-train, the BLEU-{ 1 , 2 , 3 } respectively drop by 0 . 64 , 1 . 21 , and 1 . 52 . For direct translation on raw waves, the impact is significantly larger. Without selfsupervised initialization, the BLEU-{ 1 , 2 , 3 } respectively drop by 3.93 ( ↓ 19 . 16%) , 2.40 ( ↓ 23 . 57%) , and 1.48 ( ↓ 28 . 68%) . Similar observation also appears on ROUGE scores. This phenomenon is rational since raw wave decoding requires the model to pick useful features without any help from eye fixations. The self-supervised initialization could help the model form a preliminary ability to extract time-wise or channel-wise features from raw waves.

5 Limitations

Despite DeWave's enhancements in EEG-to-Text translation using a discrete codex and raw wave encoding, its accuracy remains far from real-life scenarios compared to traditional language-tolanguage translations. Also, to keep a fair comparison with EEG-to-Text, this paper uses a teacherforcing setting in evaluation. This setting eliminates accumulation error and turns the sequence decoding task into a word-level classification task given the ground truth token from the previous step. This setting is relatively easier yet we think it is still valuable as it could suggest feature extraction quality while keeping a fair comparison.

Additionally, the experiments in this paper are restricted to public neural reading data, not fully aligning with the "silent speech" concept of direct thought translation from human brains. Instead, the current ZuCo dataset is collected by giving people reading stimuli. This paper focuses on introducing Wav2Vec formation of feature extraction on raw EEG waves and introduces discrete codex as learnable representations for the EEG-to-Text translation domain. One scientific problem in this domain is a better way of doing the 'silent speech" task, which we are exploring as on-going research.

6 Conclusion

This paper presents DeWave, a framework for the recently proposed open-vocabulary EEG-to-Text translation task [48], introducing the concept of discrete codex encoding. This approach brings enhancement in corpus text relevancy metrics, such as BLEU and ROUGE. DeWave also expands the task to decode raw EEG waves without the assistance of eye fixation markers. Despite these advancements, the quality of brain decoding results remains substantially low and remains teacher forcing setting for fair comparison. The translation of thoughts directly from the brain is a valuable yet challenging endeavor that warrants significant continued efforts. In our ongoing work, we are exploring more reasonable settings that remove teacher forcing for both training and testing. We will also include the 'neural-feedback" mechanism in this EEG-to-Text research to enhance the scientific value of this domain.

References

Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang. Speech synthesis from neural decoding of spoken sentences. Nature , 568(7753):493-498, 2019.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems , 33:12449-12460, 2020.

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 2020.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Ciaran Cooney, Rafaella Folli, and Damien Coyle. Mel frequency cepstral coefficients enhance imagined speech decoding accuracy from eeg. In 2018 29th Irish Signals and Systems Conference (ISSC) , pages 1-7. IEEE, 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, 2019.

Yiqun Duan, Zhen Wang, Yi Li, Jianhang Tang, Yu-Kai Wang, and Chin-Teng Lin. Cross task neural architecture search for eeg signal classifications. arXiv preprint arXiv:2210.06298 , 2022.

Yiqun Duan, Zhen Wang, Yi Li, Jianhang Tang, Yu-Kai Wang, and Chin-Teng Lin. Cross task neural architecture search for eeg signal recognition. Neurocomputing , 545:126260, 2023.

Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teng Lin. Domainspecific denoising diffusion probabilistic models for brain dynamics. arXiv preprint arXiv:2305.04200 , 2023.

Jiang et al. Self-supervised contrastive learning for eeg-based sleep staging. In IJCNN 2021 , pages 1-8. IEEE, 2021.

Kostas et al. Bendr: using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience , 15:653659, 2021.

Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines , 30:681-694, 2020.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100 , 2020.

Christian Herff, Dominic Heger, Adriana De Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk, and Tanja Schultz. Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers in neuroscience , 9:217, 2015.

Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading. Scientific data , 5(1):1-13, 2018.

Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. Zuco 2.0: A dataset of physiological recordings during natural reading and annotation. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020 , pages 138-146. European Language Resources Association, 2020.

Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. Zuco 2.0: A dataset of physiological recordings during natural reading and annotation. In Proceedings of the 12th Language Resources and Evaluation Conference , pages 138-146, 2020.

Amir Jahangiri and Francisco Sepulveda. The relative contribution of high-gamma linguistic processing stages of word production, and motor imagery of articulation in class separability of covert speech tasks in eeg data. Journal of medical systems , 43(2):1-9, 2019.

Robert Jenke, Angelika Peer, and Martin Buss. Feature extraction and selection for emotion recognition from eeg. IEEE Transactions on Affective computing , 5(3):327-339, 2014.

Reinmar J Kobler, Jun ichiro Hirayama, Qibin Zhao, and Motoaki Kawanabe. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7871-7880, 2020.

Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653 , 2023.

Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xiaomeng Li. Exploring visual interpretability for contrastive language-image pre-training. arXiv preprint arXiv:2209.07046 , 2022.

Lun-De Liao, Chi-Yu Chen, I-Jan Wang, Sheng-Fu Chen, Shih-Yu Li, Bo-Wei Chen, Jyh-Yeong Chang, and Chin-Teng Lin. Gaming control using a wearable and wireless eeg-based brain-computer interface device with novel dry foam-based sensors. Journal of neuroengineering and rehabilitation , 9(1):1-12, 2012.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

Joseph G Makin, David A Moses, and Edward F Chang. Machine translation of cortical activity to text with an encoder-decoder framework. Nature Neuroscience , 23(4):575-582, 2020.

Dennis J McFarland, Charles W Anderson, K-R Muller, Alois Schlogl, and Dean J Krusienski. Bci meeting 2005-workshop on bci signal processing: feature extraction and translation. IEEE transactions on neural systems and rehabilitation engineering , 14(2):135-138, 2006.

Anton Nijholt. Bci for games: A 'state of the art' survey. In Scott M. Stevens and Shirley J. Saldamarco, editors, Entertainment Computing - ICEC 2008 , pages 225-228, 2009.

Julie Onton, Marissa Westerfield, Jeanne Townsend, and Scott Makeig. Imaging human eeg dynamics using independent component analysis. Neuroscience & biobehavioral reviews , 30(6):808-822, 2006.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Yue-Ting Pan, Jing-Lun Chou, and Chun-Shu Wei. MAtt: A manifold attention network for EEG decoding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Jerrin Thomas Panachakel, AG Ramakrishnan, and TV Ananthapadmanabha. A novel deep learning architecture for decoding imagined speech from eeg. arXiv preprint arXiv:2003.09374 , 2020.

Jerrin Thomas Panachakel and Angarai Ganesan Ramakrishnan. Decoding covert speech from eeg-a comprehensive review. Frontiers in Neuroscience , 15:392, 2021.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 , 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Jaime F Delgado Saa and Müjdat Çetin. Discriminative methods for classification of asynchronous imaginary motor tasks from eeg data. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 21(5):716-724, 2013.

David Sabbagh, Pierre Ablin, Gael Varoquaux, Alexandre Gramfort, and Denis A. Engemann. Manifoldregression to predict from meg/eeg brain signals without source modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Pramit Saha, Muhammad Abdul-Mageed, and Sidney Fels. Speak your mind! towards imagined speech recognition with hierarchical deep learning. arXiv preprint arXiv:1904.05746 , 2019.

Simanto Saha and Mathias Baumert. Intra-and inter-subject variability in eeg-based sensorimotor brain computer interface: a review. Frontiers in computational neuroscience , 13:87, 2020.

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Towards sentence-level brain decoding with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 7047-7054, 2019.

Pengfei Sun, Gopala K Anumanchipalli, and Edward F Chang. Brain2char: a deep architecture for decoding text from brain recordings. Journal of Neural Engineering , 17(6):066015, 2020.

Kazuo Tanaka, Kazuyuki Matsunaga, and Hua O Wang. Electroencephalogram-based control of an electric wheelchair. IEEE transactions on robotics , 21(4):762-766, 2005.

Luca Tonin, Tom Carlson, Robert Leeb, and José del R. Millán. Brain-controlled telepresence robot by motor-disabled people. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society , pages 4227-4230, 2011.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Neeraj Wagh, Jionghao Wei, Samarth Rawal, Brent M. Berry, and Yogatheesan Varatharajah. Evaluating latent space robustness and uncertainty of EEG-ML models under realistic distribution shifts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Jiang Wang and Mei Wang. Review of the emotional feature extraction and classification using eeg signals. Cognitive Robotics , 1:29-40, 2021.

Zhenhailong Wang and Heng Ji. Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 5350-5358, 2022.

Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performance brain-to-text communication via handwriting. Nature , 593(7858):249-254, 2021.

Haoning Xi, Didier Aussel, Wei Liu, S Travis Waller, and David Rey. Single-leader multi-follower games for the regulation of two-sided mobility-as-a-service markets. European Journal of Operational Research , 2022.

Haoning Xi, Liu He, Yi Zhang, and Zhen Wang. Differentiable road pricing for environment-oriented electric vehicle and gasoline vehicle users in the bi-objective transportation network. Transportation Letters , 14(6):660-674, 2022.

Xiang Zhang, Lina Yao, Quan Z Sheng, Salil S Kanhere, Tao Gu, and Dalin Zhang. Converting your thoughts to texts: Enabling brain typing via deep feature learning of eeg signals. In 2018 IEEE international conference on pervasive computing and communications (PerCom) , pages 1-10. IEEE, 2018.

Shunan Zhao and Frank Rudzicz. Classifying phonological categories in imagined and articulated speech. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 992-996. IEEE, 2015.

Jinzhao Zhou, Yiqun Duan, Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teng Lin. Belt: Bootstrapping electroencephalography-to-language decoding and zero-shot sentiment classification by natural language supervision. arXiv preprint arXiv:2309.12056 , 2023.

Jinzhao Zhou, Yiqun Duan, Zhihong Chen, Yu-Cheng Chang, and Chin-Teng Lin. Generalizing multimodal variational methods to sets, 2022.

Jinzhao Zhou, Yiqun Duan, Yingying Zou, Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teng Lin. Speech2eeg: Leveraging pretrained speech model for eeg signal recognition. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2023.

Peer reviews:
comments: SUMMARY:
This paper proposes an EEG to text model that feeds into the raw EEG signals and predicts the corresponding words or long sentences. The model is optimized in two stages: (i) matching EEG to Text by contrastive learning with codex quantization; (ii) fintune the BART model for EEG embedding decoding to text. The model is evaluated on ZuCo 1.0 and 2.0 datasets.

STRENGTHS:
1. The EEG to text application sounds interesting and challenging. The proposed model architecture sounds reasonble by leveraging recent popular CLIP-style contrastive learning, signal quantization (from speech processing domain), and BART finetuning (LLM).
2. The paperr is easy to read, and the related works are well organized. 
3. The performance on ZuCo datasets looks good.

WEAKNESSES:
1. The reviewer has concerns on the data volume for training the CLIP model (ZuCo dataset might not be large enough). It would be good if authors can comment/explain on this.
2. It would be good to add more experiments on "whether the discrete codex" module is useful? What if we directly use the raw continuous signal? How to better learn the discreate codex (the codex grouping in [1] can potentially improve the model)?
3. BART is a generally purpose pre-trained language model, which may not be suitable for this application. In the medical domain, [2] might be more suitable given the finetuned data might be limited.
4. More datasets and baseline models should be added to systematically and rigorously demonstrate the supriority of the proposed model.

[1] Baevski et al. Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.
[2] Lee et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method, “DeWave”, for decoding text strings from EEG data recorded from subjects while reading. This method seems to differ from earlier ones in two important and useful ways: (1) it uses a discrete “codebook” to represent the EEG data, which helps to control noise in the extremely noisy setting of EEG, and (2) it is also applied to continuous, rather than word-segmented, EEG data, more closely mimicking a real-world situation. The quantitative results show a modest boost over earlier methods in the word-aligned setting, and decent performance in the novel continuous setting.\n\nSTRENGTHS:\n* This paper is the first I have seen that attempts to decode continuous language, i.e. without explicit word boundary markers, from EEG data. This is a big and important step! Decoding from word-aligned EEG snippets is clearly much easier but also much less interesting than this variant. I think pushing for this type of decoding is a great contribution of the current work.\n* The use of a vector quantization approach to represent EEG data also seems like an important step. (However, I have not read every paper in this area, so I can’t say with 100% confidence that this is the first work to apply this approach to the current problem.)\n* The method seems, overall, well-designed.\n\nWEAKNESSES:\n* The paper is at several points confusing and difficult to follow or understand. For example, in section 3.2 I found it impossible to understand the sentence “The codex contains fewer time-wise properties which could alleviate the order mismatch between event markers (eye fixations) and language outputs.” (I’ll expand on this with specific queries under “Questions”, below.)\n* Minor scientific issue: Eye movements were used to segment the EEG data into words in the segmented (traditional) condition, but were ignored for the continuous condition. However, eye movements are well known to cause HUGE transient deflections in EEG signals, to the point where much/most EEG data includes preprocessing that attempts to minimize their influence. I would imagine such preprocessing was done for this dataset (please correct me if I’m wrong), but even ideal eye movement removal cannot remove 100% of the related signals. So I think it could be important for the authors to consider (and perhaps even test) whether their continuous decoder is actually discovering latent eye movement signals.\n\nQUESTIONS:\n* In section 3.2 under “Inference”, it would be very useful to state how temporal information is encoded, e.g. with a separate x for each word in the segmented condition, and for each 200 ms time period in the continuous condition. \n* In section 3.3 under “Word-Level EEG Features…”, the preprocessing is not well explained. This section might rely on the reader knowing earlier work, but I found it difficult to understand what the 840-dimensional vector actually comprises.\n* I found it very surprising and slightly worrying that the decoder seemed to be able to retrieve both exact proper nouns (“Kerouac” in Table 3, example 3) and approximate proper nouns (“Heroughs” for “Burroughs” in same). How is this possible? The contrastive semantic alignment loss, as I understand, cannot account for this, since it would operate on discrete tokens, unless those tokens also occurred in the training set. Is the discrete codex really capturing enough information about the constituent BPE tokens for those words to reconstruct them?\n\nLIMITATIONS:\nI think the most important limitation to note is that this method applies to EEG data collected while subjects actively read text, and cannot (currently) be applied to data collected while subjects merely think words. The authors do note this in their “Limitations” section, which is great.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new framework called DeWave, which integrates discrete encoding sequences with EEG-to-text translation tasks, using a quantized variational encoder and pre-trained language models. This approach overcomes the mismatch between eye fixations and spoken words and reduces interference from individual differences in EEG waves. The model outperforms previous baselines on the ZuCo Dataset, achieving improved BLEU-1 and Rouge-F scores. Importantly, this work is the first to enable translation of entire EEG signal periods without relying on word-level order markers like eye fixations.\n\nSTRENGTHS:\n- The authors introduce a novel discrete codex encoding to EEG waves, which seems like a promising way of representing EEG signals. \n- DeWave has promising results as it achieves state of the art performances on EEG to text translation.\n\nWEAKNESSES:\n- Figure 5's caption reads \"Ablation study on different codex sizes and perception fields (raw waves),\" however, for the table on the left, the experiments are done on word level EEG features. It would be better if the table title on perception field can specify the feature type (raw waves) instead of the caption. \n- In the abstract, the authors mention large language models such as ChatGPT, but experiment only on BART. An ablation study on other LLMs such as LLaMA, BERT, or RoBERTa can be enlightening. \n‌\n\nQUESTIONS:\n- The authors seem to have conducted the experiments separately for each patient or an averaged score over all patients. In [1], they mention \"In spite of the high inter-subject variability in EEG data, it has been shown in previous research of machine learning applications (Foster et al., 2018; Hollenstein et al., 2019a), that averaging over the EEG features of all subjects yields results almost as good as the single best-performing subjects.\" Have the authors considered this method?\n- I am aware that the ZuCo dataset has sentence level features. Additionally, I am aware of some works that have experimented on concatenated word level features [2]. Have the authors considered these different EEG feature types?  \n\n\n[1] Hollenstein N, Renggli C, Glaus B, Barrett M, Troendle M, Langer N, Zhang C. Decoding EEG Brain Activity for Multi-Modal Natural Language Processing. Front Hum Neurosci. 2021 Jul 13;15:659410. doi: 10.3389/fnhum.2021.659410. PMID: 34326723; PMCID: PMC8314009.\n[2] Han, William, et al. “An Empirical Exploration of Cross-Domain Alignment between Language and Electroencephalogram.” ArXiv:2208.06348 [Cs, Q-Bio], 10 Aug. 2022, arxiv.org/abs/2208.06348.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an approach to decode language from EEG data. The proposed approach can work on both time-locked and raw data (i.e. without markers indicating when a word was read). The model is trained to (1) reconstruct its EEG input using a vector-quantized representation (learnable \"codex\") in a pretraining stage, (2) align the learned codex representations with word2vec embeddings of the corresponding words and (3) fine-tune the whole model end-to-end including a language model. Experiments on a text-EEG dataset are presented, showing the proposed approach outperforms existing baselines including other self-supervised learning objectives that do not use vector quantization on BLEU and ROUGE metrics. Ablation studies on codex size and windowing parameters are presented, along with  an analysis of cross-subject performance. Finally, examples of EEG-to-text decoding are also presented.\n\nSTRENGTHS:\nOriginality: The proposed approach combining vector quantized representation learning and \"freeform\" text generation from decoded EEG is novel. \n\nQuality: The submission appears technically sound, with claims supported by benchmark comparisons and ablation studies.\n\nClarity: The paper is mostly clear, though some details are hard to understand from the text (see questions).\n\nSignificance: I believe these results pave the way to better text decoding from EEG data. This is important for the field of brain decoding given a lot of the work on brain-to-text has been using intracranial modalities or fMRI, which are very expensive and dramatically more constraining than surface EEG.\n\nWEAKNESSES:\nThe main weakness of this submission in my opinion is that several elements remained unclear after reading the text and taking a look at the provided code. For instance, I did not understand how the alignment was achieved between raw EEG and words/text (Question 2), how the language model was fine-tuned (Question 3) and how DeWave was combined with existing SSL approached (Question 4).\n\nQUESTIONS:\n1. Section 3.3, lines 136-141: The description in this paragraph is unclear to me. What is meant by \"statistical result of four frequency bands\"? I assume that summary statistics of the power in the four different bands were used? Also, the phrase \"different fragments may have different wavelengths\" is confusing as I believe the same frequency bands are used for each EEG fragment. More generally, why is a \"handcrafted feature\" approach used for the word-level features, while a similar convolutional encoder as is used for the \"raw wave\" could instead be trained end-to-end? The ablation of Table 4 shows that pretraining the word-level models improves performance bit a small amount, and I wonder if performance would further improve with a learned tokenization.\n2. I do not understand how the sequence of embeddings obtained from the raw EEG were aligned with word2vec representations (Section 3.3, second paragraph). In the word-level experiments, my understanding is that each word was embedded with word2vec, and that the resulting embedding was used to align the EEG-based discrete representations with the loss of Eq.4. However in the \"raw\" case, how was word2vec applied to the text, i.e. was it still applied at the word level?\n3. What loss is used to fine-tune the whole system end-to-end including the language model?\n4. Table 1 shows that combining DeWave with SCL further improves performance over the base DeWave model. How were these two approaches combined, and did the SCL approach had to be modified?\n5. The approach of [2] should be written \"wav2vec\" instead of \"wave2vec\" in the text and in Figure 2, Table 1, etc.\n6. There is missing information about the dataset (number of subjects, length of data collection, description of the text corpus, etc.).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}