DeWave: Discrete Encoding of EEG Waves for EEG to Text Translation
Authors: Yiqun Duan
Charles Zhou
Zhen Wang
Yu-Kai Wang
Chin-teng Lin
Venue: NeurIPS 2023 spotlight (2023)
Keywords: EEG; Neural Encoding; Brain Computer Interface

Abstract: The translation of brain dynamics into natural language is pivotal for brain-computer interfaces (BCIs), a field that has seen substantial growth in recent years. With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. These event markers may not be readily available or could be challenging to acquire during real-time inference, and the sequence of eye fixations may not align with the order of spoken words. To tackle these issues, we introduce a novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) it alleviates the order mismatch between eye fixations and spoken words by introducing text-EEG contrastive alignment training, and 2) it minimizes the interference caused by individual differences in EEG waves through an invariant discrete codex. Our model surpasses the previous baseline (40.1 and 31.7) by 3.06% and 6.34\%, respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. Furthermore, this work is the first to facilitate the translation of entire EEG signal periods without the need for word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset, respectively.

De Wave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation
Yiqun Duan'; Jinzhao Zhou', Zhen Wang', Yu-Kai Wang', Chin-Teng Lin -GrapheneX-UTS HAI Centre, Australian Artificial Intelligence Institute,
Faculty of Engineering and Information Technology
University of Technology Sydney, Ultimo, NSW 2007
School of Computer Science, The University of Sydney, Camperdown NSW 2050
{yiqun. duan, jinzhao zhtueett uts edu. au zwan4121@umi. sydney edu. au yukai. wangeuts. edu.au, chin-teng Lintuts. edu.au
Abstract
The translation of brain dynamics into natural language is pivotal for brain- computer interfaces (BCIs). With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. To tackle these is- sues, we introduce novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) realizes translation on raw waves without marker by introducing text-EEG contrastive alignment training, and alleviates the interference caused by individual differences in EEG waves through an invariant discrete codex with or without markers. Our model surpasses the previous baseline (40.1 and 31.7) by 3.06% and 6.34%, respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. This work the first to facilitate the translation of entire EEG signal periods without word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset.
Ground Truth
Bob attended the University of Texas at Austin where he graduated, Phi Beta Kappa with Bachelor's 's degree in Latin American Studies in 1973, taking only two and a half years to complete his work, and obtaining generally excel- lent grades.
Predict
was University of California at degree of degree history American Studies 1975. and one classes half years to complete the degree. and was excellent grades.
where studied in Beta Kappa
Figure 1: Overall illustration of translating EEG waves into text hrough quantised encoding.
Introduction
Decoding brain states into comprehensible representations has long been focal point of research 20 38 54]. Electroencephalogram (EEG) signals are particularly favored by researchers due to their non con-invasive nature and ease of recording 31 46]. Traditional EEG decoding techniques
is the first author, t is the corresponding author This visualization still keeps teacher-forcing evaluation for fair comparison with previous methods.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

largely focus on classifying brain states into restricted categories like Motor Imaginary (MI) 37
Emotion [9] 47 Robotic Control 4 15 , and Gaming 28 However, these labels, bound to specific tasks, are insufficient for broad-based brain-computer communication. Consequently, there has been surge of interest in brain-to-text (speech) translation in recent years. As the current trend leans towards large language models (LLM) E 12 30
showcasing increasingly generalized intelligence capabilities, it becomes crucial to delve into ways of bridging the gap between brain signals and natural language representation. However, this area remains under-explored.
The early work in brain-to-text translation handwriting or eye-trace trackinc fixations to segment whole brain signals into fragmentary features. This methodology treated the task as word-level classification on a small, closed vocabulary set, with each time step analyzed individually. Both invasive [14 22 (ECoG) and non-invasive [33] (EEG)
14]
26 4 relied on external event markers like
brain signals have been used in these approaches. Notably, utilizing handwriting as event markers on invasive signals has led researchers + to achieve state-of-the-art (SOTA) recognition accuracy on closed-set character-level recognition. Wang A4] expanded the vocabulary size substantially and demonstrated the feasibility of open-vocabulary brain-to-text translation by employing pre-trained language models with word-level EEG features. However, limitations persist. The order of event markers, particularly eye-tracking fixations used to segment EEG waves into word-level features, may not match the natural word order in language Moreover, current methods do not have the capacity for direct text translation.
In this paper. we present Discrete EEG Waves Encoding for Brain Dynamics Text Translation (DeWave), pioneering framework depicted in Figure DeWave uses vector quantized variational encoder to transform EEG waves into discrete codex, linking EEG waves to tokens based on their proximity to codex book entries. This method offers two key advantages: it addresses significant
distribution variances in EEG waves across individuals (20] 40 [ and rectifies order mismatches between raw wave sequences and text without eye-tracking markers. Our raw waves encoder is guided by both self-reconstruction and a contrastive supervision alignment between text embeddings and vectorized raw waves. To navigate the challenges of training with limited parallel data, Wave leverages large-scale pre-trained language models [6] 3G. B specifically employing BART (1I, which combines BERT's s bidirectional context with GPT's left-to-right decoder. Notably, our discrete codex aligns more closely with actual language tokens than continuous EEG features, serving as an interpretable bridge between EEG input and the language model.
large-scale public resource that records eye-tracking and BEG during natural reading tasks. Notably, DeWave can be generalized for both word-level EEG features and raw EEG wave translation. With a robust codex representation, this work pioneers in translating the entire time period of EEG signals without the need for word-level order markers such as eye fixations. We assess the performance using standard translation metrics EE! 25 With word-level EEG features, WeWave attains 42.8 BLEU-1 and 34.9 Rouge-1, surpassing the previous baseline (40.1 and 31.7) by 6.7 and 10.09% respectively on the ZuCo Dataset. For raw EEG waves without event markers, Wa achieves 20.5 BLEU-1 and 29.5 Rouge-1. 1. This work's 's contributions can be summarized in three main points.
Experiments employ non- con-invasive EEG signals and data from the ZuCo dataset 16
This paper introduces discrete codex encoding to EEG waves and proposes a new framework, Wavave for open vocabulary EEG-to-Tex translation.
By utilizing discrete codex, Wa is the first work to realize the raw EEG wave-to-text translation, where a self-supervised wave encoding model and contrastive learning- based EEG-to-text alignment are introduced to improve the coding ability.
Experimental results suggest the De reaches SOTA performance on EEG translation, where it achieves 41 1.35 BLUE-1 and 33.71 Rouge-1, which outperforms the previous baselines by 3.06% and 6.34% respectively.
2
Related Works
The key to decoding natural language from EEG signals is good representations. Existing work for De-ttt-text representation can be categorized into hand-crafted representations and deep-learning
For example. the ZuCo Dataset collects data by simultaneously recording eye-tracking fixation and brain waves during reading tasks. However, the order of eye- fixatiosso and spoken words may not always coincide.

---

Wave2Vec
Discrete Codex
Transformor
Teant to eay hello and wave
Indexing Codex
Prefrained BART
Treinffomeer Encoders
Dacoder
Gradients
Eya Fixetton Sliced Waves
Band Fiters
Projection Layer
6
Discrete Codex
Figure 2: The Wave model structure involves vectorizing either word-level EEG features raw EEG waves into embeddings (Section 3.3). The vectorized features are then encoded into latent variable (a()), which is converted into discrete latent (X') through codex indexing. Finally, pre-trained BART model translates this discrete codex representation into texts.
representations. Earlier works make use of traditional methods to obtain hand-crafted representations from a sequence of EEG signals. continuous feature representation extracted using methods such statistical features [ correlation coefficient matrix from fast Fourier transform (FFT) compo- 50), wavelet transform features [
nents
22 54 and fel-frequency cepstral coefficients
(MFCCs) 5 These hand-crafted representations can be used to establish a mapping between EEG
segments and words using distance-basc based methods such as linear discriminative analysis (LDA) or Support Vector Machine (SVM).
On the other hand, deep learning methods, such 12 utilize a convolutional neural network (CNN) and Long Short-Term Memory (LSTM) to learn deep features and perform classification of user's instructions. However, the direct representation often insufficient to discriminate a larger number of imagined text categories. In as the intermediate state using a deep convolutional autoencoder. Then latent features are used as input for another encoder predict total of 1I speech tokens. Recently, 4 proposed framework that uses a multi-layer transformer encoder to project word-level EEG feature sequences into EEG embeddings. Then pre-trained BART model is used to decode these embeddings into words.
39 EEG signals are first classified into six phonological categories
3 Method
The overall process of DeWave illustrated Figure where the word-level or raw EEG features vectorized into sequences embedding and fed into the discrete codex. The language model generates translation output based on discrete codex representation.
3.1 Task Definition
Given sequence of word-level EEG features E, the aim is to decode the corresponding open- vocabulary text tokens W. These EEG-Text pairs (E, W) are collected during natural reading, as defined in Section 4.1 We delve into two task settings: (1) Word-level BEG-to-Texx Translation, where EEG feature sequences are fragmented and re-ranked based on eye-fixation aligned with each word token in sequence W: and (2) Raw EEG Waves to Text Translation, where EBG feature sequences & are directly vectorized into embedding sequences for translation without any event markers, more challenging but practical real-time setting. DeWave is the pioneering work in this latter task.
3.2 Discrete Codex
Discrete representation is first proposed in VQ-VAE 45
DeWave is the first work to introduce discrete encoding into EEG signal representation. The discrete representation could benefit both the word-level EEG features and the raw EEG wave tramslation. Introducing discrete encoding into brain waves could bring two aspects of advantages. 1) widely accepted that EEG features have a strong data distribution variance across different human subjects. Meanwhile, the datasets can only have samples from few human subjects due to the expense of data collection. This severely weakened the generalized ability of EEG-based deep learning models By introducing discrete encoding,

---

could alleviate the imput variance to a large degree as the encoding is based on checking the nearest neighbor in the codex book. 2) The codex contains fewer time-wise properties which could alleviate the order mismatch between event markers (eye fixations) and language outputs. Meanwhile, because of this property of codex represents, DeWave is the first work that could realize the direct translation from raw EEG waves without any event marker to give the order.
Inference: Given the EEG waves E, is firsted vectorized into embedding as introduced in Section 3.3 with (X' = ( F)) or without (X' = (EE)) eye fixations F, where X' is the embedding sequence. A codex book {c,} E R is initialized with number & of latent embedding with size m. 'Tee vectorized feature X' is encoded into (x( through transformer encoder. The discrete representation is acquired by caleulating the nearest embedding in the codex of input embedding X' as shown in Equation M
2g = (Zz(x))
2x(x)= = ck.
== argminn (zxox) -cyl2
(I)
Different from the original VQ- which decodes the original input. Dewave directly decodes the translation output given the representation (2²). Given a pre-trained language model, the decoder predicts text output with (X))
The codex is like bridge connecting the vectorized EEG feature and the language model. Compared learning direct EEG-to-text relation, DeWave learns better discrete codex for the language model. It is easier to learn since We learn the discrete codex by the combination of the loss functions in three parts,
Learn:
L loggppWW//.gg)
[|sg//c(x)] n, (x) + Bl|ze(x) - (x)21(x))11ã
(2)
where the loss maximize the log-likelihood of language outputs log (P(W2,, and minimize the distance between latent variable z and the codex value z. Here, the sg denotes the stop gradients. The learning is robust for 33 from 0.1-2.0, where we set it as 0.2 throughout the training process.
3.3 EEG Vectorization
Word-Level EEG Features With Event Markers: The EEG waves are first sliced into fragments according to the eye-tracking fixation of word sequences given in the annotation. Similar to [S], we calculate the statistical result of four frequency band filters, Theta band (5-7Hz), the Alpha band (8-13Hz), the Beta band (12-30Hz), and Gamma band (30Hz-) [27] to get the statistic frequency features of each fragment. It is noted that although different fragments may have different EEG window sizes, the statistical results are the same (embedding size 840). A multi-head transformer layer is applied to project the embedding into feature sequences with latent size 512.
Raw EEG Waves: Self-Guided Waves to Discrete Codex Our self-supervised EEG wave encoder transforms raw EEG signals into sequence of embeddings (2] 56 as illustrated in Figure 3 It has two guiding principles: Seff-Reconstruction, where the encoder is trained to transform and subsequently reconstruct the original waveforms from discrete codices; and Text Alignment, where the codices' encoding is semantically aligned with word vectors, fostering the development of text-aligned EEG signal representations.
For structure- wise,, a conformer-basee multi-layer encoder with specially designed hyperparameters is employed. The one dee-dmenniionll convolution layer processes the EEG waves to generate the embedding sequence fusing the EEG channels into unique embedding for each period. We apply bi-directional transformer attention layers to the sequence to capture temporal relations.
For the reconstruction process, decoder transformer and transpose convolution structure then convert these discrete embeddings back into raw waves. Given the reconstruction process X' ((2, (X), the self-supervised loss could be modified into:
1 Σ (o(z,())) x)² + [[sg]z.(x)) (x)] - Z, (x) + (1]ze(x) - (x)),(x)]]
Lwave
(3)
TI
where the model calculates the mean square loss between the reconstructed wave and the wave ground truth perform self-supervised training.
To obtain a semantically coherent codex, we introduce a cross- moddlltyy contrastive learning approach that contrasts CLS embeddings between
distinct from prevalent methods. Unlike CLIP 35
2
2
percepption field is roughly 200ms with overlap 100ms for each embedding

---

Discret Codex
Embedding
Eneoder
Word2Vec
Source Text
Confonterr 12
komel (10.3 3.2 unide (3.22222))
Codlor Encoders
Coler Trangooee
Transforme Attemtton
EBG
Codlx Dooster
Generato Attantion
leadens, di dis5112 leyarms
Beeeder
heads-3, 0.5-512 laye--b
(3.33) (2,23)
Traupore
Figure 3: The image demonstrates the process of self-supervised pre-training for raw waves. The
left subgraph details our strategy for directing the encoder, utilizing both relftrecontt and text alignment through contrastive learning.
sample pairs within mini-batch, our approach operates within single EEG-text pair. We contrast the EEG-tokenized codex embeddings sequence 2g with the text embeddings sequence 2,. Assuming the raw wave feature extractors can produce a token sequence in an organized chronological order, we treat the diagonal EEG codex and text word2vec encoding pairs as positive pairs within the sequences. All other pairs considered negative. The model is trained to minimize the distance between embeddings positive pairs and maximize that of the negative pairs. For a given EEG-text pair (i, j), the loss is defined in Equation
Loontrast =-1x Σ log
exp(s;s/T)
Siy 2x(x')² zz(
(4)
k= exp(sik/1)
Here, T is the temperature parameter, N is the total number of EEG-text pairs in batch, and the sum in the denominator is over all N EEG-text pairs and N text-EEG pairs. The EEG embeddings are expected to correctly match with their corresponding text while distinguishing them from mismatched EEG-text pairs. The total loss then becomes a combination of original Wave2Vec loss and contrastive loss Ltotal Lwaae + a.ootrrstt
By this means, the model not only learns to reconstruct the EEG signal but also learns robust representation of the signal that aligns with the corresponding text embeddings. This cross-modal learning can potentially improve the translation system by bridging the gap between EEG signals and the semantic content of the text.
3.4 Language Model
We used large-scale text corpus pre-trained BART 21 the generative language model for transla- tion output. As the EEG-to-text translation data is quite limited, leveraging the BART model could introduce prior knowledge of text relations. In that case, the translation system only needs to learn a codex representation for the language model, which is easier to learn. The codex representations fed into pre-trained BART model and get the output hidden states. A fully connected layer is applied on the hidden states to generate English tokens from pre-trained BART vocabulary V.
3.5 Training Paradigm
DeWave is trained through multi-stage process, where the training process is illustrated in Ap- pendix C In the first stage, we do not involve the language model in weight updates. The target the first stage is to train proper encoder projection Goodz and discrete codex representation C for the language model. In the second stage, the gradient of all weights, including language model BATT is opened to fine-tune the whole system.
4 Experiments
4.1 Dataset
DeWave utilize both ZuCo 1.0 15 and 2.0 [1 for experiments. The dataset simultancously recorded
the text and EEG corpus during Normal Reading (NR) and Task-Specific Reading (TSR) tasks. The
5

---

EEG waves are collected with 128-channel system under a sampling rate 500Hz through frequency band filter from O.IHz to 100Hz. However, after the noise canceling process, only 105 channels [T5] are used for translation. Similar to [48], we slice the EEG wave according to the eye fixation and calculate the frequency features. For raw EEG waves, the signal is normalized into value range of 0-1 for decoding. The reading task's data are divided into the train (80%), development (10%), and test (10%) respectively by 10874, 1387, and 1387 unique sentences with no intersections. Please refer to Appendix A for detailed description.
4.2 Implementation Details
For word-level EEG features, we use the 56 tokens each with an 840 embedding size. For raw EEG waves, we clip or pad the EEG waves up to sample point 5500 with a constant value of zero. A transformer layer with head number 8 and a 1 x convolutional layer are combined to fuse multiple EEG channels into an embedding sequence with size 512. Wa uses a codex with size 2048 where each codex latent is an embedding with size 512. The ablation study (Section 4.6 gives a discussion about the codex size. All models are trained on Nvidia V100 and A100 GPUs. For the self self-supervised decoding for raw waves, we use a learning rate of 5e-4 and a VQ coefficient of 0.25 for training 35 epochs. For training the codex (stage 1), DeWave uses learning rate of 5e-4 for 35 epochs. For finetuning the translation (stage 2). Wewave uses a learning rate of 5e-6 for 30 epochs. We use the SGD as the optimizer for training all the models. Due to limited space, refer to Appendix B for more details.
4.3 Evaluation Metrics
We evaluate translation performance using NLP metries, BLEU and ROUGE, as shown in Table For word-level EEG features, we compare our results to EEG-to-Tex 48), maintaining a consistent language model for fairness. In the absence of methods for raw EEG waves, we establish baseline EEG-to-Text! by segmenting the entire EEG waves into a sequence embedding using a 200ms time window with 100ms overlap. We adapt Wave2 originally developed for speech recognition, to brain waves and compare it with our approach, DeWave. Furthermore, we adapt unsupervised raw EEG waves classification methods BENDR [I] and SCL [0,, using SSL pre-training and feature extraction for comparison, underscoring the impact of discrete encoding.
Table 1:
Evaluation metrics of EEG-to-Ter Text translation under both word-level features input and raw waves input, where +Contrastive denotes boost encoder with Leontrast mentioned in Sec. 3.3
For a fair
comparison, these results keep the same teacher-forcing evaluation setting as EEG-to-Tees Text M
BLEU-N (%)
ROUGE-1 (%)
Source
Method
N=1
N=2
N=3
N=4
R
P
F
Word-level features
EEG-to-Ter
48
40.12
23.18
12.61
6.80
28.84
31.69
30.10
DeWave
41.35
24.15
13.92
8.22
28.82
33.71
30.69
EEG-to-Texi
48
13.07
5.78
2.55
1.10
15.22
18.08
16.36
Wave2Vec
2
18.15
8.94
3.89
2.04
18.96
23.86
20.07
BENDR
18.48
9.16
4.05
2.15
19.03
25.22
21.18
Raw waves
De
20.51
10.18
5.16
2.52
21.18
29.42
24.27
De Contte-Conteassive
21.09
10.69
5.88
3.04
22.01
29.95
24.68
Word-Level EEG Features: For the word-level EEG feature, we observe that introducing the dis-
crete brain representation could help We reach BLEU-(1, 2, 4) scores of 41.35, 24.15, 13.92, and 8.22, which respectively outperform the previous baseline by 1.23 (+3.06%), 0.97 (+4.11%), 1.31 (+10.38%) and 1.54 +11.77%). It is observed that the increasing ratio is more significant for larger grams evaluation. Wawave achieve ROUGE-1 score 28.84 (R), 31.69 (P), and 30.10 (F) which outperforms the previous baseline by -0.02 (-0.66%). 1.98 (+6.35%). and 0.59 (+1.9%). The increase of the matrix is higher for longer phrases (3-gram and 4-gram). This is rational since the single-word level EEG features are sliced by eye fixation during reading, which may naturally contain noises since the human subject may not really think about the corresponding words when looking at them.
Dave represents unprecedented endeavor in translating raw EEG waves directly, obviating the need for event markers. This achieved through the application of learned Weve2Vec model, which converts waves into discrete codex tokens. When benchmarked against
Raw Waves:

---

the baseline method which merely slices the wave according to time window Wewe exhibits marked improvement. It attains BLEU-1, 4 scores of 20.51, 10.18, 5.16, and 2.56, respectively. surpassing the baseline by margins of 7.44 (+ 1-5699%). 4.44 (476.1%). 2.61 (-102.35%) and 1.42 (4-12909%%) This underscores the superiority of a learned discrete embedding representation over the rudimentary time-window baseline. In comparison with cutting-edge self supervised learning (SSL) methods, Wave consistently outperforms them, including contrastive learning (CL)-based methods such as SCL [TO, the original Wave2Vec 22], and BENDR |_
4.4 Cross-Subject Performance
cross--ubbect performance is of vital importance for practical usage. To further illustrate the perfor- mance variance on different subjects, we train the model by only using the data from subject YAG and test the metrics on all other subjects. The results are illustrated in Figure where the radar chart denotes the performance is stable across different subjects. Please refer to Appendix G for more detailed results.
Subject Perfomance BLEU-2
Subject Perfomance BLEUU- 3MD
Subject Perfomance ROUGER
Subject Perfemancee ROUIGEF
IMC 195
YaM...
182
154
152
182
T:0t
1500
10A
130
VER
3347
108
Figure 4: The cross-subjects performance on ZuCo dataset.
4.5
Generated Samples
In Table 2 we display visualized examples of text generated from unseen EEG signals. Despite the challenge of thought translation and limited prior research, our model yields meaningful results, aligning keywords and forming similar sentence structures, although it may not yet match traditional language translation tasks.
The model is more adept at matching verbs than nouns. For instance, implies" vs. implies". the author who wrote it" vs. the man who wrote it" both effectively convey the intended sentiment of the sentence. However, when it comes to nouns, we observe a tendency towards synonymous pairs rather than precise translations, such as the man" vs. the author", Burroughs" vs. Heroughs", edition" " vs. version". Our analysis suggests two potential causes for this. First, when the brain processes these words, semantically similar words might produce similar brain wave patterns. Given the inherent noise in brain waves, the codex might group these features under the same value. Second, the volume of EEE--to-Tess pairs available for training is significantly smaller than that for traditional language translation. Hence, some degree of error translating unseen nouns sentences is to be expected.
Translation on raw EEG waves naturally harder than word-level translation, as it lacks eye fixation to suggest the relationship between the period of waves and the word target. Table meet our expectation that the results on raw EEG waves are not as good as those on word-level features, especially on the real semantic meaning of the sentence (sample (2) and sample (5) have the same target for comparison). However, the translation still could output the correct translation of certain words, such as "much of" vs. "much of", "individual" vs. "individual", and "more complicated story" vs. "more exciting thing". Although EEG-to-text a hard topic, Wave suggests the feasibility of translation improvements.
4.6 Ablation Study
Discrete Codex DeWave encodes EEG waves into discrete codex, aiming for a language model- friendly representation. We evaluated performance against varying codex sizes (1024 to 8192) to ascertain if larger sizes yield better results. As depicted in Figure 5 we found no strong correlation between codex size and model performance. A codex size of 2048 yielded the highest BLEU score average, and while the ROUGE score slightly improved with larger sizes, there was no clear evidence that increasing codex size consistently enhanced performance.
7

---

Table 2: Translation results the unseen EEG waves, where bold denotes correct match between ground truth and our prediction. Underline denotes a fuzzy match with similar semantic meanings. For fair comparison, these results keep the same teacher-forcing evaluation setting EEG-to- Text 48. This means the decoding process eliminates accumulated errors and just predicts the current token with the GT token from the last step.
Decoding Results with Eye Fixation Assistance
(1) Ground Truth: Everything title implies, a standard issue crime drama spat out from the Tinseltown assembly line...
Prediction: own implies, including great issue, novel of the beginningschownss...
(2) Ground Truth: "The Kid Stays in the Picture" is a great story, terrifically told by the man who wrote it but this
Cliff Notes edition is a cheat.
Prediction: The film "says in the Game" is a film about but movic was written, the author who wrote it. also its version cheat.
(3) Ground Truth: During Kerouac's
at Columbia University, Burroughs and Kerouac got into trouble with the law for failing report murder, this incident formed the basis of mystery novel
Prediction: Keouac's time the,. Heroughs and Kerouac were along for the police their to pay the murder. they led led basis of the lawsuit... ...
Decuding Results with Raw Waves
(4) Ground Truth: Every individual will see the movic through the prism of his or her own beliefs and prejudices...
Prediction: Everyday individual is their results. their eyes of his or her own personal, desires. and
Ground Truth: "The Kid Stays in the Picture" is a great story, terrifically told by the man who wrote it but this
(5)
Cliff Notes edition is a cheat.
Model Output:
The Price's says in the Middle. and a common deal. and for good. this moment. made still little.
(6) Ground Truth: much of this well-acted but dangerously slow thriller feels like preamble a bigger, complicated story.
Model Output: much of this is-being but not over- like it disaster-ble to new and more exciting thing.
Raw EEG wave performance fluctuates noticeably with codex size. Performance improves when increasing the codex size from 1024 to 2048, but any larger size reduces performance. This variation may result from the training formation; word-level EEG data, selected by eye fixations, contains less noise than raw waves. We hypothesize that our current training data may be insufficient for larger codex sizes, Additionally, our experiments indicate that the latent dimension of the codex doesn't t significantly affect performance.
Translation Metrics Variation Word EBG Feature)
Translation Metrics Variation Raw Waves)
Ablation Metrics
Perception Field
30
40
35
Metrics (96)
30
Metrics (36)
Metrics (%6
20
BLEU-1
25
BLEU-2
DLEU-1
15
BLEU-2
BLBU-1
35
BLEU-2
00-
BLEU-3
BLBU-3
BLEU-
BLEU-4
BLEU-4
ROUGE-R
15-
ROUGE-R
ROUGE-R
ROUGE-P
10-
ROUGE-P
ROUGE-P
ROOGE-F
ROUGE-F
ROUGE-F
4Dme
10me
120m
100me
200ms 240ms 500me 1000ms
2048
4008
6144
1024
6144
Perception Fleld
Codex Size
Codex Size
Figure Ablation study on different codex sizes and perception fields (raw waves).
Perception Time Window We also conduct the ablation study on the model structure for the wave2vec model illustrated in Figure 5 As the model utilize a multi-layer CNN model to slide through the raw waves, the model compresses the waves for perception. The compress ratio decided how large the perception field is for each extracted embedding feature. As described in Section 3.3 the model utilized a perception field of 200ms with an overlap of 100ms. We conduct an ablation study of different perception fields and report it in Figure 5 Where it is observed that the model performance is significantly lower when the perception field is smaller than 80ms or larger than 240ms. The model could achieve similar results in the perception field 120ms to 240ms. The model reaches small peak around 200ms to 240ms. We think this phenomenon rational since the normal reading speed for humans is around 160-400 words per minute (reading speed may vary from different

---

material and human subjects). In other words, the reading period for each word is 150-375ms on average, which roughly meets our observation between 200ms-240ms.
Self-Supervision Initialization Theoretically, we could pre-train the codex by introducing a decoder and calculate the reconstruction loss with the original input for both word-level EEG and raw EEG waves. We prefix other parameters as our best setting with codex size 2048 and compare the impact in Table
Table 3: Ablation on self-supervised pre-trained codex weights.
Pretrain
BL,EU (%)
ROUGE- I (%)
N=1
N=2
N=3
P
Word-level
41.35
24.15
13.92
28.82
33.71
30.69
features
x
40.71
22.94
12.40
28.01
34.08
30.63
20.51
10.18
5.16
21.18
29.42
24.27
Raw waves
16.58
7.78
3.68
17.86
19.84
18.33
For word-level EEG features, the impact of pre-train is mostly on BLEU scores while the ROUGE score does not have much variance. Without pre-train, the (1, respectively drop by 0.64, 1.21, and 1.52. For direct translation on raw waves, the impact is significantly larger. Without self- supervised initialization, the BLEU-(1, 3} respectively drop by 3.93 (L 19.16%), 2.40 (4 23.57%). and 1.48 (L 28.68%). Similar observation also appears on ROUGE scores. This phenomenon is rational since raw wave decoding requires the model to pick useful features without any help from eye fixations. The self-superviser initialization could help the model form a preliminary ability to extract time-wise or channel-wist -iie features from raw waves.
Limitations
Despite DeWave's 's enhancements in EEG-to-Text translation using discrete codex and raw wave encoding, its accuracy remains far from real-life scenarios compared to traditional language-to- language translations. Also, to keep fair comparison with EEG-to-Text, this paper uses a teacher- forcing setting in evaluation. This setting eliminates accumulation error and turns the sequence decoding task into a word-level classification task given the ground truth token from the previous step. This setting is relatively easier yet we think it is still valuable as it could suggest feature extraction quality while keeping a fair comparison.
Additionally, the experiments in this paper are restricted to public neural reading data, not fully aligning with the "silent speech" concept of direct thought translation from human brains. Instead, the current ZuCo dataset is collected by giving people reading stimuli. This paper focuses on introducing Wav2Vee formation of feature extraction on raw EEG waves and introduces discrete codex as learnable representations for the CEG-to-Text translation domain. One scientific problem in this domain is better way of doing the "silent speech" task, which we are exploring as on-going research.
6 Conclusion
This paper presents DeWave, framework for the recently proposed open-vocabulary EEG-to- Text translation task 48). introducing the concept of discrete codex encoding. This approach brings enhancement in corpus text relevancy metrics, such as BLEU and ROUGE. WeWave also expands the task to decode raw EEG waves without the assistance of eye fixation markers. Despite these advancements, the quality of brain decoding results remains substantially low and remains teacher forcing setting for fair comparison. The translation of thoughts directly from the brain is a valuable yet challenging endeavor that warrants significant continued efforts. In our ongoing work, we are exploring more reasonable settings that remove teacher forcing for both training and testing. We will also include the Ineural-feedbade mechanism in this EEG-to-TH research to enhance the scientific value of this domain.
Acknowledgement
This work was supported in part by the Australian Research Council (ARC) under discovery grants DP210101093 and DP220100803, and the GrapheneX-UTS Human-Centric Al Centre sponsored by GrapheneX 22023-2031).

---

References
[1] Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang. Speech synthesis from neural decoding of spoken sentences. Nature, 568(7753):493 498, 2019.
Alexei Baevski, Yuhao Zhou, Ahdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449- 12460, 2020.
[3] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advancer in Neural Information Processing Systems, 2020.
[4] Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Eee Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712. 2023.
[5] Ciaran Cooney, Rafaella Folli, and Damien Coyle. Mel frequency cepstral cocfficients enhance imagined speech decoding accuracy from eog. In 2018 29th Irish Signals and Systems Conference (ISSC), pages 1-7. IEEE, 2018.
[6] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, 2019.
[7] Yiqun Duan. Zhen Wang, Yi Li, Jianhang Tang, Yu-Kai Wang, and Chin-Teng Lin. Cross task neural architecture search for eeg signal classifications. arXiv preprint (xXiv:22/0006220 2022.
[8] Yiqun Duan. Zhen Wang, Yi Li, Jianhang Tang. Yu-Kai Wang, and Chin-Teng Lin. Cross task neural architecture search for eeg signal recognition. Neurocomputing, 545: 126260, 2023.
[9] Yiqun Duan, Jinzhao Zhou. Zhen Wang, Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teny Lin. Domain- specific denoising diffusion probabilistic models for brain dynamics. arXiv preprint arXiv:2305.042: 002.00000 2023.
[10] Jiang et al. Self-supervised contrastive learning for eeg-based sleep staging. In IJCNN 2021, pages 1-8. IEFE, 2021.
[11] Kostas et al. Bendr: using transformers and a contrastive sulf-super learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15:653659, 2021.
[12] Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
[13] Anmol Gulati, James Qin, Chung-Chare Coung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang. Yonghui Wu, et al. Conformer: Convolution-augmeedt transformer for speech recognition. arXiv preprint arXiv:2005.08100 2020.
[14] Christian Herff, Dominic Heger, Adriana De Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk, and Tanja Schultz. Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers in neuroscience, 9:217, 2015.
[15] Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. Zuco, simultaneous ceg and eye-tracking resource for natural sentence reading. Scientific data, 5(1):1-13, 2018.
[16] Nora Hollenstein, Marius Troendle, Zhang, and Nicolas Langer. Zuco 2.0: A dataset of physiological recordings during natural reading and annotation. Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerek, Sara Goggi, Hitoshi Isahara, Bente Muegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of The 12th Language Rexourves and Evaluation Conference, LREC 2020, Marseille, France, May 11-16. 2020. pages 138-146. European Language Resources Association. 2020.
[17] Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. Zuco 2.0: A dataset of physiological recordings during natural reading and annotation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 138-146, 2020.
[18] Amir Jahangiri and Francisco Sepulveda. The relative contribution of high gamma linguistic processing stages of word production, and motor imagery of articulation in class separability of covert speech tasks in eeg data. Journal of medical systems, 43(2):1-9, 2019.
[19] Robert Jenke, Angelika Peer, and Martin Buss. Feature extraction and selection for emotion recognition from eeg. IEEE Transactions on Affective computing, $(3):327-33) 2014.
10

---

[20] Reinmar J Kobler, Jun ichiro Hirayama, Qibin Zhao, and Motoaki Kawanabe. SPD domain-specific batch normalization crack interpretable unsupervised domain adaptation in EEG. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[21] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising serueenne----sqeenne pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, 2020.
[22] Yi Li, Hualiang Wang. Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.0665. 2023.
[23] Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xiaomeng Li. Exploring visual interpretability for contrastive language-image pre-training. arXiv preprint arXiv:2209.07044 2022.
[24] Lun-De Liao, Chi-Yu Chen, I-Jan Wang, Sheng- Fu Chen, Shih- Yu Li, Bo-Wei Chen, Jyh- Yeong Chang. and Chin-Tene Teng Lin. Gaming control using a wearable and wireless eeg -aaed brain- comptce interface device with novel dry foam based sensors, Journal of neuroengineering and rehabilitation, 9(1):1 122 2012.
[25] Chin- Lin. ROUGE: A package for automatic evaluation of summarics. In Text Summarization Brunches Out, pages 74-81, Barcelona, Spain, July 2004, Association for Computational Linguistics.
[26] Joseph G Makin, David A Moses, and Edward Chang. Machine translation of cortical activity to text with an mecoer-decoded framework. Nature Neuroscience, 23(4):575-3) 2020.
[27] Dennis J McFarland, Charles W Anderson, K-R Muller, Alois Schlogl, and Dean Krusienski. Bci meeting wo0S-workshop on bci signal processing: feature extraction and translation. IEEE transactions on neural systems and rehabilitation engineering, 14(2):135-133 2006.
[28] Anton Nijholt. Bci for games: A "state of the art' survey. In Scott M. Stevens and Shirley J. Saldamarco, editors, Entertainment Computing ICEC 2008, pages 225-228, 2009.
[29] Julie Onton, Marissa Westerfield, Jeanne Townsend, and Scott Makeig. Imaging human ceg dynamics using independent component analysis. Neuroscience & biobehavioral reviews, (066):80-822.. 2006.
[30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2775
2022.
[31] Yue- Ting Pan, Jing-Lam xm Chou, and Chun-Shu Wei, MAtt: A manifold attention network for EEG decoding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems. 2022.
[32] Jerrin Thomas Panachakel, AG Ramakrishnan, and TV Ananthapadmanabba. A novel deep learning architecture for decoding imagined speech from cog. arXiv preprint 2rxiv:2003.09374, 2020.
[33] Jerrin Thomas Panachakel and Angarai Ganesan Ramakrishnan. Decoding covert speech from eeg-a comprehensive review. Frontiers in Neuroscience, 15:392, 2021.
[34] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. arXiv preprint
(tt::2103,00022
2021.
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[37] Jaime F Delgado Saa and Müjjat Çetin. Discriminative methods for classification of asynchronous imaginary motor tasks from eeg data, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 21 21(5):11-7724, 2013.
[38] David Sabbagh, Pierre Ablin, Gael Varoquaux, Alexandre Gramfort, and Denis A. Engemann. Manifold- regression to predict from meg/ceg brain signals without souree modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buee Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[39] Pramit Saha, Muhammad Abdul-Mageed, and Sidney Fels. Speak your mind! towards imagined speech recognition with hierarchical deep learning. arXiv preprint 1cXvv:1904.05746 2019.
[40] Simanto Saha and Mathias Baumert. Intra-and inter subject variability in bagebased sensorimotor brain computer interface: review. Frontiers in computational neuroscience, 13:87, 2020.
11

---

[41] Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Towards sentence-level brain decod- ing with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7047-7054, 2019.
[42] Pengfei Sun, Gopala K Anumanchipalli, and Edward F Chang. Brain2char: a deep architecture for decoding text from brain recordings. Journal of Neural Engineering, 17(6):060015, 2020.
[43] Kazuo Tanaka, Kazuyuki Matsunaga, and Hua O Wang. Electroencephalogram- based control of an electric 2005.
wheelchair. IEEE transactions on robotics, 21
[44] Luca Tonin, Tom Carlson, Robert Leeb, and José del R. Millán. Brain-controlled telepresence robot by motor-disabled people. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 4227-4230, 2011.
[45] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information procesing systems, 30, 2017.
[46] Neeraj Wagh, Jionghao Wei, Samarth Rawal, Brent M. Berry, and Yogatheesan Varatharajah. Evaluating latent space robustness and uncertainty of EEG-ML models under realistic distribution shifts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[47] Jiang Wang and Mei Wang. Review of the emotional feature extraction and classification using eeg signals. Cognitive Robotics, 1:29-40 40, 2021.
[48] Zhenhailong Wang and Heng Ji. Open vocabulary electroencephalgrappy to-text decoding and zero-shot sentiment classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 3350-5358, 2022.
[49] Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performe performance brain-to-tens communication via handwriting. Nature, 593((858):249 2222449-2544444 2021.
[50] Haoning Xi, Didier Aussel, Wei Liu, Travis Waller, and David Rey. Single-leader multi-follower games for the regulation of two-sided mobility-assa-service markets. European Journal of Operational Research, 2022.
[51] Haoning Xi, Liu He, Yi Zhang. and Zhen Wang. Differentiable road pricing for environmentt-rrennted electric vehicle and gasoline vehicle users in the bi-objective transportation network. Transportation Letters, 14(6):660-674, 2022.
[52] Xiang Zhang. Lina Yao. Quan Z Sheng. Salil Kanhere, Tao Gu, and Dalin Zhang. Converting your thoughts to texts: Enabling brain typing via deep feature learning of eeg signals. In 2018 IEEE international conference pervasive computing and communications (PerCom), pages 1-10. IEEE, 2018
[53] Shunan Zhao and Frank Rudzicz. Classifying phonological categories in imagined and articulated speech. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 992 996. IEEE, 2015.
[54] Jinzhao Zhou, Yiqun Duan, Yu- Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teng Lin. Belt: Bootstrapping hwwpreeepphooggapptttttoaaaeeeee
decoding and zero-shot sentiment classification hy natural language supervision. arXiv preprint arXiv:2309. /2056, 2023.
[55] Jinzhao Zhou, Yiqun Duan, Zhihong Chen, Yu-Cheng Chang, and Chin-Teng Lin. Generalizing multimodal variational methods sets, 2022.
[56] Jinzhao Zhou, Yiqun Duan, Yingying Zou, Yu-Cheng Chang, Yu-Kai Wang. and Chin-Teng Lin. Speech2eeg: Leveraging pretrained speech model for eeg signal recognition. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023.
12


Reviews:
[{"review_id": "0pz7Jewxnb", "reviewer": "Reviewer_TbVs", "summary": "The authors present an approach to decode language from EEG data. The proposed approach can work on both time-locked and raw data (i.e. without markers indicating when a word was read). The model is trained to (1) reconstruct its EEG input using a vector-quantized representation (learnable \"codex\") in a pretraining stage, (2) align the learned codex representations with word2vec embeddings of the corresponding words and (3) fine-tune the whole model end-to-end including a language model. Experiments on a text-EEG dataset are presented, showing the proposed approach outperforms existing baselines including other self-supervised learning objectives that do not use vector quantization on BLEU and ROUGE metrics. Ablation studies on codex size and windowing parameters are presented, along with  an analysis of cross-subject performance. Finally, examples of EEG-to-text decoding are also presented.", "questions": "1. Section 3.3, lines 136-141: The description in this paragraph is unclear to me. What is meant by \"statistical result of four frequency bands\"? I assume that summary statistics of the power in the four different bands were used? Also, the phrase \"different fragments may have different wavelengths\" is confusing as I believe the same frequency bands are used for each EEG fragment. More generally, why is a \"handcrafted feature\" approach used for the word-level features, while a similar convolutional encoder as is used for the \"raw wave\" could instead be trained end-to-end? The ablation of Table 4 shows that pretraining the word-level models improves performance bit a small amount, and I wonder if performance would further improve with a learned tokenization.\n2. I do not understand how the sequence of embeddings obtained from the raw EEG were aligned with word2vec representations (Section 3.3, second paragraph). In the word-level experiments, my understanding is that each word was embedded with word2vec, and that the resulting embedding was used to align the EEG-based discrete representations with the loss of Eq.4. However in the \"raw\" case, how was word2vec applied to the text, i.e. was it still applied at the word level?\n3. What loss is used to fine-tune the whole system end-to-end including the language model?\n4. Table 1 shows that combining DeWave with SCL further improves performance over the base DeWave model. How were these two approaches combined, and did the SCL approach had to be modified?\n5. The approach of [2] should be written \"wav2vec\" instead of \"wave2vec\" in the text and in Figure 2, Table 1, etc.\n6. There is missing information about the dataset (number of subjects, length of data collection, description of the text corpus, etc.).", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The proposed approach combining vector quantized representation learning and \"freeform\" text generation from decoded EEG is novel. \n\nQuality: The submission appears technically sound, with claims supported by benchmark comparisons and ablation studies.\n\nClarity: The paper is mostly clear, though some details are hard to understand from the text (see questions).\n\nSignificance: I believe these results pave the way to better text decoding from EEG data. This is important for the field of brain decoding given a lot of the work on brain-to-text has been using intracranial modalities or fMRI, which are very expensive and dramatically more constraining than surface EEG.", "weaknesses": "The main weakness of this submission in my opinion is that several elements remained unclear after reading the text and taking a look at the provided code. For instance, I did not understand how the alignment was achieved between raw EEG and words/text (Question 2), how the language model was fine-tuned (Question 3) and how DeWave was combined with existing SSL approached (Question 4). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comprehensive review, appreciation, and insightful inquiries. We will address your concern step by step below and improve in the final version.\n\n1. **Regarding Section 3.3, lines 136-141 and beyond:** This paragraph is to introduce how we construct the word-level EEG features. The method is the same as it is in the ZuCo dataset [1] and EEG-to-Text [2] for fair comparison. \n   - \"statistical result of four frequency bands\": When the ZuCo dataset preprocesses word-level features, it averages power statistical features on all corresponding EEG fragments of four main bands (two sub-bands for each main band), including 'theta\\_{1,2} (4-6 Hz, 6.5-8 Hz)', 'alpha\\_{1,2} (8.5-10 Hz, 10.5-13 Hz)', 'beta\\_{1,2} (13.5-18 Hz, 18.5-30 Hz)', 'gamma_{1,2} (30.5-40 Hz, 40-49.5 Hz)'.\n   - By \"different fragments may have different wavelengths\", we meant to emphasize the sliced EEG fragments have **varying time lengths of waves**, not wavelengths. The varying time lengths of EEG waves caused by the eye-tracking markers length variation arises because human subjects spend different amounts of time reading different words. Apologize for the typo here, we will correct “wavelengths” to the “different lengths of EEG wave samples” throughout the whole paper.\n   - Comparing word-level handcraft features with raw waves. Though both are hard tasks, translation on word-level features is easier as :\n     - It already has an EEG feature for word alignment suggested by eye-tracking markers. Instead, the raw wave setting requires the transformer to learn the alignment through self-supervised pre-training as illustrated in Fig.3. \n     - Less noise: Each word-level feature is consistent in each sentence as it averaging feature regardless of how many times the same word appears in the sentence. The performance gap in Table 1 supports this point. \n\n    - Learned tokenization: Thank you for a good point. We think the projection layer is taking a similar role, where it `tokenizes’ handcraft word-level features into transformer space. Tab. 4 suggests that learned tokenization has improvements in the word-level setting. \n\n2. **Alignment on raw waves:** For the raw waves setting, there are no eye-tracking markers to slice EEG waves accordingly. As shown in Fig. 3, we use a wide convolutional kernel to slide on waves to sequentially slice raw waves time-wise. Then, position embeddings are added into each sliced embedding to maintain the order information. The alignment happens extrinsically sequence-wise (sentence-wise). We simultaneously optimize both the discrete coding and alignment similar to CLIP as described in Eq.4, where alignment matrix $s_{i,j}$ which contains all possible alignment is calculated between each encoded EEG embedding and text embedding in a sequence pair. The transformer attention encoders are expected to learn the alignment $s_{i,j}$ intrinsically given position embedding and output supervision. The current code only contains E2E training, discrete codex, and evaluation, code for contrastive alignment pretraining will be available after the anonymous era. \n\n3. **End-to-end fine-tuning:** As described between lines 128-132, we maximize the log-likelihood on language output similar to most of the language decoder end-to-end training as $L=-log(P(W |zq (X ))$ for both word-level EEG features and raw waves. The only difference to Eq. 2 is that the discrete codex is fixed during fine-tuning. Thank you very much for the notice, we will make these points more clear for better readability. \n\n4. **How were DeWave and SCL combined?** Here the +SCL meant we apply the additional contrastive loss proposed in their paper and combine it with a coefficient when training the transformer encoder before discrete codex. As SCL are RNN-based model, we didn’t use their model structure. However, this point is not related to our core contribution. We will clarify this part in our final version and provide a section to introduce details in supplementary details. \n\n5. **Typo Correction:** Thank you for pointing out the problem regarding \"wav2vec\". We will correct \"wave2vec\" to \"wav2vec\" throughout the text, Fig. 2, Tab. 1, and other potential problems. \n\n6. **More information about the Dataset:** We use the ZuCo (including both 1.0, 2.0) dataset [1] for our main experiments which is exactly the same as [2]. ZuCo stands for Zurich Cognitive Language Processing Corpus (ZuCo), including raw and preprocessed eye-tracking and electroencephalography (EEG) data. The data is collected by letting human subjects read the given text corpus and simultaneously recording both recording the eye-tracking signal and EEG waves. The collection device is the Biosemi-128 system. After denoising, it provides 105 of 128 channels for down-streaming tasks. The 1.0 data is collected from 12 subjects while the 2.0 data is collected from 18 subjects. \n\n   Regarding the text corpus of the ZuCo dataset, these were sourced from a diverse set of textual genres to ensure a wide variety of syntactic structures and word frequencies, which includes: 1) Wikipedia articles 2) movie reviews 3) BNC (British National Corpus). \n\n   We further apply a statistical analysis on the sentences we utilized from the dataset, and reported below.\n\n   | Feature      | ZuCo 1.0 Natural Reading | ZuCo 2.0 Natural Reading |\n   | ------------ | :----------------------: | :----------------------: |\n   | sentences    |           300            |           390            |\n   | sent. length |       21.3 (±10.6)       |       19.6 (±8.8)        |\n   | total words  |           6386           |           6828           |\n   | word length  |           6.7            |           4.9            |\n\n[1] Nora Hollenstein, et al, **ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation.** LREC 2020   \n\n[2] Zhenghailong Wang, et al, **Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification** AAAI 2022"}, {"review_id": "9kglZxPfEh", "reviewer": "Reviewer_tXBq", "summary": "The authors propose a new framework called DeWave, which integrates discrete encoding sequences with EEG-to-text translation tasks, using a quantized variational encoder and pre-trained language models. This approach overcomes the mismatch between eye fixations and spoken words and reduces interference from individual differences in EEG waves. The model outperforms previous baselines on the ZuCo Dataset, achieving improved BLEU-1 and Rouge-F scores. Importantly, this work is the first to enable translation of entire EEG signal periods without relying on word-level order markers like eye fixations.", "questions": "- The authors seem to have conducted the experiments separately for each patient or an averaged score over all patients. In [1], they mention \"In spite of the high inter-subject variability in EEG data, it has been shown in previous research of machine learning applications (Foster et al., 2018; Hollenstein et al., 2019a), that averaging over the EEG features of all subjects yields results almost as good as the single best-performing subjects.\" Have the authors considered this method?\n- I am aware that the ZuCo dataset has sentence level features. Additionally, I am aware of some works that have experimented on concatenated word level features [2]. Have the authors considered these different EEG feature types?  \n\n\n[1] Hollenstein N, Renggli C, Glaus B, Barrett M, Troendle M, Langer N, Zhang C. Decoding EEG Brain Activity for Multi-Modal Natural Language Processing. Front Hum Neurosci. 2021 Jul 13;15:659410. doi: 10.3389/fnhum.2021.659410. PMID: 34326723; PMCID: PMC8314009.\n[2] Han, William, et al. “An Empirical Exploration of Cross-Domain Alignment between Language and Electroencephalogram.” ArXiv:2208.06348 [Cs, Q-Bio], 10 Aug. 2022, arxiv.org/abs/2208.06348.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors introduce a novel discrete codex encoding to EEG waves, which seems like a promising way of representing EEG signals. \n- DeWave has promising results as it achieves state of the art performances on EEG to text translation.", "weaknesses": "- Figure 5's caption reads \"Ablation study on different codex sizes and perception fields (raw waves),\" however, for the table on the left, the experiments are done on word level EEG features. It would be better if the table title on perception field can specify the feature type (raw waves) instead of the caption. \n- In the abstract, the authors mention large language models such as ChatGPT, but experiment only on BART. An ablation study on other LLMs such as LLaMA, BERT, or RoBERTa can be enlightening. \n‌", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate your thorough review and insightful feedback. We will address each of your comments and concerns below and also in our revised manuscript. \n\n1. **Improve Figure 5’s title**: Thank you for your suggestion on slight confusion on the perception field graph. We will revise the figure title to ``Ablation Metrics on Perception Fields (on Raw Waves)” which ensures that the table title specifies the feature type (raw waves) rather than the current caption for clarity. This should provide better readability to the audience with less background knowledge. \n\n2. **Experimentation on other LLMs:** We agree with the reviewer’s suggestion that further extending the discrete codex for large language models can be enlightening. The reason why we keep the main experiments using BART is that we want to control the decoder on the same scale compared to previous works. In that case, we can better illustrate that our improvement is from discrete coding but not from a more powerful decoder. \n\n   In fact, we did have applied a larger-scale ablation in the near past. We conducted further experiments by replacing the BART decoder with OPT and Llama V1. However, the performance improvement is not as large as we expected. Bridging brain activities with LLMs and AGI is an important area worth plenty of papers to explore. **For reasons of caution, we did not include** this experiment previously in the main paper. \n\n   However, we can report our previous experiments on these three models. Limited to our computing resources, we fine-tune the OPT-1.3B model and Llama-1 7B model with half-precision each for 3 epochs using PyTorch FSDP mode. The tokenized EEG waves are prompted into LLMs according to Mini-GPT4’s method [1] of dealing visual tokens. \n\n   | Source              | Decoder             | BLEU-1 | BLEU-3 | ROUGE-R | ROUGE-P | ROUGE-F |\n   | ------------------- | ------------------- | ------ | ------ | ------- | ------- | ------- |\n   | Word-level features | DeWave              | 41.35  | 13.92  | 28.82   | 33.71   | 30.69   |\n   | Word-level features | DeWave + OPT 1.3B   | 41.97  | 14.06  | 28.98   | 33.82   | 30.86   |\n   | Word-level features | DeWave + Llama-1 7B | 42.84  | 15.03  | 29.42   | 35.43   | 32.05   |\n   |                     |                     |        |        |         |         |         |\n   | Raw Waves           | DeWave              | 20.51  | 5.16   | 21.18   | 29.42   | 24.27   |\n   | Raw Waves           | DeWave + OPT 1.3B   | 21.31  | 5.84   | 22.09   | 29.94   | 25.42   |\n   | Raw Waves           | DeWave + Llama-1 7B | 22.05  | 6.03   | 22.45   | 30.01   | 26.08   |\n\n3. **Averaging over the EEG features:** Considering potential practical use cases in the future, using averaged EEG waves might not be very valuable. This is because, in real-world scenarios, the translation from brainwaves to text is more likely to be performed on individual human subjects in real time. If our understanding is right, averaging waves across multiple subjects requires pre-collecting EEG waves offline and performing benchmark tests. Given intuition closer to realistic usage, we trained on a mixed dataset and tested on different single human subjects to better reflect real-world applicability.\n\n4. **Different EEG feature types:** Yes, we considered this method previously. There are two reasons why we didn't use it. \n\n     - As the EEG-to-Text translation is conducted on long sentences, which is different from traditional classification tasks. The EEG feature fragments according to each word have large variation according to how many times the word appears in the sentence and how many times human subjects looks at that word when reading. In that case, concatenate features lead to higher computational consumption and instability in input features. \n\n\n     - We have drafted experiments in the earlier stage. According to our previous experiments, the performance of concatenate method is significantly lower than averaging word-level feature, which is used by [2] and our paper. \n\n       \n\n\n[1] Deyao Zhu et al,  **MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models**\n\n[2] Zhenghailong Wang, et al, **Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification** AAAI 2022\n\n"}, {"review_id": "0flNEPlSHB", "reviewer": "Reviewer_iEZr", "summary": "This paper introduces a new method, “DeWave”, for decoding text strings from EEG data recorded from subjects while reading. This method seems to differ from earlier ones in two important and useful ways: (1) it uses a discrete “codebook” to represent the EEG data, which helps to control noise in the extremely noisy setting of EEG, and (2) it is also applied to continuous, rather than word-segmented, EEG data, more closely mimicking a real-world situation. The quantitative results show a modest boost over earlier methods in the word-aligned setting, and decent performance in the novel continuous setting.", "questions": "* In section 3.2 under “Inference”, it would be very useful to state how temporal information is encoded, e.g. with a separate x for each word in the segmented condition, and for each 200 ms time period in the continuous condition. \n* In section 3.3 under “Word-Level EEG Features…”, the preprocessing is not well explained. This section might rely on the reader knowing earlier work, but I found it difficult to understand what the 840-dimensional vector actually comprises.\n* I found it very surprising and slightly worrying that the decoder seemed to be able to retrieve both exact proper nouns (“Kerouac” in Table 3, example 3) and approximate proper nouns (“Heroughs” for “Burroughs” in same). How is this possible? The contrastive semantic alignment loss, as I understand, cannot account for this, since it would operate on discrete tokens, unless those tokens also occurred in the training set. Is the discrete codex really capturing enough information about the constituent BPE tokens for those words to reconstruct them?", "limitations": "I think the most important limitation to note is that this method applies to EEG data collected while subjects actively read text, and cannot (currently) be applied to data collected while subjects merely think words. The authors do note this in their “Limitations” section, which is great.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper is the first I have seen that attempts to decode continuous language, i.e. without explicit word boundary markers, from EEG data. This is a big and important step! Decoding from word-aligned EEG snippets is clearly much easier but also much less interesting than this variant. I think pushing for this type of decoding is a great contribution of the current work.\n* The use of a vector quantization approach to represent EEG data also seems like an important step. (However, I have not read every paper in this area, so I can’t say with 100% confidence that this is the first work to apply this approach to the current problem.)\n* The method seems, overall, well-designed.", "weaknesses": "* The paper is at several points confusing and difficult to follow or understand. For example, in section 3.2 I found it impossible to understand the sentence “The codex contains fewer time-wise properties which could alleviate the order mismatch between event markers (eye fixations) and language outputs.” (I’ll expand on this with specific queries under “Questions”, below.)\n* Minor scientific issue: Eye movements were used to segment the EEG data into words in the segmented (traditional) condition, but were ignored for the continuous condition. However, eye movements are well known to cause HUGE transient deflections in EEG signals, to the point where much/most EEG data includes preprocessing that attempts to minimize their influence. I would imagine such preprocessing was done for this dataset (please correct me if I’m wrong), but even ideal eye movement removal cannot remove 100% of the related signals. So I think it could be important for the authors to consider (and perhaps even test) whether their continuous decoder is actually discovering latent eye movement signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your thorough review and constructive comments on our manuscript. We genuinely appreciate your insights and will address your concerns point by point below. \n\n1. **In section 3.2 under “Inference”, it would be very useful to state how temporal information is encoded.** \n\n   - For the word level EEG features, since there are eye-tracking markers suggesting EEG wave fragments related to each word, we directly get the embedding sequence by slicing it into sequence {$x_p$},  $p$={$1,2,..M$}, where $p$ th embedding corresponding to each word. \n   - For the raw wave setting, as shown in Fig. 3, we use a wide convolutional kernel to slide through a certain stride length to sequentially slice raw waves into continuous embedding sequences {$x_p$},  $p$={$1,2,..N$}, where $p$ th embedding corresponding to continuous perception fields (take 200ms as an example). \n\n   Then we directly use the position embedding in the original transformer paper [1] for both settings. Given position (order $p$), the position embedding in dimension $i$ is calculated by $PE_{(p,2i)}=sin(p/10000^{2i/d_{model}}), \\quad PE_{(p,2i+1)}=cos(p/10000^{2i/d_{model}})$. \n   \n   As the calculated position embedding is with the same shape as the continuous embedding sequence $\\{x_p\\}$, the position embedding is directly added with each EEG embedding accordingly. The position embedding is intrinsically maintained in the sequence. \n\n   **Further question: how does codex benefit the feature noise due to time-wise properties**\n\n   When a human subject reads the same word, the EEG waves could be different, given in different sentences context, in a different order, and at different times, which leads to feature variance time-wise. The advances of discrete encoding are that it could minimize noise variances and intuitively map encoded semantic (might with noise) embedding corresponding to the same word into a stable discrete codex value representation for the decoder. This will benefit the decoder with a more stable feature representation with less time-wise variances.\n\n   We will significantly improve writing in Section 3.2 for better readability. \n\n2. **Eye Movements and EEG Data:** Yes, you are right in stating pre-processing procedures the dataset used, preprocessing steps were indeed taken to minimize the influence of eye movements. Here [2], 9 EOG channels were used for artifact removal, and additional 14 channels lying mainly on the neck and face were discarded before data analysis. Fig. 1d in paper [2] shows a very good removal performance. \n\n   However, there may remain a small noisy component related to eye tracking as the artifact removal is not 100% accurate. Yet, according to both our correlation analysis experiments and experiments reported in EEG-Text [3]’s code, we have not observed a significant component related to eye movement. \n\n   Another thing is we use a quite deep transformer encoder to extract signals to semantic space, even if it has some remained eye-movement information, it will be jointly encoded into semantic embeddings. In that case, it could be hard to separate the impact from eye movement individually. However, we are keen to explore more properties of this point and keep updating it. \n\n3. **section 3.3 under “Word-Level EEG Features...” and how the feature dimension 840 comes from:** Thank you very much for the writing suggestion to improve readability to wider audiences who are not familiar with earlier work and the dataset. The EEG waves are collected from the Biosemi-128 system, where 9 EOG channels were used for artifact removal, and additional 14 channels lying mainly on the neck and face were discarded. That led to 105 channels for EEG signals. For word-level features, the eye-tracking marker is used to slice EEG wave fragments according to each word. Then, both [2, 3] and our paper averaged the power statistical feature on 8 frequency bands. These bands are 'theta\\_{1,2} (4-6 Hz, 6.5-8 Hz)', 'alpha\\_{1,2} (8.5-10 Hz, 10.5-13 Hz)', 'beta\\_{1,2} (13.5-18 Hz, 18.5-30 Hz)', 'gamma_{1,2} (30.5-40 Hz, 40-49.5 Hz)'. In that case, the feature shape after flattening and concatenating all statistical features would be $8\\times 105=840$. \n\n   We will enhance this section by providing a step-by-step description of our preprocessing pipeline, ensuring it is self-contained and does not necessitate prior knowledge of earlier works [2, 3].\n\n4. **Decoded example text:** \n\n   According to our experiments, the decoder has better prediction ability on nouns rather than verbs and adjectives. Our observation is that when human subject reading, it normally pays more attention to nouns (names) (which are likely to be more complicated, or even rarely seen) rather than common words. It leads to enriched input feature qualities for the model to learn. \n\n   Another point is the data is recorded when the human subject is reading the text. Considering the mentioned term “Burroughs and Kerouac” vs. “Heroughs and Kerouac”, it has similar pronunciation when reading on “-roughs” part. In that case, the learned distribution might be closer between these words (adjusted from word2vec). \n\n   Meanwhile, according to our statistical results on the ZuCo dataset, it contains 6828 total words and 300 sentences (1.0) and 390 sentences (2.0). Though the training and testing text corpus is different, these corpora are sampled from the same wiki paragraph. So it is true that these two names did appear in the training set. The current sampled example text is surprising, but we think this performance is totally under control and as we expected. \n\n\n\n[1] Vaswani, Ashi et al, **Attention is all you need** , NeurIPS 2017\n\n[2] Nora Hollenstein, et al, **ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation.** LREC 2020: 138-146\n\n[3] Zhenghailong Wang, et al, **Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification** AAAI 2022"}, {"review_id": "CdtOrQSVt1", "reviewer": "Reviewer_E1CL", "summary": "This paper proposes an EEG to text model that feeds into the raw EEG signals and predicts the corresponding words or long sentences. The model is optimized in two stages: (i) matching EEG to Text by contrastive learning with codex quantization; (ii) fintune the BART model for EEG embedding decoding to text. The model is evaluated on ZuCo 1.0 and 2.0 datasets.", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The EEG to text application sounds interesting and challenging. The proposed model architecture sounds reasonble by leveraging recent popular CLIP-style contrastive learning, signal quantization (from speech processing domain), and BART finetuning (LLM).\n2. The paperr is easy to read, and the related works are well organized. \n3. The performance on ZuCo datasets looks good.\n", "weaknesses": "1. The reviewer has concerns on the data volume for training the CLIP model (ZuCo dataset might not be large enough). It would be good if authors can comment/explain on this.\n2. It would be good to add more experiments on \"whether the discrete codex\" module is useful? What if we directly use the raw continuous signal? How to better learn the discreate codex (the codex grouping in [1] can potentially improve the model)?\n3. BART is a generally purpose pre-trained language model, which may not be suitable for this application. In the medical domain, [2] might be more suitable given the finetuned data might be limited.\n4. More datasets and baseline models should be added to systematically and rigorously demonstrate the supriority of the proposed model.\n\n[1] Baevski et al. Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.\n[2] Lee et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback and recognize the significance of the highlighted concerns. Below, we address the identified weaknesses point by point:\n\n1. **Data Volume Concerns for the CLIP Model**: The limited data scale problem has been a key challenge in the BCI area. The training data is limited compared to visual-language alignment CLIP addressing. This also leads to EEG-to-Text alignment remains unexplored compared to Vision-to-Text alignment. However, this work proves that introducing contrastive alignment on a limited data scale has a clear improvement and cross-modality alignment in the BCI area is feasible, which can be a good reference/encouragement for future works. Although the text corpus is limited, each sentence has EEG waves from 18/12 human subjects reading, this still leads to a large amount of EEG data. We utilize pre-trained language models as the decoder and pre-trained word2vec embedding as the alignment guidance for the EEG encoder, which alleviates the limited training data problem. \n\n2. **Utility of the Discrete Codex Module**: This is a good point. For comparison directly using raw continuous signal, we have already reported in raw waves part in Tab. 1 in the main paper. Since the EEG-to-Text [1] is only designed for word-level eeg features. We re-implement this method and directly train it on raw waves to realize this comparison. For a fair comparison, both methods are using the same pre-trained BART decoder. We list the performance reported in Table 1 in the main paper (between line 213 to line 243) here for your reference. \n\n   | Method                              | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE-R | ROUGE-P | ROUGE-F |\n   | ----------------------------------- | ------ | ------ | ------ | ------ | ------- | ------- | ------- |\n   | EEG-to-Text (directly on raw waves) | 13.07  | 5.78   | 2.55   | 1.10   | 15.22   | 18.08   | 16.36   |\n   | DeWave                              | 20.51  | 10.18  | 5.16   | 2.52   | 21.18   | 29.42   | 24.27   |\n\n   The performance suggests a clear improvement in using discrete codex over directly using continuous raw EEG waves. \n\n   Regarding the open question of grouping the discrete codex, we think given the current limited training data further splitting the codex into sub-groups might not have a positive effect. But it could be an interesting question in the future if there is sufficient data. It is also noted that, according to Fig. 5 of the main paper, simply increasing codex size will lead to a decrease in performance. In that case, we didn’t report these aspects in the main paper. \n\n3. **Choice of BART over Domain-specific Models**:  The choice of BART has three reasons: 1) as you mentioned, it is a general-purpose language model with normal language distributions. As ZuCo uses text corpus mostly from wiki pages in the general domain, not the medical domain, the pre-trained distribution of BART is closer to the ZuCo dataset. 2) BART pretraining process added noise tokens to the encoder when training the language decoder. This is more suitable for noisy encoded EEG embedding. 3) In order to control the same decoder model with previous works [1]. In that case, we could better illustrate the impact on the proposed discrete codex with contrastive alignment. \n\n4. **Possibility of Additional Datasets and Baselines**:  We very much like to have more data on brain signal-to-text translation. However, subject to the current situation as of the submission deadline, the ZuCo dataset is the only choice. We will actively track the new datasets. Also, we are recording our own data currently and will provide more details in our future work. \n\n   For baselines, the EEG-to-Text [1] baseline in our main paper is the only valid baseline currently. We've tried our best to adopt possible baselines from other domains for this task. We adopt Wav2Vec [2] from the speech recognition area and SCL from the brain sleeping stage recognition area and BENDR [3] from EEG self-supervised training area into translation tasks as our baselines. Yet, in this case, we think the current experiment is quite completely limited to existing works. \n\nThank the reviewer for the very good advice, we will actively track the baselines of brain decoding for the community in our future works.\n\nAgain, we thank the reviewer for their constructive feedback and hope our rebuttal addresses your concern. \n\n[1] Zhenghailong Wang, et al, **Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification** AAAI 2022.\n\n[2] Baevski, Alexei, et al. **Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.** NeurIPS 2020: 12449-12460.\n\n[3] Kostas, Demetres, et al. **BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data.** Frontiers in Human Neuroscience 15 (2021)."}]

Author rebuttal:
Dear chairs and reviewers, 

We express our profound gratitude for the comprehensive feedback and comments on our manuscript. This paper receives **Accept, Weakly Accept, Borderline Accept, and Weakly Accept** during the review period. We are excited about the consensus among the reviewers regarding the novelty and potential impact of our work. 

This paper introduces vector quantized representation learning and contrastive alignment between EEG waves and natural languages. The experiments of DeWave showcased state-of-the-art performances on word-level EEG-to-text translation. Additionally, DeWave stands as the pioneering effort to achieve language decoding directly from raw waves. This advancement is a significant step towards real-world applications, which alleviate the dependence on pre-known eye-tracking markers to segment EEG waves accordingly. Meanwhile, the discrete codex encoding introduced for EEG waves also provided a new choice for follow-up works that require EEG wave vectorization. 

Building upon the above, we have acted on the reviewers' feedback to enhance the clarity and thoroughness of our paper. Please refer to the rebuttal for each specific review for step-by-step clarification on potential unclear technology inquiries below.  Additionally, we have undertaken a thorough refinement of the manuscript to rectify typos and augment its readability for a broader audience. We are committed to ensuring the manuscript's excellence and its readiness for publication.

The related code provided in the review area will be refined and be public after the anonymous phase. 

To conclude, we earnestly believe that our approach can make a good contribution to the EEG decoding realm. The feedback has encouraged us to perfect our manuscript to ensure it can contribute to the community as a solid publication. 


Best wishes,

Paper Authors





Meta-review: This paper present an ambitious BCI project that is innovative by (1) using EEG, a much more noisy yet much more feasible tool for everyday BCI than what is currently used in papers (e.g., ecog, fmri) and (2) by decoding connected input (read text) through the use of eye movements instead of doing the easier task of cutting the EEG signal into words prior to decoding. These steps make the current work very useful for future BCI technology. The reviewers have a good assessment of the paper after the responses, and the authors are requested to include the required clarifications and new experiments in their manuscript or supplementary.
Decision: Accept (spotlight)