SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Raw Waves: DeWave represents an unprecedented endeavor in translating raw EEG waves directly, obviating the need for event markers. This is achieved through the application of a learned Wave2Vec model, which converts waves into discrete codex tokens. When benchmarked against
the baseline method - which merely slices the wave according to a time window - DeWave exhibits marked improvement. It attains BLEU-1, 2, 3, 4 scores of 20.51, 10.18, 5.16, and 2.56, respectively, surpassing the baseline by margins of 7.44 (+56.92%), 4.44 (+76.1%), 2.61 (+102.35%), and 1.42 (+129.09%). This underscores the superiority of a learned discrete embedding representation over the rudimentary time-window baseline. In comparison with cutting-edge self-supervised learning (SSL) methods, DeWave consistently outperforms them, including contrastive learning (CL)-based methods such as SCL [10], the original Wave2Vec [2], and BENDR [11].
4.4 Cross-Subject Performance
Cross-subject performance is of vital importance for practical usage. To further illustrate the performance variance on different subjects, we train the model by only using the data from subject YAG and test the metrics on all other subjects. The results are illustrated in Figure 4, where the radar chart denotes the performance is stable across different subjects. Please refer to Appendix G for more detailed results.
Figure 4: The cross-subjects performance on ZuCo dataset.
4.5 Generated Samples
In Table 2, we display visualized examples of text generated from unseen EEG signals. Despite the challenge of thought translation and limited prior research, our model yields meaningful results, aligning keywords and forming similar sentence structures, although it may not yet match traditional language translation tasks.
The model is more adept at matching verbs than nouns. For instance, implies" vs. implies", the author who wrote it" vs. the man who wrote it" both effectively convey the intended sentiment of the sentence. However, when it comes to nouns, we observe a tendency towards synonymous pairs rather than precise translations, such as the man" vs. the author", Burroughs" vs. Heroughs", edition" vs. version". Our analysis suggests two potential causes for this. First, when the brain processes these words, semantically similar words might produce similar brain wave patterns. Given the inherent noise in brain waves, the codex might group these features under the same value. Second, the volume of EEG-to-Text pairs available for training is significantly smaller than that for traditional language translation. Hence, some degree of error in translating unseen nouns or sentences is to be expected.
Translation on raw EEG waves is naturally harder than word-level translation, as it lacks eye fixation to suggest the relationship between the period of waves and the word target. Table 2 meet our expectation that the results on raw EEG waves are not as good as those on word-level features, especially on the real semantic meaning of the sentence (sample (2) and sample (5) have the same target for comparison). However, the translation still could output the correct translation of certain words, such as "much of" vs. "much of", "individual" vs. "individual", and "more complicated story" vs. "more exciting thing". Although EEG-to-text is a hard topic, DeWave suggests the feasibility of translation improvements.
Discrete Codex DeWave encodes EEG waves into a discrete codex, aiming for a language model-friendly representation. We evaluated performance against varying codex sizes (1024 to 8192) to ascertain if larger sizes yield better results. As depicted in Figure 5, we found no strong correlation between codex size and model performance. A codex size of 2048 yielded the highest BLEU score average, and while the ROUGE score slightly improved with larger sizes, there was no clear evidence that increasing codex size consistently enhanced performance.
Table 2: Translation results on the unseen EEG waves, where **bold** denotes a correct match between ground truth and our prediction. **Underline** denotes a fuzzy match with similar semantic meanings. For a fair comparison, these results keep the same teacher-forcing evaluation setting as EEG-to-Text [48]. This means the decoding process eliminates accumulated errors and just predicts the current token with the GT token from the last step.
Decoding Results with Eye Fixation Assistance
(1)	Ground Truth: Everything its title implies, a standard-issue crime drama spat out from the Tinseltown assembly line...
Prediction: is own implies, including great of issue, novel of the beginningseltowns....
(2)	Ground Truth: “The Kid Stays in the Picture” is a great story, terrifically told by the man who wrote it but this Cliff Notes edition is a cheat.
Prediction: The film "says in the Game" is a film about but movie was written, the author who wrote it. also its version is a cheat.
(3)	Ground Truth: During Kerouac's time at Columbia University, Burroughs and Kerouac got into trouble with the law for failing to report a murder; this incident formed the basis of a mystery novel ...
Prediction: Keouac's time at the, Heroughs and Kerouac were along a for the police for their to pay the murder. they led led the basis of the lawsuit ...
Decoding Results with Raw Waves
(4)	Ground Truth: Every individual will see the movie through the prism of his or her own beliefs and prejudices ...
Prediction: Everyday individual is their results. their eyes of his or her own personal desires. and ...
(5)	Ground Truth: “The Kid Stays in the Picture” is a great story, terrifically told by the man who wrote it but this Cliff Notes edition is a cheat.
Model Output: The Price'says in the Middle. and a common deal. and for good. this moment. made it is still little.
(6)	Ground Truth: much of this well-acted but dangerously slow thriller feels like a preamble to a bigger, more complicated story,
Model Output: much of this is-being but not over- like it disaster-ble to a new and more exciting thing,
Raw EEG wave performance fluctuates noticeably with codex size. Performance improves when increasing the codex size from 1024 to 2048, but any larger size reduces performance. This variation may result from the training formation; word-level EEG data, selected by eye fixations, contains less noise than raw waves. We hypothesize that our current training data may be insufficient for larger codex sizes. Additionally, our experiments indicate that the latent dimension of the codex doesn't significantly affect performance.
Figure 5: Ablation study on different codex sizes and perception fields (raw waves).
Perception Time Window We also conduct the ablation study on the model structure for the wave2vec model illustrated in Figure 5. As the model utilizes a multi-layer CNN model to slide through the raw waves, the model compresses the waves for perception. The compress ratio decided how large the perception field is for each extracted embedding feature. As described in Section 3.3, the model utilized a perception field of  $200\mathrm{ms}$  with an overlap of  $100\mathrm{ms}$ . We conduct an ablation study of different perception fields and report it in Figure 5. Where it is observed that the model performance is significantly lower when the perception field is smaller than  $80\mathrm{ms}$  or larger than  $240\mathrm{ms}$ . The model could achieve similar results in the perception field  $120\mathrm{ms}$  to  $240\mathrm{ms}$ . The model reaches a small peak around  $200\mathrm{ms}$  to  $240\mathrm{ms}$ . We think this phenomenon is rational since the normal reading speed for humans is around 160-400 words per minute (reading speed may vary from different
material and human subjects). In other words, the reading period for each word is 150-375ms on average, which roughly meets our observation between 200ms-240ms.
Self-Supervision Initialization Theoretically, we could pre-train the codex by introducing a decoder and calculate the reconstruction loss with the original input for both word-level EEG and raw EEG waves. We prefix other parameters as our best setting with codex size 2048 and compare the impact in Table 3.
Table 3: Ablation on self-supervised pre-trained codex weights.
Pretrain	BLEU (%)	ROUGE-1 (%)
N=1	N=2	N=3	R	P	F
Word-level features	✓	41.35	24.15	13.92	28.82	33.71	30.69
×	40.71	22.94	12.40	28.01	34.08	30.63
Raw waves	✓	20.51	10.18	5.16	21.18	29.42	24.27
×	16.58	7.78	3.68	17.86	19.84	18.33
For word-level EEG features, the impact of pre-train is mostly on BLEU scores while the ROUGE score does not have much variance. Without pre-train, the BLEU- $\{1,2,3\}$  respectively drop by 0.64, 1.21, and 1.52. For direct translation on raw waves, the impact is significantly larger. Without self-supervised initialization, the BLEU- $\{1,2,3\}$  respectively drop by  $3.93 \ (\downarrow 19.16\%)$ ,  $2.40 \ (\downarrow 23.57\%)$ , and  $1.48 \ (\downarrow 28.68\%)$ . Similar observation also appears on ROUGE scores. This phenomenon is rational since raw wave decoding requires the model to pick useful features without any help from eye fixations. The self-supervised initialization could help the model form a preliminary ability to extract time-wise or channel-wise features from raw waves.
Despite DeWave's enhancements in EEG-to-Text translation using a discrete codex and raw wave encoding, its accuracy remains far from real-life scenarios compared to traditional language-to-language translations. Also, to keep a fair comparison with EEG-to-Text, this paper uses a teacher-forcing setting in evaluation. This setting eliminates accumulation error and turns the sequence decoding task into a word-level classification task given the ground truth token from the previous step. This setting is relatively easier yet we think it is still valuable as it could suggest feature extraction quality while keeping a fair comparison.
Additionally, the experiments in this paper are restricted to public neural reading data, not fully aligning with the "silent speech" concept of direct thought translation from human brains. Instead, the current ZuCo dataset is collected by giving people reading stimuli. This paper focuses on introducing Wav2Vec formation of feature extraction on raw EEG waves and introduces discrete codex as learnable representations for the EEG-to-Text translation domain. One scientific problem in this domain is a better way of doing the "silent speech" task, which we are exploring as on-going research.
This paper presents DeWave, a framework for the recently proposed open-vocabulary EEG-to-Text translation task [48], introducing the concept of discrete codex encoding. This approach brings enhancement in corpus text relevancy metrics, such as BLEU and ROUGE. DeWave also expands the task to decode raw EEG waves without the assistance of eye fixation markers. Despite these advancements, the quality of brain decoding results remains substantially low and remains teacher forcing setting for fair comparison. The translation of thoughts directly from the brain is a valuable yet challenging endeavor that warrants significant continued efforts. In our ongoing work, we are exploring more reasonable settings that remove teacher forcing for both training and testing. We will also include the "neural-feedback" mechanism in this EEG-to-Text research to enhance the scientific value of this domain.
This work was supported in part by the Australian Research Council (ARC) under discovery grants DP210101093 and DP220100803, and the GrapheneX-UTS Human-Centric AI Centre sponsored by GrapheneX (2023-2031).