MAD: Multi-Alignment MEG-to-Text Decoding

## Abstract

Abstract Deciphering language from brain activity is a crucial task in brain-computer interface (BCI) research. Non-invasive cerebral signaling techniques including electroencephalography (EEG) and magnetoencephalography (MEG) are becoming increasingly popular due to their safety and practicality, avoiding invasive electrode implantation. However, current works under-investigated three points: 1) a predominant focus on EEG with limited exploration of MEG, which provides superior signal quality; 2) poor performance on unseen text, indicating the need for models that can better generalize to diverse linguistic contexts; 3) insufficient integration of information from other modalities, which could potentially constrain our capacity to comprehensively understand the intricate dynamics of brain activity. This study presents a novel approach for translating MEG signals into text using a speech-decoding framework with multiple alignments. Our method is the first to introduce an end-to-end multi-alignment framework for totally unseen text generation directly from MEG signals. We achieve an impressive BLEU-1 score on the GWilliams dataset, significantly outperforming the baseline from 5.49 to 10.44 on the BLEU-1 metric. This improvement demonstrates the advancement of our model towards real-world applications and underscores its potential in advancing BCI research. Code is available at https://github.com/NeuSpeech/MAD-MEG2text .

## 1 Introduction

Decoding brain to language has emerged as a rapidly developing area of neurotechnology, offering semantic communication and control for general Brain-Computer-Interface (BCI) tasks.
This region has garnered growing focus as it may profoundly impact individuals with verbal and movement disabilities resulting from conditions such as severe spinal cord trauma or end-stage amyotrophic lateral sclerosis (ALS).
Moreover, the scope of brain-to-text technology extends to pioneer novel human-machine interfaces, allowing seamless control of prosthetic limbs, software, and virtual environments, shifting the paradigm of interaction for both able-bodied individuals and those with disabilities, and re-defining what is achievable in both everyday life and professional spheres.

Under this scope, various previous works have explored this area in multiple ways.
Pioneer researchers first verify this idea by using invasive signals such as Electrocorticography (ECoG) [, , , ].
Recently, these invasive methods [, ] concentrate on decoding speech, phonemes or letter from ECoG signals and have achieved remarkably high accuracy using limited word sets for real-time brain-to-text translation.
However, these invasive-signal-based approaches pose significant medical risks and challenges for long-term use.

Non-invasive techniques, therefore, present a safer and more sustainable alternative, albeit with their own set of challenges.
Wang et al. [] showcased a method for translating EEG signals into text with an extensive lexicon, utilizing language models that had been pre-trained on EEG data features at word-level.
Duan et al. [] progressed this methodology by interpreting raw EEG signals directly, devoid of reliance on temporal indicators, but their models still relied heavily on teacher forcing for evaluation, limiting their ability to generate meaningful sentences autonomously in real-life scenarios.
At the same time, although Magnetoencephalography (MEG) provides better signal quality, previous works [, , ] on MEG have primarily focused on decoding limited classes or short phrases from MEG signals, showing limited success in generating whole sentences and complete semantic segments.

Furthermore, as pointed out by Jo et al. [], all previous works in EEG-to-Text translation following Wang’s method [] meets the “decoder dominated" problem.
It means that given a strong decoder and noisy EEG input,
these models are more likely to memorize the text distribution corresponding to certain statistical features rather than mapping EEG to semantic texts. Thus, these models have similar performances even when we replace EEG input with random noise. Besides, due to the nature of limited data and the non-understandability of the neural signal, it is difficult to train and evaluate the model. Yang et al. [] proposed NeuSpeech model on MEG to text task, however, their model is evaluated on the text that is seen in the training set, which does not meet the need for open-vocabulary translation.
Defossez et al. [] highlighted the potential to decode speech perception from MEG signals, where they matched MEG signals with corresponding speech segments. However, their approach was limited to classification tasks and could not generate sentences directly from MEG signals. This underscores a significant gap in the current state of MEG-based brain-to-text decoding.

In this paper, our motivation is to establish an end-to-end framework for open-vocabulary MEG-to-Text translation capable of processing unseen text without relying on biomarkers, while ensuring that the encoder captures brain dynamics effectively.
We propose Multi-Alignment MEG-to-Text Decoding (MAD) with the aim of guiding the brain encoders towards learning salient representations. To achieve this, we incorporate audio as an auxiliary modality to facilitate alignment.
Here, we make a bold assumption that directly formatting noise brain signals into discrete text is difficult due to limited data.
Hence, we utilize brain module [] and an extra whisper model [] to align brain representation in three aspects as shown in Figure 1, the Mel spectrogram, hidden state, and text.
1) We first align the Brain module with audio in the Mel spectrogram feature space to learn low-level features, such as acoustic features.
2) Additionally, we align the hidden state output from both the whisper encoder and the brain module in latent space, enhancing the model’s ability to extract high-level semantic features.
3) Lastly, we align the text representation from both streams within the framework.

Our objective in incorporating textual data is to assess whether it can furnish supplementary contextual cues that enhance the correspondence between neural activity and the resulting linguistic output.

Comprehensive experiments are conducted by utilizing non-invasive public MEG data from GWilliams [] dataset, which captured MEG signals during a speech listening task.
Remarkably, MAD is capable of generalizing to unseen text. Performance is evaluated using translation text relevancy metrics [, ]. On raw MEG waves, MAD achieves 10.44 BLEU-1 on GWilliams without teacher-forcing evaluation on entirely unseen text which largely exceeds the current SOTA performance.
This paper also provides insights though numerous ablation studies to help people understand the impact of each component on aligning the MEG signal with texts.
The contributions of this research could be summarized as follows:

- •
MAD presents an end-to-end neural network design for the direct conversion of MEG signals into text in open-vocabulary, obviating the dependence on markers, teacher forcing, or pre-training, representing the initial implementation of translating raw MEG waves into text for unseen content.
- •
We are the first to investigate various alignments and demonstrate the benefits of aligning with speech modality rather than text modality in the MEG-to-text transcription task, offering significant insights for network improvement.
- •
Our extensive experimentation and thorough analysis of the proposed model showcase its effectiveness and highlight its superiority over existing methods in terms of translation accuracy, efficiency ,and reliability.

## 2 Related Works

The discipline of converting brain signals into textual output has undergone considerable development in the contemporary era. In 2019, Anumanchipalli et al. [] introduced a pioneering model capable of translating ECoG patterns into the articulatory movements necessary for speech production, subsequently generating acoustic properties such as MFCCs, leading to the production of intelligible speech. This landmark study ignited further exploration within the field. In the subsequent year, Wang et al. [] leveraged the capabilities of generative adversarial networks (GANs) to decipher ECoG data and synthesize speech. The year following, Willett et al. [] engineered a system that utilized a recurrent neural network (RNN) alongside a probabilistic language model to decode letters from neural activity during the act of handwriting. Most recently, Metzger et al. [] constructed a sequence of processes that converted ECoG signals into textual information using an RNN, enhancing the results with the GPT-2 language model.

Within the domain of open-vocabulary interpretation, Metzger et al. [] unveiled an RNN architecture capable of real-time decoding of speech, text, sentiment, and facial expressions from ECoG data. Simultaneously, Willett et al. [] managed to interpret text directly from neural activity. Liu et al. [] introduced a tripartite model designed to decode logo-syllabic languages, such as Chinese, by transforming ECoG signals into Chinese pinyin inclusive of tones and syllables, followed by speech synthesis. In a related development, Feng et al. [] achieved text interpretation from SEEG recordings. It is essential to highlight that these functional systems are predominantly reliant on invasive neural recordings.

In the domain of non-invasive neural recording, Meta unveiled a brain-to-speech system that leverages contrastive learning with MEG and EEG data []. While this system is proficient in categorizing a constrained set of sentences, it is not conducive to open-vocabulary textual interpretation. Ghazaryan et al. [] explored the decoding of a restricted vocabulary from MEG responses. Wang et al. [] crafted a mechanism for translating EEG features at the word level into text, employing a pre-trained BART model []. Subsequent investigations, including Dewave [], adopted the methodology established by Wang et al. [], proposing a schema that incorporates wave2vec [] and discrete codex for robust representations, which are subsequently funneled into a BART [] model for text synthesis. These approaches, however, are dependent on teacher-forcing and disregard the necessity of comparing results with noise-injected inputs, potentially resulting in an inflated assessment of system efficacy. Recent scholarship [] has revealed the limitations of these methods.

Yang et al. [] proposed an end-to-end paradigm for converting MEG signals to text, demonstrating high performance when training and evaluation sets were fully overlapped. However, it does not show good performance on unseen text.
Our approach diverges from these methods by employing transfer learning with assistance of extra modality(Mel spectrogram) to align the model through multiple stages with low-level and high-level features of the ground truth. This enables our model to learn more effectively and generalize better to unseen text.

## 3 Method

Figure: Figure 1: (a) Overview of model architecture. We added alignments on the Mel spectrogram, the hidden states, and the text. There are three types of alignment, which are either based on our physics world (text and speech) or a largely pre-trained model. $M_{1}$, $M_{2}$ is predicted and ground truth Mel spectrogram, $E_{1}$, $E_{2}$ is the hidden state of meg-input and speech-input encoder respectively. $T_{1}$, and $T_{2}$ are predicted and ground truth text respectively. (b) Details about the brain module.
Refer to caption: x1.png

### 3.1 Task Definition

Given a sequence of raw segment-level MEG signals $\varepsilon$, the goal is to decode the associated open-vocabulary text tokens $T$. This task also incorporates additional information in the form of speech $\Xi$. The MEG-Speech-Text pairs $\left\langle\varepsilon,\Xi,T\right\rangle$ are collected during speech perception. Our approach focuses on decoding $T$ using only the raw MEG signal $\varepsilon$, with the support of $\Xi$. MAD represents the first attempt at tackling this MEG to unseen text translation challenge.

### 3.2 Model

Figure. 1 shows the overview of our work. Our model uses some transfer learning techniques to facilitate better performance on unseen text. The encoder and decoder models are from the Whisper model [], a transformer-based encoder-decoder architecture tailored for robust speech recognition in challenging environments such as noisy conditions. The brain module [] first takes the MEG signal $\varepsilon$ of $C$ channels in the Spatial Attention layer, it adds position embedding of physical sensors to the MEG, then Initial Conv maps the MEG channel number to hidden model dimensions. After that, the Subject Layer takes the MEG feature and subject index and applies subject embedding on the MEG feature. Next, the MEG feature is input into the residual-designed module which is repeated 5 times. Finally, after Point Conv of which kernel size is 1, it maps to the Mel spectrogram $M_{1}$.

$L_{m}$ is the loss to align the Mel spectrogram, which is Clip loss [] in this situation. Then we want to make sure the encoder model can learn high-level features, so we designed to align the encoder output with $L_{e}$, which is Maximum Mean Discrepancy (MMD) loss []. We used LoRA [] module to train our architecture for saving memory. Last but not least, we have the cross entropy loss $L_{t}$ for predicted text and ground truth text. The overall loss $L$ is below:

$$ $\displaystyle L=\lambda_{m}\cdot L_{m}+\lambda_{e}\cdot L_{e}+\lambda_{t}\cdot L _{t}$ (1) $$

Recall the clip loss [] function, it takes two feature representations from each modality. These features are then used to calculate the similarity scores between the representations of the image and text modalities. The Clip loss function aims to minimize the distance between matching pairs of image and text representations while maximizing the distance between non-matching pairs. This approach allows the CLIP model to learn a joint embedding space where semantically similar image-text pairs are close together, enabling tasks like zero-shot image classification and text-based image retrieval. Here the clip loss is applied on the Mel spectrogram, which is of 3 dimensions, so we flattened the batch size and time length dimensions as the first dimension, and then the loss is calculated as follows:

Figure: Algorithm 1 CLIP-like Loss Calculation

The MMD loss (Maximum Mean Discrepancy loss) is a measure of the discrepancy between two probability distributions. It is commonly used in domain adaptation and generative modeling to encourage the distributions of source and target data to be similar. If we flatten the hidden state $E$ of the batch size $n$, time dimension $t_{d}$ and feature dimension $d_{e}$, it will run out of memory if we input full length into the model, so we randomly select features time wise $t_{r}$, therefore the selected features is $E_{r}$ shape is $[n,t_{r},d_{e}]$ The formula for the MMD loss is:

$$ $\displaystyle\operatorname{MMD}^{2}(E_{1},E_{2})=\frac{1}{n}\left\|\sum_{i=1}^ {n}\phi(E_{1r}(i))-\sum_{i=1}^{n}\phi(E_{2r}(i))\right\|_{\mathcal{H}}^{2}$ (2) $$

For an Automatic Speech Recognition (ASR) system, the cross-entropy loss is commonly used as a loss function to train the model. The basic idea is similar to the general cross-entropy loss but adapted for the ASR context where the inputs are speech features and the outputs are text transcriptions. The cross-entropy loss in the context of ASR can be defined as follows:

$$ CrossEntropyLoss $\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{T}\sum_{c=1}^{C}T_{1,i,t,c }\log(T_{2,i,t,c})$ (3) $$

### 3.3 Evaluation

We evaluate transcribing performance on GWilliams dataset [] using NLP metrics, BLEU [] is used to evaluate the accuracy of machine-translated text, ROUGE-1-F [] is to measure the quality of automatic summarization, BertScore [] is a measurement of semantic similarity, CER [] is used to evaluate the accuracy of speech recognition and self-BLEU [] is used to assess the diversity of generated text.

- 1.
BLEU is used to evaluate the accuracy of machine-translated text.
- 2.
ROUGE-1-F is to measure the quality of automatic summarization.
- 3.
BertScore is a measurement of semantic similarity.
- 4.
CER is used to evaluate the accuracy of speech recognition.
- 5.
Self-BLEU is used to assess the diversity of generated text.

## 4 Experiments

### 4.1 Dataset

The GWilliams dataset [] is a magnetoencephalography (MEG) dataset designed for assessing natural speech comprehension. It features authentic MEG recordings from 27 participants proficient in English. These participants engaged in two separate sessions, each involving two hours of listening to four stories, which are “cable spool fort",“easy money",“lw1",“the black willow". To get a fair evaluation, we split our dataset directly on stories, we test on “cable spool fort", validate on “lw1" and train on other stories. Details are in Table. 1. For more details about the dataset, please refer to Supp. A.

**Table 1: Details about the dataset splits, we ensured the three splits are totally separated. Unique sentences means the sentences that are different with other sentences, same meaning for unique words. There is no overlap sentence between train and test set. 371(46%) means 371 words in test set is also in train set, accounting for 46 percentage.**
| Split | Segments | Unique sentences | Words | Unique words | Overlap sentence | Overlap words |
| --- | --- | --- | --- | --- | --- | --- |
| train | 133966 | 13266 | 150497 | 2776 | - | - |
| validation | 14896 | 1387 | 156027 | 478 | - | - |
| test | 31115 | 3151 | 355654 | 805 | 0 | 371(46%) |

For preprocessing, we used first band pass filter the MEG signal $\varepsilon$ between 1 Hz and 40 Hz, then it is resampled to 100Hz to reduce computing. We ensure that we separated training, evaluation, testing set totally since we used one story for testing, another story for evaluation, last two ones for training. We extract 4-second windows from the MEG-speech-text pairs, sliding every second and randomly shifting the window by ±0.5 seconds to generate samples. Speech $\Xi$ is then transformed to Mel $M$ with window length of 400, hop length of 160, which is the original configuration in Whisper model [], since the setted speech sampling rate is 16kHz, after conversion, $M$ is of shape [400, 80] time and feature wise for 4 second speech, then it is matched with $\varepsilon$ of time length 400.

### 4.2 Implementation details

All models were trained using Nvidia 4090 (24GB) GPUs. Training was conducted with a learning rate of 3e-4 and a batch size of 32 over 5 epochs, selecting the best-performing model based on evaluation loss. AdamW was employed as the optimizer across all models. Each experiment takes about 18 hours on signal GPU with 8 workers to finish. Lambda value in all experiment on MAD model set as follows: $\lambda_{m}$ = 1, $\lambda_{e}$ = 0.01, $\lambda_{t}$ = 1.

### 4.3 Evaluation Metrics

**Table 2: Comparison with other models. Lo is LoRA, B is brain module. Bert here means Bertscore. Results is obtained without teacher forcing in evaluation. Here, Tr stands for trainable modules. B-1 stands for BLEU-1. R-1 stands for ROUGE-1-F. SB stands for Self-BLEU. RS means randomly selecting sentences from test set as predictions. As we can see, only MAD is much higher than RS on BLEU-1 score.**
| Modality | Method | Tr | Loss | B-1(%)$\uparrow$ | R-1 (%)$\uparrow$ | Bert(%)$\uparrow$ | CER(%)$\downarrow$ | SB(%)$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| - | RS | - | - | 5.86 | 7.20 | 83.73 | 87.30 | 96.12 |
| MEG | NeuSpeech [] | Lo | $L_{t}$ | 5.49 | 8.43 | 83.98 | 77.02 | 99.7 |
| MEG | Wav2vec2CTC [] | B | $L_{m}$ | 0.55 | 1.44 | 76.02 | 152.23 | 92.67 |
| MEG | MAD | B | $L_{m}+L_{e}$ | 10.44 | 6.93 | 83.39 | 89.82 | 85.66 |
| Noise | MAD | B | $L_{m}+L_{e}$ | 3.87 | 3.16 | 83.20 | 126.95 | 87.54 |
| MEG | MAD w/tf | B | $L_{m}+L_{e}$ | 12.93 | 18.28 | 82.87 | 74.31 | 83.35 |
| Noise | MAD w/tf | B | $L_{m}+L_{e}$ | 0.19 | 6.68 | 59.92 | 87.57 | 68.63 |

The performance comparison of our proposed MAD model with other state-of-the-art models is summarized in Table 2. The table highlights various configurations and the corresponding evaluation metrics: BLEU-1, ROUGE-1, BertScore, CER and self-BLEU. Each model’s performance is evaluated on MEG data, with results illustrating the impact of different loss functions and modules on decoding accuracy.

We compare the performance of our proposed model, MAD, against existing state-of-the-art methods, NeuSpeech [] and Wav2vec2CTC [], for decoding MEG signals into text. The performance metrics used for evaluation include BLEU-1, ROUGE-1-F, Bertscore, and Character Error Rate (CER). The results are summarized in Table 2. We find out BLEU-1 seems to be the most effective measurement in this situation.

NeuSpeech [] is a encoder-decoder framework model used for MEG, utilizing the Low-Rank Adaptation (LoRA) method with a text-based loss ($L_{t}$), achieves best scores on ROUGE-1-F, BertScore, and CER. However, the self-bleu score is almost 100%, which means the generation always repeat same thing. Besides, the BLEU-1 score is lower than RS, which means these three metrics are not reliable, which is further discussed in Supp. B.

Wav2vec2CTC []: The original model predicts the output of the Wav2vec2 [] encoder with brain module. We add the pretrained language model head in the Wav2vec2CTC [] model as another baseline. This model shows significantly lower performance across all metrics, which is not effective.

Our MAD model, which integrates the brain module with a combined loss ($L_{m}+L_{e}$), demonstrates superior performance with a BLEU-1 score of 10.44% which is about 5 points higher than NeuSpeech [] and RS. Besides, we compared the performance of our model when it receives pure Gaussian noise which is the shape of the MEG signal to show that our model is generating text based on MEG signal. For noise input, MAD’s performance BLEU-1 dropped to 3.87%, indicating that MAD model has learned from the MEG signal rather than just noise. Additionally, we evaluated MAD with teacher forcing. When teacher forcing was applied (MAD w/tf), the model’s performance significantly improved, achieving a BLEU-1 score of 12.93% and a ROUGE-1-F score of 18.28%, confirming the effectiveness of teacher forcing in enhancing model performance. Similarly, the BLEU-1 score for noise w/tf is low too (0.19%), further indicating our model can distinguish noise and MEG. In addition, our model has low Self-BLEU which means our model is generate diverse sentences according to MEG signal rather than simply repeating.

Overall, our MAD model achieved state-of-the-art (SOTA) performance for MEG-to-text decoding compared to previous SOTA models, demonstrating significant progress in MEG-to-text translation. Additionally, we performed a fair comparison with noise and RS, which served as two error bars to validate the robustness and reliability of our model’s performance. Furthermore, the self-BLEU scores indicated the diversity of our model’s generated text, demonstrating its ability to truly learn and generalize from the data. Next section, we will show the generated sample along with the Mel spectrogram to further show the effectiveness of our MAD model.

### 4.4 Generated Samples

#### 4.4.1 Text

**Table 3: Transcription results. These are some results obtained with teacher forcing evaluation. Bold for exact matched words, underline for similar semantic words.**
| Decoding Results on GWilliams [] |
| --- |
| Ground Truth: corner of his eyes two forts stood on the playground and a hot |
| Prediction: the own of his in the ground for and the few spot hot |
| Ground Truth: of the top of the black fort like a gold headed monster tucker |
| Prediction: of the top hole a giant medal to is |
| Ground Truth: he knew it wasn’t going to be as relax easy as just pretending he was too |
| Prediction: he wast a to be a bad as as as it a to was a lazy |

We showed the text result in table 3.
It presents the transcription results obtained using the teacher forcing evaluation method.
The transcription results indicate that while the model can generate segments of text that partially match the ground truth, there are significant gaps in overall accuracy and coherence. Specifically, for the first example, the ground truth is “corner of his eyes two forts stood on the playground and a hot" and the model’s prediction is “the own of his in the ground for and the few spot hot". Although the model captures some keywords like “ground" and “hot," the overall sentence diverges significantly from the ground truth, exhibiting repetition and grammatical errors. This outcome highlights the model’s struggle with complex sentence structures and semantic relationships.

In the second example, the ground truth is “of the top of the black fort like a gold headed monster tucker," while the model predicts “of the top hole a giant medal to is." Here, the model successfully identifies “of the top," and the subsequent key word matches the ground truth in semantics, particularly "medal" which is semantically similar to “gold" in the ground truth sentence. Though it ignores the “monster tucker", this suggests that, despite the MAD model’s failure in maintaining coherence and context understanding between words, it can yield some keywords which are semantically similar to the keywords in the original sentence.

For the third example, the ground truth is “he knew it wasn’t going to be as relax easy as just pretending he was too," whereas the model predicts “he wast a to be a bad as as as it a to was a lazy." Although the initial word “he" matches the ground truth, the following prediction includes repeated words and grammatical errors, making it difficult to form a meaningful sentence.
However, we should notice that “lazy" may be similar to “relax" in meaning.

For these three texts, we can observe that MAD can generate semantically similar words though they may not be coherent. Besides, the noise inputs generated only blanks. This demonstrates that our MAD model can capture the semantic meaning in the MEG signals, rather than only relying on the decoder.

Overall, these results reflect the inherent difficulties in directly decoding MEG signals into natural language text. While the model demonstrates some capability in recognizing individual words, there is substantial room for improvement in generating coherent and accurate sentences. Particularly, the model struggles with complex grammatical structures and longer sentences. These findings underscore the necessity of further optimizing the decoding model, especially in enhancing contextual awareness and semantic coherence. Future work should focus on improving the model’s ability to understand context and the relationships between words to enhance the overall accuracy and readability of the transcriptions.

#### 4.4.2 Mel spectrogram

More than text, we showed the Mel spectrogram in Figure 2. It presents the Mel spectrogram of the two sample sentences in the test set. In this context, it is employed to compare the predicted audio signal generated by the model with the actual ground truth audio signal in the form of Mel spectrogram.

Upon examining the spectrograms of two samples, several observations can be made regarding the model’s capabilities and performance. 1) There is a general similarity between prediction and ground truth in the overall structure, 2) the model learns some fine-grained details such as temporal variations in the low-frequency regions which have bigger energy than the high-frequency region, 3) the model can predict the speech signal’s temporal blanks, proving it understands the MEG features associated with the absence of speech. However, significant discrepancies are apparent. While the ground truth spectrogram displays a more complex and detailed pattern with distinct frequency bands and variations over time, the predicted spectrogram seems less detailed and exhibits more uniform and repetitive patterns.

These discrepancies highlight the current limitations of the model in producing high-quality, accurate, natural audio signals from MEG data. Future work can introduce pre-trained generative models in speech modality to improve the model’s ability to learn and represent these fine-grained details, which is important for accurate speech recognition.

Figure: Figure 2: Two sample examples from the test set. Predictions refer to Mel spectrograms generated by the brain module. Ground truth refers to Mel spectrograms of the audio signal processed by the whisper processor. The predicted text was generated using teacher forcing.
Refer to caption: x2.png

### 4.5 Model Ablation

**Table 4: Here shows the comparison of using different modules and loss. B means brain module, Lo means LoRA applied on encoder. E means encoder, D means decoder. These results are obtained without teacher forcing in evaluation. $L_{m}(mmd)$ is the mmd loss for aligning mel spectrogram instead of Clip loss. B-1 is the abbreviation of BLEU-1. R-1 is the ROUGE-1-F. SB is self-BLEU.**
| Loss | Trainables | Architect. | B-1 (%)$\uparrow$ | R-1 (%)$\uparrow$ | Bert (%)$\uparrow$ | CER (%)$\downarrow$ | SB (%)$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $L_{e}$ | B | B+D | 10.09 | 6.29 | 82.74 | 88.84 | 83.62 |
| $L_{e}+L_{t}$ | B | B+D | 6.15 | 4.81 | 84.43 | 80.33 | 95.32 |
| $L_{m}$ | B | B+E+D | 1.88 | 2.24 | 79.83 | 83.65 | 99.03 |
| $L_{m}+L_{e}$ | B | B+E+D | 10.44 | 6.93 | 83.39 | 89.82 | 85.28 |
| $L_{m}(mmd)+L_{e}$ | B | B+E+D | 9.64 | 5.71 | 81.62 | 87.95 | 80.55 |
| $L_{m}+L_{e}+L_{t}$ | B | B+E+D | 7.14 | 4.37 | 82.29 | 88.40 | 83.95 |
| $L_{m}+L_{e}$ | B+Lo | B+E+D | 1.13 | 0.79 | 81.17 | 87.65 | 99.98 |
| $L_{m}+L_{e}+L_{t}$ | B+Lo | B+E+D | 8.33 | 6.40 | 83.14 | 91.43 | 99.11 |

Table 4 presents a comparison of various configurations of trainable modules and loss functions in the brain-to-text decoding model, evaluated under teacher forcing conditions. The configurations include different combinations of the brain module (B), LoRA applied to the encoder (Lo), the encoder (E), and the decoder (D). The evaluation metrics used are BLEU-1, ROUGE-1, Bert score, and CER (Character Error Rate), Self-BLEU.

The baseline configuration, using only the brain module with the loss function $L_{m}$, achieves a BLEU-1 score of 1.88 and a ROUGE-1 score of 2.24, with Bert and CER scores of 79.83 and 83.65, respectively. Self-BLEU scores of 99.03 indicate the model generated almost identical sentences, showing that using only the brain module results in significant errors and inaccurate content.

Adding the encoder loss $L_{e}$ to $L_{m}$ while maintaining the same modules (B+E+D) significantly improves performance, yielding a BLEU-1 score of 10.44 and a ROUGE-1 score of 6.93. The Bert and CER scores were 83.39 and 89.82, respectively. In this configuration, we changed the alignment loss from clip to MMD($L_{m}(mmd)+L_{e}$), resulting in BLEU-1 scores of 9.64 and ROUGE-1 scores of 5.71. Similarly, the configuration using the brain module with $L_{e}$ (B+D) achieves the following scores: a BLEU-1 score of 10.09, a ROUGE-1 score of 6.29, a BERT score of 82.74, and a CER of 88.84. This indicates enhanced decoding accuracy when using encoder loss. Adding the triplet loss $L_{t}$ to this configuration decreases the BLEU-1 and ROUGE-1 scores to 6.15 and 4.81, respectively.

Using LoRA with the combination of $L_{m}$ and $L_{e}$ results in significantly poor performance, with BLEU-1 and ROUGE-1 scores of 1.13 and 0.79, and Bert scores of 81.17. The Self-BLEU score of 99.98 indicates that this configuration is highly ineffective, likely due to an incompatibility between LoRA and the task requirements.
Incorporating the triplet loss $L_{t}$ along with $L_{m}$ and $L_{e}$ for the same architecture (B+E+D) resulted in a Self-BLEU score of 99.11, indicating that the model generated almost identical sentences.

The results indicate that the brain module (B) is crucial for effective brain-to-text decoding, and the combination of multiple loss functions, particularly with the inclusion of the encoder loss ($L_{e}$), enhances the model’s performance. Configurations involving LoRA applied to the encoder are generally less effective unless complemented with the $L_{t}$, highlighting the need for carefully designed adaptation strategies for optimal performance in this context.

## 5 Limitation

Although our MAD model outperforms previous SOTA models, we have to point out that this model’s generation is far from practical utilization in reality since the performance is much lower than speech recognition models. Besides, this work is implemented on listening datasets, which is different from silent speech.

## 6 Conclusion

In this paper, we presented MAD, a novel end-to-end training framework for MEG-to-Text translation.
Our model leverages a multi-stage alignment utilizing auxiliary audio modality, which aligns brain activity data more effectively with corresponding textual outputs.
Experimental results suggest that the newly proposed MAD framework achieves 10.44 BLEU-1 on GWilliams without teacher-forcing evaluation on entirely unseen text which largely exceeds the current SOTA performance.
Through comprehensive ablation studies, we demonstrated the performance of our approach in various situations. Our results indicate that the brain module, in conjunction with appropriate loss functions, substantially enhances decoding performance.
The inclusion of encoder and decoder modules further refines the text generation process, with the triplet loss playing a crucial role in improving the model’s robustness and accuracy.
Particularly, the combination of the brain module with both the encoder and decoder, enhanced by multiple loss functions, shows marked improvements in BLEU-1 and ROUGE-1 scores, while reducing word and character error rates.
The insights gained from this research underline the potential of the MAD framework in the realm of neural decoding.
By effectively capturing the complex patterns in MEG signals and translating them into coherent text, our approach offers a promising solution for brain-to-text applications. This work sets the stage for further exploration into multi-modal alignments and their impact on neural decoding systems.

In conclusion, our proposed MAD framework significantly advances the state-of-the-art in brain-to-text decoding, offering new avenues for enhancing communication tools for individuals with severe speech and motor impairments. Future work will focus on refining alignment mechanisms and extending the application of our model to more diverse linguistic tasks.

## Supplementary Material for MAD: Multi-alignment MEG-text decoding

## Appendix A Dataset

The Gwilliams [] dataset is described below:

### A.1 Participants

- •
Total Participants: 27 English-speaking adults (15 females)
- •
Age: Mean = 24.8 years, SD = 6.4 years
- •
Recruitment: Subject pool of NYU Abu Dhabi
- •
Consent and Compensation: All provided written informed consent and were compensated
- •
Health: Reported normal hearing and no history of neurological disorders
- •
Language: All but one participant (S20) were native English speakers
- •
Sessions:
–
Majority (22 participants) performed two identical one-hour-long sessions
–
Sessions were separated by 1 day to 2 months
- •
Ethics Approval: Approved by the IRB ethics committee of NYU Abu Dhabi

### A.2 Procedure

- •
Recording Sessions:
–
Duration: Each session lasted approximately 1 hour.
–
Equipment: Recorded with a 208 axial-gradiometer MEG scanner (Kanazawa Institute of Technology).
–
Sampling Rate: 1,000 Hz.
–
Filtering: Online band-pass filtered between 0.01 and 200 Hz.
–
Task: Participants listened to four distinct stories through binaural tube earphones (Aero Technologies) at a mean level of 70 dB sound pressure level.
- •
Pre-Experiment Exposure:
–
Participants were exposed to 20 seconds of each distinct speaker voice.
–
Purpose: To clarify session structure and familiarize participants with the voices.
- •
Story Presentation Order:
–
Assigned pseudo-randomly using a "Latin-square design."
–
Same order used for both recording sessions for each participant.
- •
Attention Check:
–
Participants answered a two-alternative forced-choice question every  3 minutes.
–
Example Question: "What precious material had Chuck found? Diamonds or Gold."
–
Average Accuracy: 98%, confirming engagement and comprehension.
- •
MRI Scans:
–
T1-weighted anatomical scans were performed after MEG recording if not already available.
–
Six participants did not return for their T1 scan.
- •
Head Shape Digitization:
–
Head shape digitized with a hand-held FastSCAN laser scanner (Polhemus).
–
Co-registered with five head-position coils.
–
Coil positions collected before and after each recording, stored in the ’marker’ file.
–
Experimenter continuously monitored head position to minimize movement.

### A.3 Stimuli

- •
Stories: Four English fictional stories selected from the Open American National Corpus:
–
‘Cable spool boy’: 1,948 words, narrating two young brothers playing in the woods.
–
‘LW1’: 861 words, narrating an alien spaceship trying to find its way home.
–
‘Black willow’: 4,652 words, narrating the difficulties an author encounters during writing.
–
‘Easy money’: 3,541 words, narrating two friends using a magical trick to make money.
- •
Audio Tracks:
–
Synthesized using Mac OS Mojave’s (c) text-to-speech.
–
Voices and speech rates varied every 5-20 sentences to decorrelate language from acoustic representations.
–
Voices used: ‘Ava’, ‘Samantha’, and ‘Allison’.
–
Speech rate: Between 145 and 205 words per minute.
–
Silence between sentences: Varied between 0 and 1,000 ms.
- •
Story Segments:
–
Each story divided into  5-minute sound files.
–
Random word list played approximately every 30 seconds, generated from unique content words of the preceding segment.
–
Very small fraction (<1%) of non-words introduced in natural sentences.
- •
Task Definition:
–
Each "task" corresponds to the concatenation of sentences and word lists.
–
All subjects listened to the same set of four tasks, in different block orders.

## Appendix B Discussion about the main table

We used BLEU-1 [],ROUGE-1-F [],BertScore [],CER [],Self-BLEU [] as metrics in the main table to show the capability of previous models and our models. However, as observed, NeuSpeech [] model has the best score for ROUGE-1,Bert,CER, which is incredible, therefore we measured the Self-BLEU of this model, which is almost 100%, and found out NeuSpeech predicts almost the same sentence “He looked at me and said to me" all the time for different sentences in Supp. 1. Generation of this bad quality has best score on these three metrics, which means these three metrics are not effective in measuring the generation quality. Therefore, we think BLEU-1 the most reliable metric in this task for now. Besides, we randomly selected sentences, which is RS in the table, from the test set as another baseline, we found out that the BLEU-1 score is higher than NeuSpeech, which means the NeuSpeech model is not effective, which is very reasonable.
After all, it seems that using BLEU score is the only reasonable metric of evaluating the quality of generated text.

As observed in the table, it is very clear that our MAD model is significantly higher than RS and NeuSpeech and Wav2vec2CTC on BLEU-1, which means our MAD model is effective on unseen text.

## Appendix C More generated samples

We showed more generate samples here to show that we are not cherry-picking.

Figure: Figure 3: Eight sample examples of the test set. Prediction refers to Mel spectrograms generated by the brain module. Ground truth refers to Mel spectrograms of the audio signal processed by the whisper processor. The predicted text was generated using teacher forcing. These examples were produced using $L_{m}(mmd)$ with only a trainable brain module.
Refer to caption: x3.png

Figure: Listing 1: NeuSpeech [] generation without teacher forcing.

Figure: Listing 2: Wav2vec2CTC [] generation.

Figure: Listing 3: MAD generation with teacher forcing.

## Appendix D Ethics

### D.1 Safety

Our MEG-to-text translation technology is designed to assist individuals with severe speech and motor impairments by translating brain signals into text. While this technology has the potential to greatly improve quality of life, we acknowledge the possibility of misuse. However, there are no foreseeable situations where the direct application of our technology could harm, injure, or kill people. We do not develop or intend to develop applications that increase the lethality of weapons systems.

### D.2 Security

We recognize the importance of securing brain-computer interface systems. Future research should include thorough risk assessments to identify and mitigate potential security vulnerabilities. We recommend employing robust encryption methods and secure data transmission protocols to protect against unauthorized access and ensure the safety of users’ neural data.

### D.3 Discrimination

Our technology aims to provide equal accessibility to communication for individuals with speech and motor impairments. We are committed to ensuring that our MEG-to-text translation system does not facilitate discrimination or exclusion. We will continuously monitor and test our models to prevent biases that could negatively impact service provision in healthcare, education, or financial sectors.

### D.4 Surveillance

We adhere strictly to local laws and ethical guidelines regarding data collection and analysis. Our research does not involve bulk surveillance data. We obtained data from public dataset [], and we do not predict protected categories or use data in ways that endanger individual well-being.

### D.5 Deception & Harassment

Our technology is designed with safeguards to prevent its misuse in deceptive or harmful interactions. We implement verification mechanisms to detect and prevent impersonation and fraudulent activities. We actively work to ensure our system cannot be used to promote hate speech, abuse, or influence political processes maliciously.

### D.6 Environment

We are mindful of the environmental impact of our research. While our work primarily involves computational resources, we strive to optimize our algorithms to minimize energy consumption. We do not engage in activities that promote fossil fuel extraction or increase societal consumption. Our focus is on developing sustainable and efficient technologies.

### D.7 Human Rights

Our research adheres to ethical standards and legal requirements, ensuring that it does not facilitate illegal activities or deny individuals their rights to privacy, speech, health, liberty, security, legal personhood, or freedom of conscience or religion. We are committed to protecting and promoting human rights through our work.

### D.8 Bias and Fairness

Our goal is to create fair and inclusive technology that benefits all users equally, regardless of their background.

## Appendix E Broader Impacts

### E.1 Positive Impacts

Improved Communication for Individuals with Disabilities
Our MEG-to-text translation technology has the potential to significantly improve the quality of life for individuals with severe speech and motor impairments. By enabling these individuals to communicate effectively, we can help them achieve greater independence, participate more fully in society, and reduce their reliance on caregivers.

Advancements in Neurotechnology
This research contributes to the broader field of neurotechnology, advancing our understanding of brain activity and its relationship to language. These advancements could lead to new therapies and interventions for a variety of neurological conditions, potentially benefiting a wide range of patients.

Innovation and Economic Growth
The development and commercialization of advanced neurotechnologies can stimulate economic growth by creating new industries and job opportunities. This innovation can drive progress in related fields such as healthcare, robotics, and artificial intelligence, fostering a collaborative and dynamic technological ecosystem.

### E.2 Negative Impacts

Privacy and Security Risks
The collection and analysis of neural data pose significant privacy and security concerns. Unauthorized access to such sensitive information could lead to misuse, including identity theft or unauthorized surveillance. Ensuring robust data protection measures is crucial to mitigate these risks.

Potential for Misuse
There is a risk that the technology could be misused for purposes other than its intended therapeutic applications. For instance, it could be exploited for invasive surveillance or to manipulate individuals by decoding their thoughts without consent. Strict ethical guidelines and regulations are necessary to prevent such misuse.

Bias and Discrimination
If not carefully managed, the technology could inadvertently encode or exacerbate existing biases. For example, if the training data is not representative of diverse populations, the system might perform poorly for certain groups, leading to unequal access to its benefits. Ongoing efforts to ensure fairness and inclusivity are essential to address these concerns.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - **Dataset**: Could you clarify your choice of dataset? Why did you select GWilliams instead of alternatives like Armeni (https://www.nature.com/articles/s41597-022-01382-7)? If the model is SOTA on unseen text, why not test it on other available datasets?

- **Inputs**: Can you explain how you transform a shifted 4-second window into a fixed-size sample for the Mel spectrogram?

- **Splitting**: Would it be possible to include experiments that demonstrate performance using different data splits (i.e., different combinations of the 4 stories)?

- **Multi-Modality Integration**: Could you elaborate on the challenges encountered when aligning MEG data with Mel spectrograms and hidden states?

- **Losses**: Can you provide the performance results when only using the Lt loss? Additionally, could you expand on the significance of the Le loss in this setup?

- **Discussion**: Could you elaborate on the other metrics used in your analysis? How does the performance behavior differ among them, and what insights do they provide to enhance the overall analysis?

- **Noise**: Would you consider experimenting with a model trained on noise, rather than merely evaluating it on noisy data?
soundness: 3
strengths: - **Clarity**: The paper is exceptionally well-written and easy to understand.
- **Novelty**: It introduces a novel multi-alignment approach that utilizes auxiliary modalities, such as Mel spectrograms, to enhance the translation of MEG signals into text. This work is notable for being one of the few that reports results without using teacher forcing.
- **Performance**: The method achieves state-of-the-art results on the BLEU-1 metric for this dataset.
- **Ablation Studies**: The paper includes comprehensive experiments and ablation studies that clarify the contributions of each model component, especially the loss functions and alignment mechanisms.
summary: This paper presents **MAD**, an end-to-end framework for decoding MEG signals into text. They use a novel multi-alignment approach with audio and text representations. They present their results utilizing the GWilliams dataset showing high BLEU score on entirely unseen text. They highlight the framework's potential for generalization. They conduct comprehensive ablation studies that highlight the important of high dimensional representations alignment over direct text alignment for robust performance.
weaknesses: - **Dataset**: The reliance on a single dataset reduces the impact of the results. Although the authors claim good performance on entirely unseen text, the word overlap is 46% on stories belonging to the same corpus and same subjects. This assumption should be validated across other datasets.
  
- **Losses**: The authors indicate that the Le loss is the most important in the study, but they do not explain why. A part-of-speech analysis could clarify the performance differences between content and function words. They assert that high-level speech is better decoded than low-level speech; however, this has been demonstrated in previous studies (https://www.biorxiv.org/content/10.1101/2024.04.19.590280v1). Additionally, the authors do not provide a solid hypothesis for why the Lt loss negatively impacts the BLEU-1 score. They also do not show the performance when only using Lt, as they do for Lm and Le.

- **Noise Study**: In the Evaluation Metrics section, noise is employed to demonstrate that the model effectively learns from MEG signals. It would be more insightful to train the model on noise rather than simply evaluating it.

- **Metrics**: Although the paper mentions five different metrics, the discussion predominantly focuses on the BLEU-1 score, while the ROUGE-1 and CER scores exhibit different (opposite?) behaviors that do not show SOTA results. Notably, one of the main findings regarding the detrimental effect of Lt is not corroborated by the CER metrics. While it is commendable to use five metrics, the authors primarily discuss BLEU-1 without explaining why it is the most relevant metric.

- **Remarks**:
  - **Introduction and Related Work**: Consider integrating the related work section into the introduction for better flow, or relocating some introductory elements to the related work section.
  - **Dataset Description**: The paper states 2 hours per session, but it should be corrected to 1 hour per session, as per the MEG-MASC paper (Methods/Participants section, https://www.nature.com/articles/s41597-023-02752-5).
  - **Limitations**: The limitations section is underdeveloped and should be expanded.
  - **Typos**: Correct "setted" to "set" (section 4.1), "primarized" to "prioritized" (section 4.4.1), add a missing space between "state-of-the-art" and "(SOTA)" (conclusion), and address incorrect references (Supplementary A1).
  - **References**: On two different computers, the PDF displays a "double" reference in the Related Work section, mentioning the same studies twice in succession.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.	What is the purpose of random window shifting in line 304?
2.	How is cross-subject implemented in Section A.1? Why does it perform similarly to intra-subject despite having a different subject index input to the brain module?
soundness: 2
strengths: 1. The incorporation of a new modality in the MEG2Text translation is desirable.
2. High-level L_e loss is proven to be effective.
summary: This paper proposes a MEG2Text framework, which consists of a Brain Module and a Whisper Codec. MEG is initially decoded into a Mel spectrogram and subsequently translated into text using the Whisper model. The framework incorporates three alignments: low-level, high-level, and text-level alignments, which have potential inspiration for neural transcription.
weaknesses: The residual connection in Figure 1 (b) is missing an arrow.
	The main experimental results in Table 2 have no advantage over RS and NeuSpeech, and are almost all lower except for B-1 and self-B.
	The brain module within Wav2vecCTC is not trained on text, so it exhibits poor performance and struggles to generate coherent words. The comparison is unfair.
	Baseline models should be compared both w/ and w/o tf.
	The paper aims to showcase the model's ability to generalize to unseen text, yet this attribute is not evident within the experimental setup presented. The paper solely conducts experiments on a test set with 46% overlap in words, which fails to represent unseen text. For a more thorough evaluation, it is crucial to compare the model with the baselines on both seen and unseen text separately. Furthermore, experiments should be conducted on additional datasets to enhance the validation of generalization.
	Ablation studies have demonstrated that L_t and LoRA are redundant and should be omitted from the model structure.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: ### Methods
- Please use consistent symbols for the same concept (for example either $n$ or $N$ for the batch size)
- $\phi$ is not defined in (2)
- line 304: define the random shifts. Were they samples from the interval $[-0.5, 0.5]$ or the set $\{-0.5, 0.5\}$?
- the considered baseline method `Wav2vec2CTC` was not proposed in (Défossez et al. 2023). 


### Wording, Grammar and Organization
- citation formatting: please use the `\citep` and `\citet` latex commands appropriately.
- line 42: "letter" -> "letters"
- line 201: "don't" -> "do not"
- lines 207 to 208: grammar issues
- line 231: "in default" -> "by default"
- some references miss important information (e.g., the journal). Please check thoroughly.


### References

A. Défossez, C. Caucheteux, J. Rapin, O. Kabeli, and J.-R. King, “Decoding speech perception from non-invasive brain recordings,” Nat Mach Intell, Oct. 2023, doi: 10.1038/s42256-023-00714-5.
soundness: 2
strengths: The paper can be classified as a combination of existing ideas.
In terms of architecture, the authors follow the idea of (Yang et al. 2024) and combine a convolutional network with a pre-trained whisper speech decoder; instead of standard convolution layers they propose to use the brain module of (Défossez et al. 2023).
As training objectives, they compare combinations of similar (and related) loss functions as proposed in (Défossez et al. 2023) and (Yang et al. 2024).

### Quality

I appreciate that the authors evaluated the models on unseen stories and put substantial effort in assessing the significance of the results. They compare the obtained results against shuffled data and models with random inputs.

### Clarity

The problem setting and the overall approach of the contribution are clearly communicated.

### Significance

The work confirms that pre-trained speech models (e.g., whisper here or wave2vec in (Défossez et al. 2023)) are helpful to extract a significant amount of language information from non-invasive brain activity (MEG here). Yet, the reported generalization results to unseen stories (Table 2) and subjects (Table 5) are discouraging.

### References

A. Défossez, C. Caucheteux, J. Rapin, O. Kabeli, and J.-R. King, “Decoding speech perception from non-invasive brain recordings,” Nat Mach Intell, Oct. 2023, doi: 10.1038/s42256-023-00714-5.

Y. Yang et al., “NeuSpeech: Decode Neural signal as Speech,” Jun. 03, 2024, arXiv: 2403.01748.[Online]. Available: http://arxiv.org/abs/2403.01748
summary: The submission studies text decoding from brain activity based on a publicly available dataset that contains magnetoencephalography (MEG) recording of 27 subjects while they were passively listening to different stories.
Building upon recent prior work, the authors utilize a brain module and train it to predict mel spectrograms derived from the audio signals.
These MEG-derived mel spectrograms are then fed through a publicly available, pre-trained speech decoding model (whisper, OpenAI) to complete the MEG-to-text decoder.
The authors propose a combination of three loss terms with the aim to align mel spectrograms, latent speech encoder representations as well as decoder outputs and labels.
Given the small amount of data, they find that training merely the brain module parameters and freezing the remaining modules yielded best results (BLEU-1 of 10.44 % without teacher forcing) if the model was trained to align the latent speech encoder representations and the mel spectrograms.
Although the proposed method outperforms the considered baseline models, the overall poor performance (BLEU-1 values of approx. 10%) prevents practical utilization.
weaknesses: ### Originality
- Brain activity (invasive and non-invasive) to speech/text translation has already been introduced in prior works, as summarized in section 2 (related work).

- The methodological contribution is incremental. 
Défossez et al. (2023) introduced the idea of aligning M/EEG signals with latent representations of a pre-trained ASR model (wav2vec2.0) with the CLIP loss. In the submission, the authors use a related approach (whisper encoder instead of wav2vce2.0, and MMD loss instead of CLIP).
Yang et al. (2024) combined the whisper model with a convolutional adapter network (trained with AdaLoRa). In this submission, the authors propose a modified architecture that uses the brain module of (Défossez et al. 2023) as adapter network.

### Quality
- While I appreciate that the authors report results of the considered performance metrics for random effects (i.e., random shuffling and noise as input), their approach is insufficient to actually estimate the significance of the results. Instead of a single evaluation they should have used permutation testing (see e.g., Maris 2012) to estimate the metrics' distribution under the null hypothesis (i.e., no-relation between the MEG-derived text and the ground truth).

- Unlike (Défossez et al. 2023) and (Yang et al. 2024) the authors decided to analyze only one public dataset. I think the authors should have also analyzed the results for the (Schoffelen et al. 2019) dataset. Beyond that, the results lack quantification of the stability/variability of the results across random weight initializations.

### Clarity
- Figure 1(a) does not correspond to the proposed MAD model in Table 2.
This discrepancy suggests that the study might be affected by the double dipping problem (Kriegeskorte et al. 2009) in the sense that the authors tested many different configurations and then picked the final MAD model based on the test set result.

### References

A. Défossez, C. Caucheteux, J. Rapin, O. Kabeli, and J.-R. King, “Decoding speech perception from non-invasive brain recordings,” Nat Mach Intell, Oct. 2023, doi: 10.1038/s42256-023-00714-5.

Y. Yang et al., “NeuSpeech: Decode Neural signal as Speech,” Jun. 03, 2024, arXiv: 2403.01748.[Online]. Available: http://arxiv.org/abs/2403.01748

J.-M. Schoffelen, R. Oostenveld, N. H. L. Lam, J. Uddén, A. Hultén, and P. Hagoort, “A 204-subject multimodal neuroimaging dataset to study language processing,” Sci Data, vol. 6, no. 1, p. 17, Apr. 2019, doi: 10.1038/s41597-019-0020-y.

E. Maris, “Statistical testing in electrophysiological studies,” Psychophysiology, vol. 49, no. 4, pp. 549–565, 2012, doi: 10.1111/j.1469-8986.2011.01320.x.

N. Kriegeskorte, W. K. Simmons, P. S. F. Bellgowan, and C. I. Baker, “Circular analysis in systems neuroscience: the dangers of double dipping,” Nat Neurosci, vol. 12, no. 5, pp. 535–540, May 2009, doi: 10.1038/nn.2303.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Questions:
- Are there results of benchmarking of the proposed model against other SOTA models using different test splits (i.e. not for entirely unseen data)? or with the Défossez et al. model as originally proposed (i.e. classification with the unseen test audio)?

Suggestions:
- fix the citation styling (see section 4.1 of the Formatting Instructions for ICLR 2025 Conference Submissions)
soundness: 2
strengths: The approach as a whole is novel and the reported performance is strong across metrics for both high level accuracy and low level semantic content. The inclusion of random gaussian baselines should be noted and adopted as standard for future decoding studies. The push for evaluations on unseen text is also laudable.
summary: This paper proposes a novel brain-to-text perceived speech decoding pipeline (MAD) that leverages alignments from multiple modalities. The key contributions highlighted by the authors are the ability of their framework to generalize to unseen text, eliminating the need for word time segmentation (e.g. teacher-forcing, pre-training, or via eye-tracker), and a performance analysis with a suite of metrics, ablation testing, and benchmarking against other models. A comparison of different modalities is possible due to the modular structure of their design.
weaknesses: There appears to be some issue with citation formatting that should be fixed (see ICLR 2025 style guide), as well as a few grammatical issues. Additionally, the benchmarking comparison with Défossez et al. (2023) appears to be in bad faith. Instead of comparing the performance of their decoding framework against Défossez et al.'s model as originally designed, it seems as if the authors of this paper use only the brain model and then apply a decoding head in the style of their framework. Thus, it results in many of the generated unigrams being gibberish as opposed to actual words and greatly reduces whatever the real performance of the Défossez et al. model should be across all reported metrics. The original model from Défossez et al. returns the most likely segment of audio given the meg input and therefore always returns real words. The paper by Défossez et al. explicitly mentions the ability to decode segments not present in the training set. Thus, it is still reasonable to benchmark with the Défossez et al. model as originally designed. If the intention was to underscore that the model from Défossez et al. needs access to the segmented test audio while the proposed framework operates with MEG data alone, this could have been stated and the Défossez et al. model omitted from bechmarking comparison.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Lack of Multi-Dataset Testing: The model is only evaluated on the GWilliams dataset, unlike related works such as NeuSpeech and Wav2vecCTC, which test on additional datasets like Schoffelen. Although the authors acknowledge risks of overfitting on small datasets and discuss generalizability, a practical demonstration on other datasets like Schoffelen would better validate the model’s transferability and robustness.
2.	Inference Process for Sliding Windows: The model uses 4-second sliding windows with a shift of 1 second, incorporating random shifts of ±0.5 seconds to generate training samples. However, it remains unclear whether the inference stage aggregates outputs across windows or applies additional processing steps. Moreover, this raises concerns about the effect of different window sizes on the decoding results, an aspect not explored in the paper.
3.	Ablation Study on Loss Functions: The ablation studies discuss three loss functions—high-level feature alignment ($L_e$), low-level feature alignment ($L_m$), and text alignment ($L_t$)—but do not provide concrete evidence of how each influences decoding results. Adding examples of decoding results under different combinations of these loss functions would provide valuable insight, especially in validating the importance of high-level and low-level feature alignments as argued by the authors.
soundness: 3
strengths: - This paper introduces a new open-vocabulary MEG-to-Text translation framework MAD, avoiding the teacher-forcing problem seen in previous models and using a more reasonable evaluation method.

- The experimental design is robust, including comprehensive ablation studies that underscore the advantages of multi-modal alignment.

- Explanations of the model architecture, data, and experimental setup are clear and well-structured.

- The model sets a new standard in MEG-to-Text decoding accuracy, showcasing the valuable potential for real-world BCI applications.
summary: The paper introduces MAD, an end-to-end multi-alignment framework for translating MEG signals to text. This framework addresses challenges related to decoding unseen text by aligning MEG data with speech and text modalities. Using the GWilliams dataset, MAD achieves improved performance in MEG-to-text generation compared to prior approaches. The study also examines the impact of different loss functions on decoding performance, offering insights that may aid in future optimization of MEG-based language decoding.
weaknesses: The primary concern is the limited dataset, as the experiments were only conducted on the GWilliams dataset, while the model was trained with MEG-Speech-Text modality pairs. Testing on a single dataset limits the model's robustness and generalizability claims.