Can Brain Signals Reveal Inner Alignment with Human Languages?
Authors: Jielin Qiu
William Han
Jiacheng Zhu
Mengdi Xu
Douglas J Weber
Bo Li
Ding Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: EEG
Human Languages
Inner Alignment
Interpretation
TL;DR: We propose to use a computational model to understand the inner alignment between brain signals and human languages.

Abstract: Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced \textbf{MTAM}, a \textbf{M}ultimodal \textbf{T}ransformer \textbf{A}lignment \textbf{M}odel, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7\% on K-EmoCon and 9.3\% on Zuco datasets for sentiment analysis, and 7.4\% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at \url{https://github.com/Jason-Qiu/EEG_Language_Alignment}.

Can Brain Signals Reveal Inner Alignment with Human Languages?
William Han" Jielin Qiu';, Jiacheng Zhu Mengdi Xu',
Douglas Weber', Bo Li,, Ding Zhao'
'Carnegie Mellon University, MIT CSAIL, University of Chicago
Abstract
have made large efforts in using brain-recording technologies to analyze cognitive activity during language-rolatee tasks and observed that these tech- nologies added value in terms of understanding language (Stemmer and Connolly, 2012).
Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream lasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency be- tween EEG and language. To study at the representation level, we introduced MTAM, a Multimodal Transformer Alignment Model, observe coordinated representations between the two modalities. We used various relation- techniques, such as Canonical Correlation Analysis Wasser- stein Distance, loss functions to transfig- ure features. On downstream applications, sentiment analysis and relation detection, achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an Fl-score improvement of 1.7% K- L-EmoCon and 9.3% on Zuco datasets for sen- timent analysis, and 7.4% on ZuCo for rela- tion detection. In addition, tations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and en- coding the relationship between EEG and lan- guage; (2) aligmment weights show the influ- ence of different language semantics as well as EEG frequency features; (3) brain topograph- ical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code available at https: /gi comb Laaoor/ouu/EE_LLnguugeeelllll
Basic linguistic rules seem to be effortlessly un- derstood by humans in contrast machinery. Re- cent advances in natural language processing (NLP) models (Vaswani et al., 2017) have enabled com- puters to maintain long and contextual informa- tion through self-attention mechanisms. This at- tention mechanism has been maneuvered to create robust language models but at the cost of tremen- dous amounts of data (Devlin et al., 2019; Liu et al., 2019b; Lewis et al., 2020; Brown al., 2020; Yang et al., 2019). Although performance has signif- icantly improved by using modern NLP models, they are still seen to be suboptimal compared to the human brain. In this study, we explore the re- lationship and dependencies of EEG and language. We apply EEG, popularized routine in cognitive research, for its accessibility and practicality, along with language to discover connectivity.
ship alignment-seekinn
provide interpre-
Our contributions are summarized follows:
To the best of knowledge, this is the first work to explore the fundamental relationship and connectivity between EEG and language through computational multimodal methods.
We introduced MTAM, a Multimodal Transformer Alignment Model, that learns coordinated representations by hierarchical transformer encoders. The transformed rep- resentations showed tremendous performance improvements and state-of-the-ar results in downstream applications, i.e., sentiment anal- ysis and relation detection, on two datasets, ZuCo 1.0/2.0 and K-EmoCon.
1 Introduction
Brain activity is an important parameter in fur- thering our knowledge of how human language is represented and interpreted (Toneva al., 2020; Williams and Wehbe, 2021; Reddy and Wehbe. 2021; Wehbe et al., 2020; Deniz al., 2021). Re- scarchers from domains such as linguistics, psy- chology, cognitive science, and computer science marked as equal contribution
We carried out experiments with multiple aligmment mechanisms, i.c., canonical corre- lation analysis and Wasserstein distance, and

---

Downstream Tasks
umbedd
Hightioull
Cross Alignment
Hanarchizi Torractrmer
Herrarchccl Transfomerr
EBG Encoder
Fnonder
Ensale
Figure The architecture of our model, where EEG and language features coordinately explored by encoders. The EEG encoder and language encoder are shown on the left and right, respectively. The cross-alignment module is used to explore the connectivity and relationship within two domains, while the transformed features are used for downstream tasks.
proved that relation-seed seeekig loss functions are helpful in downstream tasks.
dict specific values of each dimension in a word vector through regression models. Wang and (2021) used word-level EEG features to decode cor- responding text tokens through an open vocabulary, sequence-to-sequene framework. Hollenstein al. (2021) focused multimodal approach by uti- lizing combination of EEG, eye-tracking, and text data improve NLP tasks, but did not explore the relationship between EEG and language. More related work can be found in Appendix E.
We provided interpretations of the perfor- mance improvement by visualizing the origi- nal & transformed feature distribution, show- ing the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language.
Our findings on word-level and sentence-level EEG-language alignment showed the influ- ence of different language semantics as well as EEG frequency features, which provided additional explanations.
3
Methods
3.1 Overview of Model Architecture
The brain topographical maps delivered an in- tuitive demonstration of the connectivity of EEG and language response in the brain re- gions, which issues physiological basis for our discovery.
The architecture of our model is shown in Fig. 1. The bi-encoder architecture is helpful in project- ing embeddings into vector space for methodical analysis (Liu et al., 2019a; Hollenstein et al., 2021; Choi et al., 2021). Thus in study, we adopt the bi-encoder approach to effectively reveal hidden relations between language and EEG. The MTAM, Multimodal Transformer Alignment Model, con- tains several modules. We use dual-encoder ar- chitecture, where each view contains hierarchical transformer encoders. The inputs of each encoder are EEG and language, respectively, For EEG hi- erarchical encoders, each encoder shares the same architecture as the encoder module Vaswani al. (2017). In the current literature, researchers as- sume that the brain acts as an encoder for high- dimensional semantic representations (Wang and Ji, 2021; Gauthier and Ivanova, 2018; Correia et al., 2013). Based on this assumption, the EEG sig- nals act as low-level embeddings. By feeding it into its respective hierarchical encoder, extract transformed EEG embeddings input for the cross alignment module. As for the language path, the language encoder is slightly different from the EEG encoder. We first process the text with a pretrained large language model (LLM) to extract text em-
2 Related Work
Multimodal Learning of Language and Other Brain Signals Wehbe et al. (2014) used recur- rent neural network to perform word alignment between MEG activity and the generated word em- beddings. Toneva and Wehbe (2019) utilized word- level MEG and fMRI recordings to compare word embeddings from large language models. Schwartz al. (2019) used MEG and fMRI data fine-tune a BERT language model (Devlin et al., 2019) and found the relationships between these two modal- ities were generalized across participants. Huang et al. (2020) leveraged CT images and text from electronic health records to classify pulmonary em- bolism cases and observed that the multimodal model with late fusion achieved the best perfor- mance. However, the relationship between lan- guage and EEG has not been explored before.
Multimodal Learning of EEG and Language Foster et al. (2021) applied EEG signals to pre-

---

Table I: Comparison with baselines on Zuco dataset for Sentiment Analysis (SA) and Relation Detection (SD).
Sentence Level
Word Level
Concut Word Level
Model
MILSEEEG
0:183
0:477
0.480
0.199
0.451
0.483
0.167
0.973
0.117
0.164
0:455
0.450
MLP-Jes: Jest
0.259
0.257
0.257
0.373
0.390
0.388
0.384
0.387
0.210
0.243
0.225
0.228
B-- STM-RR
0.506
0.492
0.49%
0.499
0.506
0.494
0.501
0.503
0.459
0.457
0:457
0.456
Bi-I Tea-
0-420
0.347
0.380
0.371
0.335
0.326
0.350
0.329
0.341
0.322
0.331
0.329
Transformer BEG
0.065
0.669
0.562
0.562
0.624
0.630
0.627
0.664
0.624
0.630
0.627
0.514
Senkiment Amilais
Transfomms-Tes Text
ResNes
0.548
0.508
0.546
0.547
0.507
0,527
0.530
0.539
0.533
0.534
0,550
0.532
0.582
0.518
0.558
0.517
0.547
0.552
0.518
0.550
Restet-Test
0.202
0.206
RNN- Multimodal (Hellenstein 2021)
CNN Multinodaal (Hollenstein et 2021)
0.724
0.723
Ours-BRG
0.779
0.783
0.781
1.779
0,783
0.7%6
0.784
0.781
0.783
0.790
0.791
0.792
Oum-Tza
IL763
B.762
0.762
11735
9749
0.748
0.753
0.740
0.764
1.763
0.772
Oun- Oun-Muakimdaal
0.822
0.829
0.826
0,836
08821
0,812
0,816
0.827
0.802
0.509
0.506
0,513
MLP-EEG
0.265
0.262
0.255
LSTM
0333
0.324
0.331
BL-LSTMM-Test
0:126
6,200
0.19
0.19
0:00
0:133
01154
0.148
Trandormer--e FFGG
0.502
0,468
0,299
03341
0,40
0.588
0.100
0.116
0.113
0.315
Tranfomma-Tes
0.428
0.487
01444
(1430
0.468
0.491
0.489
00447
0.301
0.209
0.300
0.300
Relation Desection
RecNe-EEBO
0.359
0.599
0.433
0.590
0,349
0.350
0.350
0,550
0.352
0.363
0.358
0.152
ResNet Text
0.214
0.283
0.265
0.112
0,311
0,336
0.321
0.326
0.278
0.289
0.283
0.288
CNX. Multimodal (Hollenstein et 2023)
0.647
0.664
0,650
RNK Multimahall (Ho (Holleendein 2021)
0,653
0.690
0.668
0.455
0.472
Cun--Mllinodal
IL744
0.746
0.737
0.734
IL736
beddings and then use hierarchical transformer encoders to transform the raw text embeddings into high-level features. The mechanism of the cross aligmment module is to explore the inner re- lationship between EEG and language through connectivity-basel loss function. In our study, we investigate two alignment methods, i.e., Canoni- cal Correlation Analysis (CCA) and Wasserstein Distance (WD). The output features from the cross alignment module can be used for downstream ap- plications. The details of each part are introduced in Appendix B.3.
alignment between EEG and language. The results of the K-EmoCon dataset are listed in Appendix D
4.2 Ablation Study
To further investigate the performance of different mechanisms in the CAM, we carried out ablation experiments on the Zuco dataset, and the results are shown in Table 6 in Appendix D.2. The combina- tion of CCA and WD performed better compared to using only one mechanism for sentiment analysis and relation detection in all model settings. We also conducted experiments on word-level, sentence- level, and concat word-level inputs, and the re- sults are also shown in Table 6. We observe that word-level EEG features paired with their respec- tive word generally outperform sentence-leved and concat word-level in both tasks.
4 Experiments
4.1 Experimental Results and Discussions
In this study, we evaluate our method on two down- stream tasks: Sentiment Analysis (SA) and Rela- tion Detection (RD) of two datasets: K-EmoCon (Park et al., 2020) and ZuCo 1.0/2.0 Dataset (Hol- lenstein et al., 2018, 2020b). Given succession of word-level or sentence-level EEG fcatures and their corresponding language, Sentiment Analysis (SA) task aims to predict the sentiment label. For Relation Detection (RD), the goal is to extract se- mantic relations between entities in a given text.
4.3 Analysis
To understand the alignment between language and EEG, visualize the alignment weights of word- level EEG-language alignment the ZuCo dataset. Fig. 2 and Fig. 3 show examples of negative & pos- itive sentence word-level alignment, respectively. The sentence-level alignment visualizations are shown in Appendix D.5.
More details about the tasks, data processing, and experimental settings can be found in Appendix C. In Table 1, we show the comparison results of the ZuCo dataset for Sentiment Analysis and Re- lation Detection, respectively. Our method outper- forms all baselines, and the multimodal approach outperforms unimodal approaches, which further demonstrates the importance of exploring the inner
From word level alignment in Fig. 2 and 3, beta2 and gammal waves are most active. This is consis- tent with the literature, which showed that gamma waves are seen to be active in detecting emotions (Li and Lu, 2009) and beta waves have been in- volved in higher-order linguistic functions (c.g., dis- crimination of word categories). Hollenstein et al.

---

06
onap
nowee
42
00
642
-42
-4
up
-44
Figure 2: Negative word- level alignment.
Figure 3: Positive word-level alignment.
Figure 4: Brain topologies.
(2021) found that beta and theta waves were most useful in terms of model performance sentiment analysis. In Kensinger (2009), Kensinger explained that generally, negative events are more likely to be remembered than positive events. Building off of Kensinger (2009), negative words can embed more significant and long-lasting memory than pos- itive words, and thus may have higher activation in the occipital and inferior parietal lobes.
tivity Wernicke's 's area is seen most visibly in the beta2, gammal, and gamma2 bands for the words 'Upscale' and 'Will', For the word 'Upscale," we also saw activity around Broca's ' area for alphal, al- pha2, betal, beta2, thetal, and theta2 bands. An in- teresting observation is that for the negative words, 'Lame' and 'Someone', we see very low activation in Broca's and Wernicke's areas. Instead, we see most activity in the occipital lobes and slightly over the inferior parietal lobes. The occipital lobes are noted as the visual processing area of the brain and are associated with memory formation, face recog- nition, distance and depth interpretation, and vi- suospatial perception (Rehman and Khalili, 2019). The inferior parietal lobes generally found be key actors visuospatial attention and semantic memory (Numssen et al., 2021).
We performed an analysis of which EEG feature refined the model's performance since different neurocognitive factors during language processing are associated with brain oscillations miscella- neous frequencies. The beta and theta bands have positively contributed the most, which is due the theta band power expected rise with increased language processing activity and the band's s relation to semantic memory retrieval (Kosch et al., 2020; Hollenstein et al., 2021). The beta's 's contribution can be best explained by the effect of emotional connotations of the text (Bastiaansen et al., 2005; Hollenstein et al., 2021).
Conclusion
In this study, we explore the relationship between EEG and language. We propose MTAM, Mul- timodal Transformer Alignment Model, to ob- serve coordinated representations between the modalities and employ the transformed represen- tations for downstream applications. Our method achieved state-of-the-rs performance on sentiment analysis and relation detection tasks on two public datasets, ZuCo and K-EmoCon. Furthermore, we carried out comprehensive study to analyze the connectivity and alignment between EEG and lan- guage. We observed that the transformed features show less randomness and sparsity. The word-level language-FEG alignment clearly demonstrated the importance of the explored connectivity. We also provided brain topologies as an intuitive under- standing of the corresponding activity regions in the brain, which could build the empirical neu- ropsychological basis for understanding the rela- tionship between EEG and language through com- putational models.
In Fig. 4, we visualized the brain topologies with word-level EEG features for important and unim- portant words from positive and negative sentences in the ZuCo dataset. We deemed word important if the definition had positive or negative connota- tion. 'Upscale' and 'lame' are important positive and negative words, respectively, while 'will' and "somcone" are unimportant positive and negative words, respectively. There are areas in the brain that are heavily associated with language pro- cessing: Broca's area and Wernicke's area. Broca's area is assumed to be located in the left frontal lobe, and this region is concerned with the production of speech (Nasios et al., 2019). The left posterior superior temporal gyrus is typically assumed as Wernicke's area, and this locale is involved with the comprehension of speech (Nasios al., 2019). Similar to Fig. 2,3, we can observe that beta2, gammal, and gamma2 frequency bands have the most powerful signals for all words. In Fig. 4, ac-

---

6 Limitations
not conduct any human-involved experiments by ourselves. Additionally, we do not implement any technologies on the human brain. The datasets can be found in Park et al. (2020); Hollenstein et al. (2018, 2020b)
Since we proposed a new task of exploring the re- lationship between EEG and language, we believe there are several limitations that can be focused on in future work.
We believe this study can empirically provide findings about the connection between natural lan- guage and the human brain. To our best knowledge, we do not foresee any harmful uses of this scientific study.
The size of the datasets may not be large enough. Due to the difficulty and time- consumption of collecting human-related data (in addition, to privacy concerns), there are few publicly available datasets that have EEG recordings with corresponding natural lan- guage. When compared to other mature tasks, (i.e. image classification, object detection, etc), datasets that have a combination of EEG signals and different modalities are rare. In the future, we would like to collect more data EEG signals with natural language hance innovation in this direction.
References
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. 28.
Lan- nan--Qnng Bao, Jie-Lin Qiu. Hao Tang. Wei-Long Zheng, and Bao-Liang Lu. 2019. Investigating sex differ- ences in classification and eye movement signals. 2019 41 Annual In- ternational Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 6746- 6749.
five emotions from eeg
The computational architecture, the MTAM model, is relatively straightforward. We agree the dual-encoder architecture one of the standard paradigms in multimodal learning. Since our target is to explore the connectiv- ity and relationship between EEG and lan- guage, we used a straightforward paradigm. Our model's 's architecture may be less complex compared to others in different tasks, such as image-text pre-training. However, we pur- posely avoid complicating the model's struc- ture due to the size of the training data. We noticed when adding more layers complex- ity, the model was more prone to overfitting.
Khayrul Bashar. 2018. Eeg and eeg based multimodal biometrics for human identification.
Marcel C. M. Bastiaansen, Marieke van der Linden, Mariken Ter Keurs, Ton Dijkstra, and Peter Ha- goort. 2005. Theta responses involved in lexical- semantic retrieval during language processing. Jour- nal of Cognitive Neuroscience, 17:530-541.
Ahmed Ben Said, Amr Mohamed, Tarek Elfouly, Khaled Harras, and Z. Jane Wang. 2017. Multimodal deep learning approach for joint eeg-emg data com- pression and classification.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. ArXiv, abs/2005. 14165.
The literature lacks available published base- lines. As shown in our paper, since the task is new, there are not enough published works that provide comparable basclines. We under- stand that the comparison is important, so we implemented several baselines by ourselves, including MLP, Bi-LSTM, Transformer, and ResNet, to provide more convincing judgment and support future work in this area.
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. 2020. Graph optimal transport for cross-domain alignment. ArXiv. abs/2006.14744.
7 Ethics Statement
The goal of our study is to explore the connectivity between EEG and language, which involves human subjects' data and may inflect cognition in the brain, so we would like to provide an ethics discussion.
Jackeol Choi, Euna Jung, Jangwon Suh, and Wonjong Rhee. 2021, Improving bi-encoder document ranking models with two rankers and multi-teacher distilla- tion. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Reirieval, pages 2192-2196.
First, all the data used in our paper are publicly available datasets: K- K-FmoCon and Zuco. We did

---

J. Correia, E. Formisano, G. Valente, L. Hausfeld, B. Jansma, and M. Bonte. 2013. Brain-based transla- tion: fmri decoding of spoken words in bilinguals re- veals
Jiang-Jian Guo, Rong Zhou, Li-Ming Zhao, and Bao- Liang Lu, 2019. Multimodal emotion recognition from eye image, eye movement and eeg using deep neural networks.
semantic representations in anterior temporal lobe. Journal of Neuroscience, 34:332-338.
John Hale, Adhiguna Kuncoro, Keith B. Hall, Chris Dyer, and Jonathan Brennan. 2019, Text genre and training data size in human-like parsing, In EMNLP.
Aron Culotta, Andrew McCallum, and Jonathan Betz. 2006. Integrating probabilistic extraction models and data mining discover relations and patterns in text. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 296-303, New York City, USA. Association for Com- putational Linguistics.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 770-778.
Nora Hollenstein, Maria Barrett, and Lisa Beinborn. 2020a. Towards best practices for leveraging human language processing signals for natural language pro- cessing. In LINCR.
Pinar Demetci. Rebecca Santorella, Bjöm Sandstede.
William Stafford Noble, and Ritambhara Singh. 2020. Gromov-wassestein optimal transport align single- cell multi-omics data. bioRxiv.
Nora Hollenstein, Maria Barrett, Marius Troendle. Francesco Bigiolli, Nicolas Langer, and Ce Zhang. 2019. Advancing nlp with cognitive language pro- cessing signals. ArXiv, abs/1904.02683 02682.
Fatma Deniz, Christine Tseng, Leila Wehbe, and Jack L. Gallant. 2021. Semantic representations during language comprehension are affected by context. bioRxiv.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, abs/1810.04805.
Nora Hollenstein, Cédric Renggli, Benjamin James Glaus, Maria Barrett, Marius Troendle, Nicolas Langer, and Ce Zhang. 2021. Decoding eeg brain activity for multi-modal natural language processing. Frontiers in Human Neuroscience, 15.
Vipula Dissanayake, Sachith Seneviratne, Rajib Rana, Elliott Wen, Tharindu Kaluarachchi, and Suranga Nanayakkara. 2022. Sigrep: Toward robust wearable emotion recognition with contrastive representation learning. JEEE Access, 10:18105-18120.
Nora Hollenstein, Jonathan Rotsztejn, Marius Troen- dle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. Zuco, simultaneous eeg and eye-tracking re- source for natural sentence reading. Scientific Data, 5.
Yang Du, Yongling Xu, Xiaoan Wang, Li Liu, and Pengcheng Ma. 2022. Etst; Eeg transformer for per- identification.
Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020b. Zuco 2.0: A dataset of physi- ological recordings during natural reading and anno- tation. arXiv: 1912.00903 [cs].
Javier Fdez, Nicholas Guttenberg, Olaf Witkowski, and Antoine Pasquali. 2021. Cross-subject eeg-based emotion recognition through neural networks with stratified normalization. Frontiers in Neuroscience, 15.
Shenda Hong. Yanbo Xu, Alind Khare, Satria Pri- ambada, Kevin Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov. 2020. Holmes: Health online model ensemble serving for deep learning models in intensive care units. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1614-1624.
Chris Foster, Chad C. Williams, Olave E. Krigolson, and Alona Fyshe. 2021. Using ceg to decode semanties during an artificial language learning task. Brain and Behavior, II.
Jon Gauthier and Anna A. Ivanova. 2018 Does the brain represent words? an evaluation of brain de- coding studies of language understanding. ArXiv, abs/1806.00591.
Shih-Cheng Huang, Anuj Pareek, Roham Zamanian, Imon Banerjee, and Matthew P. Lungren. 2020. Mul- timodal fusion with deep neural networks for lever- aging et imaging and electronic health record: a case- study in pulmonary embolism detection. Scientific Reports, 10:22147.
Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. 2013. A multi-view embedding space for modeling internet images, tags, and their semantics. International Journal of Computer Vision, 106:210- 233.
Lorenz Hübschle- Schheeider and Peter Sanders. 2019. Parallel weighted random sampling. ACM Transac- tions on Mathematical Software (TOMS).
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional 1stm and other neural network architectures. Neural networks : the official journal of the International Neural Net- work Society, 18 5-6:602-10.
Effects of emotion. Emotion Review, 1:113 99. Elizabeth A. Kensinger. 2009. Remembering the details:
Thomas Kosch, Albrecht Schmidt, Simon Thanheiser. and Lewis L. Chuang. 2020. One does not simply

---

Isvp: Mental workload select speed reading pa- rameters using dletroencephalography Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
Jie-Lin Qiu, W. Liu, and Bao-Liang Lu. 2018. Multi- view emotion recognition using deep canonical cor- relation analysis. In ICONIP.
Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Der- noncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, and Hailin Jin. 2022. Mhms: Multimodal hierarchical multimedia summarization. ArXiv, abs/2204.032 033334
John Lee, Max Dabagia. Eva Dyer, and Christo- pher J. Rozell. 2019. Hierarchical optimal trans- port for multimodal distribution alignment. ArXiv, abs/1906. 1768.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeutlmoyer. 2020. Bart: Denoising peuuennc-to-sqquene pre-training for nat- ural language generation, translation, and compre- hension. In ACI.
Jingyu Quan, Yoshihiro Miyake, and Takayuki Nozawa. 2021. Incorporating interpersonal synchronization features for automatic emotion recognition from vi- sual and audio data during communication. Sensors, 21:5317.
Aniketh Janardhan Reddy and Leila Wehbe, 2021. Can fmri reveal the representation of syntactic structure in the brain? bioRxiv.
Mu Li and Bao-Liang Lu. 2009. Emotion classification based on gamma-band eeg. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference. 19991132-1322
Amma Rehman and Yasir Al Khalili, 2019. Neu- roanatomy, occipital lobe.
Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao- Liang Lu. 2019a. Multimodal emotion recognition using deep canonical correlation analysis. ArXiv, abs/1908.05339.
David Ruppert. 2004, The elements of statistical learn- ing: Data mining, inference, and prediction. Journal of the American Statistical Association, 99:567 567.
Enas S. Yousif, Azmi Shawkat Abdulbaqi. Abdulad- heem Zaily Hameed, and Saif Al-din M. N. 2020. Ellettnencephalograa signals classification based on feature normalization. IOP Conference Series: Mate- rials Science and Engineering. 928:032028.
Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. 2022. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEF Transactions on Cognitive and Developmental Systems, 14:715-
729.
Maham
Waldemar Karwowski. Farzad Vasheghani Farahani, Krzysztof Fiok, Redha Taiar, Peter A. Hancock, and Awad Al-Juaid. 2021. Neural decoding of eeg signals with machine learning: A systematic review. Brain Sciences, 11.
Saeidi,
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 20196. Roberta; robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11662.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. 2019. Inducing brain-relevant bias in natural language pro- cessing models. In NeurIPS.
Alex Murphy, Bernd Bohnet, Ryan T. McDonald, and Uta Noppeney. 2022. Decoding part-of-speech from human ceg signals. In ACL.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642. Seattle, Washington, USA. Association for Computational Linguistics.
Grigorios Nasios, Efthymios Dardiotis, and Lambros Messinis. 2019. From broca and wernicke to the neuromodulation era: Insights of brain language net- works for neurorehabllitation. Behavioural Neurol- ogy, 2019.
Ole Numssen, Danilo Bzdok, and Gesa Hartwigsen. 2021. Functional specialization within the inferior parietal lobes across cognitive domains. eLife, 10.
Brigitte Stemmer and John Connolly. 2012. The eeg/erp technologies in linguistic research.
Pablo Ortega and Aldo Faisal. 2021. Deep learning multimodal fnirs and eeg signals for bimanual grip force decoding. Journal of Neural Engineering, 18.
Mariya Toneva, Tom. Mitchell, and Leila Wehbe. 2020. Combining computational controls with natu- ral text reveals new aspects of meaning composition. biokxiv.
Cheul Young Park, Narac Cha. Soowon Kang, Auk Kim, Ahsan Habib Khandoker, Leontios Hadjileon- tiadis, Alice Oh, Yong Jeong, and Uichin Lee. 2020. K-emocon, multimodal sensor dataset for continu- ous emotion recognition in naturalistic conversations. Scientific Data, 7.
Mariya Toneva and Leila Wehhe. 2019. Interpret- ing and improving natural-languas processing (in machines) with natural waguage-processing (in the brain). ArXiv, 1911.10511333

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Zhenhailong Wang and Heng Ji. 2021. Open vocabulary elettreencephaalggaayy--eee
decoding and zero- shot sentiment classification. ArXiv, abs/21 12.02690.
Leila Wehbe, Idan Asher Blank, Cory Shain, Richard Futrell, Roger Philip Levy, Titus von der Malsburg, Nathaniel J. Smith, Edward Gibson, and Evelina Fe- dorenko. 2020. Incremental language comprchension difficulty predicts activity in the language network but not the multiple demand network. bioRxiv.
Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. 2014. Aligning context-hased statistical models of language with brain activity during read- ing.
Jennifer Williams and Leila Wehbe. 2021. havior measures are predicted by how informa- tion is encoded in an individual's brain. ArXiv, abs/21 12.06048.
Kangning Yang, Benjamin Tag, Yue Gu, Chaofan Wang. Tilman Dingler. Greg Wadley, and Jorge Goncalves. 2022. Mobile emotion recognition via multiple phys- iological signals using convoluionnaaggennte trans- former. Proceedings of the 2022 International Con- ference Multimedia Retrieval.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoe Lc. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In NeurlPS.
Siyang Yuan, Ke Bai, Liqun Chen, Yizhe Zhang, Chenyang Tao, Chunyuan Li, Guoyin Wang. Ri- cardo Henao. and Lawrence Carin. 2020. Advancing weakly supervised cross-domain alignment with opti- mal transport. ArXiv, 0bs/2008.06597..
P. Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics.
Jiacheng Zhu, Jielin Qiu, Zhuolin Yang, Douglas We- ber, Michael A. Rosenberg. Emerson Liu, Bo Li, and Ding Zhao. 2022. Geoecg: Data augmentation via wasserstein geodesic perturbation for robust electro- cardiogram prediction. In MLHC.

---

A
Three paradigms of EEG and language alignment
Cross Alignment Module
Cross Alignment Module
Cross Alignment Module
EEG Encoder
Text Encoder
EEG Encoder
Text Encoder
EBG Encoder
Text Encoder
106
105
105
840
840
theta
alpha
beta
One word
BEG
LEG
BDG
EEG for the whole
One senience
frequency bands' LIG
LIXSs
(a) Word-level
(b) Concal Ward-lovel
(c) Sentence-level
Figure 5: Three paradigms of EEG and language alignment.
B More Details about our Model
B.1 Hierarchical Transformer Encoders
Let X. E Re and X₁ R: be the two normalized input feature matrices for EEG and text, respectively, where De and D, describes the dimensions of the feature matrices. To encode the two feature vectors, we feed them to their hierarchical transformer encoders: Ve (Xx: W.): V (x;; Wt), where Egg and Et denotes the separate encoders, Ve and Vi symbolizes the outputs for the transformed low-level features and W, and W, denotes the trainable weights for EEG and text respectively. The outputs of these two encoders can be further expanded by stating Ve = [+' u "e..... enl E R" and V [ut vf ui...., rf] E R& where n and k denotes the number of instances in a given output vector and oo and of denotes the instance itself. The details about Transformer encoders are introduced in the section below.
B.2 Transformer Encoders
The transformer is based on the attention mechanism and outperforms previous models in accuracy and performance. The original transformer model is composed of an encoder and decoder. The encoder maps input sequence into latent representation, and the decoder uses the representation along with other inputs to generate target sequence. Our model only adopts the encoder, since aim at learning the representations of features.
First, we feed out the input into an embedding layer, which a learned vector representation. Then we inject positional information into the embeddings by:
(p0 sin 10000
P'(poaa2t11) cos (pos) 2000/0
(1)
The attention model contains two sub-modules, a multi-headed attention model and fully connected network. The multi-headed attention computes the attention weights for the input and produces an output vector with encoded information on how each feature should attend to all other features in the sequence. There are residual connections around each of the sub-layers followed by a layer normalization. where the residual connection means adding the multi-headed attention output vector the original positional input embedding, which helps the network train by allowing gradients to flow through the networks directly. Multi-headed attention applies a self-attention mechanism, where the input goes into three distinct fully connected layers to create the query, key, and value vectors. The output the residual connection goes through layer normalization.
In model, our attention model contains N same layers, and each layer contains two sub-layers, which multi-head self-attention model and fully connected feed-forward network. Residual connection and normalization added in each sub-layer. So the output of the sub-layer can be expressed as: Output LayerNorm(z
(aubLayer(x))) For the Multi-head self-attention module, the attention can be expressed as: attention = Attention( K, V), where multi-heal attention uses different linear transformations to project query, key, and value, which are Q, K, and V. respectively, and finally

---

concatenate different attention results:
MultHHcad(Q.K.VV Concat(head ..... heady
(2)
head; Attention(QW, , vWW)
(3)
where the projections are parameter matrices:
w&e R Amsdd de w E IR Amaidd de W.Y. E IR Ammdi du 1o E h
(4)
where the computation of attention adopted scaled dot- dot-product: Attention Q, K, V) = softmax For the output,
QK
V vig
ID convolutional layer and softmax layer to caleulate the final output.
B.3 Cross Alignment Module
As shown in Fig. 5, there are three paradigms of EEG and language alignment. For word level, the EEG features are divided by each word, and the objective of the aligmment is to find the comnectivity of different frequencies with the corresponding word. For the concat-word level, the frequencies' EEG features are concatenated as a whole, and then concatenated again to match the corresponding sentence, so the alignment find out the relationship within the sentence. As for sentence level, the EEG features are calculated as an average over the word-level EEG features. There no boundary for the word, so the alignment module tries to encode the embeddings as a whole, and explore the general representations. In the Cross Alignment Module (CAM), we introduced a new loss function in addition to the original cross-entropy loss. The new loss is based on Canonical Correlation Analysis (CCA) (Andrew et al., 2013) and Optimal Transport (Wasserstein Distance), As in Andrew al. (2013), CCA aims to concurrently learn the parameters of two networks to maximize the correlation between them. Wasserstein Distance (WD), which originates from Optimal Transport (OT), has the ability to align embeddings from different domains explore the relationship (Chen et al., 2020).
Canonical Correlation Analysis (CCA) is method for exploring the relationships between two multivariate sets of variables. It learns the lincar transformation of two vectors maximize the correlation between them, which is used in many multimodal problems (Andrew et al., 2013; Qiu et al., 2018; Gong et al., 2013). this work, we apply CCA to capture the cross-domain relationship. Let low-level transformed HEG features be Ve and low-level language features be I,, We assume (Ve, V/) E IR" x Rh= has covariances (E11, Σx2) and cross-covariance Σ1g- CCA finds pairs of linear projections of the two views, (w1 Ve, Vi) that are maximally correlated:
w/ E12us
(wf, wS) argmax corr (wi Ve, w/Vi) argmax W1,02
(5)
w Σ11 (11ti w w/ E22
our study, we modified the structure of Andrew et al. (2013) while honoring its duty by replacing the neural networks with Transformer encoders. wi and w denote the high-level, transformed weights from the low-level text and EEG features, respectively.
Wasserstein Distance (WD) is introduced in Optimal Transport (OT), which a natural type of divergence for registration problems as it accounts for the underlying geometry of the space, and has been used for multimodal data matching and alignment tasks (Chen et al., 2020; Yuan et al., 2020: Lee et al., 2019; Demetci et al., 2020; Qiu et al., 2022; Zhu et al., 2022). In Euclidean settings, OT introduces WD W(H, r), which measures the minimum effort required to "displace" points across measures μ and v, where y and are values observed in the empirical distribution. In our setting, we compute the temporal- pairwise Wasserstein Distance EEG features and language features, which are (p, v) (Ve, Vt). For simplicity without loss of generality, assume C P(X) and v E P(W) denote the two discrete distributions, formulated as - Σn uzdz, and // = > Uz0y, with ò as the Dirac function centered on x. I[(,, v) denotes all the joint distributions (x. y). with marginals (xx) and vly). The weight vectors

---

u -- E An and {u.]1=1 Am belong the 7t and m midimmesionnal simplex, respectively.
The WD between the two discrete distributions μa and defined as:
WD(,
inf ((x))) PEl((M))
min Tell(uy))
Tg (g)
(6)
5=1
where I[(,,))((CR?)) TImmu "mm 1. In denotes an dimensional all-one vector, and (274, Uj) is the cost function evaluating the distance between z; and Vj-
Loss Objective The loss objective for the CAM module can be formalized as: Loss = ICE + anloCA + a2/WD. where ag E {0, 10,1 (1, controls the weights of different parts of alignment-ban based loss objective.
Experimental Settings
C.1 Downstream Tasks
In this study, we evaluate our method two downstream tasks: Sentiment Analysis (SA) and Relation Detection (RD) of two datasets: K-EmoCon (Park et al., 2020) and ZuCo 1.0/2.0 Dataset (Hollenstein al., 2018, 2020b).
Sentiment Analysis (SA) Given a succession of word-level or sentence-level EEG features and their corresponding language, the task to predict the sentiment label. The ZuCo 1.0 dataset consists sentences from the Stanford Sentiment Treebank, which contains movic reviews and their corresponding sentiment label (i.e., positive, neutral, negative) (Socher et al., 2013). The K-EmoCon dataset categorizes emotion annotations valence, arousal, happy, sad, nervous, and angry. For each emotion, the participant labeled the extent of the given emotion felt by following Likert-scale paradigm. Arousal and valence are rated 5 (1: very low; 5: very high). Happy, sad, nervous, and angry emotions are rated 4, where means very low and means very high. The ratings are dominantly labeled as very low and neutral. Therefore to combat class imbalance, we collapse the labels to binary and ternary settings.
Relation Detection (RD) The goal of relation detection (also known relation extraction or entity association) is to extract semantic relations between entities in given text. For example, in the sentence, "June Huh won the 2022 Fields Medal.", the relation AWARD connects the two entities "June Huh" and "Fields Medal" together. The ZuCo 1.0/2.0 datasets provide the ground truth labels and texts for this task. We use texts from the Wikipedia relation extraction dataset (Culotta et al., 2006) that has 10 relation categories: award, control, education, employer, founder, job title, nationality, political affiliation, visited, and wife (Hollenstein et al., 2018, 2020b).
C.2 Datasets and Data Processing
K-EmoCon Dataset K-EmoCon (Park et al., 2020) is multimodal dataset including videos, speech audio, accelerometer, and physiological signals during a naturalistic conversation. After the conversation. each participant watched recording of themselves and annotated their own and partner's 's emotions. Five external annotators
recruited to annotate both parties' emotions, six emotions in total (Arousal, Valence, Happy, Sad, Angry, Nervous). The NeuroSky MindWave headset captured EEG signals from the left prefrontal lob (FP1) at a sampling rate of 125 Hz in 8 frequency bands: delta (0.5-2.75H)) theta (3.5-).75Hz), low-alpha 7.5-9.25522) high-alpha (10-11.75H2) low-beta (13-16.75H2) high-beta beta (18-29.75Ha) 29.75Hz), low-gamma (31539.75Hz), and middle-gamma (41-49.75Hz). We used Google Cloud's Speech-to-Texx -extt API to transcribe the audio data into text.
The ZuCo Dataset (Hollenstein al., 2018, 2020b) corpus of EEG signals and eye-tracking data during natural reading. The tasks during natural reading can be separated into three categories: sentiment analysis, natural reading, and task-specified reading. During sentiment analysis, the participant was presented with 400 positive, neutral, and negative labeled sentences from the Stanford Sentiment Treebank (Socher et al., 2013). The EEG data used in this study can be categorized into
ZuCo Dataset

---

sentence-level and word-level features. The sentence-level features are the averaged word-level EEG features for the entire sentence duration. The word-level EEG features are for the first fixation duration (FFD) of a specific word, meaning when the participant's 's eye met the word, the EEG signals were recorded. For both word and sentence-level features, 8 frequency bands were recorded at sampling frequency of 500 Hz and denoted as the following: thetal (4-6Hz), theta2 (6.5-8Hz), alphal (8.5-10Hz). alpha2 (10.5-13H2), betal (13.5-18Hz), beta2 (18.5-30Hz) gammal (30.5 -00,), and garmaa2 (40-49.5H2)
C.3 Experimental Setup
The hierarchical transformer encoders follow the standard skeleton from Vaswani et al. (2017), excluding its complexity. To avoid overfitting, we adopt the oversampling strategy for data augmentation (Hübschle- Schneider and Sanders, 2019), which ensures a balanced distribution of classes included in each batch. The train/test/validation splitting is (80%. 10%, 10%) as in Hollenstein et al. (2021). The EEG features are extracted from the datasets in 8 frequency bands and normalized with Z-score according to previous work (S. Yousif ct al., 2020; Fdez et al., 2021; Du et al., 2022) over cach frequency band. To preserve relatability, the word and sentence embeddings language models to generate text features (Devlin et al., 2019), where all texts are tokenized and embedded using the BERT-uncased-baas model. Each sentence has an average length of tokens, so we instantiate a max length of 32 with padding. In the case of word-level, we use an average length of tokens for each word and establish a max length of 10 with padding. The token vectors' from the four last hidden layers of the pre-trained model are withdrawn and averaged to get a final sentence or word embedding. These embeddings are used during the sentence-level and word-level settings. For concat word-level, we simply concatenate the word embeddings for their respective sentence. All the experimental parameters are listed in Appendix C.4.
also normalized with Z-scores. We use pre-trained
C.4 Experiment Parameters, Code and Dataset
Our model's parameters used in the experiments are listed in Table 2. Parameters with the best performance are marked in bold. Our anonymous code is available at https: http////oonnomuuuu.//// wwwpeymmouuuuuunnn.oeooeo wwwpsscceieeeeeieieeeeeeeeeee 2023 BBBDD.
Table 2: Experiment parameters used in the paper, where the best ones are marked in bold
Task
Batch Size
Encoder Layers
Att. Heads
Channel Size
Out Channel Size
Sentiment Analysis
[8, 16, 32, 64, 128]
2, 6, 12) 21, 12]
[8. 16. 32, 64, 256]
[8. 16. 32. 64, 128, 256] 16, 32, 64, 128, 256]
Relation Detection
[8, 16, 64, 128]
122.3.3........1
[8, 161 32,64, 64, 128, 256]
Task
Kernel Sizes
Dropout
Epochs
Warmup Steps
Sentiment Analysis
[1, 3, 3, 5]
00.1, 05, 0.7] [0.1, 0.3, 0.6.0.
[10, 20, 50, 100, 200]
[1000, 2000, 4000] (1000, 2000, 4000]
Relation Detection
[10, 20, 50, 100, 200)
C.5 Baselines
The area of multimodal learning of EEG and language is not well explored, and to the best of our knowledge, only Hollenstein al. (2021)'s approach was directly comparable our study. However, to make fair evaluation, we implemented the following ofate-of-the- art representative approaches as basclines for verification: MLP (Ruppert, 2004), Bi-LSTM (Graves and Schmidhuber, 2005; Zhou et al., 2016), Transformer (Vaswani et al., 2017), and ResNet (He et al., 2016).
In this section, we present implementation details for our multilayer perceptron (MLP), ResNet, and BiLSTM models during baseline retrieval. Throughout all baseline results, we used pre-trained BERT- umcased-base model extract useful features for text. In the case of EEG features, we used the signals is. Both text and EEG features were normalized with Z-score before inputting them into the models. We also used the cross-entropy loss function for all baseline results. We configure the MLP with 6 hidden layers. every step before the last output layer, we established a rectified linear unit activation function and dropout rate of 0.3. Starting from the input layer, we use a hidden layer sizes of 256, 128, and 64 for our baseline results. Our ID ResNet architecture has 34 layers (Hong et al., 2020). The BILSTM

---

model has 4 layers with size of 128 and 64, respectively. Once the initial embedding is fed into the BiLSTM model, we use a pack padded sequence function ignore the padded elements. The comparison of implementation details of baseline methods and our methods is shown in Table 3
Table 3: Implementation details for the baselines models and the comparison with our methods.
Architecture
Parameters
Model
Num. of I Layers
Size of layers
Dmpout
Batch Size
MLP-EEG
[2. 4.
[64, 128, 256]
[0.1. 0.3, 0.5]
[8. 16,
MLP-Text
[2. 4.
[64, 128, 256]
[0.1. 0.3.0
[8. 16, 32]
BILSTM-EEG
[2. 3,4]
[32, 64, 128, 256]
[0.1. 0.3, 0.3.0.3.0.55
[8. 16, 32]
BiLSTM- Text
[2.
[32, 64, 128, 256]
[0.1. 0.3, 0.5)
[8. 16, 32]
Transformer-EED
[1.2.4]
[8,
64]
[0.1. 0.3.
16,
Transfomer-Tees
2,
[8, 16, 32, 64]
10.1, 0.3,
[8. 16, 32, 64)
ResNett-Ril
[34, 50, 96]
[32, 64, 128, 256]
[0.1, 0.3, 0.5]
[8, 16, 32]
ResNet-Text
[34, 50, 96]
[32, 64, 128, 256]
[0.1, 0.3, 0.5]
[8, 16, 32)
Our--EEG
[1.
[8, 16, 32, 64]
[0.1, 0.3,
[8. 16, 32]
Ours-Text
(1, 4]
[8, 16, 32, 64]
[0.11 0.3, 0.5]
[8, 16, 32]
Ours- Ours-Multimodal
2,,
[8, 16, 32, 641
[0.1, 0.3,
[8, 16, 32, 64)
D Additional Experimental Results
D.I Experimental Results on K-EmoCon dataset
To the best of our knowledge, there is no existing work where EEG or text is used for the K-EmoCon dataset. However, other modalities such as audio, video, blood volume pulse (BVP), electrodermal activity (EDA), body temperature (TEMP), skin temperature (SKT), accelerometer (ACC) and heart rate (HR) have been used to perform sentiment analysis. As shown in Table 4, our model outperforms previous method, with even less domains' data, showing the connectivity between EEG and language and also the advantages of exploring them for downstream applications.
Table 4: Comparison of performance on K-EmoCon dataset with different physiological signals as inputs on the Sentiment Analysis task.
Model
Modalitics
Rec
F1
Acc
CNN + Transformer (Quan et al., 2021)
Video and Audio
0.693
0.712
0.725
CNN Fusion (Dissanayake et al., 2022)
ACC, BVP, EDA, TEMP
NA
0.562
0.591
Convolution-augmeette Transformer (Yang al., 2022)
BCP, EDA, HR, SKT
0.635
0.564
NA
Transformer (Yang et al., 2022)
BCP EDA, HR. SKT
0.628
0.518
NA
BiLSTM (Yang al., 2022)
BCP, EDA, HR, SKT
0.563
0.473
NA
Ours
Text, EEG
0.720
0.729
0.733
Table 5: Comparison with baselines on K-EmoCon dataset for Sentiment Analysis.
Model
Prec
Rec
F1
Acc
MLP-EEG
0.295
0.317
0.222
0.231
MI.P-Toxt
0.263
0.272
0.182
0,180
Bi-LSTM-EEG
0.340
0.354
0.226
0.220
Bi-LSTM-TeN
0,241
0.329
0.125
0.224
Transformer EEG
0.399
0.411
0.405
0.484
Transformer Text
0.454
0.492
0.472
0.443
ResNet-EEG
0.456
0.389
0.202
0.229
ResNet-Text
0.133
0.348
0.169
0.224
Ours-EEG
0.591
0,516
0.551
0.591
Ours-Text
0.524
0.561
0.509
0.542
Ours-Multimodal
0.739
0.720
0.729
0.733
In Table 5, we show the comparison results of different methods on the K-EmoCon dataset. From Table 5, we can see that our method outperforms the other baselines, and the multimodal approach outperforms the unimodal approach, which also demonstrates the effectiveness of our method.

---

D.2 Ablation results on the components in the CAM module
To further investigate the performance of different mechanisms in CAM, we carried out ablation experiments on the Zuco dataset, and the results are shown in Table 6 in Appendix D.2. The combination CCA and WD performed better compared to using only one mechanism for sentiment analysis and relation detection in all model settings. We also conducted experiments word-level, sentence-level. and concat word-level inputs, and the results also shown in Table 6. We observe that word-level BEG features paired with their respective word generally outperform sentence-level and concat word-level in both tasks.
Table 6: Ablation results on the components in the CAM module (best results in bold).
Senfence Level
Word Level
Concat Word Level
Model
Pree
Ree
F1
Ace
Pree
Rec
F1
Acc
Pree
Rec
Acc
Ours CCA- Text
0.748
0.746
0.747
0.707
0.701
0.733
0.717
0.769
0.752
0.787
0.769
0.744
Ours-CAAEEEG
0.758
0.763
0.761
0.758
0.761
0.763
0.761
0.758
0.748
0.751
0.749
0.748
Ours-CCA-All
0.790
0.765
0.777
0.793
0.791
0.783
0.787
0.793
0.767
0.778
0.773
0.778
Ours- -entWt
0.718
0.704
0.711
0.724
0.753
0.747
0.750
0.770
0.740
0.731
0.735
0.733
ZuCo (SA)
Ours BGG
0.710
Ours WD-All
0.774
0.733
0.752
0.804
0.817
0.809
0.803
0.805
0.781
0.784
0.837
0.780
Oury CCA+-DDTTe..
0.763
0.762
0.762
0.735
0.747
0.749
0.748
0.753
0.720
0.744
0.723
0.742
Ous-CCA-WDD-BEC
0.781
Ours Dus-CCA-WD-AI
0.822
0.829
0.826
0.826
0.821
0.812
0.816
0,827
0,802
0.809
0.806
0.813
Ouss-CA--Tet
0.525
0.524
0.524
0.502
0.489
0.463
0.475
0.455
0.386
0.459
0.419
0.421
Ours-CCA-EEC
0.596
0.648
0.612
0.604
0.546
0.547
0.546
0.547
0.520
0.561
0.540
0.525
Ours-CCA-All
0.624
0.651
0.620
0.617
0.638
0.646
0.642
0.633
0.596
0.606
0.600
0.599
Ous-WD-Text
0.539
0.514
0.526
0.534
0.537
0.499
0.518
0,521
0.478
0.462
0.470
0.479
ZuCo (RD)
Ours-WD- WD-BEG
0.626
0.625
0.625
0.610
0.647
0.653
0.650
0.648
0.642
0.666
0.653
0.648
OD--WWD-AIl
0.642
0.686
0.663
0.704
0.718
0.754
0.736
0,731
0.718
0.693
0.706
0.733
Oury CCA+WD-Test
0.534
0.538
0.536
0.526
0.534
0,565
0.552
0.549
0.455
0.494
0.472
0.473
MMs-CCC-WD-EEG
0.676
0.698
0.687
0.673
0.669
0.681
0.678
0.679
0.658
0.690
0.669
0.669
Ours- Ous--CCA-WD-All
0.742
0.746
0,742
0.744
0.741
0.743
0.742
0.746
0.737
0.734
0.736
0.737
D.3 Full Brain Topological Maps
In the paper, we only showed the brain topological maps for three frequency bands due page limit, here we provide full brain topological maps for all the eight frequency bands, and the results are shown Figure 6. We can observe the beta2, gammal, and gamma2 frequency bands having the most powerful signals for all words.
LV
WI
Someore
aphat
alpaa
busa
garenst
ganorad
Sustd
Figure 6: Positive and Negative word brain topologics (Sentiment Analysis)

---

D.4 1-SNE Feature Projections
order to interpret the performance improvement, we visualized the original feature distribution and the transformed feature distribution. As shown in Fig. 7, the transformed feature distribution makes better clusters than the original one. The features learned by CAM can be more easily separable, showing the effectiveness of discovering and encoding the relationship between EEG and language. Figures 8,9,10 show more t-SNE projection results of the K- K-EmoCon dataset on Sentiment Analysis task.
Plen BBC
Powe That
PLace
-
-00
CCA CEG
CCA Tuadon
-00
EEG
Tasi
WD EEG
WO Teat
VD Fusor
Figure 7: TSNE projection comparison of untransformed & transformed features of ZuCo dataset, where different colors represent different classes.
Figure 8: Transformed feauure projections of K-EmoCon dataset on Sentiment Analysis, where different colors represent different classes.
1100
-004
ock teo
OCA
Goa
-5
-0 -3
Transformed feature projections of ZuCo Figure 10: Transformed feature projections of ZuCo dataset on Sentiment Analysis, word-level, where dif- ferent colors represent different classes.
Figure 9:
dataset on Sentiment Analysis, concat word-level. where different colors represent different classes.
D.5 Sentence-level Alignment
Figure 11 shows the negative and positive sentence-level alignment weights of ZuCo dataset. In Figure/ 11, we can find that alphal, betal,and gammal frequency bands show larger different response between negative and positive sentences.
Beariaay crtied...
025
0.00
Alamo smorii....
-0.25
thebx1
thelac
alphal
atchal)
swaal
bebs?
gamma11 gaimaa
Figure 11: Negative and Positive sentence-level alignment of ZuCo dataset.
D.6 Baseline Results
this section, we provided baseline results that directly used either EEG, language, or fusion as input for the downstream applications. The results are shown in Table 7 and Table

---

Table 7: Baseline results
Dataset
Modal
Task
Senlence Level
Word Lavel
Concat Word Lavel
Prec
Re:
PT
Aue
Plec
Rec
F1
Ace
Fiet
Rec
F1
Ace
Tranformnc-
Sentiment Amalsii. Tornry
0.548
0.546
0.550
7aCo
TrandeemarPPe Transforme Text
Sentinent Auilysis, Tersary
Relation Detection
0:428
0.639
0:187
0.662
0.164
0.120
0.662
0.188
0.624
0.191
0.630
0.189
0.627
0:487
0.30L
0.298
0.300
0.300
Transfomuu-FE FFG
Rolation Detection
0.502
0:440
0.468
0,479
0.339
0,34]
0.340
0.358
0.310
0.316
0.313
0.315
Transformer-
Sentimeut Andlyiis Termary
0.454
0,472
0.443
K-EmoCon
Tranfeemm=FFC
Senient Analysis, Terrary
0.309
Trandormer-
Sentiment Analy Binury
0.597
0.557
0.500
Transformee EEG
Sentien Analysis, Binayy
0.814
0.877
0.808
0.890
Table 8: Baseline fusion results.
Sentence Level
Word Level
Concat Word Level
Dutaset
Model
Proc
Prec
7aCo
Fusion
Sentiment Analysis, Ternry
0.674
0.686
0,680
0.677
0.706
0.707
0.707
0.706
0.654
0.671
0.662
0.674
Fusion
Relation Detection
0.541
0.491
0.489
0.398
0.402
K. EmeCon
Fusion Sentiment Analysis, Ternrry
0,512
0.520
0,516
0.515
Pasion
Sentinent Analysis, Rinary
0.982
0.933
0.957
E More Related Work
Multimodal Learning of EEG and Other Domains EEG signal is a popular choice as modality in multimodal learning. Ben Said et al. (2017) used EMG signals jointly with EEG in bi-autoencoder architecture and increased accuracies for sentiment analysis. Bashar (2018) integrated ECG and EEG signals in a human identification task, where fused classifiers produced the highest score. Liu al. (2019a, 2022); Bao et (2019) extracted correlated features between EEG and eye movement data for emotion classification, showing transformed features are more homogeneous and discriminative. Guo et al. (2019) collected eye images, eye movement data, and EEG signals, and encoded the three modalities with a bimodal deep autoencoder. Ortega and Faisal (2021) fed INIRS and EEG to decode bimanual grip force and resulted in increased performance, compared to single modality models. There are also efforts to find correlations between EEG and visual stimulus frequencies (Sacidi et al., 2021). A common theme occurring among these works showed EEG paired with other domains could boost performance.
Multimodal Learning of Language and Other Brain Signals Recently, language and cognitive data were also used together in multimodal settings to complete desirable tasks (Wang and Ji, 2021; Hollenstein et al., 2019, 2021, 2020a). Wehbe et al. (2014) used a recurrent neural network to perform word alignment between MEG activity and the generated word embeddings. Toneva and Wehbe (2019) utilized word-level MEG and fMRI recordings to compare word embeddings from large language models. Schwartz et al. (2019) used MEG and fMRI data to fine-tune BERT language model (Devlin et al., 2019) and found that the relationships between these two modalities were generalized across participants. Huang et al. (2020) leveraged CT images and text from electronic health records to classify pulmonary embolism cases and observed that the multimodal model with late fusion achieved the best performance. Murphy al. (2022) found semantic categories between MEG and language. However, the relationship between language and EEG has not been explored before.
Multimodal Learning of EEG and Language Hale et al. (2019) related EEG signals to the states of a neural phrase structure parser and showed that through EEG signals, models were correlating syntactic properties specific genre of text. Foster al. (2021) applied EEG signals to predict specific values of each dimension word vector through regression models. Wang and Ji (2021) used word-level EEG features to decode corresponding text tokens through an open vocabulary, sequence-to-sequemn framework. Hollenstein et al. (2021) focused on a multimodal approach by utilizing a combination of EEG, eye-tracking, and text data to improve NLP tasks. They used a variation of LSTM and CNN to decode the EEG features, but did not explore the relationship between EEG and language. Their proposed multimodal framework follows the bi-encoder approach (Choi et al., 2021) where the two modalities are encoded separately (Hollenstein et al., 2021).


Reviews:
[{"review_id": "inGGfZHfgf", "reviewer": "Reviewer_fE1A", "summary": "", "questions": "Have mentioned earlier.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! We appreciate it for recognizing our work “The paper is of course one of the firsts, to relate EEG to language. The merit of being the first in the area seems to be useful in the first sight”. Please find our detailed response to your concerns outlined below.\n\n1. ***Relationship of the brain signals such as EEG to human language***\n\n- Thank you for the valuable question!\n- Concerning the enhanced outcomes from the fusion of EEG and text data compared to the results of each modality individually, this is a key observation outlined in our work. Furthermore, it's noteworthy that the reported outcomes solely based on EEG data demonstrate remarkable performance, potentially indicating the quality of EEG signals within this dataset is very good. These signals could already offer utility in subsequent classification tasks. However, it's important to highlight that our primary objective doesn't revolve around introducing a model to achieve state-of-the-art results. Instead, our central goal is to uncover the inherent relationship between EEG patterns and human language, showcasing the potential for exploration using computational models.\n- Regarding our analysis, we have observed a distinct functional division where Broca's area is implicated in speech production, while Wernicke's area is linked to speech comprehension. These conclusions are based on the outcomes of our experimental investigations. It's worth acknowledging, however, that the field of research surrounding this matter might not have reached a definitive consensus. In fact, multiple researchers have put forth contrasting hypotheses. Pulvermüller (2018), Strijkers & Costa (2016), Pylkkänen (2019), and Friederici (2011) [1-4] have each proposed brain-language models that suggest both frontal regions, like Broca's area, and temporal regions, like Wernicke's area, play active roles in both speech production and perception. Conversely, Matchin and Hickok (2020) [5] propose an alternative viewpoint, suggesting that Broca's area is exclusively involved in production and not perception, whereas temporal regions are posited to contribute to both production and perception.\n-  As discussed in [6], Kensinger outlined the general tendency for negative events to be more memorable compared to positive events. Expanding upon the concepts presented in [1], it is suggested that negative words possess the capability to create stronger and enduring memories than positive words. Consequently, these negative words might exhibit heightened activation within the occipital and inferior parietal lobes.\n- In our examination, we identified heightened activation in both the occipital lobes and slightly extending over the inferior parietal lobes in response to negative words, similar to the patterns observed for positive words. These specific brain regions have been previously associated with functions such as visual processing, visuospatial attention, and semantic memory.\n- The intricate connection between brain signals like EEG and human language is undeniably complex. To address this complexity, we introduced a computational approach with the intention of delving into this relationship and offering a novel avenue for its exploration.\n\n>[1] Friederici, A. D. (2011). The brain basis of language processing: From structure to function. Physiological Reviews, 91(4), 1357–1392.\nMatchin, W., & Hickok, G. (2020). The Cortical Organization of Syntax. Cerebral Cortex, 30(3), 1481–1498.  \n> [2] Pulvermüller, F. (2018). Neural reuse of action perception circuits for language, concepts and communication. Progress in Neurobiology, 160, 1–44.  \n> [3] Pylkkänen, L. (2019). The neural basis of combinatory syntax and semantics. Science, 366(6461), 62–66.  \n> [4] Strijkers, K., & Costa, A. (2016). The cortical dynamics of speaking: present shortcomings and future avenues. Language, Cognition and Neuroscience, 31(4), 484–503.  \n> [5] Matchin, W., & Hickok, G. (2019). The Cortical Organization of Syntax. Cerebral cortex.  \n> [6] Kensinger, Elizabeth A.. “Remembering the Details: Effects of Emotion.’’ Emotion Review 1 (2009): 113 - 99.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is of course one of the firsts, to relate EEG to language. Codes are available and as far as I checked they look clear and enough for repro. So the merit of being the first in the area seems to be useful in the first sight. ", "reasons_to_reject": "How you argue table 1, in the sense that values reported even with only EEG are very high, and together with text they become close to 100 % accuracy on SA and SD tasks?\nThere are speculations about relationship of the brain signals such as EEG to computerized information as in the tokens of the language. There is a little information on how these signals can expose information about what the person is thinking or talking. \nI would expect to see some discussions on this aspect at least to show some human studies or refer for interested readers.", "questions_to_authors": "Have mentioned earlier.", "paper_topic_and_main_contributions": "Authors propose a transformer-based multi-modal model that exploits language and EEG information to perform downstream tasks, such as sentiment analysis and relation detection. The study claims to be the first, after a prior work that is relatively related, cited as y Hollenstein et al. (2021)’. This is as baseline, and some other works prior to 2016 are also used to compare the method with them.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA as far as I know.", "Typos_Grammar_Style_and_Presentation_Improvements": "NA as far as I know."}, "extra_text": {}}, {"review_id": "ChtzrQSMDN", "reviewer": "Reviewer_9CNo", "summary": "", "questions": "A. How are multimodal features incorporated into MTAM?\n\nB. Do you think this architecture is particularly well-suited for EEG? What are the benefits of specifically analyzing EEG data, rather than using datasets in other neuroimaging modalities (with perhaps a larger number of baselines)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback and encouraging review! Please find our detailed response to your concerns outlined below.\n\n1. ***Clarity of contributions***\n\nWe would like to emphasize our primary contributions and provide some clarifications:\n\n> ***Our contributions.***\n\n- Our main goal is to use computational models to discover the connection between language and brain signals, which proves the existence and shows meaningful findings related to cognition.\n\n> ***What is NOT our main contribution***\n\n- Developing the most advanced or elaborate model to attain state-of-the-art (SOTA) outcomes through EEG and text-based classification.\n- While our current model does outperform baselines, our primary emphasis is not centered around achieving optimal classification results. Instead, our intention is to employ these outcomes to illustrate the genuine presence of a connection between EEG patterns and human language. We aim to demonstrate that computational models offer a valuable avenue for exploring this connection.\n\n\n2. ***\"Ours-EEG\", \"Ours-Text\", and \"Ours-Multimodal\"***\n\nWe appreciate your inquiry. Just like demonstrated in Figure 2 of the paper, our proposed approach encompasses two distinct encoders: one dedicated to EEG processing, and another for text processing. In this context, the terms \"Ours-EEG\" and \"Ours-Text\" refer to instances where the model exclusively receives EEG or text inputs respectively. Conversely, \"Ours-Multimodal\" indicates that the model processes both inputs simultaneously. By examining \"Ours-EEG\" and \"Ours-Text,\" we conduct an ablation study to assess the impact of utilizing only one modality input. This comparison helps unveil insights into our findings. We will add more clarifications in the Appendix in our revision.\n\n3. ***Soudness and novelty of conclusions***\n\n> How positive and negative aonnotations were labeled?\n\nThe annotations indicating positive and negative sentiments were provided by the authors of the datasets [1-2] that we employed in our research. It's important to note that we didn't undertake the task of labeling the data personally.\n\n[1] Park, C.Y., Cha, N., Kang, S., Kim, A., Khandoker, A.H., Hadjileontiadis, L.J., Oh, A.H., Jeong, Y., & Lee, U. (2020). K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations. Scientific Data, 7.  \n[2] Hollenstein, N., Troendle, M., Zhang, C., & Langer, N. (2019). ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation. International Conference on Language Resources and Evaluation. \n\n> How the set of active frequency bands were determined for each word category (e.g., what level of increase in activity was deemed a reliable difference?)\n\nAs detailed in Section 4.3, we meticulously examined all the provided frequency bands of the dataset, as evidenced in Appendix Section D.3. Our objective was to visualize the brain regions exhibiting the most significant activity, particularly within the regions of interest – Wernicke's and Broca's areas. These areas were specifically chosen due to their prominent involvement in language processing. After scrutinizing the complete brain topological maps, it became evident that beta2, gamma1, and gamma2 frequencies exhibited the highest amplitude in microvolts within these regions. Consequently, we opted to showcase these three frequencies as representative maps in our paper.\n\n> How brain areas were segmented\n\nIn terms of the segmentation of the brain areas, we closely followed the description of credible sources [3-5] for the locations of the areas of interest (Broca's area, Wernicke's area, occipital lobes, inferior parietal lobes).\n\n[3] University of California San Francisco. “Speech & Language.” Memory and Aging Center, 2019, memory.ucsf.edu/symptoms/speech-language.  \n[4] “Inferior Parietal Lobule - an Overview | ScienceDirect Topics.” Sciencedirect.com, 2014, www.sciencedirect.com/topics/neuroscience/inferior-parietal-lobule.  \n[5] “Brain Map: Occipital Lobes | Queensland Health.” Www.health.qld.gov.au, 22 Jan. 2021, www.health.qld.gov.au/abios/asp/boccipital.\n‌\n\n\n4. ***How are multimodal features incorporated into MTAM?***\n\nPlease refer to Figure 1 in our paper. Our model intricately investigates EEG and language features through the utilization of two encoders. On the left, we have the EEG encoder, while on the right, the language encoder is depicted. To establish connections and relationships between the two domains, we introduce the cross-alignment module. This module effectively learns the connectivity and associations between EEG and language, resulting in the generation of multimodal features.\n\n5. ***Do you think this architecture is particularly well-suited for EEG? What are the benefits of specifically analyzing EEG data, rather than using datasets in other neuroimaging modalities (with perhaps a larger number of baselines)?***\n\n- Thank you for the question. Our choice to focus on EEG signals stems from its simplicity as a means of uncovering brain activity. Collecting signals from sources like brain-MRI can be considerably more expensive, posing potential challenges for subsequent research. Furthermore, EEG signals hold a paramount position in cognitive and computational neuroscience studies, being the most extensively utilized brain signal. This very reason underpins our decision to employ EEG as the material in our work, representing the first effort to delve into the intricate interplay between brain signals and human language via a computational model.\n- Another crucial aspect we wish to emphasize is the availability of a substantially larger number of existing EEG datasets in comparison to neuroimaging datasets. Within our specific context, it's essential to collect brain signals in tandem with human language during the course of human subject experiments. This simultaneous collection of brain signals and human language is conducted within highly granular timestamps to ensure a detailed alignment between the two. \n- Under the stringent criterion above, a significant proportion of the available EEG datasets fall short of meeting these specific requirements. Moreover, the situation is even more constrained for neuroimaging datasets, which primarily encompass brain signals without the incorporation of human languages. Even within the EEG modality, our research has led us to identify merely two publicly accessible datasets that conform to these stringent prerequisites – these are precisely the two datasets [6,7] that we've utilized in our study.\n- We appreciate the reviewer for the great question. Taking feasibility into careful consideration, EEG emerges as the optimal choice for conducting this study.\n\n> [6] Park, C.Y., Cha, N., Kang, S., Kim, A., Khandoker, A.H., Hadjileontiadis, L.J., Oh, A.H., Jeong, Y., & Lee, U. (2020). K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations. Scientific Data, 7.  \n> [7] Hollenstein, N., Troendle, M., Zhang, C., & Langer, N. (2019). ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation. International Conference on Language Resources and Evaluation.\n\n6. ***Terminology: \"connectivity\"***\n\nThank you for the question. In our context, the term \"connectivity\" pertains to the interrelation and interdependence between EEG signals and language processing (L006-L007). Given the innovative nature of our research, which delves into the uncharted territory of exploring the link between EEG and human language, it's possible that an exact term from existing literature might not precisely encapsulate our setting.\n\n7. ***Reproducibility***\n\nWe would respectfully inquire whether the reviewer identifies any aspects they believe are insufficient for reproducing our work. The code and data necessary for replication can be accessed through an anonymous link: [https://anonymous.4open.science/r/EMNLP_2023-B08D/](https://anonymous.4open.science/r/EMNLP_2023-B08D/) (mentioned in paper lines L033-L034). This link encompasses the complete setup, including environment configuration, raw and processed data, along with scripts for processing, training, testing, and visualization. We are confident that these resources facilitate result reproduction. If there are any additional materials you require, please do not hesitate to inform us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper includes a comparison to multiple baseline models. The results show better predictions for the authors' model across different task conditions.", "reasons_to_reject": "Clarity of contributions:\nIn general, it is unclear what the presented results tell us about language processing in the brain, or about language representations in artificial models. For instance, it is unclear what the significance of the MTAM architecture is. What is the purpose of comparing different alignment mechanisms, and of using this particular model architecture? Without further analyses/discussion, the contribution of the model itself is unclear. Although the improvement in performance holds over a number of task conditions, without further analyses/discussion SoTA performance alone does not clarify the purpose of this model architecture. Moreover, given that most of the baselines were not developed for this task, it is unclear whether the improvement is itself a substantial improvement.\n\nClarity of method:\nThe model that the authors propose is relatively straightforward, but the explanation of the model (particularly in the main paper) is unclear. For instance, in Table 1 the authors reference three versions of their model (\"Ours-EEG\", \"Ours-Text\", and \"Ours-Multimodal\"), but I was not able to find an explanation of the differences in the inputs for each of these models. Does \"Ours-Multimodal\" incorporate multimodal stimulus features in the Text Encoder of MTAM, or does \"Our-Multimodal\" refer to a different type of model? \n\nSoudness and novelty of conclusions:\nThe authors make some claims as to the brain areas and EEG frequency bands that are aligned/activated in response to positive and negative words. However, the descriptions of the analysis method are too vague to determine whether the conclusions are sound. For instance, it is unclear how positive and negative connotations were labeled, how the set of active frequency bands were determined for each word category (e.g., what level of increase in activity was deemed a reliable difference?), and how brain areas were segmented.", "questions_to_authors": "A. How are multimodal features incorporated into MTAM?\n\nB. Do you think this architecture is particularly well-suited for EEG? What are the benefits of specifically analyzing EEG data, rather than using datasets in other neuroimaging modalities (with perhaps a larger number of baselines)?", "paper_topic_and_main_contributions": "This paper proposes a model for predicting stimulus features (e.g, emotion categories and relations between entities) from EEG brain responses. The model embeds stimulus words with a pre-trained language model (BERT-base), uses a series of transformer blocks to encode the stimulus word embeddings and EEG recordings, and then predicts stimulus features from the outputs of the transformer blocks. The authors compare to a set of other models, and show that their models performs better on the task of predicting stimulus features. The model includes a module that computes an alignment between EEG and stimulus words, and the authors analyze this module to determine the types of words that are most \"aligned\" to different sets of EEG frequency bands. The authors also perform analyses to determine which EEG frequency bands contribute most to prediction performance, and which brain areas are activated in response to more positive or more negative words.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In general, the methods are underspecified (please see \"Reasons to Reject\" for more details).\n\nSome of the terminology is unclear. For instance, the term \"connectivity\" is used many times. Given other popular uses of the word \"connectivity\" in the context of neuroimaging data (e.g., \"functional connectivity\" or \"model connectivity\"), this term is a bit misleading, especially without a definition."}}, {"review_id": "2zIvgGVo9G", "reviewer": "Reviewer_VHS7", "summary": "", "questions": "1. It was confusing that words that can be used negatively or positively are included in the negative or positive: e.g. an \"about\" in Negative and an \"a\" in Positive. What does this mean and what is the interpretation? I found it very difficult to understand how Figures 2 to 4 were made and what the authors were trying to say (partly because of no statistical analysis as I described above).\n2. How the authors preprocessed EEG features? It is surprising to obtain near perfect accuracy using EEG (even exceeds that of text information). Could factors other than brain activity be contributing? Is there any possibility that potential artifacts contributed? I would like to see a discussion on how the artefacts are being removed.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We appreciate it for recognizing our work “A multi-modal alignment between brain features and text description has been rightly emphasized for its practical relevance”. Please find our detailed response to your concerns outlined below.\n\n1. ***Statistical analysis of Figs 2, 3, 4, and more clarifications***\n\n- Thank you for the suggestion.\n- Figures 2 and 3 illustrate the alignment between EEG signals and individual words at the word level. Given that the datasets solely offer sentence-level annotations and lack word-level annotations, we proceeded by presuming that entire sentences are labeled as \"Positive/Negative.\" However, we acknowledge that certain words likely contribute more informative value than others within these sentences. For instance, words like \"romantic\" are likely more relevant for EEG alignment than generic words like \"be.\" Regrettably, due to the absence of granularity in the dataset annotations, we were compelled to assume uniform labeling (\"Positive/Negative\") for all words within sentences. It's important to note that this approach may not capture nuances accurately and could be considered a limitation originating from the dataset's characteristics.\n-To enhance the clarity of our explanations, we have introduced an additional statistical analysis focused on the alignment of words labeled as \"Positive/Negative\" at the word level. We present the outcome of this analysis in the following Table A.\n\nTable A. Statistical analysis of EEG-word alignment.\n|Method | theta1 | theta2 | alpha1 | alpha2 | beta1  | beta2 |gamma1 |gamma2|  \n|:--------: |:--------:| :--------:|:--------:|:--------:|:--------:|:--------:|:--------:| :--------:|\n|Negative| -0.16| -0.32| -0.15| -0.09| -0.18| 0.45| 0.62| 0.67| \n|Positive| -0.18| -0.35| -0.13| -0.08| -0.29| 0.08| 0.39| 0.43|\n|Difference (Negative - Positive)| 0.02| 0.03| -0.02| -0.01| 0.11| 0.37| 0.23| 0.22|\n\n- Table A presents the averaged results of EEG-word alignment across 8 frequency bands, obtained by averaging over 100 EEG-sentence pairs. The analysis highlights noteworthy differences in the alignment outcomes for the \"Positive\" and \"Negative\" categories, particularly within the beta2, gamma1, and gamma2 frequency bands. These findings suggest that these specific frequency bands might hold significance, as they exhibit more pronounced differences between the positive and negative sentiments. This observation could potentially stem from the anticipated rise in theta band power during heightened language processing activities, coupled with the theta band's known association with the retrieval of semantic memories.\n\n2. ***EEG preprocessing and EEG results***\n\n- Regarding the K-Emocon dataset, it's worth noting that the authors of the original work did not disclose the precise preprocessing techniques applied to the EEG waves due to \"privacy concerns extending beyond the scope of shared information\" [1]. However, we are aware that the EEG signals were captured using the Neurosky device. This device inherently conducts internal preprocessing on raw EEG signals to derive attention and meditation levels, along with brainwave band information. Furthermore, the authors incorporated a Fast Fourier Transform analysis to the EEG signals. In our approach, we followed a similar path. We began by normalizing the amplitudes of the EEG signals using the Z-score technique for each frequency. This normalization process was adopted from previous research [2, 3, 4].\n- In the case of the ZuCo Dataset, the original authors performed a comprehensive preprocessing of the EEG signals utilizing various tools and techniques. Notably, they employed Automagic [5] for initial preprocessing steps. To ensure data quality, they detected and removed problematic electrodes that exhibited noise, flatline, or low-frequency signals. The clean_rawdata function from the EEGLab library was used for this purpose. Further processing included high-pass filtering the EEG signals at 0.5 Hz and applying a notch filter in the range of 49-51 Hz to eliminate power line interference. Eye-related artifacts were addressed by regression of the EOG channels. The authors also integrated the MARA tool to automate the rejection of artifacts, and any electrodes presenting issues were interpolated. A critical alignment step was carried out by synchronizing the EEG data with eye-tracking information using the \"EYE EEG extension,\" a process crucial for precise temporal alignment. For a more in-depth analysis, the EEG signals underwent frequency band filtering, followed by a Hilbert transform to derive power dynamics. A notable analysis point was the comparison of power between frontal left and right electrodes. Stringent criteria were established to identify and exclude artifacts exceeding ±90μV. Following these steps, akin to the K-EmoCon dataset and prior research [2, 3, 4], we proceeded to normalize the preprocessed EEG signals using the Z-score method.\n- The unexpectedly high performance achieved by the EEG signals in our results has sparked our curiosity as well. We postulated that this success might be attributed to the high quality of the collected EEG signals present in the datasets. Additionally, the relatively modest size of the EEG datasets could potentially contribute to the favorable outcomes we observed. This combination of high-quality data and a manageable dataset size likely plays a role in enabling our transformer-based architecture to excel in performance. We are of the opinion that the notable results observed on the Zuco dataset can primarily be attributed to the meticulous collection and processing of the EEG data. In comparison to the baseline established by Hollenstein et al. (2021), where a single RNN/CNN model yielded scores surpassing 70%, the introduction of more powerful transformer architectures naturally holds the potential for even more impressive outcomes. It's worth noting that the K-EmoCon dataset might entail more inherent noise when compared to the Zuco dataset, which could contribute to the slightly lower achieved score of around 70%. This suggests the existence of opportunities for further enhancement in our results.\n\n> [1] Park, Cheul Young, et al. “K-EmoCon, a Multimodal Sensor Dataset for Continuous Emotion Recognition in Naturalistic Conversations.” Scientific Data, vol. 7, no. 1, 8 Sept. 2020, https://doi.org/10.1038/s41597-020-00630-y.  \n> [2] Du, Yang, et al. “EEG Temporal–Spatial Transformer for Person Identification.” Scientific Reports, vol. 12, no. 1, 23 Aug. 2022, https://doi.org/10.1038/s41598-022-18502-3.  \n> [3] Fdez, Javier, et al. “Cross-Subject EEG-Based Emotion Recognition through Neural Networks with Stratified Normalization.” Frontiers in Neuroscience, vol. 15, 3 Feb. 2021, https://doi.org/10.3389/fnins.2021.626277. Accessed 27 Aug. 2021.  \n> [4] S. Yousif, Enas, et al. “Electroencephalogram Signals Classification Based on Feature Normalization.” IOP Conference Series: Materials Science and Engineering, vol. 928, no. 3, 1 Nov. 2020, p. 032028, https://doi.org/10.1088/1757-899x/928/3/032028.  \n> [5] Automagic [https://github.com/methlabUZH/automagic](https://github.com/methlabUZH/automagic) \n\n3. ***Missing References***\n\nThank you for pointing it out! We will add it in our revision!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A multi-modal alignment between brain features and text description has been rightly emphasized for its practical relevance.\n2. The paper demonstrates that the accuracy in downstream tasks exceeds other non-alignment methods.\n3. The authors have provided the code for analyses.\n", "reasons_to_reject": "The interpretation section may benefit from further clarity and depth. \n1. While qualitative interpretations are present in Figs 2, 3, and 4, it is hard to understand the implications of these figures. It is nice to include statistical analysis (currently nothing) to support the authors' argument. \n2. Furthermore, although the authors provided (only qualitative) interpretations about brain function from the obtained EEG topography, it is well known that such an locational interpretations are inherently difficult in EEG. This is a limitation of this study. \n", "questions_to_authors": "1. It was confusing that words that can be used negatively or positively are included in the negative or positive: e.g. an \"about\" in Negative and an \"a\" in Positive. What does this mean and what is the interpretation? I found it very difficult to understand how Figures 2 to 4 were made and what the authors were trying to say (partly because of no statistical analysis as I described above).\n2. How the authors preprocessed EEG features? It is surprising to obtain near perfect accuracy using EEG (even exceeds that of text information). Could factors other than brain activity be contributing? Is there any possibility that potential artifacts contributed? I would like to see a discussion on how the artefacts are being removed.\n", "paper_topic_and_main_contributions": "In this paper, the authors present an approach by aligning EEG and text information to create new features, subsequently utilizing these aligned features to carry out downstream tasks. \nThe authors conclude that the accuracy of downstream tasks has been enhanced in comparison to other methods that do not employ alignment, and provide some interpretations regarding the aligned EEG features.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Ren and Xiong, ACL 2021: Methods for dealing with both cognitive indicators (EEG) and text. The accuracy of this paper should also be included in the comparison.\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The attainment of near perfect accuracy using EEG (even exceeds that of text information) is, to be honest, super surprising given that brain signal from EEG is known as very noisy measurement. A discussion on the author's rationale behind this remarkable result would be helpful. \n"}}, {"review_id": "UNAu6SVDeC", "reviewer": "Reviewer_HtV4", "summary": "", "questions": "Here, I copy and past the questions covered in Reasons To Reject to make authors to readily state the answers. \n\nQ1. Language is discrete data and EEG is continuous data. How did you explain this difference to align between them?\n\nQ2. EEG is highly random in nature and very subjective between people which leads to hard to extract the consistent features. How did you deal with this problem?\n\nQ3. Why the authors consider CCA and WD? How about KL divergence or Euclidean distance?\n\nQ4. How did you combine CCA and WD? This explanation should be included in main paper, not Appendix.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We appreciate it for recognizing our work “Initial work to explore the fundamental relationship and connectivity between EEG and language through computational multimodal methods”. Please find our detailed response to your concerns outlined below.\n\n1. ***Language is discrete data and EEG is continuous data. How did you explain this difference to align between them?***\n\n- Thank you for the question.  We incorporated this component in both Appendix Section A and Section B.3, as a result of page limit. Illustrated in Figure 5 within the Appendix are three EEG and language alignment paradigms: word level, concatenated word level, and sentence-level.\n- For the word level approach, the EEG features are segregated on a per-word basis, aiming to establish connections between different frequencies and their corresponding words. On the other hand, in the concat-word level approach, the EEG features from the 8 frequencies are combined into a unified entity, then concatenated again to correspond with the sentence. This alignment strategy seeks to uncover relationships within the sentence structure. For the sentence level perspective, the EEG features are computed as an average across the word-level EEG features. In this context, the alignment module operates without word boundaries, endeavoring to encode the embeddings as a cohesive entity, thereby delving into overarching representations.\n\n2. ***EEG is highly random in nature and very subjective between people which leads to hard to extract the consistent features. How did you deal with this problem?***\n\n\n- Regarding the K-Emocon dataset, it's worth noting that the authors of the original work did not disclose the precise preprocessing techniques applied to the EEG waves due to privacy concerns extending beyond the scope of shared information [1]. However, we are aware that the EEG signals were captured using the Neurosky device. This device inherently conducts internal preprocessing on raw EEG signals to derive attention and meditation levels, along with brainwave band information. Furthermore, the authors incorporated a Fast Fourier Transform analysis to the EEG signals. In our approach, we followed a similar path. We began by normalizing the amplitudes of the EEG signals using the Z-score technique for each frequency. This normalization process was adopted from previous research [2, 3, 4].\n- In the case of the ZuCo Dataset, the original authors performed a comprehensive preprocessing of the EEG signals utilizing various tools and techniques. Notably, they employed Automagic [5] for initial preprocessing steps. To ensure data quality, they detected and removed problematic electrodes that exhibited noise, flatline, or low-frequency signals. The clean_rawdata function from the EEGLab library was used for this purpose. Further processing included high-pass filtering the EEG signals at 0.5 Hz and applying a notch filter in the range of 49-51 Hz to eliminate power line interference. Eye-related artifacts were addressed by regression of the EOG channels. The authors also integrated the MARA tool to automate the rejection of artifacts, and any electrodes presenting issues were interpolated. A critical alignment step was carried out by synchronizing the EEG data with eye-tracking information using the \"EYE EEG extension,\" a process crucial for precise temporal alignment. For a more in-depth analysis, the EEG signals underwent frequency band filtering, followed by a Hilbert transform to derive power dynamics. A notable analysis point was the comparison of power between frontal left and right electrodes. Stringent criteria were established to identify and exclude artifacts exceeding ±90μV. Following these steps, akin to the K-EmoCon dataset and prior research [2, 3, 4], we proceeded to normalize the preprocessed EEG signals using the Z-score method.\n\n> [1] Park, Cheul Young, et al. “K-EmoCon, a Multimodal Sensor Dataset for Continuous Emotion Recognition in Naturalistic Conversations.” Scientific Data, vol. 7, no. 1, 8 Sept. 2020, https://doi.org/10.1038/s41597-020-00630-y.\n> [2] Du, Yang, et al. “EEG Temporal–Spatial Transformer for Person Identification.” Scientific Reports, vol. 12, no. 1, 23 Aug. 2022, https://doi.org/10.1038/s41598-022-18502-3.  \n> [3] Fdez, Javier, et al. “Cross-Subject EEG-Based Emotion Recognition through Neural Networks with Stratified Normalization.” Frontiers in Neuroscience, vol. 15, 3 Feb. 2021, https://doi.org/10.3389/fnins.2021.626277. Accessed 27 Aug. 2021.    \n> [4] S. Yousif, Enas, et al. “Electroencephalogram Signals Classification Based on Feature Normalization.” IOP Conference Series: Materials Science and Engineering, vol. 928, no. 3, 1 Nov. 2020, p. 032028, https://doi.org/10.1088/1757-899x/928/3/032028.  \n> [5] Automagic [https://github.com/methlabUZH/automagic](https://github.com/methlabUZH/automagic) \n\n3.  ***Why the authors consider CCA and WD? How about KL divergence or Euclidean distance?***\n\n- Thank you for the question.\n- WD can be conceptualized as a metric representing the minimal \"cost\" required to transform one distribution of points into another. Conversely, CCA identifies a collection of canonical variates – which are orthogonal linear combinations of variables within each set – that optimally elucidate variability within and between sets. In broad terms, WD primarily emphasizes disparities in distributions, whereas CCA centers on canonical variables, thus exhibiting synergistic benefits when employed in conjunction.\n- We experimented with the Euclidean distance; however, the outcomes were unsatisfactory. Consequently, we chose not to include these results in the paper.\n- When juxtaposed with WD and CCA, KL divergence might introduce added complexity. Given that we've already incorporated WD and CCA as two complementary loss objectives within our existing framework, we decided against introducing a third loss. However, we acknowledge the potential of KL divergence and plan to explore its applicability in our future research.\n\n4. ***How did you combine CCA and WD? This explanation should be included in main paper, not Appendix.***\n\nThank you for the question. We detailed the entire loss objective, which outlines the integration of CCA and WD, in Appendix pages L750-L752, constrained by page limitations. We will add it to the main paper in our revision. Thank you very much for the suggestion.\n\n\n5. ***More introduction of physiological signals***\n\nThank you for the suggestion! In our revision, we will expand on the introduction of the concept of physiological signals, encompassing distinctions between EEG and MEG, as well as providing additional details about the datasets employed.\n\n6. ***Typos in writing***\n\nThank you so much! We will correct them in our revision!\n\n7. ***Reproducibility***\n\nWe would respectfully inquire whether the reviewer identifies any aspects they believe are insufficient for reproducing our work. The code and data necessary for replication can be accessed through an anonymous link: [https://anonymous.4open.science/r/EMNLP_2023-B08D/](https://anonymous.4open.science/r/EMNLP_2023-B08D/) (mentioned in paper lines L033-L034). This link encompasses the complete setup, including environment configuration, raw and processed data, along with scripts for processing, training, testing, and visualization. We are confident that these resources facilitate result reproduction. If there are any additional materials you require, please do not hesitate to inform us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Initial work to explore the fundamental relationship and connectivity between EEG and language through computational multimodal methods. \n\n- Suggest the simple dual-encoder model (MTAM) which reveals great performances in two different tasks.", "reasons_to_reject": "- This paper should explain two high-level questions before suggesting author's approach (at least literature works should be covered).\n(1) Language is discrete data and EEG is continuous data. How did you explain this difference to align between them?\n(2) EEG is highly random in nature and very subjective between people which leads to hard to extract the consistent features. How did you deal with this problem?\n\n- This is NLP conference which means many people are not familiar with the concept of physiological signals. The authors need to kindly explain the concepts in the work.\n(1) Explain a difference between EEG and MEG. \n(2) Explain more details about the used datasets.\n\n- The above two reasons suggest the author to prepare a long paper to cover all the details. For example, \n(1) Why the authors consider CCA and WD? How about KL divergence or Euclidean distance?\n(2) How did you combine CCA and WD? This explanation should be included in main paper, not Appendix. \n\n- This paper needs to be edited/reviewed again by authors before submission since there are many mistakes. Few of them are shared here.\n(1) Please explain what MEG means (line 105).\n(2) were generalized -> which were generalized (line 112).\n(3) several -> two (line165)\n(4) Appendix ?? -> Appendix D.2 (line 197).\n(5) Correct quotation marks (line 249-251).", "questions_to_authors": "Here, I copy and past the questions covered in Reasons To Reject to make authors to readily state the answers. \n\nQ1. Language is discrete data and EEG is continuous data. How did you explain this difference to align between them?\n\nQ2. EEG is highly random in nature and very subjective between people which leads to hard to extract the consistent features. How did you deal with this problem?\n\nQ3. Why the authors consider CCA and WD? How about KL divergence or Euclidean distance?\n\nQ4. How did you combine CCA and WD? This explanation should be included in main paper, not Appendix.", "paper_topic_and_main_contributions": "This paper explores the relationship and dependency between EEG and languages. Author suggests a Multimodal Transformer Alignment Model (MTAM) which basically consists of dual-encoder with cross-alignment between EEG and languages. Also, canonical correlation analysis (CCA) and Wasserstein distance (WD) are considered as alignment mechanism where the combined one works the best. Two different downstream tasks are covered to reveal the novelty of work. Also, the authors share the publicly available software.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See Reasons To Reject."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper in submission explores the connection between EEG (Electroencephalogram) signals and language, making it one of the first studies to do so in the field. The authors provide code for analyses, making the research reproducible. The study uses multiple baseline models for comparison and shows that their model outperforms others across different task conditions. A significant strength lies in emphasizing the multi-modal alignment between brain features and text descriptions, highlighting its practical relevance. The paper demonstrates superior accuracy in downstream tasks compared to non-alignment methods.

However, the paper also has several shortcomings that raise questions about its overall contribution and clarity. One major concern is the near-100% accuracy reported for certain tasks, which seems implausible and is not adequately explained. There is also a lack of information regarding how EEG signals can be reliably interpreted to understand what a person is thinking or talking about. The study lacks a clear articulation of its contributions to our understanding of language processing in the brain or language representations in artificial models.

The methodological presentation in the paper is somewhat weak. It references three versions of their model ("Ours-EEG," "Ours-Text," and "Ours-Multimodal") without clearly explaining the differences in their inputs. Questions about the model architecture remain unanswered, reducing the clarity of the paper's contributions. Concerns are also raised about the soundness and novelty of the conclusions, particularly relating to the identification of brain areas and EEG frequency bands responding to positive and negative words. The description of the methodology used is considered too vague to affirm whether the conclusions are valid.

The paper could benefit from additional statistical analyses to support its arguments, as only qualitative interpretations are currently provided. The limitations of using EEG for locational interpretations are acknowledged but not thoroughly discussed. The paper also fails to answer some high-level questions critical to the field, such as how the continuous EEG data align with the discrete nature of language and how individual subjectivity in EEG signals is addressed.

Further, given that the paper is intended for an NLP (Natural Language Processing) conference, the authors should include explanations of physiological signals like EEG and MEG (Magnetoencephalography) for readers unfamiliar with these concepts. The paper is also advised to delve into why specific metrics like CCA (Canonical Correlation Analysis) and WD (Wasserstein Distance) were chosen over others like KL divergence or Euclidean distance.

Lastly, there are several mistakes in the paper, ranging from undefined acronyms to grammatical errors, that need correction before final submission.

Overall, while the paper makes a pioneering attempt to link EEG with language and offers promising results, it suffers from issues related to clarity, methodology, and the soundness of its conclusions. I would suggest the author submit this work to journals instead of presented in EMNLP 2023.
Decision: Accept-Findings