Computational Linguistics for Brain Encoding and Decoding: Principles, Practices and Beyond (2024)

Abstract: Computational linguistics (CL) has witnessed tremendous advancements in recent years, with models such as large language models demonstrating exceptional performance in various natural language processing tasks. These advancements highlight their potential to help understand brain language processing, especially through the lens of brain encoding and decoding. Brain encoding involves the mapping of linguistic stimuli to brain activity, while brain decoding is the process of reconstructing linguistic stimuli from observed brain activities. CL models that excel at capturing and manipulating linguistic features are crucial for mapping linguistic stimuli to brain activities and vice versa. Brain encoding and decoding have vast applications, from enhancing human-computer interaction to developing assistive technologies for individuals with communication impairments. This tutorial will focus on elucidating how computational linguistics can facilitate brain encoding and decoding. We will delve into the principles and practices of using computational linguistics methods for brain encoding and decoding. We will also discuss the challenges and future directions of brain encoding and decoding. Through this tutorial, we aim to provide a comprehensive and informative overview of the intersection between computational linguistics and cognitive neuroscience, inspiring future research in this exciting and rapidly evolving field.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 5: Tutorial Abstracts), pages 8–9 August 11-16, 2024 ©2024 Association for Computational Linguistics
Computational Linguistics for Brain Encoding and Decoding: Principles, Practices and Beyond
Jingyuan Sun Shaonan Wang Zijiao Chen Jixing Li Marie-Francine Moens
Computational linguistics (CL) has witnessed tremendous advancements in recent years, with models such as large language models demonstrating exceptional performance in various natural language processing tasks. These advancements highlight their potential to help understand brain language processing, especially through the lens of brain encoding and decoding. Brain encoding involves the mapping of linguistic stimuli to brain activity, while brain decoding is the process of reconstructing linguistic stimuli from observed brain activities. CL models that excel at capturing and manipulating linguistic features are crucial for mapping linguistic stimuli to brain activities and vice versa. Brain encoding and decoding have vast applications, from enhancing human-computer interaction to developing assistive technologies for individuals with communication impairments. This tutorial will focus on elucidating how computational linguistics can facilitate brain encoding and decoding. We will delve into the principles and practices of using computational linguistics methods for brain encoding and decoding. We will also discuss the challenges and future directions of brain encoding and decoding. Through this tutorial, we aim to provide a comprehensive and informative overview of the intersection between computational linguistics and cognitive neuroscience, inspiring future research in this exciting and rapidly evolving field.
Jingyuan Sun, Postdoc researcher in the Department of Computer Science, KU Leuven, Belgium email: jingyuan.sun@kuleuven.be website: https://www.kuleuven.be/wieiswie/ en/person/00155742 Jingyuan Sun has published papers in artificial intelligence and natural language processing journals and conferences such as TNNLS, Scientific Data, AAAI, IJCAI, EMNLP, COLING, ECAI, etc. He also serves as a (senior) PC member for these
above conferences. He is a reviewer of TPAMI.
Shaonan Wang, Associate Professor in the State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences email: shaonan.wang@nlpr.ia.ac.cn website: https://wangshaonan.github.io/ Shaonan Wang has contributed papers to natural language processing journals and conferences such as Information Sciences, TNNLS, ACL, EMNLP, AAAI, and IJCAI. Her work also spans neurolinguistics, with publications in Scientific Data, Brain and language, Cognitive, Affective, and Behavioral Neuroscience, and more. She serves as an editor for Neurobiology of Language and TALLIP and holds the role of curriculum chair at Neuromatch Academy.
Jixing Li, Assistant Professor at in the Department of Linguistics and Translation at the City University of Hong Kong email: jixingli@cityu.edu.hk website: https://jixing-li.github.io/ Jixing Li is an Assistant Professor at in the Department of Linguistics and Translation at the City University of Hong Kong. Her research combines NLP models and neuroimaging methods to examine syntactic and semantic analyses in the brain. Her work has been published in the Journal of Neuroscience, Brain and Language, Annual Review of Linguistics, etc. She serves as an ad-hoc reviewer for top journals in the field of neurolinguistics and is on the editorial board of Communications Psychology.
Zijiao Chen, PhD candidate in the Multimodal Neuroimaging in Neuropsychiatric Disorders Laboratory at the National University of Singapore email: zijiao.chen@u.nus.edu Zijiao Chen is a PhD candidate in the Multimodal
8
Neuroimaging in Neuropsychiatric Disorders Laboratory at the National University of Singapore. Her research primarily centers on representation learning within neuroimaging data and brain decoding. She has published papers in artificial intelligence and neuroimage conferences and journals such as CVPR, OHBM, NCAA, and AD.
Marie-Francine Moens, Full Professor in the Department of Computer Science, KU Leuven email: sien.moens@cs.kuleuven.be website: https://people.cs.kuleuven.be/ ~sien.moens/ Marie-Francine Moens is a Full Professor in the Department of Computer Science, KU Leuven. She is a fellow of the European Laboratory for Learning and Intelligent Systems (ELLIS). She is an associate editor of the journal IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). She holds the ERC Advanced Grant CALCULUS (2018-2024) granted by the European Research Council. She used to be the chair of the European Chapter of the Association for Computational Linguistics (EACL) and was a member of the executive board of the Association for Computational Linguistics (ACL). From 2012 to 2016 she was the coordinator of the MUSE project financed by Future and Emerging Technologies (FET) - Open of the European Commission.
9

heading: Future Work
text: We will also discuss the challenges and future directions of brain encoding and decoding. Through this tutorial, we aim to provide a comprehensive and informative overview of the intersection between computational linguistics and cognitive neuroscience, inspiring future research in this exciting and rapidly evolving field. Jingyuan Sun, Postdoc researcher in the Department of Computer Science, KU Leuven, Belgium email: jingyuan. sun@kuleuven. be website: https://www. kuleuven. be/wieiswie/ en/person/00155742 Jingyuan Sun has published papers in artificial intelligence and natural language processing journals and conferences such as TNNLS, Scientific Data, AAAI, IJCAI, EMNLP, COLING, ECAI, etc. He also serves as a (senior) PC member for these
above conferences. He is a reviewer of TPAMI. Shaonan Wang, Associate Professor in the State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences email: shaonan. wang@nlpr. ia. ac. cn website: https://wangshaonan. github. io/ Shaonan Wang has contributed papers to natural language processing journals and conferences such as Information Sciences, TNNLS, ACL, EMNLP, AAAI, and IJCAI. Her work also spans neurolinguistics, with publications in Scientific Data, Brain and language, Cognitive, Affective, and Behavioral Neuroscience, and more. She serves as an editor for Neurobiology of Language and TALLIP and holds the role of curriculum chair at Neuromatch Academy. Jixing Li, Assistant Professor at in the Department of Linguistics and Translation at the City University of Hong Kong email: jixingli@cityu. edu. hk website: https://jixing-li. github. io/ Jixing Li is an Assistant Professor at in the Department of Linguistics and Translation at the City University of Hong Kong. Her research combines NLP models and neuroimaging methods to examine syntactic and semantic analyses in the brain. Her work has been published in the Journal of Neuroscience, Brain and Language, Annual Review of Linguistics, etc. She serves as an ad-hoc reviewer for top journals in the field of neurolinguistics and is on the editorial board of Communications Psychology. Zijiao Chen, PhD candidate in the Multimodal Neuroimaging in Neuropsychiatric Disorders Laboratory at the National University of Singapore email: zijiao. chen@u. nus. edu Zijiao Chen is a PhD candidate in the Multimodal
8
Neuroimaging in Neuropsychiatric Disorders Laboratory at the National University of Singapore. Her research primarily centers on representation learning within neuroimaging data and brain decoding. She has published papers in artificial intelligence and neuroimage conferences and journals such as CVPR, OHBM, NCAA, and AD. Marie-Francine Moens, Full Professor in the Department of Computer Science, KU Leuven email: sien. moens@cs. kuleuven. be website: https://people. cs. kuleuven. be/ ~sien. moens/ Marie-Francine Moens is a Full Professor in the Department of Computer Science, KU Leuven. She is a fellow of the European Laboratory for Learning and Intelligent Systems (ELLIS). She is an associate editor of the journal IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). She holds the ERC Advanced Grant CALCULUS (2018-2024) granted by the European Research Council. She used to be the chair of the European Chapter of the Association for Computational Linguistics (EACL) and was a member of the executive board of the Association for Computational Linguistics (ACL). From 2012 to 2016 she was the coordinator of the MUSE project financed by Future and Emerging Technologies (FET) - Open of the European Commission. 9