UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language
Year: 2023

Abstract:
Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on decoding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first openvocabulary task aiming to bridge fMRI time series and human language. Furthermore, to explore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its efficacy in decoding coherent text from fMRI series across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEGto-text decoding, thereby surpassing the former baseline. Experimental results indicate the feasibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cognitive signals using a unified structure.

heading: 1 Introduction
text: Language serves as a window into the cognitive processes unfolding within our minds, communicating a vast amount of information through its syntax and semantics (Pagel, 2017). Advances in cognitive neuroscience have enabled us to directly observe the cognitive processes that underlie language use through the analysis of non-invasive cognitive signals, such as functional Magnetic Resonance Imaging (fMRI) and electroencephalogram (EEG). However, this also poses a challenge in understanding the relationship between these signals and the external stimuli that give rise to them
∗Corresponding author
within the mind. Deciphering cognitive signals into human language not only enhances our grasp of the linguistic system, but also facilitates the development of practical brain-computer interfaces (BCIs) by leveraging our comprehension of decoded signals (Wolpaw, 2007; Mudgal et al., 2020).
Although brain decoding has gained great success from word-level to sentence-level decoding on EEG (Panachakel and Ramakrishnan, 2021; Wang and Ji, 2022), relatively little research has been dedicated to directly generating text, particularly complete sentences, from fMRI volumes. This is largely attributed to the challenges posed by the relatively low temporal resolution of fMRI, which makes it challenging to acquire word-level fMRI frames within a sentence. In this study, we propose fMRI2text, the first open-vocabulary task that decodes fMRI time series into the corresponding texts under naturalistic settings.
Despite the early efforts in fMRI decoding (Mitchell et al., 2008; Palatucci et al., 2009; Wang et al., 2020; Zou et al., 2021), these methods are limited in the ways that they: (1) primarily rely on predefined regions of interest (ROIs) for feature extraction, underutilizing the rich spatial data inherent in full fMRI volumes. This may oversimplify the complex, distributed nature of cognitive processes (Ruiz et al., 2014). (2) do not effectively leverage the sequential information embedded in fMRI time series, missing valuable insights into the dynamics of cognitive processes (Du et al., 2022). (3) prioritize the role of the decoder while overlooking the importance of efficient encoding, particularly for high-dimensional signals like fMRI. These limitations extend beyond fMRI decoding and apply to other cognitive signal decoding methods as well. To address these issues and obviate the need for separate, complex pipelines to decode specific cognitive signals, we propose UniCoRN (Unified Cognitive signal ReconstructioN for brain decoding), a versatile brain decoding pipeline that
13277
can be applied to various types of cognitive signals. As a standard encoder-decoder framework, UniCoRN leverages the robust decoding abilities of pre-trained language models. Crucially, it constructs an effective encoder through both snapshot and series reconstructions, harnessing the power of seq2seq models. This allows UniCoRN to analyze individual signal “snapshots” (such as a single fMRI volume or an EEG time point) and capture the “series” or temporal dependencies among these snapshots, thus maximizing the information extracted from the cognitive signals.
In summary, our contributions are as follows:
• We introduce a novel task, designated as fMRI2text, which is the first open-vocabulary task that decodes fMRI time series into human language in a naturalistic context.
• We present a baseline solution to further elucidate the potential of fMRI2text and demonstrate that our proposed method is effective across various split settings.
• We propose a unified framework UniCoRN (Unified Cognitive signal ReconstructioN for brain decoding) to translate cognitive signals into human language, and validate its effectiveness on both EEG and fMRI.

heading: 2 Related Work
text: Cognitive Signals Cognitive signals represent the dynamic neural activity associated with information processing and cognitive functions, and are crucial in building BCI systems (Mudgal et al., 2020). These signals are captured at individual time points or as part of a time series, with each data point providing a snapshot of brain activity at a specific point in time. While EcoG is often used in high-performance BCI systems (Akbari et al., 2019; Rapeaux and Constandinou, 2021; Metzger et al., 2022), its semi-invasive nature limits its potential for widespread application in healthy individuals. In non-invasive BCI systems, EEG is most commonly used due to its high temporal resolution and cost-effectiveness, while other techniques such as fMRI have also been employed in recent years (Saha et al., 2021; Martinek et al., 2021; Pitt and Dietz, 2022). In spite of its relatively lower temporal resolution, fMRI allows for the mapping of brain-wide responses to linguistic stimuli at a highly detailed spatial resolution of millimeters
(Vouloumanos et al., 2001; Noppeney and Price, 2004; Binder et al., 2009). This makes fMRI particularly ideal for BCI systems that translate brain signals into text, a process that involves the participation of multiple brain regions (Ruiz et al., 2014).
Brain Decoding Recent research has been directed towards resolving the issue of decoding cognitive signals into human language through the introduction of new multi-modal tasks and models. Most recently-proposed tasks in this field focus on aligning cognitive signals with a limited vocabulary up to a thousand for word-level decoding (Bhattasali et al., 2019; Affolter et al., 2020; Défossez et al., 2022) or incorporating them into sentence embeddings for sentence-level decoding using pairwise classification (Pereira et al., 2018; Sun et al., 2019). Wang and Ji introduce a novel brain decoding task called EEG-To-Text decoding (EEG2text for short), which achieves sentence-level decoding by converting each word-level EEG signal into corresponding text stimuli using pre-trained language models, thereby extending the problem from a closed vocabulary to an open vocabulary.

heading: 3 Task Definition
text: As shown in Figure 1, the subject is instructed to read or listen to the text stimuli, while an fMRI volume is acquired every fixed repetition time (TR). Given an fMRI time series of length T , F := {f1, f2, ..., fT }, the task is to decode the corresponding text tokens W := {w1, w2, ..., wn} of the stimuli used during the acquisition of the fMRI volumes from an open vocabulary V .
As mentioned in Section 2, many studies have aimed to link cognitive signals with human language. We summarize three related tasks and compare them to fMRI2text in Table 1.
The three representative tasks share a common characteristic of relying on cognitive signals that operate at the word level. However, this approach may not be practical for real-world fMRI applications, as the poor temporal resolution of fMRI necessitates tightly controlled experimental manipulations under these settings (Nastase et al., 2020; Hamilton and Huth, 2020). In contrast, fMRI2text leverages cognitive signals from more naturalistic settings, using text and speech as stimuli in a manner closer to real-world language use (Huth et al., 2016). Here, each fMRI frame corresponds to a specific timeframe, and is aligned with an undetermined number of tokens rather than a fixed one,
better reflecting the variable and dynamic nature of natural language processing.
Another distinct feature that differentiates fMRI2text from prior fMRI-related tasks is its incorporation of multiple sequential frames as input. The inherent low signal-to-noise ratio of fMRI has directed prior studies towards a focus on individual frames. However, this approach overlooks the valuable temporal information embedded within the interrelations of successive frames, which is particularly crucial when dealing with continuous data streams such as cognitive signals.

heading: 4 Method
text: In this section, we introduce the UniCoRN structure and use the fMRI2text task as an explicit demonstration. As shown in Figure 2, UniCoRN consists of two stages: (1) the cognitive signal reconstruction to train the encoder specifically for cognitive signals, and (2) the cog2text decoding to convert the embeddings of the cognitive signals
from the first stage to human language.

heading: 4.1 Cognitive Signal Reconstruction
text: The cognitive signal reconstruction consists of two phases, snapshot reconstruction and series reconstruction, aiming to train the encoder of UniCoRN to integrate the individual characteristics of each fMRI volume (intra-volume information), as well as the temporal relationships among volumes in a time series (inter-volume information).
As shown in Figure 2, during the snapshot reconstruction, each fMRI frame is input into the Snapshot Encoder Er (Ereconstruction) respectively to obtain the snapshot embedding Ei, which will be used later for series reconstruction. In our case, we use a CNN-based model similar to Malkiel et al. (2021) as the Snapshot Encoder. During this phase, Ei is then fed to the Snapshot Decoder Dr (Dreconstruction) to reconstruct the original fMRI frame fk (The k-th frame in fMRI time series). Note that Dr is also CNN-based but simpler than
Er in structure, to ensure that the reconstruction of fMRI snapshots does not mostly rely on the decoding ability of Dr. We use mean average error (MAE) as the loss function for both phases of cognitive signal reconstruction. Phase 1 can be formulated as follows:
Eik = Er(fk) (1)
Er = argmin E MAE(Dr(Er(fk)), fk) (2)
During Phase 2, Series Encoder Es (Eserialized) takes the snapshot embedding Ei of T sequential fMRI frames to generate the corresponding serialized embedding Ee. We use multi-layer transformer encoder (Vaswani et al., 2017) as Es to obtain information in time domain by applying self-attention to fMRI series. Serialized embedding Ee is then input into the same decoder as Phase 1 for series reconstruction. We continue using Dr as the decoder to keep minimal effect of decoding process to signal reconstruction, as we will only be using Er and Es in the next
stage. Denote {Eek, Eek+1, ..., Eek+T −1} as Eek∼T , {Eik, Eik+1, ..., Eik+T −1} as Eik∼T .
Eek∼T = Es(Eik∼T ) (3)
Es = argmin E MAE(Dr(Es(Eik∼T )), Eik∼T ) (4)

heading: 4.2 Cog2text Decoding
text: The motivation of cognitive signal reconstruction is to get a decent representation of fMRI, which is quite so different from and more difficult than EEG since each fMRI frame has more spatial information as a 3D signal. Similar to Wang and Ji (2022), we use this representation as primary word embeddings for language models, except that these embeddings have been denoised and condensed through reconstruction. The high-level idea here is that we consider each original frame of fMRI as a word-level representation of “the foreign language spoken by the human brain”, and use the encoder constructed in Section 4.1 to obtain the embeddings of this “language”, which will be then
decoded to real human language (English in our case) like traditional machine translation tasks.
Figure 2 gives a detailed demonstration of how fMRI embedding is acquired and how the two stages are concatenated together. After the two phases of cognitive signal reconstruction, the decoder Dr used in stage one is replaced with the fMRI-Text decoder Dt (Dtranslation) for text generation. The serialized embeddings Ee are then projected into fMRI embedding E as the final representation of fMRI, which contains both intra-volume information and inter-volume information and will be used as the input for Dt to convert to texts. Here we use BART (Lewis et al., 2019) as the fMRIText decoder Dt and cross-entropy loss (CE) like most seq2seq tasks as the training target. Denote {Ek, Ek+1, ..., Ek+T −1} as Ek∼T , and the projection layer matrix as WP .
Ek∼T = E e k∼T W P (5)
Dt = argmin D CE(D(Ek∼T ),W) (6)

heading: 4.3 UniCoRN Structure
text: Other than fMRI, UniCoRN is also capable of decoding other cognitive signals into human language. We generalize the same pipeline to EEG2text, without changing the overall structure but only moderately modifying the snapshot encoder Er and snapshot decoder Dr due to the difference in spatial structure between EEG and fMRI. The detailed illustration is provided in Appendix D.

heading: 5 Experiments

heading: 5.1 Dataset
text: The “Narratives” dataset (Nastase et al., 2021) encompasses a range of fMRI data from individuals who were engaged in listening to spoken stories in the real-world setting. Given that various fMRI machines produce frames of different sizes, and considering the “Narratives” dataset comprises data from multiple machines, we focus solely on data with dimensions of 64× 64× 27 voxels. The detailed information of the “Narratives” dataset we used in this paper is provided in Appendix C.
Most cognitive signals require pre-processing before putting into use. For fMRI, We follow the same pre-processing procedure as provided in Nastase et al. (2021). As for EEG, we use the same waves as in Wang and Ji (2022) for comparison.
Given that the “Narratives” dataset does not offer any pre-determined splits and the appropriate
method for splitting fMRI data for this task is a matter of debate, we conduct experiments utilizing a variety of different split configurations.
Denote all subjects as S := {S1, S2, ..., Sn}, all stimuli as C := {C1, C2, ..., Cm}, where n and m stands for the total number of subjects and stimuli respectively. Note that the total number of stimuli given to individual subjects may vary. The fMRI series of subject Si receiving stimuli Cj is represented as F ij := {f ij1 , f ij2 , ..., f ijTj}. Tj here represents the total number of fMRI frames of stimuli j. For briefings, we use F ijk∼T to represent the fMRI series of length T starting at the k th frame {f ijk , f ij k+1, ..., f ij k+T −1}. Different split methods are formulated in detail in Table 2. As for EEG2text, We use ZuCo1.0 datasets (Hollenstein et al., 2018), which comprises EEG recordings obtained from natural reading tasks, including both Normal Reading (NR) and Task-Specific Reading (TSR). The reading materials utilized for these tasks were sourced from movie reviews (Socher et al., 2013) and Wikipedia articles. The ZuCo1.0 dataset comprises a total of 1,107 unique sentences across 12 subjects, yielding a total of 10,258 samples. Given the limited number of training samples, we utilize a split method similar to the random method described above.

heading: 5.2 Implementation
text: Our model utilizes the Pytorch-based (Paszke et al., 2019) Huggingface Transformers (Wolf et al., 2020) packages and is designed to reconstruct sequences with a length of 5 for fMRI2text and 10 for EEG2text in Phase 2. Additional hyperparameters can be found in Appendix A. Both datasets are split into train, validation, test sets with a ratio of 70%, 15%, 15% respectively. We follow the same evaluation strategy as Wang and Ji (2022) to establish a fair comparison and gain insights into the optimal performance scenario of UniCoRN. The
results reported are the average of three separate runs. All experiments were conducted on NVIDIA A100-80GB-PCIe GPUs.

heading: 5.3 UniCoRN Structure for fMRI2text
text: fMRI2text across Different Splits We experiment with series length T of 10 and report the BLEU scores and ROUGE-1 scores for fMRI2text across different splits. As shown in Table 3, UniCoRN achieves fairly effective results across all splitting methods introduced in Section 5.1. Meanwhile, to have an intuitive grasp of the decoding quality, we present a few cases comparing the target tokens and the predicted tokens in Table 5.
The experiments conducted under the random, random time, and by subject settings resulted in BLEU-4 scores of 39.74%, 34.77%, and 37.78% respectively. These results shed light on the prospect of fMRI2text when viewing it as a translation-like task, particularly in comparison to state-of-the-art results in machine translation, such as 46.40% for English-French translation as reported by Liu et al. (2020) and 15.20% for English-Arabic translation as reported by Provilkov et al. (2019).
In contrast, the results obtained under the by stimuli and consecutive time settings are less so ideal. This may be attributed to the fact that the input fMRI frames do not correspond to a fixed
and predetermined set of words. Consequently, the fMRI embeddings learned by the model may represent an imprecise combination of words rather than specific, individual words. Such variability might pose a challenge when the model encounters frames paired with unique word combinations unseen during training. Nonetheless, this does not preclude UniCoRN’s ability to extract meaningful information under these conditions. As shown in Table 5, despite a decline in decoding quality under these two methods, UniCoRN is still successful in identifying key words within the text fragments, and maintains a semblance of polarity and structure that resonates with the target sentence.
One thing to notice is that the results under the by subject split setting do not show a significant deviation from those under the random and random time settings. This contrasts with previous studies that relied on individual fMRI frames for decoding, which suggests that UniCoRN’s incorporation of inter-volume information can mitigate the effects of inter-subject variability on decoding performance.
Another interesting anomaly is that, despite that both the random time and consecutive time configurations have distinct text content across their train, validation, and constest sets, the former setting performs significantly better than the latter. This discrepancy may be attributed to the robust
decoding capabilities of BART, which effectively bridges the gap between frames that UniCoRN did not encounter during training.
The above results demonstrate an intrinsic characteristic when interpreting the fMRI2text task as a translation-like endeavor. The fMRI time series of different subjects can be likened to the unique accent or speaking style that each individual possesses. While variations among individuals exist, they usually do not present significant challenges in discerning the overall meaning, especially when contextual information is provided. This analogy extends to the case of random time and consecutive time: when a non-native speaker attempts to comprehend a foreign language, the chances of comprehending key information increase significantly when interpretation can be made from a broader context, as opposed to deciphering a sentence without any foresight of what follows.
Effect of Series Length T To further demonstrate the effectiveness of decoding fMRI by series, we conduct experiments on different series length
T under random time split setting. As shown in Table 4, the length of fMRI series does have a major impact on decoding results when T is relatively small. However, this impact seems to reach a plateau and might even turn adverse as T increases. Such trend could be attributed to the inherent limitations of the transformer model in effectively learning long-term dependencies.
Meanwhile, although decoding results tend to be less optimal when T is small, experiments indicate that apart from frequently used phrases (such as catchphrases during pauses), UniCoRN can still decode semantically and syntactically similar tokens. This capability aligns with previous studies, affirming the feasibility of bridging fMRI and human language under naturalistic settings.

heading: 5.4 UniCoRN Structure for EEG2text
text: As shown in Table 7, the UniCoRN structure surpasses the former baseline on all metrics except when solely using snapshot reconstruction, which will be further discussed in Section 6.
Here we take a closer look at the performance of UniCoRN on EEG2text in Table 6 and compare with the former baseline in Wang and Ji (2022). The results illustrate that UniCoRN outperforms the previous baseline in terms of capturing semantics and syntax in target tokens. Specifically, UniCoRN not only enhances the decoding accuracy of individual words but also maintains superior coherence in sentence structure, resulting in more fluent and comprehensible decoding outputs

heading: 6 Ablation Study
text: To further validate the effectiveness of UniCoRN, we conduct ablation studies on both fMRI2text and EEG2text, to assess how the two phases of signal reconstruction affect the model’s performance.
As shown in Table 8, fMRI2text greatly benefits from both phases of fMRI reconstruction, resulting in an improvement of the BLEU score by approximately 20% when reconstruction is included. This indicates that for cognitive signals that are rich in spatial information like fMRI, it is important for the encoder to have a thorough understanding of these signals themselves, but not mainly rely on the ability of decoder. Comparatively, series reconstruction proves to be slightly more effective than snapshot reconstruction, which may be attributed to the nature of seq2seq tasks as the input of series reconstruction is more similar to that of cog2text decoding than snapshot reconstruction.
Conversely, Table 7 shows a decline in overall metrics when only Phase 1 is used for EEG2text.
This could be attributed to the noise introduced by the snapshot reconstruction, which might potentially compromise the ability of the model to process EEG sequences — a crucial aspect for cognitive signals with high temporal resolution like EEG. However, this doesn’t undermine the importance of snapshot reconstruction for such signals. As evident in the results, combining snapshot and series reconstruction increases the BLEU-4 score from 36.51% to 37.04%, suggesting an enhancement in the model’s performance for predicting longer ngrams. Thus, while the impact may vary depending on the spatial and temporal resolution of different cognitive signals, integrating both phases generally enhances the model’s overall performance by developing a more sophisticated encoder.

heading: 7 Conclusion
text: In this paper, we introduce a novel open-vocabulary brain decoding task fMRI2text, aiming to decode linguistic stimuli from multiple fMRI frames collected under naturalistic conditions. Building upon this, we present UniCoRN, a two-stage framework that integrates both temporal and spatial aspects of cognitive signals through snapshot and series reconstruction. The efficacy of UniCoRN is validated under various split settings, illuminating the opportunities that this task provides. Furthermore, we adapt the framework to EEG2text, demonstrating its capacity to generate semantically and syntactically more accurate results, thereby introducing a fresh perspective to brain decoding tasks.
Limitation
The “Narratives” dataset provides a valuable fMRI resource, stimulated by language and obtained under naturalistic conditions. Further research opportunities can be pursued with the availability of more detailed datasets. For instance, comparative studies between instances of stuttering and nonstuttering in text stimuli can be conducted, as our experiments demonstrate that the model tends to retain frequently-used filler words (such as “um” and “like,”) as a shortcut for higher accuracy. Meanwhile, the evaluation strategy applied for current research of open-vocabulary brain decoding presents an idealized condition and and serves as a starting point from which further exploration of how existing methods might perform under more real-world scenarios can commence. Although we use this setting for baseline comparison purposes and a testament to the feasibility of our fMRI2text task, additional tests under more practical conditions could be an essential step in future work, further elucidating the applicability and robustness of the methods. Furthermore, the structure of the snapshot encoder can be explored further, as exemplified by the use of transformer-based Vision Transformer (ViT) in Chen et al. (2022) for fMRI encoding.
Ethical Considerations
In this work, we introduce a new NLP task related to fMRI and a unified approach for decoding various types of cognitive signals into human language. We conduct our experiments on the public cognition datasets Narratives and ZuCo1.0 with the authorization from the respective maintainers of the datasets. All experimental datasets involved have been de-identified by dataset providers and used for research only.

heading: Acknowledgements
text: We express our sincere gratitude to the anonymous reviewers for their professional, insightful and constructive comments and gratefully acknowledge the support of the National Key R&D Program of China [2021ZD0113302]; and the National Natural Science Foundation of China [62206079]; and Heilongjiang Provincial Natural Science Foundation of China [YQ2022F006].

heading: B Notation Table
text: The notation for the variables mentioned in this paper is presented in Table 10.

heading: C Details of Dataset
text: The detailed information of the “Narratives” datasets that are used for fMRI2text experiments in this paper is shown in Table 11.

heading: D Details of UniCoRN for EEG2text
text: As depicted in Figure 3, the snapshot encoder Er begins by partitioning the original EEG signal into smaller patches. Subsequently, a multi-layer transformer encoder is utilized to analyze the connections between these patches. The resulting output of Er is then concatenated and transformed into a vector with a dimensionality of 1024, serving as the snapshot embedding. The subsequent steps in the process are analogous to those used in the fMRI2text scenario.

heading: E Case Analysis
text: In this section, we present several cases from our ablation study in Table 12 for fMRI2text and Table 13 for EEG2text to provide a more comprehensive understanding of the variations in decoding quality and the impact of different phases.
As demonstrated in Table 12, UniCoRN effectively decodes “key information” ranging from verbs (such as “swallowing” and “smiled”) to nouns (“chocolate” in this example). Without the series reconstruction in Phase 2, the model still demonstrates the ability to decode some nouns, but its performance in predicting verbs is significantly impaired. The performance further deteriorates when the snapshot reconstruction in Phase 1 is removed, although the model still retains sentence structure that is more similar to the target sentence than the model without Phase 1 and Phase 2.
In contrast, the differences in decoding quality are less pronounced in the case of EEG2text. Although UniCoRN is still able to decode some accurate information such as “Einstein” and “Soviet”, it fails to correctly decode “physicist” like other methods, and instead generates “government”. This discrepancy could be attributed to the fact that EEG signals are aligned at the word level, making the task of decoding EEG less challenging than fMRI2text and thus not showcasing the superiority of UniCoRN as much. Additionally, it could be attributed to UniCoRN’s efficient encoder which allows for better utilization of pre-trained language models, since “government” might be mentioned more frequently in the context of “Soviet” than “physicist”.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
the limitation section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? the abstrct and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 5
3 B1. Did you cite the creators of artifacts you used? 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. The detailed of the datasets used for the paper is explicitly explained in the original dataset paper, which are cited in section 5
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Although we use this setting for baseline comparison purposes and a testament to the feasibility of our fMRI2text task, additional tests under more practical conditions could be an essential step in future work, further elucidating the applicability and robustness of the methods. Furthermore, the structure of the snapshot encoder can be explored further, as exemplified by the use of transformer-based Vision Transformer (ViT) in Chen et al. (2022) for fMRI encoding. Ethical Considerations
In this work, we introduce a new NLP task related to fMRI and a unified approach for decoding various types of cognitive signals into human language. We conduct our experiments on the public cognition datasets Narratives and ZuCo1. 0 with the authorization from the respective maintainers of the datasets. All experimental datasets involved have been de-identified by dataset providers and used for research only