SlowGuess/NAACL_2022_OCR huggingface dataset:
show low scores in the early auditory cortex (EAC), auditory association cortex (AAC); average scores in TPOJ and DFL; and superior scores in PMC. This aligns with the known language hierarchy for spoken language understanding (Nastase et al., 2020). Tasks such as PD, Summarization, and NLI seem to yield better performance in predicting the brain responses than the other NLP tasks across all the ROIs. These Pearson correlation  $(\tau)$  results are comparatively much higher compared to those obtained using pretrained (task-agnostic) GPT2 model in (Caucheteux et al., 2021a)  $(\tau$  ranging from  $0.02 - 0.06)$ . As shown in Fig. 2, our method obtains much higher correlations  $(\tau$  ranging from  $0.02 - 0.229)$ . Similar to the Pereira dataset, we estimate the statistical significance of the performance differences using the one-way ANOVA test. The main effect of task was significant for all the ROIs with  $\mathrm{p} \leq 10^{-3}$  with confidence  $95\%$  (see Appendix for detailed ANOVA results). Also, Post hoc pairwise comparisons (Ruxton and Beauchamp, 2008) revealed that on both 2V2 accuracy and Pearson correlation measures, tasks such as PD, Sum, and NLI performed significantly better compared to other tasks (see Appendix for pairwise comparison results).
Further, from Fig. 2, we see that the bilateral posterior medial cortex (PMC) associated with higher language function exhibits a higher correlation among all the brain ROIs. ROIs, including bilateral TPOJ and bilateral DFL, yield higher correlations with the five NLP tasks, which is in line with the language processing hierarchy in the human brain. Finally, across all regions, pretrained BERT model has worse correlation compared to at least 5 other task models.
In summary, different and distinct language Taskonomy features seem to be related to the encoding performance in reading versus listening tasks. CR, NER, SRL, and SS perform better for reading. PD, Sum, and NLI perform better for listening. While listening the subject is cognitively more involved in the activity compared to reading (Buchweitz et al., 2009). Thus, it makes sense that shallow tasks like NER and SS are useful for reading while more complex NLP tasks like PD, Sum and NLI are effective for encoding listening stimuli.
5.2 Language Task Similarity Computation
Pearson correlation values between predicted responses for each pair of tasks were used to con
Figure 3: Pereira - Prediction Similarity Matrix constructed from the task-wise brain response predictions across 10 tasks averaged across all subjects.
Figure 4: Narratives-Pieman - Prediction Task Similarity constructed from the task-wise brain response predictions across 10 tasks averaged across all subjects.
struct the similarity matrix with heatmap for both Pereira and Narratives-Pieman datasets, as shown in Figs. 3 and 4. We observe that the following task pairs are highly correlated for the Pereira dataset: (NER and CR), (SS and CR) and (PD and Sum). Also these task pairs are highly correlated for the Narratives-Pieman dataset: (CR and NLI), (NLI and SA) and (PD and Sum). Similarities are relatively higher for Narratives-Pieman compared to the Pereira dataset. Surprisingly, the (NLI, SA) pair has lowest similarity for Pereira (reading) and close to highest in Narratives-Pieman (listening). We hypothesize that this is because sentiment is best conveyed while the subject is listening.
Reading sentences (Pereira): The stimulus sentences from the Pereira dataset were fed as input to each of the 10 task Transformers. The similarity among the resulting representations was analyzed using hierarchical clustering, and the clusters are visualized as dendrograms in Fig. 5 (left). We observe that the tasks are clustered into three groups denoted using red, green, and blue colors. Next, we wished to check if similar task grouping is observed on brain activations predicted by ridge regression trained on task-specific representations. Hence, similar clustering analysis was conducted on the neural space representations, and the clusters are visualized as dendrograms in Fig. 5 (right)
Figure 5: Left: Pereira Dendrogram constructed using similarity on representations from task-specific Transformer encoder models with stimuli from the dataset passed as input. Right: Pereira Dendrogram constructed using similarity matrix shown in Fig. 3.
Figure 6: Left: Narratives-Pieman Dendrogram constructed using similarity on representations from task-specific Transformer encoder models with stimuli from the dataset passed as input. Right: Narratives-Pieman Dendrogram constructed using similarity matrix shown in Fig. 4.
across all subjects. Interestingly, the tree derived from brain representation also shows a similar distribution of tasks across the three groups. Similar dendrograms for individual subjects are illustrated in Appendix-Fig. 11.
Listening Stories (Narratives-Pieman): Fig. 6 compares the task similarity tree based on the patterns from the pretrained task Transformers, with the task similarity tree generated based on similarity in brain response prediction performance averaged across all subjects. We observe that the tasks are clustered into three groups denoted using red, green, and blue colors. Again, the tree derived from brain representation also shows a similar distribution of tasks across the three groups. Dendrograms for individual subjects are in the Appendix-Fig. 12.
5.3 Brain maps for whole brain predictions
The mean absolute error (MAE) between predictive and actual responses is obtained using individual task features from the taskonomy. MAE values are obtained for all the voxels in the brain for both the reading (Fig. 7) and listening datasets (Fig. 8).
In the reading task, we observe from Fig. 7 that CR has lower MAE compared to PD which in turn has lower MAE compared to the NLI task (brain maps for the other tasks are reported in Fig. 17 in the Appendix). Overall, for the reading stimuli, tasks such as NLI, QA, and SA display higher MAE values. To further investigate which sub
ROIs (LPTG, LMTG, LATG, LFus, Lpar, Lang, LIFGorb, LIFG, LaMFG, LpMFG, and LmMFG) of the Language network are related to the predictive task features, we train encoding models for all the sub ROIs for the best encoding task, i.e., for the CR task (see Fig. 14 in Appendix). We notice that both LMTG (middle temporal gyrus) and LPTG (posterior temporal gyrus) are more accurately predicted than the other sub ROIs. On the other hand, LIFG-orb displays a lower Pearson correlation for the CR task. The presence of superior encoding information in the ROIs in the temporal gyrus as compared to those in the inferior frontal gyrus seems to mirror similar observations seen in decoder performance (Anderson et al., 2017b).
On the other hand, in the listening task, we observe from Fig. 8 that Paraphrase and WSD display lower MAE values compared to QA task (brain maps for the other tasks are reported in Fig. 18 in the Appendix). Taken together, for listening stimuli, tasks such as NER, QA, SA, CR, and SS display higher MAE values. From Fig. 8, we see that ROIs such as EAC and AAC have higher MAE compared to PMC and TPOJ brain ROIs.
We further demonstrate the prediction performance of the encoder model trained on sub ROIs for the paraphrase task in Fig. 15 in the Appendix. It can be observed that sub ROIs such as Pos1 and Pos2 have a higher Pearson correlation than other sub ROIs of the PMC region. Both sfl and l55b display a higher correlation among all the sub ROIs for the DFL ROI. However, all the sub ROIs in the TPOJ yield higher correlation, as shown in Fig. 15. The control and attention ROIs in the posterior cingulate cortex (for ex., POS1 in PMC), together with the superior frontal language region (sfl in DFL) and TPOJ, are part of the well-known language network associated with narrative comprehension (Nastase et al., 2020), and it is heartening to see that task features from PD task also relate to semantic analysis of the ongoing narrative.
(1) We used a ridge regression model instead of more complicated models for encoding. We believe that more complex models can lead to further exciting insights. (2) We experimented with 10 NLP tasks. Models can be pretrained for more such tasks to check if other tasks are better predictive of voxel activations. (3) We leveraged models finetuned using datasets of different sizes across
Figure 7: Pereira BrainMaps: Mean absolute error (MAE) between predictive voxels and actual voxels using task features from Taskonomy in one sample subject (subject 1). Predictive regions of different tasks are dissimilar across tasks. The MAE values of each brain ROI are: CR (Language: 0.64, Visual: 0.57, DMN: 1.19, TP: 0.67), PD (Language: 0.81, Visual: 0.74, DMN: 1.34, TP: 0.87) and NLI (Language: 1.9, Visual: 1.88, DMN: 2.1, TP: 2.03).
Figure 8: Narratives-Pieman BrainMaps: Mean absolute error (MAE) between predictive voxels and actual voxels using task features from Taskonomy in one sample subject (subject 1) of PieMan dataset. Predictive regions of various tasks are different across tasks. The MAE values of each brain ROI: PD task (EAC: 0.74, AAC: 0.66, PMC: 0.60, TPOJ: 0.61, and DFL: 0.694), WSD task(EAC: 0.83, AAC: 0.75, PMC: 0.68, TPOJ: 0.68, and DFL: 0.76), QA task (EAC: 0.92, AAC: 0.83, PMC: 0.74, TPOJ: 0.75, and DFL: 0.76).
tasks. While a fair comparison of dataset sizes across tasks is impossible, we understand that this could have resulted in some bias in our results. (4) We used a different dataset for reading vs listening. While we believe that the differences in task-specific model performances across reading and listening are mainly due to the learned stimulus representations, but they could also arise from other factors such as experimental conditions, the text domain of the stimuli or number of voxels, etc. (5) On Natural Language Understanding tasks such as NLI, SA, QA and PD, Gauthier and Levy (2019) observed that scrambled sentence representations gave better decoding performance. But encoding models (especially for the listening task), scrambled order would be detrimental to making sense of what is being heard. It is an interesting future task to see if the opposite result is seen in the case of brain encoding models. It is plausible that brain uses encoding models in a flexible way when it comes to decoding (Kriegeskorte and Douglas,
2019). Kriegeskorte and Douglas (2019) mention that "Decoding models can help reveal whether particular information is present in a brain region in a format the decoder can exploit. Encoding models make comprehensive predictions about representational spaces." In this sense, results of current work are not directly comparable to those of Gauthier and Levy (2019).
In this paper, we studied the effectiveness of task specific NLP models for brain encoding. We observe that building individual encoding models and exploiting existing relationships among models can provide a more in-depth understanding of the neural representation of language information. Our experiments on Pereira and Narrative datasets lead to interesting cognitive insights.
7 Ethical Statement
We reused publicly available datasets for this work: Pereira and Narratives. We did not collect any new dataset.
Pereira dataset can be downloaded from https://osf.io/crwz7/. Please read their terms of use<sup>2</sup> for more details.
Narratives dataset can be downloaded from https://datasets.datalad.org/?dir=/labs/hasson/narratives. Please read their terms of use<sup>3</sup> for more details.
We do not foresee any harmful uses of this technology.
Andrew J Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. 2017a. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. Transactions of the Association for Computational Linguistics, 5:17-30.
Andrew James Anderson, Jeffrey R Binder, Leonardo Fernandino, Colin J Humphries, Lisa L Conant, Mario Aguilar, Xixi Wang, Donias Doko, and Rajeev DS Raizada. 2017b. Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation. Cerebral Cortex, 27(9):4379-4395.
Andrew James Anderson, Jeffrey R Binder, Leonardo Fernandino, Colin J Humphries, Lisa L Conant, Rajeev DS Raizada, Feng Lin, and Edmund C Lalor. 2019. An integrated neural decoder of linguistic and experiential meaning. Journal of Neuroscience, 39(45):8969-8987.
Andrew James Anderson, Kelsey McDermott, Brian Rooks, Kathi L Heffner, David Dodell-Feder, and Feng V Lin. 2020. Decoding individual identity from brain activity elicited in imagining common experiences. Nature communications, 11(1):1-14.
Richard Antonello, Javier Turek, Vy Vo, and Alexander Huth. 2021. Low-dimensional structure in the space of language representations is reflected in brain responses. arXiv preprint arXiv:2106.05426.
Pinglei Bao, Liang She, Mason McGill, and Doris Y Tsao. 2020. A map of object space in primate inferotemporal cortex. Nature, 583(7814):103-108.
Julia Berezutskaya, Zachary V Freudenburg, Luca Ambrogioni, Umut Güçlü, Marcel AJ van Gerven, and
Nick F Ramsey. 2020. Cortical network responses map onto data-driven features that capture visual semantics of movie fragments. Scientific reports, 10(1):1-21.
Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. *Cerebral cortex*, 19(12):2767-2796.
Augusto Buchweitz, Robert A Mason, Leda Tomitch, and Marcel Adam Just. 2009. Brain activation for reading and listening comprehension: An fmri study of modality effects and individual differences in language comprehension. Psychology & neuroscience, 2(2):111-123.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. 2021a. Disentangling syntax and semantics in the brain with deep networks. In International Conference on Machine Learning, pages 1336–1348. PMLR.
Charlotte Caucheteux, Alexandre Gramfort, and JeanRémi King. 2021b. Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects. arXiv preprint arXiv:2110.06078.
R Todd Constable, Kenneth R Pugh, Ella Berroya, W Einar Mencl, Michael Westerveld, Weijia Ni, and Donald Shankweiler. 2004. Sentence complexity and input modality effects in sentence comprehension: an fmri study. NeuroImage, 22(1):11-21.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Evelina Fedorenko, Idan Asher Blank, Matthew Siegelman, and Zachary Mineroff. 2020. Lack of selectivity for syntax relative to word meanings throughout the language network. Cognition, 203:104348.
Evelina Fedorenko, Alfonso Nieto-Castanon, and Nancy Kanwisher. 2012. Lexical and syntactic representations in the brain: an fmri investigation with multivoxel pattern analyses. Neuropsychologia, 50(4):499-513.
Angela D Friederici, Shirley-Ann Ruschemeyer, Anja Hahne, and Christian J Fiebach. 2003. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cerebral cortex, 13(2):170-177.
Jon Gauthier and Roger Levy. 2019. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 529-539.
$^{2}$ https://github.com/  
CenterForOpenScience/cos.io/blob/master/TERMS_OF_USE.md  
 $^{3}$ https://datasets.datalad.org/labs/hasson/narratives/stimuli/README
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. 2016. A multimodal parcellation of human cerebral cortex. Nature, 536(7615):171-178.
Giacomo Handjaras, Emiliano Ricciardi, Andrea Leo, Alessandro Lenci, Luca Cecchetti, Mirco Cosottini, Giovanna Marotta, and Pietro Pietrini. 2016. How concepts are encoded in the human brain: a modality independent, category-based cortical organization of semantic knowledge. Neuroimage, 135:232-242.
Nora Hollenstein, Antonio de la Torre, Nicolas Langer, and Ce Zhang. 2019. Cognival: A framework for cognitive word embedding evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 538-549.
Alexander G Huth, Wendy A De Heer, Thomas L Grifths, Frédéric E Theunissen, and Jack L Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453-458.
Shailee Jain and Alexander G Huth. 2018. Incorporating context into language encoding models for fmri. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6629-6638.
S Jat, H Tang, P Talukdar, and T Mitchel. 2020. Relating simple sentence representations in deep neural networks and the brain. In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 5137-5154. Association for Computational Linguistics (ACL).
Tim C Kietzmann, Courtney J Spoerer, Lynn KA Sorensen, Radoslaw M Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. 2019. Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences, 116(43):21854-21863.
Nikolaus Kriegeskorte and Pamela K Douglas. 2019. Interpreting encoding and decoding models. Current opinion in neurobiology, 55:167-179.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. science, 320(5880):1191-1195.
Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Kenneth A Norman, and Uri Hasson. 2020. Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space. NeuroImage, 217:116865.
Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor
Regev, et al. 2021. Narratives: fmri data for evaluating models of naturalistic language comprehension. bioRxiv, pages 2020-12.
Satoshi Nishida, Alexander G Huth, Jack L Gallant, and Shinji Nishimoto. 2015. Word statistics in large-scale texts explain the human cortical semantic representation of objects, actions, and impressions. In *Society for Neuroscience Annual Meeting*, volume 333.
Subba Reddy Oota, Jashn Arora, Manish Gupta, and Raju S Bapi. 2022a. Cross-view brain decoding. arXiv preprint arXiv:2204.09564.
Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S Bapi. 2022b. Visio-linguistic brain encoding. arXiv preprint arXiv:2204.08261.
Subba Reddy Oota, Naresh Manwani, and Raju S Bapi. 2018. fmri semantic category decoding using linguistic encoding of word embeddings. In International Conference on Neural Information Processing, pages 3-15. Springer.
Subba Reddy Oota, Vijay Rowtula, Manish Gupta, and Raju S Bapi. 2019. Stepencog: A convolutional LSTM autoencoder for near-perfect fmri encoding. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Francisco Pereira, Matthew Botvinick, and Greg Detre. 2013. Using wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments. Artificial intelligence, 194:240-252.
Francisco Pereira, Bin Lou, Brianna Pritchett, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2016. Decoding of generic mental representations from functional mri data using word embeddings. bioRxiv, page 057216.
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):1-13.
Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James S Gao, Anwar O Nunez-Elizalde, and Jack L Gallant. 2021. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience, 24(11):1628-1636.
Graeme D Ruxton and Guy Beauchamp. 2008. Time for some a priori thinking about post hoc testing. Behavioral ecology, 19(3):690-693.
Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing. PNAS, Vol:To appear.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. 2019. Inducing brain-relevant bias in natural language processing models. Advances in Neural Information Processing Systems, 32:14123-14133.
Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019. Towards sentence-level brain decoding with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7047-7054.
Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2020. Neural encoding and decoding with distributed sentence representations. IEEE Transactions on Neural Networks and Learning Systems, 32(2):589-603.
Mariya Toneva, Otilia Stretcu, Barnabás Póczos, Leila Wehbe, and Tom M Mitchell. 2020. Modeling task effects on meaning representation in the brain via zero-shot meg prediction. Advances in Neural Information Processing Systems, 33.
Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). arXiv preprint arXiv:1905.11833.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Aria Wang, Michael Tarr, and Leila Wehbe. 2019. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. Advances in Neural Information Processing Systems, 32:15501-15511.
Jing Wang, Vladimir L Cherkassky, and Marcel Adam Just. 2017. Predicting the brain activation pattern associated with the propositional content of a sentence: Modeling neural representations of events and states. Human brain mapping, 38(10):4865-4881.
Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. 2020. Fine-grained neural decoding with distributed word representations. Information Sciences, 507:256-272.
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. 2014. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. in press.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. 2014. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619-8624.
A Details of the Finetuned Models
We selected tasks for which BERT-base-cased finetuned models were available. Note that we did not finetune any of these models ourselves but leveraged the state-of-the-art finetuned models available on Huggingface. Details of the specific finetuned model checkpoints are mentioned in Table 3.
B ANOVA test results
B.1 Pereira dataset
The main effect of model was significant for the ROIs with  $95\%$  confidence with these statistics:
- Language_LH:  $\left[\mathrm{F}(9,40) = 3.95, p = 0.0052\right]$
- Language_RH:  $[F(9, 40) = 4.53, p = 0.0015]$
- Vision_Body:  $[F(9, 40) = 4.397, p = 0.00227]$
- Vision_Face:  $[F(9, 40) = 3.46, p = 0.0085]$
- Vision_Object:  $\left[\mathrm{F}(9,40) = 3.40, p = 0.0121\right]$
- Vision_Scenes:  $[F(9, 40) = 4.917, p = 0.0007]$
- Vision:  $\left[\mathrm{F}(9,40) = 3.945, p = 0.00385\right]$
DMN:  $\left[\mathrm{F}(9,40) = 6.28,p = 0.00034\right]$
- TP:  $\left[\mathrm{F}(9,40) = 6.54, p = 0.00042\right]$
B.2 Narratives-Pieman dataset