Venue: NAACL 2022

# Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

Subba Reddy Oota\({}^{1,2*}\), Jashn Arora\({}^{2*}\), Veeral Agarwal\({}^{2}\), Mounika Marreddy\({}^{2}\)

**Manish Gupta\({}^{2,3}\) and Bapi Raju Surampudi\({}^{2}\)**

\({}^{1}\)INRIA, Bordeaux, France; \({}^{2}\)IIIT Hyderabad, India; \({}^{3}\)Microsoft, India subba-reddy.oota@inria.fr, jashn.arora@research.iiit.ac.in veeral.agarwal@research.iiit.ac.in, mounika.marreddy@research.iiit.ac.cin gmanish@microsoft.com, raju.bapi@iiit.ac.in

###### Abstract

Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporoparieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Brain encoding aims at constructing neural brain activity given an input stimulus. Since the discovery of the relationship between language stimuli and functions of brain networks using fMRI [for ex., Constable et al. (2004)], researchers have been interested in understanding how the neural encoding models predict the fMRI brain activity. Several brain encoding models have been developed to (i) understand the ventral stream in biological vision Yamins et al. (2014); Kietzmann et al. (2019); Bao et al. (2020), and (ii) to study the higher-level cognition like language processing Gauthier and Levy (2019); Schrimpf et al. (2021); Schwartz et al. (2019).

Some recent studies Nishida et al. (2015); Huth et al. (2016) have been able to identify brain ROIs (Region of Interest) that respond to words that have a similar meaning and have thus built a "semantic atlas" of how the human brain organizes language. Further, several studies Oota et al. (2018); Jain and Huth (2018); Hollenstein et al. (2019) have used a wide variety of word embeddings where words represented as vectors in an embedding space are mapped to brain activation for improved neural coding.

Recently, Transformer Vaswani et al. (2017) based models like BERT Devlin et al. (2019) have been found to be very effective across a large number of natural language processing (NLP) tasks. These Transformer based models have been pretrained on millions of text instances in an unsupervised manner and further finetuned to specialize for various NLP tasks. Natural language understanding requires integrating several cognitive skills like syntactic parsing of the language structure, identifying the named entities, capturing the word meaning in the context, coreference resolution, etc. Learning from massive corpora enables these models to excel at cognitive skills required for language understanding. Interestingly, such Transformer-based neural representations have been found to be very effective for brain encoding as well Schrimpf et al. (2021).

Despite the recent advances in mapping between language Transformers and the brain activity recorded with reading Schrimpf et al. (2021), the Transformer features themselves are notoriously difficult to interpret. In recent works, Caucheteux et al. (2021a); Antonello et al. (2021) address this

issue by disentangling the high-dimensional Transformer representations of language models into four combinatorial classes: lexical, compositional, syntactic, and semantic representations to explore which class is highly associated with language cortical ROIs. Representations do not exist in a vacuum but become meaningful only when they accomplish a task. Therefore, the next logical step is to see which of these Transformer representations most effectively drive the linear mapping between language models and the brain in the context of NLP tasks. Gauthier and Levy (2019) fine-tune a pretrained BERT model on multiple tasks to find tasks best correlated with high _decoding_ performance. In this study, we investigate the correlation between brain activation and feature representations learned by different task-specific networks, and ask which tasks lead to improvements in _brainencoding_ performance.

Recently, a study using multiple computer vision tasks has shown that 3D vision task models predict better fMRI brain activity than 2D vision task models Wang et al. (2019) for visual stimuli. Inspired by the success of correlations in the vision field Wang et al. (2019), and brain encoding study of a variety of language Transformer models Schrimpf et al. (2021); Caucheteux et al. (2021b, a), we build neural language taskonomy models for brain encoding and aim to find NLP tasks that are most explanatory of brain activations for reading and listening tasks.

In this paper, we uncover insights about the association between fMRI voxel activations and representations of diverse NLP tasks representations. The predictive power of task-specific representations with brain activation is ascertained by (1) using ridge regression on such representations and predicting activations and (2) computing popular metrics like 2V2 accuracy and Pearson correlation between actual and predicted activations.

Specifically, we make the following contributions in this paper.

* Given Transformer models finetuned for various NLP tasks, we propose the problem of finding which of these are the most predictive of fMRI brain activity for reading and listening tasks.
* Our language taskonomy results reveal that Coreference Resolution, Named Entity Recognition, and Shallow Syntax Parsing tasks have higher predictive performance while reading the text. On the other hand, paraphrase detection, summarization, and Natural Language Inference tasks display better correlation during listening.
* We also perform similarity analysis between task representations from transfer learning and neural taksonomy and derive interesting cognitive insights from brain maps.

## 2 Related Work

Older methods for text-based stimulus representation include text corpus co-occurrence counts Mitchell et al. (2008); Pereira et al. (2013); Huth et al. (2016), syntactic and discourse features Wehbe et al. (2014). In recent times, both semantic and experiential attribute models have been explored for text-based stimuli. Semantic representation models include distributed word embeddings Pereira et al. (2016); Anderson et al. (2017a); Pereira et al. (2018); Toneva and Wehbe (2019); Holenstein et al. (2019); Wang et al. (2020), sentence representation models Sun et al. (2019); Toneva and Wehbe (2019); Sun et al. (2020), recurrent neural networks Jain and Huth (2018); Oota et al. (2019), and Transformer-based language models Gauthier and Levy (2019); Toneva and Wehbe (2019); Schwartz et al. (2019); Oota et al. (2022a, b). Experiential attribute models represent words in terms of human ratings of their degree of association with different attributes of experience, typically on a scale of 0-6 Anderson et al. (2019, 2020); Berezutskaya et al. (2020); Jat et al. (2020); Caucheteux et al. (2021); Antonello et al. (2021) or binary Handjaras et al. (2016); Wang et al. (2017). Fine-grained details such as lexical, compositional, syntactic, and semantic representations of narratives are factorized from Transformer-based models and utilized for training encoding models. The resulting models are better able to disentangle the corresponding brain responses in fMRI Caucheteux et al. (2021a).

In this paper, we focus on Transformer-based linguistic stimuli representations since they have been found to be most effective. Unlike previous studies which directly used existing task-agnostic pre-trained models, we train task-specific Transformer models and aim to find which model leads to the best encoding accuracy given reading and listening language stimuli 



## 3 Brain Imaging Datasets

We work with two datasets: Pereira and Narratives-Pieman. Results on Narratives-Lucy and Narratives-SlumLord show similar trends. Hence, we also show results on Narratives-Lucy and Narratives-SlumLord in the appendix.

**Pereira Dataset (Reading Sentences from Passages)** For the Pereira dataset, similar to earlier work Sun et al. (2019, 2020), we combine the data from sentence-based experiments (experiments-2 and 3) from Pereira et al. (2018). Five subjects were presented a total of 627 sentences from 48 broad topics, spanning over 168 passages, where each passage consists of 3-4 sentences. As in Pereira et al. (2018), we focused on nine brain ROIs (regions of interest) corresponding to four brain networks: (i) Default Mode Network (DMN) (linked to the functionality of semantic processing), (ii) Language Network (related to language processing, understanding, word meaning, and sentence comprehension), (iii) Task Positive Network (TP) (related to attention, salience information), and (iv) Visual Network (related to the processing of visual objects, object recognition). We briefly summarize the details of the dataset and the number of voxels corresponding to each ROI in Table 1. We use the AAL parcellation Atlas (116 \(\times\) 116 brain ROIs) to present the brain map results, since Pereira dataset contains annotations tied to this atlas.

* fMRI recorded every 1.5 sec.). We list number of voxels per ROI in this dataset in Table 2. We use the multimodal parcellation of the human cerebral cortex (Glassar Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps Glasser et al. (2016), since Narratives dataset contains annotations tied to this atlas. The data covers ten brain ROIs in the human brain, i.e., Left hemisphere (L), and Right hemisphere (R) for each of the following: (i) early auditory cortex (EAC: A1, LBelt, MBelt, PBelt, and R1) which plays a key role for sound perception since it represents one of the first cortical processing stations for sounds; (ii) auditory association cortex (AAC: A4, A5, STStdp, STSda, STSvp, STSva, STGa, and TA2) which is concerned with the memory and classification of sounds; (iii) posterior medial cortex (PMC: POS1, POS2, v23ab, d23ab, 31pv, 31pd, 7m); (iv) the temporo parieto occipital junction (TPOJ: TPOJ1, TPOJ2, TPOJ3, STV, PSL) which is a complex brain territory heavily involved in several high-level neurological functions, such as language, visuo-spatial recognition, writing, reading, symbol processing, calculation, self-processing, working memory, musical memory, and face and object recognition; and (v) the dorsal frontal lobe (DFL: L_55b, SFL, L_44, L_45, IFJA, IFSP) which covers the aspects of pragmatic processing such as discourse management, integration of prosody, interpretation of nonliteral meanings, inference making, ambiguity resolution, and error repair.

## 4 Encoding Model

To explore how and where contextual language features are represented in the brain when reading sentences and listening to stories, we extract different features spaces describing each stimulus sentence and use them in an encoding model to predict brain responses. Our reasoning is as follows. If a feature is a good predictor of a specific brain region, information about that feature is likely encoded in that region. In this paper, for both datasets, we train fMRI encoding models using Ridge regression on stimuli representations obtained using a variety of NLP tasks. The main goal of each fMRI encoder model is to predict brain

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|} \hline HOIs\(\rightarrow\) & Language & \multicolumn{4}{c|}{Vision} & \multicolumn{4}{c|}{DMN Task Positive} \\ \hline ISabb & LH & RH & BGdY & Face & Object & Scene & Vision & RH & LH \\ \hline POI & 5265[61] & 3727[43] & 478[35] & 808[41] & 12829[17] & 1900[35] \\ M02 & 4939[56] & 507[38] & 374[78] & 752[37] & 1712[15] & 1900[35] & 0594[30] \\ M04 & 5906[50] & 401[386] & 480[37] & 812[362] & 1278[18] & 1801[340] \\ M07 & 5629[50] & 5001[419] & 409[493] & 8617[37] & 3721[1245] & 1702[30] & 30408[30] \\ MI15 & 5135[61] & 4112[49] & 4941[32] & 3349[6] & 1283[159] & 31610[36] \\ \hline \end{tabular}
\end{table}
Table 1: # Voxels in each ROI in the Pereira Dataset. LH - Left Hemisphere. RH - Right Hemisphere.



responses associated with each brain region given a stimuli. In all cases, we train a model per subject separately. Following literature on brain encoding (Caucheteux et al., 2021; Toneva et al., 2020), we choose to use a ridge regression model instead of more complicated models. We plan to explore more such models as part of future work. We follow K-fold (K=10) cross-validation. All the data samples from K-1 folds were used for training, and the model was tested on samples of the left-out fold. We used sklearn's ridge-regression with default parameters, 10-fold cross-validation, Stochastic-Average-Gradient Descent Optimizer, Huggingface for Transformer models, MSE loss function, and L2-decay (\(\lambda\)) as 1.0. We used BERT Word-Piece tokenizer for the linguistic Transformer input. All experiments were conducted on a machine with 1 NVIDIA GEFORCE-GTX GPU with 16GB GPU RAM. We make the code publicly available1.

Footnote 1: [https://tinyurl.com/langTask](https://tinyurl.com/langTask)

### Feature Spaces

To simultaneously test representations from multiple NLP tasks, we used the latent space features from each of the following ten popular NLP tasks: coreference resolution (CR), named entity recognition (NER), natural language inference (NLI), paraphrase detection (PD), question answering (QA), sentiment analysis (SA), semantic role labeling (SRL), shallow syntax parsing (SS), summarization (Sum) and word sense disambiguation (WSD). All of these are discriminative NLP tasks, and thus we use models obtained by task-specific finetuning of the same pretrained Transformer encoder model (BERT-base-cased with dimensionality=768). Given an input sentence, each task Transformer outputs token representations at the final layer. We use the #tokens \(\times\) 768 dimension vector obtained from the last hidden layer to obtain latent features for the stimuli. We then build individual ridge regression models with the extracted latent features to predict brain responses and measure the correlation between the prediction and the true response.

**Pereira:** Since individual sentences were presented to the subjects while modeling, sentences were passed one by one to the task Transformer model, and average-pooled representations were used to encode the sentence stimuli.

**Narratives-Pieman:** Due to the constraint on input sequence length for BERT (512), we considered a window size of 10 sentences with the last two sentences of one window overlapping with the next to be given as input to the BERT model. We use the average-pool representation from BERT to encode text stimuli. To get the representation for a TR, we pooled the representations of only those words of the sentences in that TR.

### Task Descriptions

Here we describe the functionality of each NLP task that we used for fMRI encoding. **CR:** involves finding all expressions that refer to the same entity in a text. **PD:** involves taking a passage - either spoken or written - and rewording it in shorter or own words. **Summarization (Sum):** involves selecting a few important sentences from a document or paragraph. **NER:** involves detection of the named entities such as person names, location names, company names from a given text. **NLI:** investigates the entailment relationship between two texts: premise and hypothesis. **QA:** aims to select an answer given a passage, a question, and a set of candidate answers. **SA:** involves determining whether a piece of text is positive, negative, or neutral. **SRL:** assigns labels to words or phrases in a sentence that indicates their semantic role in the sentence, such as that of an agent, goal, or result. **SS:** provides an approximation of phrase-syntactic structure of sentences. **WSD:** involves determining which sense (meaning) of a word is activated by the use of the word in a particular context.

Syntactic reasoning is rather shallow compared to deep semantic reasoning. Syntactic reasoning follows somewhat objective grammar rules. Comparatively semantic reasoning is often subjective in nature and complex. The emerging evidence from fMRI studies (Fedorenko et al., 2020, 2012) also points out that processing of both syntax and semantics is distributed in the brain and it is only when violations of these processes are probed, we see localization of function (Friederici et al., 2003). Thus, in this work, we explore syntactic and semantic tasks separately. Of the above mentioned tasks, NER and SS are syntactic, while the others involve semantic reasoning.

Our selection of these tasks was based on the following design principles: (1) We wanted to select a set of tasks covering diverse cognitive-linguistic skills. (2) We wanted to select tasks that are a part of popular NLP benchmarks like GLUE (Wang et al., 2018). (3) We selected tasks for which 

BERT-base-cased finetuned models were available. Note that we did not finetune any of these models ourselves but leveraged the state-of-the-art fine-tuned models available on Huggingface. Details of the specific finetuned model checkpoints are mentioned in Table 3 in the Appendix.

### Evaluation Metrics

We evaluate our models using popular brain encoding evaluation metrics described in the following. Given a subject and a brain region, let \(N\) be the number of samples. Let \(\{Y_{i}\}^{N}_{i=1}\) and \(\{\hat{Y}_{i}\}^{N}_{i=1}\) denote the actual and predicted voxel value vectors for the \(i^{th}\) sample. Thus, \(Y\in R^{N\times V}\) and \(\hat{Y}\in R^{N\times V}\) where \(V\) is the number of voxels in that region.

**2V2**Accuracy is computed as 2V2Acc=\(\frac{1}{Nc_{2}}\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}I[cosD(Y_{i},\hat{Y}_{i})+ cosD(Y_{j},\hat{Y}_{j})<cosD(Y_{i},\hat{Y}_{j})+cosD(Y_{j},\hat{Y}_{i})]\) where \(cosD\) is the cosine distance function. \(I[c]\) is an indicator function such that \(I[c]=1\) if \(c\) is true, else it is 0. The higher the 2V2 accuracy, the better.

**Pearson Correlation (PC)** is computed as PC=\(\frac{1}{N}\sum_{i=1}^{n}corr[Y_{i},\hat{Y}_{i}]\) where corr is the correlation function.

**Mean Absolute Error (MAE)** is computed as MAE=\(\frac{1}{N}\sum_{i=1}^{n}||Y_{i}-\hat{Y}_{i}|\).

**Statistical Significance**: In order to estimate the statistical significance of the performance differences (across all results), we performed one-way ANOVA on the mean values for the subjects. In all such cases we report p-values corrected using Bonferroni correction.

### Neural Language Tasks Similarity Computation

To estimate the similarity between 10 language tasks, we took the prediction performance scores across all the voxels in Pereira (97,539) and Narratives-Pieman datasets (10,732). To analyze the relationship between tasks based on neural representations, we calculated the Pearson correlation between predicted voxels of each task with the remaining tasks. These Pearson correlation values were used to construct heatmaps and the task similarity trees(dendograms) using hierarchical clustering for Pereira and Narratives-Pieman datasets.

## 5 Results

In order to assess the performance of the fMRI encoder models learned using the representations

Figure 1: Pereira – 2V2 Accuracy (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of NLP tasks. Results are averaged across all participants. CR, NER, and SS perform the best.

 

from a variety of NLP tasks, we computed the 2V2 accuracy and Pearson correlation coefficient between the predicted and true responses across various ROIs for both the reading (Pereira) dataset (Fig. 1) as well as the listening (Narratives-Pieman) dataset (Fig. 2).

### Encoding performance of Language Task models for reading vs listening tasks

**Reading Sentences (Pereira):** From Fig. 1, we observe that tasks such as CR, NER, SRL, and SS appear to have a better correlation to the brain responses compared to the other tasks. In order to estimate the statistical significance of the performance differences, we performed one-way ANOVA on the mean correlation values for the subjects across the ten language tasks for the nine brain ROIs. The main effect of the ANOVA test was significant for all the ROIs with p\(\leq 10^{-2}\) with confidence 95% (see Appendix for detailed ANOVA results). Further, _post hoc_ pairwise comparisons [34] confirmed the visual observations that on both 2V2 accuracy and Pearson correlation measures, tasks such as CR, NER, SRL, and SS performed significantly better compared to other tasks (see Appendix for pairwise comparison results). These results demonstrate that when reading a sentence, information processing operations related to recognizing named entities, labeling semantic roles to the constituents of a sentence, identifying the references from a sentence to the given topic (concept), and syntactic processing may be engaged.

Further, we observe that the ROI corresponding to language processing in the left hemisphere (Language_LH) has higher encoding performance than that of the right hemisphere (Language_RH). This is in line with the left hemisphere dominance for language processing [68]. Also, lateral visual ROIs such as Vision_Object, Vision_Body, Vision_Face, and Vision ROIs display higher correlation with the language tasks associated with named entities (NER), relating the entities (CR), and syntax processing (SS). Higher correlations with all the visual brain regions point to the possible alignment of visual and language regions for semantic understanding [75] in a reading task. Finally, across all regions, pretrained BERT model has worse correlation compared to at least 5 other task models.

**Listening Stories (Narratives-Pieman):** From Fig. 2, we observe that the profiles of performance

Figure 2: Narratives-Pieman – 2V2 Accuracy (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of NLP tasks. Results are averaged across all participants. NLI, PD, and Summarization perform the best.

 

show low scores in the early auditory cortex (EAC), auditory association cortex (AAC); average scores in TPOJ and DFL; and superior scores in PMC. This aligns with the known language hierarchy for spoken language understanding Nastase et al. (2020). Tasks such as PD, Summarization, and NLI seem to yield better performance in predicting the brain responses than the other NLP tasks across all the ROIs. These Pearson correlation (\(\tau\)) results are comparatively much higher compared to those obtained using pretrained (task-agnostic) GPT2 model in Caucheteux et al. (2021a) (\(\tau\) ranging from \(0.02-0.06\)). As shown in Fig. 2, our method obtains much higher correlations (\(\tau\) ranging from \(0.02-0.229\)). Similar to the Pereira dataset, we estimate the statistical significance of the performance differences using the one-way ANOVA test. The main effect of task was significant for all the ROIs with p\(\leq 10^{-3}\) with confidence 95% (see Appendix for detailed ANOVA results). Also, _Post hoc_ pairwise comparisons Ruxton and Beauchamp (2008) revealed that on both 2V2 accuracy and Pearson correlation measures, tasks such as PD, Sum, and NLI performed significantly better compared to other tasks (see Appendix for pairwise comparison results).

Further, from Fig. 2, we see that the bilateral posterior medial cortex (PMC) associated with higher language function exhibits a higher correlation among all the brain ROIs. ROIs, including bilateral TPOJ and bilateral DFL, yield higher correlations with the five NLP tasks, which is in line with the language processing hierarchy in the human brain. Finally, across all regions, pretrained BERT model has worse correlation compared to at least 5 other task models.

In summary, different and distinct language Taskonomy features seem to be related to the encoding performance in reading versus listening tasks. CR, NER, SRL, and SS perform better for reading. PD, Sum, and NLI perform better for listening. While listening the subject is cognitively more involved in the activity compared to reading Buchweitz et al. (2009). Thus, it makes sense that shallow tasks like NER and SS are useful for reading while more complex NLP tasks like PD, Sum and NLI are effective for encoding listening stimuli.

### Language Task Similarity Computation

Pearson correlation values between predicted responses for each pair of tasks were used to construct the similarity matrix with heatmap for both Pereira and Narratives-Pieman datasets, as shown in Figs. 3 and 4. We observe that the following task pairs are highly correlated for the Pereira dataset: (NER and CR), (SS and CR) and (PD and Sum). Also these task pairs are highly correlated for the Narratives-Pieman dataset: (CR and NLI), (NLI and SA) and (PD and Sum). Similarities are relatively higher for Narratives-Pieman compared to the Pereira dataset. Surprisingly, the (NLI, SA) pair has lowest similarity for Pereira (reading) and close to highest in Narratives-Pieman (listening). We hypothesize that this is because sentiment is best conveyed while the subject is listening.

**Reading sentences (Pereira):** The stimulus sentences from the Pereira dataset were fed as input to each of the 10 task Transformers. The similarity among the resulting representations was analyzed using hierarchical clustering, and the clusters are visualized as dendrograms in Fig. 5 (left). We observe that the tasks are clustered into three groups denoted using red, green, and blue colors. Next, we wished to check if similar task grouping is observed on brain activations predicted by ridge regression trained on task-specific representations. Hence, similar clustering analysis was conducted on the neural space representations, and the clusters are visualized as dendrograms in Fig. 5 (right)

Figure 4: Narratives-Pieman – Prediction Task Similarity constructed from the task-wise brain response predictions across 10 tasks averaged across all subjects.

Figure 3: Pereira – Prediction Similarity Matrix constructed from the task-wise brain response predictions across 10 tasks averaged across all subjects.

 

across all subjects. Interestingly, the tree derived from brain representation also shows a similar distribution of tasks across the three groups. Similar dendrograms for individual subjects are illustrated in Appendix-Fig. 11.

**Listening Stories (Narratives-Pieman):** Fig. 6 compares the task similarity tree based on the patterns from the pretrained task Transformers, with the task similarity tree generated based on similarity in brain response prediction performance averaged across all subjects. We observe that the tasks are clustered into three groups denoted using red, green, and blue colors. Again, the tree derived from brain representation also shows a similar distribution of tasks across the three groups. Dendrograms for individual subjects are in the Appendix-Fig. 12.

### Brain maps for whole brain predictions

The mean absolute error (MAE) between predictive and actual responses is obtained using individual task features from the taskonomy. MAE values are obtained for all the voxels in the brain for both the reading (Fig. 7) and listening datasets (Fig. 8).

In the **reading task**, we observe from Fig. 7 that CR has lower MAE compared to PD which in turn has lower MAE compared to the NLI task (brain maps for the other tasks are reported in Fig. 17 in the Appendix). Overall, for the reading stimuli, tasks such as NLI, QA, and SA display higher MAE values. To further investigate which sub ROIs (LPTG, LMTG, LATG, LFus, Lpar, Lang, LIFGorb, LIFG, LaMFG, LpMFG, and LmMFG) of the Language network are related to the predictive task features, we train encoding models for all the sub ROIs for the best encoding task, i.e., for the CR task (see Fig. 14 in Appendix). We notice that both LMTG (middle temporal gyrus) and LPTG (posterior temporal gyrus) are more accurately predicted than the other sub ROIs. On the other hand, LIFG-orb displays a lower Pearson correlation for the CR task. The presence of superior encoding information in the ROIs in the temporal gyrus as compared to those in the inferior frontal gyrus seems to mirror similar observations seen in decoder performance Anderson et al. (2017b).

On the other hand, in the **listening task**, we observe from Fig. 8 that Paraphrase and WSD display lower MAE values compared to QA task (brain maps for the other tasks are reported in Fig. 18 in the Appendix). Taken together, for listening stimuli, tasks such as NER, QA, SA, CR, and SS display higher MAE values. From Fig. 8, we see that ROIs such as EAC and AAC have higher MAE compared to PMC and TPOJ brain ROIs.

We further demonstrate the prediction performance of the encoder model trained on sub ROIs for the paraphrase task in Fig. 15 in the Appendix. It can be observed that sub ROIs such as Pos1 and Pos2 have a higher Pearson correlation than other sub ROIs of the PMC region. Both sfl and l55b display a higher correlation among all the sub ROIs for the DFL ROI. However, all the sub ROIs in the TPOJ yield higher correlation, as shown in Fig. 15. The control and attention ROIs in the posterior cingulate cortex (for ex., POS1 in PMC), together with the superior frontal language region (sfl in DFL) and TPOJ, are part of the well-known language network associated with narrative comprehension Mastase et al. (2020), and it is heartening to see that task features from PD task also relate to semantic analysis of the ongoing narrative.

### Discussion

(1) We used a ridge regression model instead of more complicated models for encoding. We believe that more complex models can lead to further exciting insights. (2) We experimented with 10 NLP tasks. Models can be pretrained for more such tasks to check if other tasks are better predictive of voxel activations. (3) We leveraged models finetuned using datasets of different sizes across

Figure 5: Left: Pereira Dendrogram constructed using similarity on representations from task-specific Transformer encoder models with stimuli from the dataset passed as input. Right: Pereira Dendrogram constructed using similarity matrix shown in Fig. 3.

Figure 6: Left: Narratives-Pieman Dendrogram constructed using similarity on representations from task-specific Transformer encoder models with stimuli from the dataset passed as input. Right: Narratives-Pieman Dendrogram constructed using similarity matrix shown in Fig. 4.

 

tasks. While a fair comparison of dataset sizes across tasks is impossible, we understand that this could have resulted in some bias in our results. (4) We used a different dataset for reading vs listening. While we believe that the differences in task-specific model performances across reading and listening are mainly due to the learned stimulus representations, but they could also arise from other factors such as experimental conditions, the text domain of the stimuli or number of voxels, etc. (5) On Natural Language Understanding tasks such as NLI, SA, QA and PD, Gauthier and Levy (2019) observed that scrambled sentence representations gave better decoding performance. But encoding models (especially for the listening task), scrambled order would be detrimental to making sense of what is being heard. It is an interesting future task to see if the opposite result is seen in the case of brain encoding models. It is plausible that brain uses encoding models in a flexible way when it comes to decoding Kriegeskorte and Douglas (2019). Kriegeskorte and Douglas (2019) mention that "Decoding models can help reveal whether particular information is present in a brain region in a format the decoder can exploit. Encoding models make comprehensive predictions about representational spaces." In this sense, results of current work are not directly comparable to those of Gauthier and Levy (2019).

## 6 Conclusion

In this paper, we studied the effectiveness of task specific NLP models for brain encoding. We observe that building individual encoding models and exploiting existing relationships among models can provide a more in-depth understanding of the neural representation of language information. Our experiments on Pereira and Narrative datasets lead to interesting cognitive insights.

Figure 8: Narratives-Pieman BrainMaps: Mean absolute error (MAE) between predictive voxels and actual voxels using task features from Taskonomy in one sample subject (subject 1) of PieMan dataset. Predictive regions of various tasks are different across tasks. The MAE values of each brain ROI: PD task (EAC: 0.74, AAC: 0.66, PMC: 0.60, TPOJ: 0.61, and DFL: 0.694), WSD task(EAC: 0.83, AAC: 0.75, PMC: 0.68, TPOJ: 0.68, and DFL: 0.76), QA task (EAC: 0.92, AAC: 0.83, PMC: 0.74, TPOJ: 0.75, and DFL: 0.76).

Figure 7: Pereira BrainMaps: Mean absolute error (MAE) between predictive voxels and actual voxels using task features from Taskonomy in one sample subject (subject 1). Predictive regions of different tasks are dissimilar across tasks. The MAE values of each brain ROI are: CR (Language: 0.64, Visual: 0.57, DMN: 1.19, TP: 0.67), PD (Language: 0.81, Visual: 0.74, DMN: 1.34, TP: 0.87) and NLI (language: 1.9, Visual: 1.88, DMN: 2.1, TP: 2.03).

 



## 7 Ethical Statement

We reused publicly available datasets for this work: Pereira and Narratives. We did not collect any new dataset.

Pereira dataset can be downloaded from [https://osf.io/crwz7/](https://osf.io/crwz7/). Please read their terms of use2 for more details.

Footnote 2: [https://github.com/CenterForOpenScience/cos.io/blob/master/TERMS_OF_USE.md](https://github.com/CenterForOpenScience/cos.io/blob/master/TERMS_OF_USE.md)

Narratives dataset can be dowloaded from [https://datasets.datalad.org/](https://datasets.datalad.org/)

?dir=/labs/hasson/narratives. Please read their terms of use3 for more details.

Footnote 3: [https://datasets.datalad.org/labs/hasson/narratives/stimuli/README](https://datasets.datalad.org/labs/hasson/narratives/stimuli/README)

We do not foresee any harmful uses of this technology.

## References

* Anderson et al. (2017a) Andrew J Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. 2017a. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. _Transactions of the Association for Computational Linguistics_, 5:17-30.
* Anderson et al. (2017b) Andrew James Anderson, Jeffrey R Binder, Leonardo Fernandino, Colin J Humphries, Lisa L Conant, Mario Aguilar, Xixi Wang, Donias Doko, and Rajeev DS Raizada. 2017b. Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation. _Cerebral Cortex_, 27(9):4379-4395.
* Anderson et al. (2019) Andrew James Anderson, Jeffrey R Binder, Leonardo Fernandino, Colin J Humphries, Lisa L Conant, Rajeev DS Raizada, Feng Lin, and Edmund C Lalor. 2019. An integrated neural decoder of linguistic and experiential meaning. _Journal of Neuroscience_, 39(45):8969-8987.
* Anderson et al. (2020) Andrew James Anderson, Kelsey McDermott, Brian Rooks, Kathi L Heffner, David Dodell-Feder, and Feng V Lin. 2020. Decoding individual identity from brain activity elicited in imagining common experiences. _Nature communications_, 11(1):1-14.
* Antonello et al. (2021) Richard Antonello, Javier Turek, Vy Vo, and Alexander Huth. 2021. Low-dimensional structure in the space of language representations is reflected in brain responses. _arXiv preprint arXiv:2106.05426_.
* Bao et al. (2020) Pinglei Bao, Liang She, Mason McGill, and Doris Y Tsao. 2020. A map of object space in primate inferotemporal cortex. _Nature_, 583(7814):103-108.
* Berezutskaya et al. (2020) Julia Berezutskaya, Zachary V Freudenburg, Luca Ambrogioni, Umut Guclu, Marcel AJ van Gerven, and Nick F Ramsey. 2020. Cortical network responses map onto data-driven features that capture visual semantics of movie fragments. _Scientific reports_, 10(1):1-21.
* Binder et al. (2009) Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. _Cerebral cortex_, 19(12):2767-2796.
* Buchweitz et al. (2009) Augusto Buchweitz, Robert A Mason, Leda Tomitch, and Marcel Adam Just. 2009. Brain activation for reading and listening comprehension: an fmri study of modality effects and individual differences in language comprehension. _Psychology & neuroscience_, 2(2):111-123.
* Caucheteux et al. (2021a) Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. 2021a. Disentangling syntax and semantics in the brain with deep networks. In _International Conference on Machine Learning_, pages 1336-1348. PMLR.
* Caucheteux et al. (2021b) Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. 2021b. Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects. _arXiv preprint arXiv:2110.06078_.
* Constable et al. (2004) R Todd Constable, Kenneth R Pugh, Ella Berroya, W Einar Mencl, Michael Westerveld, Weijia Ni, and Donald Shankweiler. 2004. Sentence complexity and input modality effects in sentence comprehension: an fmri study. _NeurolImage_, 22(1):11-21.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Fedorenko et al. (2020) Evelina Fedorenko, Idan Asher Blank, Matthew Siegelman, and Zachary Mineroff. 2020. Lack of selectivity for syntax relative to word meanings throughout the language network. _Cognition_, 203:104348.
* Fedorenko et al. (2012) Evelina Fedorenko, Alfonso Nieto-Castanon, and Nancy Kanvisher. 2012. Lexical and syntactic representations in the brain: an fmri investigation with multi-voxel pattern analyses. _Neuropsychologia_, 50(4):499-513.
* Friederici et al. (2003) Angela D Friedericti, Shirley-Ann Ruschemeyer, Anja Hanne, and Christian J Fiebach. 2003. The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. _Cerebral cortex_, 13(2):170-177.
* Gauthier and Levy (2019) Jon Gauthier and Roger Levy. 2019. Linking artificial and human neural representations of language. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 529-539.



Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Ya-coub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. 2016. A multimodal parcellation of human cerebral cortex. _Nature_, 536(7615):171-178.
* Handjaras et al. [2016] Giacomo Handjaras, Emiliano Ricciardi, Andrea Leo, Alessandro Lenci, Luca Cecchetti, Mirco Cosotini, Giovanna Marotta, and Pietro Pietrini. 2016. How concepts are encoded in the human brain: a modality independent, category-based cortical organization of semantic knowledge. _Neuroimage_, 135:232-242.
* Hollenstein et al. [2019] Nora Hollenstein, Antonio de la Torre, Nicolas Langer, and Ce Zhang. 2019. Cognival: A framework for cognitive word embedding evaluation. In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_, pages 538-549.
* Huh et al. [2016] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frederic E Theunissen, and Jack L Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. _Nature_, 532(7600):453-458.
* Jain and Huth [2018] Shailee Jain and Alexander G Huth. 2018. Incorporating context into language encoding models for fmri. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_, pages 6629-6638.
* Jat et al. [2020] S Jat, H Tang, P Talukdar, and T Mitchel. 2020. Relating simple sentence representations in deep neural networks and the brain. In _ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_, pages 5137-5154. Association for Computational Linguistics (ACL).
* Kietzmann et al. [2019] Tim C Kietzmann, Courtney J Spoerer, Lynn KA Sorensen, Radoslaw M Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. 2019. Recurrence is required to capture the representational dynamics of the human visual system. _Proceedings of the National Academy of Sciences_, 116(43):21854-21863.
* Kriegeskorte and Douglas [2019] Nikolaus Kriegeskorte and Pamela K Douglas. 2019. Interpreting encoding and decoding models. _Current opinion in neurobiology_, 55:167-179.
* Mitchell et al. [2008] Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. _science_, 320(5880):1191-1195.
* Nastase et al. [2020] Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Kenneth A Norman, and Uri Hasson. 2020. Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space. _NeuroImage_, 217:116865.
* Navstase et al. [2021] Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshuran, Mor Regev, et al. 2021. Narratives: fmri data for evaluating models of naturalistic language comprehension. _bioRxiv_, pages 2020-12.
* Nishida et al. [2015] Satoshi Nishida, Alexander G Huth, Jack L Gallant, and Shinji Nishimoto. 2015. Word statistics in large-scale texts explain the human cortical semantic representation of objects, actions, and impressions. In _Society for Neuroscience Annual Meeting_, volume 333.
* Oota et al. [2022a] Subba Reddy Oota, Jashn Arora, Manish Gupta, and Raju S Bapi. 2022a. Cross-view brain decoding. _arXiv preprint arXiv:2204.09564_.
* Oota et al. [2022b] Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S Bapi. 2022b. Visio-linguistic brain encoding. _arXiv preprint arXiv:2204.08261_.
* Oota et al. [2018] Subba Reddy Oota, Naresh Manwani, and Raju S Bapi. 2018. fmri semantic category decoding using linguistic encoding of word embeddings. In _International Conference on Neural Information Processing_, pages 3-15. Springer.
* Oota et al. [2019] Subba Reddy Oota, Vijay Rowtula, Manish Gupta, and Raju S Bapi. 2019. Stepencog: A convolutional lstm autoencoder for near-perfect fmri encoding. In _2019 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8. IEEE.
* Pereira et al. [2013] Francisco Pereira, Matthew Botvinick, and Greg Detre. 2013. Using wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments. _Artificial intelligence_, 194:240-252.
* Pereira et al. [2016] Francisco Pereira, Bin Lou, Brianna Pritchett, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2016. Decoding of generic mental representations from functional mri data using word embeddings. _bioRxiv_, page 057216.
* Pereira et al. [2018] Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. _Nature communications_, 9(1):1-13.
* Popham et al. [2021] Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James S Gao, Anwar O Nunez-Elizalde, and Jack L Gallant. 2021. Visual and linguistic semantic representations are aligned at the border of human visual cortex. _Nature Neuroscience_, 24(11):1628-1636.
* Ruxton and Beauchamp [2008] Graeme D Ruxton and Guy Beauchamp. 2008. Time for some a priori thinking about post hoc testing. _Behavioral ecology_, 19(3):690-693.
* Schrimpf et al. [2021] Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing. _PNAS_, Vol:To appear 

Dani Schwartz, Mariya Toneva, and Leila Wehbe. 2019. Inducing brain-relevant bias in natural language processing models. _Advances in Neural Information Processing Systems_, 32:14123-14133.
* Sun et al. (2019) Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019. Towards sentence-level brain decoding with distributed representations. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 7047-7054.
* Sun et al. (2020) Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2020. Neural encoding and decoding with distributed sentence representations. _IEEE Transactions on Neural Networks and Learning Systems_, 32(2):589-603.
* Toneva et al. (2020) Mariya Toneva, Otilia Stretcu, Barnabas Poczos, Leila Wehpe, and Tom M Mitchell. 2020. Modeling task effects on meaning representation in the brain via zero-shot meg prediction. _Advances in Neural Information Processing Systems_, 33.
* Toneva and Webhe (2019) Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). _arXiv preprint arXiv:1905.11833_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355.
* Wang et al. (2019) Aria Wang, Michael Tarr, and Leila Wehbe. 2019. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. _Advances in Neural Information Processing Systems_, 32:15501-15511.
* Wang et al. (2017) Jing Wang, Vladimir L Cherkassky, and Marcel Adam Just. 2017. Predicting the brain activation pattern associated with the propositional content of a sentence: Modeling neural representations of events and states. _Human brain mapping_, 38(10):4865-4881.
* Wang et al. (2020) Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. 2020. Fine-grained neural decoding with distributed word representations. _Information Sciences_, 507:256-272.
* Wehbe et al. (2014) Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. 2014. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. _in press_.
* Yamins et al. (2014) Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. 2014. Performance-optimized hierarchical models predict neural responses in higher visual cortex. _Proceedings of the national academy of sciences_, 111(23):8619-8624.

## Appendix A Details of the Finetuned Models

We selected tasks for which BERT-base-cased fine-tuned models were available. Note that we did not finetune any of these models ourselves but leveraged the state-of-the-art finetuned models available on Huggingface. Details of the specific finetuned model checkpoints are mentioned in Table 3.

## Appendix B ANOVA test results

### Pereira dataset

The main effect of model was significant for the ROIs with 95% confidence with these statistics:

* Language_LH: [F(9, 40) = 3.95, \(p\)=0.0052]
* Language_RH: [F(9, 40) = 4.53, \(p\)=0.0015]
* Vision_Body: [F(9, 40) = 4.397, \(p\)=0.00227]
* Vision_Face: [F(9, 40) = 3.46, \(p\)=0.0085]
* Vision_Object: [F(9, 40) = 3.40, \(p\)=0.0121]
* Vision_Scenes: [F(9, 40) = 4.917, \(p\)=0.0007]
* Vision: [F(9, 40) = 3.945, \(p\)=0.00385]
* DMN: [F(9, 40) = 6.28, \(p\)=0.00034]
* TP: [F(9, 40) = 6.54, \(p\)=0.00042]

### Narratives-Pieman dataset

The main effect of model was significant for the ROIs with 95% confidence with these statistics:

* EAC_L [F(9,810)=3.88, \(p\)=0.00009]
* EAC_R [F(9,810)=3.34, \(p\)=0.00055]
* AAC_L [F(9,810)=5.37, \(p\)=0.000007]
* AAC_R [F(9,810)=6.955, \(p\)=0.00000]
* PMC_L [F(9,810)=37.21, \(p\)=0.00000]
* PMC_R [F(9,810)=31.62, \(p\)=0.00000]
* TPCJ_L [F(9,810)=9.166, \(p\)=0.00000]
* TPCJ_R [F(9,810)=7.797, \(p\)=0.00000]
* DFL_L [F(9,810)=12.445, \(p\)=0.00000]
* DFL_R [F(9,810)=12.27, \(p\)=0.00000] 



\begin{table}
\begin{tabular}{|l|c|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.027 \\ \hline CR & Sum & 0.008 \\ \hline CR & P/D & 0.0417 \\ \hline NI,I SA & 0.056 \\ \hline NLJ & SS & 0.00011 \\ \hline NAI & SS & 0.00011 \\ \hline SA & SP & 0.0322 \\ \hline SS & SSU & 0.00002 \\ \hline SN & SP & 0.00002 \\ \hline SSL & SP & 0.0059 \\ \hline SSL & SP & 0.00004 \\ \hline SSLL & Sum & 0.0545 \\ \hline SSLL & PD & 0.08876 \\ \hline \end{tabular}
\end{table}
Table 10: Pairwise comparison one-way ANOVA results for Fusion region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline NNLS & SS & 0.00015 \\ \hline TD & SS & 0.002 \\ \hline SAL & SS & 0.0958 \\ \hline SS & WSD & 0.052 \\ \hline \end{tabular}
\end{table}
Table 11: Pairwise comparison one-way ANOVA results for EAC-L region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CNSL & SS & 0.00157 \\ \hline DD & SS & 0.0020 \\ \hline SAR & SS & 0.0958 \\ \hline SS & WSD & 0.052 \\ \hline \end{tabular}
\end{table}
Table 12: Pairwise comparison one-way Anova results for EAC-R region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline TNLS & SS & 0.00157 \\ \hline ND & SS & 0.0015 \\ \hline DT & SS & 0.0020 \\ \hline SD & SS & 0.0020 \\ \hline SSL & SRL & 0.0084 \\ \hline SSD & PR & 0.00025 \\ \hline SSL & QA & 0.00128 \\ \hline \end{tabular}
\end{table}
Table 13: Pairwise comparison one-way Anova results for AAC-L region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CNSL & SS & 0.00157 \\ \hline UD & SS & 0.0002 \\ \hline SD & SS & 0.0020 \\ \hline SSL & SRL & 0.0084 \\ \hline SS & PD & 0.00025 \\ \hline SSL & QA & 0.00128 \\ \hline \end{tabular}
\end{table}
Table 10: Pairwise comparison one-way ANOVA results for Vision region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CNSL & SS & 0.00157 \\ \hline CD & SS & 0.0015 \\ \hline DR & SS & 0.0002 \\ \hline SL & PS & 0.00020 \\ \hline SSL & PS & 0.00002 \\ \hline SSL & PS & 0.00002 \\ \hline SSL & PS & 0.00002 \\ \hline SSL & PS & 0.00545 \\ \hline SSL & PSD & 0.08876 \\ \hline \end{tabular}
\end{table}
Table 11: Pairwise comparison one-way ANOVA results for EAC-L region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CNSL & SS & 0.00157 \\ \hline RD & SS & 0.00020 \\ \hline SD & SS & 0.00020 \\ \hline SSL & MS & 0.00157 \\ \hline SD & PS & 0.00002 \\ \hline SSL & PS & 0.00002 \\ \hline SSL & PS & 0.00002 \\ \hline SSL & PS & 0.00002 \\ \hline SSL & PS & 0.00545 \\ \hline SSL & PS & 0.08876 \\ \hline \end{tabular}
\end{table}
Table 11: Pairwise comparison one-way ANOVA results for EAC-L region

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.027 \\ \hline CR & Sum & 0.008 \\ \hline IN & SA & 0.056 \\ \hline NLJ & SS & 0.0001 \\ \hline NLJ & SS & 0.00011 \\ \hline NLJ & SS & 0.00545 \\ \hline SSL & PS & 0.08876 \\ \hline \end{tabular}
\end{table}
Table 6: Pairwise comparison one-way ANOVA results for Vision_body region

\begin{table}
\begin{tabular}{|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.027 \\ \hline CR & Sum & 0.0020 \\ \hline SC & SB & 0.0084 \\ \hline SS & PD & 0.00025 \\ \hline SSL & QA & 0.00128 \\ \hline \end{tabular}
\end{table}
Table 12: Pairwise comparison one-way ANOVA results for EAC-R region

\begin{table}
\begin{tabular}{|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.027 \\ \hline CR & Sum & 0.008 \\ \hline RL & Sum(0.0431) \\ \hline NLJ & SS & 0.00011 \\ \hline NLJ & SS & 0.00011 \\ \hline SLA & Sum & 0.0188 \\ \hline SSL & PN & 0.00002 \\ \hline SSL & PN & 0.00002 \\ \hline SSL & PN & 0.00002 \\ \hline SSL & PN & 0.0545 \\ \hline SSL & PN & 0.08876 \\ \hline \end{tabular}
\end{table}
Table 10: Pairwise comparison one-way ANOVA results for Fusion region

\begin{table}
\begin{tabular}{|p{113.8pt}|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.027 \\ \hline CR & Sum & 0.008 \\ \hline ML & SA & 0.056 \\ \hline NLJ & SS & 0.00011 \\ \hline NLJ & SS & 0.00011 \\ \hline SSL & PN & 0.00002 \\ \hline SSL & PN & 0.0545 \\ \hline SSL & PN & 0.052 \\ \hline \end{tabular}
\end{table}
Table 11: Pairwise comparison one-way ANOVA results for EAC-L region



\begin{table}
\begin{tabular}{|c|l|r|} \hline T1 & T2 & p-value \\ \hline CR & NER & 1.5273e-9 \\ \hline CR & NLI & 0.000042 \\ \hline CR & NLI & 0.0000042 \\ \hline CR & SS & 0.00039 \\ \hline CRF & PD & 0.00011 \\ \hline CR & QA & 0.010101 \\ \hline NER & NLI & 8.66062e-II \\ \hline NER & SA & 8.67732e-11 \\ \hline NER & SRL & 8.88714e-11 \\ \hline NER & PS & 8.86062e-II \\ \hline NER & sum & 1.6034e-10 \\ \hline NER/NSD & 1.01319e-10 \\ \hline NER & QA & 8.86657e-11 \\ \hline NER & NAIL & SSA & 0.0059 \\ \hline NNLI & SS & 8.87086e-11 \\ \hline SNL & Sun & 0.000371 \\ \hline NER & SSN & 0.000191 \\ \hline NER & SSN & 0.0000021 \\ \hline SNL & OPR & 8.87554e-11 \\ \hline SNS & SumD & 0.000126402 \\ \hline SNS & WSD & 0.000128239 \\ \hline SNS & OA & 1.3129e-10 \\ \hline FP & SUN & 0.00619 \\ \hline PN & WSD & 0.0036 \\ \hline \end{tabular}
\end{table}
Table 16: Pairwise comparison one-way Anova results for PMC-R region

\begin{table}
\begin{tabular}{|c|l l|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.00030 \\ \hline CR & PR & 0.0072 \\ \hline NER & FD & 0.0291 \\ \hline NI & SSI & 0.000013 \\ \hline SSA & SS & 0.0299 \\ \hline SS & SRL & 0.0011 \\ \hline SSD & PP & 2.97929e-7 \\ \hline SSS & WSBD & 0.0444 \\ \hline SS & QA & 0.00099 \\ \hline PD & Sum & 0.039 \\ \hline \end{tabular}
\end{table}
Table 14: Pairwise comparison one-way Anova results for AAC-R region

Figure 9: Narratives-Lucy Dataset: 2V2 Accuracy (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of NLP tasks. Results are averaged across all participants. NLI, Paraphrase, and Summarisation perform the best.





\begin{table}
\begin{tabular}{|c|l||r|} \hline T1 & T2 & p-value \\ \hline CR & NLLI & 0.00069 \\ \hline CR & PD & 0.00395 \\ \hline NER & NLI & 0.0051 \\ NEIE & SS & 0.028644 \\ \hline NER & PD & 0.0235 \\ \hline NLLS & SS & 4.34704e-10 \\ \hline NNI & Somm & 0.0068997 \\ \hline NAIL & WSD & 0.02709 \\ \hline SA & SS & 0.0001732 \\ \hline SS & SRL & 0.000530 \\ \hline SS & FD & 4.37008ee-9 \\ \hline SSS & Sum & 0.0219850 \\ \hline SS & WSD & 0.005447 \\ \hline SS & QA & 0.0000016 \\ \hline PP & Sum & 0.00306 \\ \hline \end{tabular}
\end{table}
Table 17: Pairwise comparison one-way Anova results for TPOJ-L region

Figure 10: Narratives-Slumford Dataset: 2V2 Accuracy (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of NLP tasks. Results are averaged across all participants. NLI, Paraphrase, and Summarisation perform the best.

\begin{table}
\begin{tabular}{|c|l l|} \hline T1 & T2 & p-value \\ \hline CR & NLLI & 0.000619 \\ \hline CR & PJD & 0.00010 \\ \hline NER & NLI & 0.0168115 \\ \hline NER & SS & 0.0168115 \\ \hline NER & SP & 0.00674 \\ \hline NER & PD & 0.000674 \\ \hline NLLS & SS & 0.05897e-10 \\ \hline NLLS & Sum & 0.001194 \\ \hline SLLS & WSD & 0.003894 \\ \hline SA & SS & 0.0000256 \\ \hline SSL & SRL & 0.00000224 \\ \hline SLS & PB & 9.11706-11 \\ \hline SES & Sum & 0.0168866 \\ \hline SES & WSD & 0.0057237 \\ \hline SPS & QA & 2.98083e-27 \\ \hline PP & Sum & 0.000857 \\ \hline PP & WSD & 0.000857 \\ \hline \end{tabular}
\end{table}
Table 20: Pairwise comparison one-way Anova results for DFL-R region

\begin{table}
\begin{tabular}{|c|l l|} \hline T1 & T2 & p-value \\ \hline CR & NLLI & 0.000619 \\ \hline CR & PNLI & 0.00061987 \\ \hline NER & SS & 0.0040 \\ \hline NER & PD & 0.000399 \\ \hline NER & SB & 0.0028644 \\ \hline NER & PD & 0.0235 \\ \hline NLLS & 4.34704e-10 \\ \hline NILS & Spom & 0.0068997 \\ \hline NATWSO & 0.02709 \\ \hline SA & SS & 0.0001732 \\ \hline SAS & SRL & 0.000530 \\ \hline SES & PD & 4.37008ee-9 \\ \hline SES & Sum & 0.0219850 \\ \hline SES & WSD & 0.005447 \\ \hline PS & QA & 0.0000016 \\ \hline PP & Sum & 0.00306 \\ \hline \end{tabular}
\end{table}
Table 19: Pairwise comparison one-way Anova results for DFL-L region



\begin{table}
\begin{tabular}{|c|l|l|} \hline T1 & T2 & p-value \\ \hline CR & NLI & 0.081070 \\ CR & PD & 0.0099703 \\ SER & PD & 0.06321010 \\ NLI & SS & 6.37020e-7 \\ SA & SS & 0.00642888 \\ SS & SRL & 0.0005118 \\ SSE & PD & 2.42829e-7 \\ S SQA & QA & 0.000162 \\ PDB & WSD & 0.0476 \\ \hline \end{tabular}
\end{table}
Table 21: Pairwise comparison one-way Anova results for VC-L region

Figure 11: Dendrogram constructed using similarity matrix constructed from the task-wise brain response predictions across 10 tasks for subjects 1, 2 and 7 in Pereira Dataset

Figure 12: Dendrogram constructed using similarity matrix constructed from the task-wise brain response predictions across 10 tasks for subjects 1, 21 and 31 in Narratives Dataset

Figure 13: Pereira Dataset – Pearson correlation coefficient between predicted and true responses across different sub ROIs of the Language Network using SRL task. Results are averaged across all participants.



Figure 16: Narratives-Pieman – Pearson correlation coefficient between predicted and true responses across different sub ROIs of 5 brain ROIs using summarization task. Results are averaged across all participants.

Figure 14: Pereira Dataset – Pearson correlation coefficient between predicted and true responses across different sub ROIs of the Language Network using CR task. Results are averaged across all participants.

Figure 15: Narratives-Pieman – Pearson correlation coefficient between predicted and true responses across different sub ROIs of 5 brain ROIs using paraphrase task. Results are averaged across all participants.

 

Figure 17: Pereira BrainMaps: Mean absolute error (MAE) between predictive voxels and actual voxels using task features from Taskonomy in one sample subject (subject 1). Predictive regions of different tasks are dissimilar across tasks.

Figure 18: Narratives BrainMaps: Mean absolute error (MAE) between predictive voxels and actual voxels using task features from Taskonomy in one sample subject (subject 1) of PieMan dataset. Predictive regions of various tasks are different across tasks.