SlowGuess/NAACL_2022_OCR huggingface dataset:
Subba Reddy Oota $^{1,2*}$ , Jashn Arora $^{2*}$ , Veeral Agarwal $^{2}$ , Mounika Marreddy $^{2}$ , Manish Gupta $^{2,3}$  and Bapi Raju Surampudi $^{2}$
<sup>1</sup>INRIA, Bordeaux, France; <sup>2</sup>IIIT Hyderabad, India; <sup>3</sup>Microsoft, India
subba-reddy.oota@inria.fr, jashn.arora@research.iit.ac.in
veeral.agarwal@research.iit.ac.in, mounika.marreddy@research.iit.ac.in
gmanish@microsoft.com, raju.bapi@iiit.ac.in
Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporoparieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.
Brain encoding aims at constructing neural brain activity given an input stimulus. Since the discovery of the relationship between language stimuli and functions of brain networks using fMRI [for ex., (Constable et al., 2004)], researchers have been interested in understanding how the neural encoding models predict the fMRI brain activity. Several brain encoding models have been developed
to (i) understand the ventral stream in biological vision (Yamins et al., 2014; Kietzmann et al., 2019; Bao et al., 2020), and (ii) to study the higher-level cognition like language processing (Gauthier and Levy, 2019; Schrimpf et al., 2021; Schwartz et al., 2019).
Some recent studies (Nishida et al., 2015; Huth et al., 2016) have been able to identify brain ROIs (Region of Interest) that respond to words that have a similar meaning and have thus built a "semantic atlas" of how the human brain organizes language. Further, several studies (Oota et al., 2018; Jain and Huth, 2018; Hollenstein et al., 2019) have used a wide variety of word embeddings where words represented as vectors in an embedding space are mapped to brain activation for improved neural coding.
Recently, Transformer (Vaswani et al., 2017) based models like BERT (Devlin et al., 2019) have been found to be very effective across a large number of natural language processing (NLP) tasks. These Transformer based models have been pretrained on millions of text instances in an unsupervised manner and further finetuned to specialize for various NLP tasks. Natural language understanding requires integrating several cognitive skills like syntactic parsing of the language structure, identifying the named entities, capturing the word meaning in the context, coreference resolution, etc. Learning from massive corpora enables these models to excel at cognitive skills required for language understanding. Interestingly, such Transformer-based neural representations have been found to be very effective for brain encoding as well (Schrimpf et al., 2021).
Despite the recent advances in mapping between language Transformers and the brain activity recorded with reading (Schrimpf et al., 2021), the Transformer features themselves are notoriously difficult to interpret. In recent works, Caucheteux et al. (2021a); Antonello et al. (2021) address this
Human Language Technologies, pages 3220 - 3237
issue by disentangling the high-dimensional Transformer representations of language models into four combinatorial classes: lexical, compositional, syntactic, and semantic representations to explore which class is highly associated with language cortical ROIs. Representations do not exist in a vacuum but become meaningful only when they accomplish a task. Therefore, the next logical step is to see which of these Transformer representations most effectively drive the linear mapping between language models and the brain in the context of NLP tasks. Gauthier and Levy (2019) fine-tune a pretrained BERT model on multiple tasks to find tasks best correlated with high decoding performance. In this study, we investigate the correlation between brain activation and feature representations learned by different task-specific networks, and ask which tasks lead to improvements in brain-encoding performance.
Recently, a study using multiple computer vision tasks has shown that 3D vision task models predict better fMRI brain activity than 2D vision task models (Wang et al., 2019) for visual stimuli. Inspired by the success of correlations in the vision field (Wang et al., 2019), and brain encoding study of a variety of language Transformer models (Schrimpf et al., 2021; Caucheteux et al., 2021b,a), we build neural language taskonomy models for brain encoding and aim to find NLP tasks that are most explanatory of brain activations for reading and listening tasks.
In this paper, we uncover insights about the association between fMRI voxel activations and representations of diverse NLP tasks representations. The predictive power of task-specific representations with brain activation is ascertained by (1) using ridge regression on such representations and predicting activations and (2) computing popular metrics like 2V2 accuracy and Pearson correlation between actual and predicted activations.
Specifically, we make the following contributions in this paper.
- Given Transformer models finetuned for various NLP tasks, we propose the problem of finding which of these are the most predictive of fMRI brain activity for reading and listening tasks.
- Our language taskonomy results reveal that Coreference Resolution, Named Entity Recognition, and Shallow Syntax Parsing tasks have
higher predictive performance while reading the text. On the other hand, paraphrase detection, summarization, and Natural Language Inference tasks display better correlation during listening.
- We also perform similarity analysis between task representations from transfer learning and neural taskonomy and derive interesting cognitive insights from brain maps.
Older methods for text-based stimulus representation include text corpus co-occurrence counts (Mitchell et al., 2008; Pereira et al., 2013; Huth et al., 2016), syntactic and discourse features (Wehbe et al., 2014). In recent times, both semantic and experiential attribute models have been explored for text-based stimuli. Semantic representation models include distributed word embeddings (Pereira et al., 2016; Anderson et al., 2017a; Pereira et al., 2018; Toneva and Wehbe, 2019; Hollenstein et al., 2019; Wang et al., 2020), sentence representation models (Sun et al., 2019; Toneva and Wehbe, 2019; Sun et al., 2020), recurrent neural networks (Jain and Huth, 2018; Oota et al., 2019), and Transformer-based language models (Gauthier and Levy, 2019; Toneva and Wehbe, 2019; Schwartz et al., 2019; Oota et al., 2022a,b). Experiential attribute models represent words in terms of human ratings of their degree of association with different attributes of experience, typically on a scale of 0-6 (Anderson et al., 2019, 2020; Berezutskaya et al., 2020; Jat et al., 2020; Caucheteux et al., 2021a; Antonello et al., 2021) or binary (Handjaras et al., 2016; Wang et al., 2017). Fine-grained details such as lexical, compositional, syntactic, and semantic representations of narratives are factorized from Transformer-based models and utilized for training encoding models. The resulting models are better able to disentangle the corresponding brain responses in fMRI (Caucheteux et al., 2021a).
In this paper, we focus on Transformer-based linguistic stimuli representations since they have been found to be most effective. Unlike previous studies which directly used existing task-agnostic pretrained models, we train task-specific Transformer models and aim to find which model leads to the best encoding accuracy given reading and listening language stimuli.
3 Brain Imaging Datasets
We work with two datasets: Pereira and Narratives-Pieman. Results on Narratives-Lucy and Narratives-SlumLord show similar trends. Hence, we also show results on Narratives-Lucy and Narratives-SlumLord in the appendix.
Pereira Dataset (Reading Sentences from Passages) For the Pereira dataset, similar to earlier work (Sun et al., 2019, 2020), we combine the data from sentence-based experiments (experiments-2 and 3) from Pereira et al. (2018). Five subjects were presented a total of 627 sentences from 48 broad topics, spanning over 168 passages, where each passage consists of 3-4 sentences. As in (Pereira et al., 2018), we focused on nine brain ROIs (regions of interest) corresponding to four brain networks: (i) Default Mode Network (DMN) (linked to the functionality of semantic processing), (ii) Language Network (related to language processing, understanding, word meaning, and sentence comprehension), (iii) Task Positive Network (TP) (related to attention, salience information), and (iv) Visual Network (related to the processing of visual objects, object recognition). We briefly summarize the details of the dataset and the number of voxels corresponding to each ROI in Table 1. We use the AAL parcellation Atlas  $(116\times 116$  brain ROIs) to present the brain map results, since Pereira dataset contains annotations tied to this atlas.
ROIs→	Language	Vision	DMN	Task Positive
↓Subj	LH	RH	Body	Face	Object	Scene	Vision	RH	LH
P01	5265	6172	3774	4963	8085	4141	12829	17190	35120
M02	4930	5861	3873	4782	7552	3173	11729	15070	30594
M04	5906	5401	3867	4803	7812	3602	12278	18011	34024
M07	5629	5001	4190	4993	8617	3721	12454	17020	30408
M15	5315	6141	4112	4941	8323	3496	12383	15995	31610
Table 1: # Voxels in each ROI in the Pereira Dataset. LH - Left Hemisphere. RH - Right Hemisphere.
ROIs→	EAC	AAC	PMC	TPOJ	DFL
LH	RH	LH	RH	LH	RH	LH	RH	LH	RH
# Voxels	808	638	1420	1493	1198	1204	847	1188	1061	875
Table 2: # Voxels in each ROI in the Narratives Dataset. LH - Left Hemisphere. RH - Right Hemisphere. Pieman has 82, Lucy has 16 and SlumLord has 18 subjects. # Voxels across ROIs are same for all the three.
Narratives-Pieman (Listening to Stories) The "Narratives" collection aggregates a variety of fMRI datasets collected while human subjects listened to naturalistic spoken stories. The Narratives dataset that includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling
$\sim 4.6$  hours of unique stimuli ( $\sim 43,000$  words) was proposed in (Nastase et al., 2021). Similar to earlier works (Caucheteux et al., 2021b), we analyze data from 82 subjects listening to the story titled 'PieMan' with 259 TRs (repetition time - fMRI recorded every 1.5 sec.). We list number of voxels per ROI in this dataset in Table 2. We use the multimodal parcellation of the human cerebral cortex (Glassar Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps (Glasser et al., 2016), since Narratives dataset contains annotations tied to this atlas. The data covers ten brain ROIs in the human brain, i.e., Left hemisphere (L), and Right hemisphere (R) for each of the following: (i) early auditory cortex (EAC: A1, L Belt, MBelt, PBelt, and R1) which plays a key role for sound perception since it represents one of the first cortical processing stations for sounds; (ii) auditory association cortex (AAC: A4, A5, STSdp, STSda, STSvp, STSva, STGa, and TA2) which is concerned with the memory and classification of sounds; (iii) posterior medial cortex (PMC: POS1, POS2, v23ab, d23ab, 31pv, 31pd, 7m); (iv) the temporo parieto occipital junction (TPOJ: TPOJ1, TPOJ2, TPOJ3, STV, PSL) which is a complex brain territory heavily involved in several high-level neurological functions, such as language, visuo-spatial recognition, writing, reading, symbol processing, calculation, self-processing, working memory, musical memory, and face and object recognition; and (v) the dorsal frontal lobe (DFL: L_55b, SFL, L_44, L_45, IFJA, IFSP) which covers the aspects of pragmatic processing such as discourse management, integration of prosody, interpretation of nonliteral meanings, inference making, ambiguity resolution, and error repair.
4 Encoding Model
To explore how and where contextual language features are represented in the brain when reading sentences and listening to stories, we extract different features spaces describing each stimulus sentence and use them in an encoding model to predict brain responses. Our reasoning is as follows. If a feature is a good predictor of a specific brain region, information about that feature is likely encoded in that region. In this paper, for both datasets, we train fMRI encoding models using Ridge regression on stimuli representations obtained using a variety of NLP tasks. The main goal of each fMRI encoder model is to predict brain
responses associated with each brain region given a stimuli. In all cases, we train a model per subject separately. Following literature on brain encoding (Caucheteux et al., 2021b; Toneva et al., 2020), we choose to use a ridge regression model instead of more complicated models. We plan to explore more such models as part of future work. We follow K-fold  $(\mathrm{K} = 10)$  cross-validation. All the data samples from K-1 folds were used for training, and the model was tested on samples of the left-out fold. We used sklearn's ridge-regression with default parameters, 10-fold cross-validation, Stochastic-Average-Gradient Descent Optimizer, Huggingface for Transformer models, MSE loss function, and L2-decay  $(\lambda)$  as 1.0. We used BERT Word-Piece tokenizer for the linguistic Transformer input. All experiments were conducted on a machine with 1 NVIDIA GEFORCE-GTX GPU with 16GB GPU RAM. We make the code publicly available<sup>1</sup>.
4.1 Feature Spaces
To simultaneously test representations from multiple NLP tasks, we used the latent space features from each of the following ten popular NLP tasks: coreference resolution (CR), named entity recognition (NER), natural language inference (NLI), paraphrase detection (PD), question answering (QA), sentiment analysis (SA), semantic role labeling (SRL), shallow syntax parsing (SS), summarization (Sum) and word sense disambiguation (WSD). All of these are discriminative NLP tasks, and thus we use models obtained by task-specific finetuning of the same pretrained Transformer encoder model (BERT-base-cased with dimensionality=768). Given an input sentence, each task Transformer outputs token representations at the final layer. We use the #tokens  $\times$  768 dimension vector obtained from the last hidden layer to obtain latent features for the stimuli. We then build individual ridge regression models with the extracted latent features to predict brain responses and measure the correlation between the prediction and the true response.
Pereira: Since individual sentences were presented to the subjects while modeling, sentences were passed one by one to the task Transformer model, and average-pooled representations were used to encode the sentence stimuli.
Narratives-Pieman: Due to the constraint on input sequence length for BERT (512), we considered
a window size of 10 sentences with the last two sentences of one window overlapping with the next to be given as input to the BERT model. We use the average-pooled representation from BERT to encode text stimuli. To get the representation for a TR, we pooled the representations of only those words of the sentences in that TR.
4.2 Task Descriptions
Here we describe the functionality of each NLP task that we used for fMRI encoding. CR: involves finding all expressions that refer to the same entity in a text. PD: involves taking a passage – either spoken or written – and rewording it in shorter or own words. Summarization (Sum): involves selecting a few important sentences from a document or paragraph. NER: involves detection of the named entities such as person names, location names, company names from a given text. NLI: investigates the entailment relationship between two texts: premise and hypothesis. QA: aims to select an answer given a passage, a question, and a set of candidate answers. SA: involves determining whether a piece of text is positive, negative, or neutral. SRL: assigns labels to words or phrases in a sentence that indicates their semantic role in the sentence, such as that of an agent, goal, or result. SS: provides an approximation of phrase-syntactic structure of sentences. WSD: involves determining which sense (meaning) of a word is activated by the use of the word in a particular context.
Syntactic reasoning is rather shallow compared to deep semantic reasoning. Syntactic reasoning follows somewhat objective grammar rules. Comparatively semantic reasoning is often subjective in nature and complex. The emerging evidence from fMRI studies (Fedorenko et al., 2020, 2012) also points out that processing of both syntax and semantics is distributed in the brain and it is only when violations of these processes are probed, we see localization of function (Friederici et al., 2003). Thus, in this work, we explore syntactic and semantic tasks separately. Of the above mentioned tasks, NER and SS are syntactic, while the others involve semantic reasoning.
Our selection of these tasks was based on the following design principles: (1) We wanted to select a set of tasks covering diverse cognitive-linguistic skills. (2) We wanted to select tasks that are a part of popular NLP benchmarks like GLUE (Wang et al., 2018). (3) We selected tasks for which
'https://tinyurl.com/langTask
Figure 1: Pereira - 2V2 Accuracy (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of NLP tasks. Results are averaged across all participants. CR, NER, and SS perform the best.
BERT-base-cased finetuned models were available. Note that we did not finetune any of these models ourselves but leveraged the state-of-the-art finetuned models available on Huggingface. Details of the specific finetuned model checkpoints are mentioned in Table 3 in the Appendix.
We evaluate our models using popular brain encoding evaluation metrics described in the following. Given a subject and a brain region, let  $N$  be the number of samples. Let  $\{Y_i\}_{i=1}^N$  and  $\{\hat{Y}_i\}_{i=1}^N$  denote the actual and predicted voxel value vectors for the  $i^{th}$  sample. Thus,  $Y \in R^{N \times V}$  and  $\hat{Y} \in R^{N \times V}$  where  $V$  is the number of voxels in that region.
2V2 Accuracy is computed as 2V2Acc =  $\frac{1}{N_{C_2}}\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}I[\cos D(Y_i,\hat{Y}_i)+$  cosD(Yj,Yj) < cosD(Yi,Yj) + cosD(Yj,Yi)] where cosD is the cosine distance function. I[c] is an indicator function such that I[c] = 1 if c is true, else it is 0. The higher the 2V2 accuracy, the better. Pearson Correlation (PC) is computed as PC =  $\frac{1}{N}\sum_{i=1}^{n}\operatorname{corr}[Y_i,\hat{Y}_i]$  where corr is the correlation function. Mean Absolute Error (MAE) is computed as
$$
\mathrm {M A E} = \frac {1}{N} \sum_ {i = 1} ^ {n} | [ Y _ {i} - \hat {Y} _ {i} ] |.
$$
Statistical Significance: In order to estimate the statistical significance of the performance differences (across all results), we performed one-way ANOVA on the mean values for the subjects. In all such cases we report p-values corrected using Bonferroni correction.
4.4 Neural Language Tasks Similarity Computation
To estimate the similarity between 10 language tasks, we took the prediction performance scores across all the voxels in Pereira (97,539) and Narratives-Pieman datasets (10,732). To analyze the relationship between tasks based on neural representations, we calculated the Pearson correlation between predicted voxels of each task with the remaining tasks. These Pearson correlation values were used to construct heatmaps and the task similarity trees(dendograms) using hierarchical clustering for Pereira and Narratives-Pieman datasets.
In order to assess the performance of the fMRI encoder models learned using the representations
Figure 2: Narratives-Pieman - 2V2 Accuracy (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of NLP tasks. Results are averaged across all participants. NLI, PD, and Summarization perform the best.
from a variety of NLP tasks, we computed the 2V2 accuracy and Pearson correlation coefficient between the predicted and true responses across various ROIs for both the reading (Pereira) dataset (Fig. 1) as well as the listening (Narratives-Pieman) dataset (Fig. 2).
5.1 Encoding performance of Language Task models for reading vs listening tasks
Reading Sentences (Pereira): From Fig. 1, we observe that tasks such as CR, NER, SRL, and SS appear to have a better correlation to the brain responses compared to the other tasks. In order to estimate the statistical significance of the performance differences, we performed one-way ANOVA on the mean correlation values for the subjects across the ten language tasks for the nine brain ROIs. The main effect of the ANOVA test was significant for all the ROIs with  $p \leq 10^{-2}$  with confidence  $95\%$  (see Appendix for detailed ANOVA results). Further, post hoc pairwise comparisons (Ruxton and Beauchamp, 2008) confirmed the visual observations that on both 2V2 accuracy and Pearson correlation measures, tasks such as CR, NER, SRL, and SS performed significantly better compared to other tasks (see Appendix for
pairwise comparison results). These results demonstrate that when reading a sentence, information processing operations related to recognizing named entities, labeling semantic roles to the constituents of a sentence, identifying the references from a sentence to the given topic (concept), and syntactic processing may be engaged.
Further, we observe that the ROI corresponding to language processing in the left hemisphere (Language_LH) has higher encoding performance than that of the right hemisphere (Language_RH). This is in line with the left hemisphere dominance for language processing (Binder et al., 2009). Also, lateral visual ROIs such as Vision_Object, Vision_Body, Vision_Face, and Vision ROIs display higher correlation with the language tasks associated with named entities (NER), relating the entities (CR), and syntax processing (SS). Higher correlations with all the visual brain regions point to the possible alignment of visual and language regions for semantic understanding (Popham et al., 2021) in a reading task. Finally, across all regions, pretrained BERT model has worse correlation compared to at least 5 other task models.
Listening Stories (Narratives-Pieman): From Fig. 2, we observe that the profiles of performance