SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jerry Tang
jerrytang@utexas.edu
Meng Du
Intel Labs, UCLA
mengdu@ucla.edu
Vy A. Vo
Intel Labs
vy.vo@intel.com
Vasudev Lal
vasudev.lal@intel.com
Alexander G. Huth
huth@cs.utexas.edu
Encoding models have been used to assess how the human brain represents concepts in language and vision. While language and vision rely on similar concept representations, current encoding models are typically trained and tested on brain responses to each modality in isolation. Recent advances in multimodal pretraining have produced transformers that can extract aligned representations of concepts in language and vision. In this work, we used representations from multimodal transformers to train encoding models that can transfer across fMRI responses to stories and movies. We found that encoding models trained on brain responses to one modality can successfully predict brain responses to the other modality, particularly in cortical regions that represent conceptual meaning. Further analysis of these encoding models revealed shared semantic dimensions that underlie concept representations in language and vision. Comparing encoding models trained using representations from multimodal and unimodal transformers, we found that multimodal transformers learn more aligned representations of concepts in language and vision. Our results demonstrate how multimodal transformers can provide insights into the brain's capacity for multimodal processing.
Encoding models predict brain responses from quantitative features of the stimuli that elicited them [1]. In recent years, fitting encoding models to data from functional magnetic resonance imaging (fMRI) experiments has become a powerful approach for understanding information processing in the brain. While many studies have shown that encoding models achieve good performance when trained and tested on brain responses to a single stimulus modality, such as language [2-9] or vision [10-15], the human brain is remarkable in its ability to integrate information across multiple modalities. There is growing evidence that this capacity for multimodal processing is supported by aligned cortical representations of the same concepts in different modalities—for instance, hearing "a dog is chasing a cat" and seeing a dog chasing a cat may elicit similar patterns of brain activity [16-21].
In this work, we investigated the alignment between language and visual representations in the brain by training encoding models on fMRI responses to one modality and testing them on fMRI responses to the other modality. Encoding models that successfully transfer across modalities can provide insights into how the two modalities are related [20]. Although previous work has compared language and vision encoding models, human annotations were required to map language and visual stimuli into a shared semantic space [20]. To our knowledge, cross-modality transfer has yet to be
Figure 1: Cross-modality encoding model transfer. (a) Multimodal transformers were used to extract features of story and movie stimuli. Language encoding models were fit (using L2-regularized linear regression) to predict story fMRI responses from story stimuli; vision encoding models were fit to predict movie fMRI responses from movie stimuli. (b) As the language and vision encoding models share a representational space, language encoding models could be used to predict fMRI responses to movie stimuli, and vision encoding models to predict fMRI responses to story stimuli. Encoding model performance was quantified by the linear correlation between the predicted and the actual response time-courses in each voxel.
demonstrated using encoding models trained on stimulus-computable features that capture the rich connections between language and vision.
One way to extract aligned features of language and visual stimuli is using transformer models trained on multimodal objectives like image-text matching [22-28]. Recent studies have shown that multimodal transformers can model brain responses to language and visual stimuli [29], often outperforming unimodal transformers [30-32], suggesting that multimodal training enables transformers to learn brain-like representations. However, these studies do not assess whether representations from multimodal transformers can be used to train encoding models that transfer across modalities. Since multimodal transformers are trained to process paired language and visual inputs, the representations learned for a concept in language could be correlated with the representations learned for that concept in vision. This alignment between language and visual representations in multimodal transformers could facilitate the transfer of encoding models across modalities, which relies on the alignment between language and visual representations in the brain.
To test this, we used the BridgeTower [28] multimodal transformer to model fMRI responses to naturalistic stories [9] and movies [12]. We separately obtained quantitative features of story and movie stimuli by extracting latent representations from BridgeTower. We estimated language encoding models using story features and story fMRI responses, and vision encoding models using movie features and movie fMRI responses (Figure 1a). We evaluated how well the language encoding models can predict movie fMRI responses from movie features  $(story \to movie)$  and how well the vision encoding models can predict story fMRI responses from story features  $(movie \to story)$  (Figure 1b). We compared this to how well the language encoding models can predict story fMRI responses from story features  $(story \to story)$  and how well the vision encoding models can predict movie fMRI responses from movie features  $(movie \to movie)$ .
We found that encoding models trained on brain responses to one modality could accurately predict brain responses to the other modality. In many brain regions outside of sensory and motor cortex, story  $\rightarrow$  movie performance approached movie  $\rightarrow$  movie performance, suggesting that these regions encode highly similar representations of concepts in language and vision. To assess these representations, we performed principal components analysis on the encoding model weights and identified semantic dimensions that are shared between concept representations in language and vision. Finally, we found that cross-modality performance was higher for features extracted from multimodal transformers than for linearly aligned features extracted from unimodal transformers.
Our results characterize how concepts in language and vision are aligned in the brain and demonstrate that multimodal transformers can learn representations that reflect this alignment.
2 Multimodal transformers
Multimodal transformers are trained on paired language and visual data to perform self-supervised tasks such as image-text matching. Typically, these models are used to extract representations of paired language and visual input for downstream tasks such as visual question answering. However, since multimodal training objectives may impose some degree of alignment between language and visual tokens for the same concept, these models could also be used to extract aligned representations of language and visual input in isolation. For instance, latent representations extracted from the sentence "a dog is chasing a cat" may be correlated with latent representations extracted from a picture of a dog chasing a cat.
2.1 Feature extraction
In this study, we extracted stimulus features using a pretrained BridgeTower model [28]. BridgeTower is a vision-language transformer trained on image-caption pairs from the Conceptual Captions [33], SBU Captions [34], MS COCO Captions [35], and Visual Genome [36] datasets. For each image-caption pair, the caption is processed using a language encoder initialized with pretrained RoBERTa parameters [37] while the image is processed using a vision encoder initialized with pretrained ViT parameters [38]. The early layers of BridgeTower process language and visual tokens independently, while the later layers of BridgeTower are cross-modal layers that process language and visual tokens together. Results are shown for the BridgeTower-Base model; corresponding results for the BridgeTower-Large model are shown in Appendix F.
We used BridgeTower to extract features from the story (Section 3.1) and movie (Section 3.2) stimuli that were used in the fMRI experiments. Each story and movie stimulus was separately processed using BridgeTower by running forward passes with input from the corresponding modality. Hidden representations were extracted from each layer of BridgeTower as the inputs were processed.
For stories, segments of transcripts were provided to the model without accompanying visual inputs. A feature vector was obtained for every word by padding the target word with a context of 20 words both before and after. For movies, single frames were provided to the model without accompanying language inputs. Movies were presented at 15 frames per second, and a feature vector was obtained for every 2-second segment by averaging latent representations across the 30 corresponding frames. Feature extraction was done on a node with 10 Intel Xeon Platinum 8180 CPUs and an Nvidia Quadro RTX 6000 GPU.
2.2 Feature alignment
Transformers compute representations for each layer by attending to different combinations of the input tokens. While multimodal training tasks may require models to align language and visual tokens for the same concept, the nature of this alignment depends on the type of attention mechanism used to combine language and visual tokens [39]. BridgeTower uses a co-attention mechanism wherein language and visual tokens are passed through different projection matrices, and query vectors from each modality are only scored against key vectors from the other modality. As a consequence, the language and visual feature spaces extracted from each layer of BridgeTower may differ up to a linear transformation.
To correct for these potential transformations, we used the Flickr30K dataset [40]—which consists of paired captions and images—to estimate linear transformation matrices that explicitly align the language and visual feature spaces extracted from BridgeTower. We first used BridgeTower to separately extract language features of each caption and visual features of each image. We then used L2-regularized linear regression to estimate  $\text{image} \rightarrow \text{caption}$  matrices that predict each language feature from the visual features, and  $\text{caption} \rightarrow \text{image}$  matrices that predict each visual feature from the language features. We evaluated the matrices by measuring how well they align language and visual features for held out samples from the Flickr30K dataset, and found that the matrices achieved significant alignment of language and visual features (Appendix C).
Before using the language encoding model to predict movie fMRI responses, we first used the image  $\rightarrow$  caption matrix to project the movie features into the language feature space. Similarly, before using the vision encoding model to predict story fMRI responses, we first used the caption  $\rightarrow$  image matrix to project the story features into the visual feature space.
To show that this explicit feature alignment is only necessary for certain attention mechanisms, we repeated our analyses using a KD-VLP [24] vision-language transformer (Appendix G). KD-VLP uses a merged attention mechanism wherein language and visual tokens are passed through the same projection matrices and query vectors are scored against all key vectors, making explicit feature alignment unnecessary.
3 fMRI experiments
We analyzed publicly available fMRI data from five subjects (2 female, 3 male) who participated in a story listening experiment and a movie watching experiment [20]. Previous studies have modeled the story fMRI data as a function of the story stimuli [9, 4, 41] and the movie fMRI data as a function of the movie stimuli [12]. Blood-oxygen level dependent (BOLD) brain signals were recorded using gradient-echo EPI on a 3T Siemens TIM Trio scanner at the UC Berkeley Brain Imaging Center with a 32-channel volume coil,  $\mathrm{TR} = 2.0045$  seconds,  $\mathrm{TE} = 31$  ms, flip angle  $= 70$  degrees, voxel size  $= 2.24 \times 2.24 \times 4.1$  mm (slice thickness  $= 3.5$  mm with 18 percent slice gap), matrix size  $= 100 \times 100$ , and 30 axial slices. All experiments and subject compensation were approved by the UC Berkeley Committee for the Protection of Human Subjects. Further details about the data are provided in Appendix A.
3.1 Story experiment
Stimuli for the story experiment consisted of 10 naturally spoken narrative stories from The Moth Radio Hour totaling just over 2 hours [9]. The stories were presented over Sensimetrics S14 headphones. Subjects were instructed to listen to the stories with their eyes closed. Each story was played during a single fMRI scan.
3.2 Movie experiment
Stimuli for the movie experiment consisted of 12 videos totaling 2 hours [12, 14]. Each video was made by concatenating a sequence of  $10 - 20\mathrm{~s~}$  clips from movies drawn from the Apple QuickTime HD gallery and YouTube. The videos were presented at 15 frames per second. The videos originally contained dialogue and music, but were presented silently to the subjects. Subjects were instructed to fixate on a dot at the center of the screen. Each video was played during a single fMRI scan.
4 Voxelwise encoding models
Our study builds upon the voxelwise modeling framework used in many previous fMRI studies [1]. Voxelwise encoding models learn a mapping from stimuli to the brain responses that they elicit in each individual subject [9]. Brain images recorded at times  $t = 1\dots T$  are given by  $y(t)\in \mathbb{R}^{m}$  where  $m$  is the number of voxels in the cerebral cortex. Responses for one subject are represented by the response matrices  $Y_{story}\in \mathbb{R}^{T_{story}\times m}$  and  $Y_{movie}\in \mathbb{R}^{T_{movie}\times m}$ .
The story and movie features were resampled to the fMRI acquisition times using a Lanczos filter. To account for the hemodynamic response, a finite impulse response model with 4 delays (2, 4, 6, and 8 seconds) was applied to the downsampled features. This resulted in the delayed stimulus matrices  $X_{story} \in \mathbb{R}^{T_{story} \times 4k}$  and  $X_{movie} \in \mathbb{R}^{T_{movie} \times 4k}$  where  $k = 768$  is the dimensionality of the BridgeTower features.
We modeled the mapping between stimulus features and brain responses with a linear model  $Y = X\beta$ . Each column of  $\beta$  represents a voxel's linear weights on the  $4k$  delayed features. The weights  $\beta$  were estimated using L2-regularized linear regression. Regularization parameters were independently selected for each voxel using 50 iterations of a cross-validation procedure.
Voxelwise modeling was done on a workstation with an Intel Core i9-7900X CPU. Further details about voxelwise modeling are provided in Appendix B.
4.1 Evaluation
Voxelwise encoding models were evaluated by predicting the response matrices  $Y_{test}$  from the stimulus matrices  $X_{test}$  for stimuli that were excluded from model estimation. Prediction performance for each voxel was quantified by the linear correlation between the predicted and the actual response time-courses [9, 4, 6, 2].
To quantify source  $\rightarrow$  target performance from a source modality to a target modality, we estimated encoding models using all source scans and evaluated prediction performance on each target scan. We averaged linear correlations across the target scans to obtain a score  $r_{source\rightarrow target}$  for each voxel. A high  $r_{source\rightarrow target}$  score indicates that a voxel's tuning for concepts in the target modality is aligned with its tuning for concepts in the source modality.
We compared this cross-modality performance against the within-modality performance of an encoding model trained on the target modality. To quantify  $target \rightarrow$  target performance, we held out each target scan, estimated encoding models using the remaining target scans, and evaluated prediction performance on the held out target scan. We averaged linear correlations across the held out target scans to obtain a score  $r_{\text{target} \rightarrow \text{target}}$  for each voxel.
Many previous studies evaluated encoding models on averaged responses to a single test stimulus [9, 42, 4]. Here we evaluated encoding models on single repetition responses to many test stimuli, as our stimuli vary in semantic content. While our evaluation produces less biased estimates of cross-modality and within-modality performance, the correlation values will be lower than previously reported results due to the lower signal-to-noise ratio of single repetition response data.
We separately identified voxels with statistically significant story  $\rightarrow$  story and movie  $\rightarrow$  movie within-modality scores. We computed null distributions of the within-modality scores using a blockwise permutation test to account for autocorrelation in the voxel responses (Appendix C).
4.2 Layer selection
Separate encoding models were trained using stimulus features extracted from each layer of BridgeTower. We summarized performance across layers by estimating the best layer for each voxel using a bootstrap procedure. For each test scan, we estimated the best layer for each voxel based on mean prediction performance across the remaining test scans. We then used the selected layer for each voxel to compute prediction performance for that voxel on the held out test scan. We used this procedure for all analyses unless noted otherwise.
We separately estimated language and vision encoding models for each subject. We used these models to compute  $r_{story \to movie}$ ,  $r_{movie \to story}$ ,  $r_{story \to story}$ , and  $r_{movie \to movie}$  scores for each voxel.
5.1 Cross-modality performance
Cross-modality performance was visualized by projecting  $r_{story \to movie}$  and  $r_{movie \to story}$  scores for each voxel in one subject onto a flattened cortical surface (Figure 2a; see Appendix E for other subjects; see Appendix F for BridgeTower-Large model results and Appendix G for KD-VLP model results). We found positive  $r_{story \to movie}$  and  $r_{movie \to story}$  scores in many parietal, temporal, and frontal regions, which have previously been shown to represent the meaning of concepts in language [9] and vision [12]. The high story  $\rightarrow$  movie scores and positive (albeit lower) movie  $\rightarrow$  story scores suggest that these voxels have similar tuning for the same concepts across modalities [20].
Conversely, we found negative  $r_{story \to movie}$  and  $r_{movie \to story}$  scores in visual cortex. Previous studies have reported that the tuning to perceptual information in visual cortex may be inverted during conceptual processing in the absence of perception [43, 44]. If there is systematically inverted tuning between language and vision, it should be possible to first estimate which voxels would have negative cross-modality scores using separate validation data, and then multiply their weights by  $-1$  before computing  $r_{story \to movie}$  and  $r_{movie \to story}$  on test data. We performed this correction using a bootstrap procedure across the test scans. For each test scan, we estimated which voxels have inverted
b
Figure 2: Cross-modality prediction performance. Encoding models estimated on brain responses to one modality were evaluated on brain responses to the other modality. Cross-modality performance is measured by the linear correlation  $(r)$  between predicted and actual responses. (a)  $r_{story\rightarrow movie}$  and  $r_{movie\rightarrow story}$  scores for each voxel in one subject are displayed on the subject's cortical surface. A voxel appears red if its score is positive, blue if its score is negative, and white if its score is zero. White outlines show regions of interest (ROIs) identified using separate localizer data. AC and VC denote auditory cortex and visual cortex.  $r_{story\rightarrow movie}$  and  $r_{movie\rightarrow story}$  scores were positive in regions that have previously been found to represent the meaning of concepts in language and vision, but negative in visual cortex. (b) Prediction performance for each layer of BridgeTower. Negative scores were corrected by using held out data to fit a one-parameter model for each voxel that predicts whether the encoding model weights should be negated before computing transfer performance. Scores were averaged across voxels and then across subjects to provide unbiased albeit conservatively low summaries of model performance. Error bars indicate standard error of the mean across subjects.
tuning based on the mean prediction performance across the remaining test scans. We then multiplied the weights of these voxels by  $-1$  before computing prediction performance on the held out test scan.
Figure 2b shows story  $\rightarrow$  movie and movie  $\rightarrow$  story performance across cortex before and after this correction. We summarized performance for each layer of BridgeTower by averaging the linear correlations across all cortical voxels and subjects. Averaging across cortical voxels is an unbiased way to compare different encoding models [4], but it produces conservatively low correlation values since many cortical voxels are not involved in processing language or vision. Across layers, the correction significantly improved story  $\rightarrow$  movie performance (one-sided paired t-test;  $p < 0.05$ ,  $t(4) = 7.5295$ ,  $\overline{r}_{corrected} = 0.0230$ ,  $\overline{r}_{uncorrected} = 0.0053$ ) and movie  $\rightarrow$  story performance (one-sided paired t-test;  $p < 0.05$ ,  $t(4) = 6.6356$ ,  $\overline{r}_{corrected} = 0.0097$ ,  $\overline{r}_{uncorrected} = 0.0031$ ), providing evidence for systematically inverted tuning for the same concepts across modalities.
5.2 Comparing cross-modality and within-modality performance
While the previous analysis identified voxels with similar tuning for concepts in language and vision, it did not characterize the extent of this cross-modal similarity. To do this, we next compared cross-modality performance to within-modality performance for each voxel (Figure 3a).
To quantify the amount of information that the language encoding model learns about tuning for movies, we divided  $r_{story \rightarrow movie}$  by  $r_{movie \rightarrow movie}$  for each voxel. If the movie responses in a voxel are well-predicted by the vision encoding model but poorly predicted by the language encoding model, this value should be low. Conversely, if the movie responses are predicted about as well using both the vision and language encoding models, then this value should be close to 1. In visual cortex, which represents structural features of visual stimuli [14], the language encoding model performed much worse than the vision encoding model at predicting movie responses. In significantly
Figure 3: Comparing cross-modality and within-modality prediction performance. Cross-modality scores were compared against within-modality scores in voxels with statistically significant within-modality scores. (a) Cross-modality scores and within-modality scores for each voxel in one subject are projected onto the subject's flattened cortical surface. A voxel appears dark if its cross-modality score is much lower than its within-modality score, and bright if its cross-modality score approaches its within-modality score. Only well-predicted voxels under the within-modality model  $(q(\mathrm{FDR}) < 0.05$ , one-sided permutation test) are shown. (b) Histograms compare cross-modality scores to within-modality scores. For movie responses,  $r_{story} \rightarrow \text{movie}$  scores were much lower than  $r_{movie} \rightarrow \text{movie}$  scores near visual cortex but approached  $r_{movie} \rightarrow \text{movie}$  scores in other regions. For story responses,  $r_{movie} \rightarrow \text{story}$  scores were generally much lower than  $r_{story} \rightarrow \text{story}$  scores across cortex.