SlowGuess/NeurIPS_2023_OCR huggingface dataset:
predicted voxels outside of visual cortex, which represent the meaning of visual stimuli [45, 12, 20], the language encoding model often approached the performance of the vision encoding model (Figure 3b).
Similarly, to quantify the amount of information that the vision encoding model learns about tuning for stories, we divided  $r_{movie \rightarrow story}$  by  $r_{story \rightarrow story}$  for each voxel. In auditory cortex, which represents acoustic and articulatory features of language stimuli [42], the vision encoding model performed much worse than the language encoding model at predicting story responses. In some significantly predicted voxels outside of auditory cortex, which have been shown to represent the meaning of language stimuli [42, 9], the vision encoding model performed relatively better, but still did not approach the performance of the language encoding model (Figure 3b).
These results suggest that visual tuning can often be estimated solely based on how a voxel responds to stories, while it is much harder to estimate language tuning solely based on how a voxel responds to movies [46]. One potential confound that could contribute to this asymmetry is that the story stimuli contain both concrete concepts (such as places) and abstract concepts (such as emotions) while the movie stimuli mostly contain concrete concepts. Another potential confound is that the story stimuli contain information at a longer timescale than the movie stimuli, which consist only of 10-20 second clips. To isolate whether the asymmetry in Figure 3 is driven by differences between language and visual representations in the brain, future work could use story and movie stimuli that are matched in terms of semantics and timescale coverage.
5.3 Encoding model principal components
The earlier analyses showed that encoding models can readily transfer across modalities, at least in the direction from language to vision. But what kind of information is it that these cross-modal models are capturing? To understand the semantic dimensions that underlie the shared tuning for concepts in
a
Figure 4: Encoding model principal components. Principal components analysis identified the first 10 principal components (PCs) of language encoding model weights. (a) Each caption in Flickr30k was projected onto language PC 1. This PC distinguishes captions that refer to people and social interactions—which are represented in inferior parietal cortex, precuneus, temporal cortex, and frontal cortex—from places and objects—which are represented in superior parietal cortex and middle frontal cortex. (b) Each image in Flickr30k was projected onto language PC 1. Here language PC 1 distinguishes images of people—which are represented in EBA, OFA, FFA, inferior parietal cortex, precuneus, temporal cortex, and frontal cortex—from images of places—which are represented in superior parietal cortex and middle frontal cortex. (c) In voxels that were well predicted by both the language and the vision encoding models (red on inset heatmap), projections of language and vision encoding model weights were significantly correlated  $(^{*})$  for several language PCs, indicating semantic dimensions that are shared between language and visual representations. (d) In voxels that were well predicted by only the language or the vision encoding models (blue on the inset heatmap), projections of language and vision encoding model weights were not significantly correlated for any language PCs. For all results, error bars indicate standard error of the mean across subjects.
Language PC 1 highest captions
b
Language PC 1 highest images
Language PC 1 lowest captions
Language PC 1 lowest images
Language PC 1 projection (vision model)
language and vision, we next examined the principal components of the language encoding model weights. We first determined the 10,000 best predicted voxels in each subject using a bootstrap procedure (Appendix C). We next averaged the encoding model weights across the 4 delays for each feature to remove temporal information. We finally applied principal components analysis to the averaged encoding model weights, producing 768 orthogonal principal components (PCs) that are ordered by the amount of variance they explain across the voxels. We projected stimulus features onto each PC to interpret the semantic dimension that the PC captures, and we projected encoding model weights onto each PC to assess how the corresponding semantic dimension is represented across cortex. We estimated encoding models using layer 8 of BridgeTower, which has the highest average performance across cortex, and did not correct for negative cross-modality scores.
Projecting Flickr30k caption features onto the first PC of the language encoding model weights (language PC 1), we found that phrases with positive language PC 1 projections tend to refer to people and social interactions, while phrases with negative language PC 1 projections tend to refer to places and objects (Figure 4a). Projecting the language encoding model weights onto language PC 1, we found that voxels with positive projections were mostly located in inferior parietal cortex, precuneus, temporal cortex, and regions of frontal cortex; voxels with negative projections were mostly located in superior parietal cortex and middle frontal cortex. These findings are consistent with previous studies that mapped how different concepts in language are represented across cortex [9, 41].
Since our previous results show that many voxels have shared tuning for concepts in language and vision, the semantic dimensions captured by the language PCs may also underlie the space of visual representations. Projecting Flickr30k image features onto language PC 1, we found that images with positive language PC 1 projections tend to contain people, while images with negative language PC 1 projections tend to contain places such as natural scenes (Figure 4b). Projecting the vision
encoding model weights onto language PC 1, we found similar patterns to the language encoding model projections outside of visual cortex. However, we additionally found voxels with positive projections in visual cortex regions known to represent faces (OFA, FFA) and body parts (EBA) in vision. These results suggest that the semantic dimension captured by language PC 1 is partially shared between language and visual representations.
We next quantified the degree to which each of the top 10 language PCs is shared between language and visual representations. For each PC, we spatially correlated the projections of the language and the vision encoding model weights. We separately computed spatial correlations across multimodal voxels that were well predicted by both the language and vision encoding models—operationalized as the 10,000 voxels with the highest  $\min(r_{story} \rightarrow story, r_{movie} \rightarrow movie)$ —as well as across unimodal voxels that were well predicted by either the language or vision encoding model but not both—operationalized as the 10,000 remaining voxels with the highest  $\max(r_{story} \rightarrow story, r_{movie} \rightarrow movie)$ . The spatial correlations quantify how similarly each population of voxels represents each semantic dimension in language and vision.
We tested the significance of these correlations using a blockwise permutation test (Appendix C). For multimodal voxels (Figure 4c), the projections of the language and the vision encoding model weights were significantly correlated for language PCs 1, 3, and 5  $(\mathrm{q(FDR)} < 0.05$ ; see Appendix D for further analyses). For unimodal voxels (Figure 4d), the projections of the language and the vision encoding model weights were not significantly correlated for any of the language PCs.
5.4 Comparing transfer performance using multimodal and unimodal transformers
Finally, we isolated the effects of multimodal training on cross-modality performance. To provide a unimodal baseline, we estimated language encoding models using RoBERTa [37] and vision encoding models using ViT [38]. Since these unimodal transformers were used to initialize BridgeTower, they provide a baseline for how well language and visual features are aligned prior to multimodal training.
To perform cross-modal transfer with features from the unimodal transformers, we first estimated linear alignment matrices (Section 2.2) on the Flickr30k dataset. We estimated image  $\rightarrow$  caption matrices that predict each RoBERTa language feature from the ViT visual features, and caption  $\rightarrow$  image matrices that predict each ViT visual feature from the RoBERTa language features. We then used these alignment matrices to evaluate how well a RoBERTa language encoding model can predict movie fMRI responses using ViT movie features, and how well a ViT vision encoding model can predict story fMRI responses using RoBERTa story features.
Across cortex, we found that multimodal features led to significantly higher story  $\rightarrow$  movie performance (one-sided paired t-test;  $p < 0.05$ ,  $t(4) = 2.1377$ ,  $\overline{r}_{\text{multimodal}} = 0.0230$ ,  $\overline{r}_{\text{unimodal}} = 0.0219$ ) and movie  $\rightarrow$  story performance (one-sided paired t-test;  $p < 0.05$ ,  $t(4) = 5.3746$ ,  $\overline{r}_{\text{multimodal}} = 0.0097$ ,  $\overline{r}_{\text{unimodal}} = 0.0074$ ) than unimodal features (Figure 5a). In particular, multimodal features led to higher story  $\rightarrow$  movie performance outside of visual cortex, and higher movie  $\rightarrow$  story performance outside of auditory cortex (Figure 5b). These results suggest that multimodal training objectives induce the BridgeTower model to learn more complex connections between language and visual representations than a simple linear alignment between modalities.
To isolate the impact of any kind of training on transfer performance, we compared cross-modality performance for a trained BridgeTower model to cross-modality performance for a randomly initialized BridgeTower model. We found that the trained model substantially outperformed the randomly initialized model (Appendix H).
Our study demonstrates that encoding models trained on brain responses to language or visual stimuli can be used to predict brain responses to stimuli in the other modality, indicating similar conceptual representations of language and visual stimuli in the brain [21, 20]. Our analyses identified the regions in which these representations are aligned, as well as the semantic dimensions underlying this alignment. Notably, however, while tuning for concepts in language and vision is positively correlated in most regions outside of visual cortex, it is negatively correlated in visual cortex. Understanding the nature of this inverted tuning is an important direction for future work that could provide deeper insights into the relationship between language and vision [43, 44].
b
Figure 5: Transfer performance using features from multimodal and unimodal transformers. Cross-modality scores were compared between multimodal encoding models that extract features using BridgeTower and unimodal encoding models that extract features using RoBERTa and ViT. (a) The difference between the multimodal score and the unimodal score for each voxel in one subject is projected on the subject's flattened cortical surface. A voxel appears red if it is better predicted by multimodal features, blue if it is better predicted by unimodal features, white if it is well predicted by both, and black if it is well predicted by neither. (b) Histograms compare multimodal scores to unimodal scores. For story  $\rightarrow$  movie transfer, multimodal features outperform unimodal features in regions outside of visual cortex.
To estimate the cross-modal encoding models, we used the BridgeTower multimodal transformer to extract features of the story and movie stimuli. The successful transfer performance demonstrates that multimodal transformers learn aligned representations of language and visual input. Moreover, stimulus features extracted from multimodal transformers led to better cross-modality performance than linearly aligned stimulus features extracted from unimodal transformers, suggesting that multimodal training tasks enable BridgeTower to learn connections between language and visual concepts that go beyond a simple linear alignment between unimodal representations.
One limitation of our study is the size of the fMRI dataset. Many naturalistic neuroimaging experiments collect a large amount of brain data from a small number of subjects, which allows for the replication of the effects in each individual subject [12, 8, 9, 47, 48]. While our results are consistent across subjects (Appendix E), it would be ideal to replicate these effects in a larger and more diverse population. A second limitation of our study is that the story and movie stimuli are not matched in terms of semantics and timescale coverage, which could lead to asymmetries between story  $\rightarrow$  movie and movie  $\rightarrow$  story performance. Finally, a third limitation of our study is that there may be relevant stimulus features that are not captured by current multimodal transformers. This could lead us to underestimate the degree of multimodality in the brain. As future multimodal transformers learn to extract increasingly relevant features from story and movie stimuli, new experiments similar to our study might reveal additional brain regions with shared tuning across modalities.
This work was supported by the National Institute on Deafness and Other Communication Disorders under award number 1R01DC020088-001 (A.G.H.), the Whitehall Foundation (A.G.H.), the Alfred P. Sloan Foundation (A.G.H.) and the Burroughs Wellcome Fund (A.G.H.).