Brain encoding models based on multimodal transformers can transfer across language and vision
Authors: Jerry Tang
Meng Du
Vy A. Vo
Vasudev Lal
Alexander Huth
Venue: NeurIPS 2023 poster (2023)
Keywords: fMRI
neuroscience
encoding models
multimodal transformers
language
vision
TL;DR: Multimodal transformers were used to estimate encoding models on fMRI responses to language and visual stimuli that can predict fMRI responses to stimuli in the other modality.

Abstract: Encoding models have been used to assess how the human brain represents concepts in language and vision. While language and vision rely on similar concept representations, current encoding models are typically trained and tested on brain responses to each modality in isolation. Recent advances in multimodal pretraining have produced transformers that can extract aligned representations of concepts in language and vision. In this work, we used representations from multimodal transformers to train encoding models that can transfer across fMRI responses to stories and movies. We found that encoding models trained on brain responses to one modality can successfully predict brain responses to the other modality, particularly in cortical regions that represent conceptual meaning. Further analysis of these encoding models revealed shared semantic dimensions that underlie concept representations in language and vision. Comparing encoding models trained using representations from multimodal and unimodal transformers, we found that multimodal transformers learn more aligned representations of concepts in language and vision. Our results demonstrate how multimodal transformers can provide insights into the brain’s capacity for multimodal processing.

Brain encoding models based on multimodal transformers can transfer across language and vision
Jerry Tang UT Austin jerrytangdutexas.cdu
Meng Du Intel Labs, UCLA mengduOucla. edu
Vy A. Vo Intel Labs vy. vy.vo@intel.co
Vasudev Lal
Alexander G. Huth UT Austin
Intel Labs
vasudev. Laaidde..a@@intele com
huth@cs. www.useexeeco edu
Abstract
Encoding models have been used to assess how the human brain represents con- cepts in language and vision. While language and vision rely on similar concept representations, current encoding models are typically trained and tested on brain responses to each modality in isolation. Recent advances multimodal pretraining have produced transformers that can extract aligned representations of concepts in language and vision. this work, we used representations from multimodal transformers train encoding models that can transfer across fMRI responses stories and movies. We found that encoding models trained on brain responses to modality successfully predict brain responses the other modality, partic- ularly in cortical regions that represent conceptual meaning. Further analysis these encoding models revealed shared semantic dimensions that underlie concept representations in language and vision. Comparing encoding models trained using representations from multimodal and unimodal transformers, we found that multi- modal transformers learn more aligned representations of concepts in language and vision. Our results demonstrate how multimodal transformers can provide insights into the brain's 's capacity for multimodal processing.
Introduction
Encoding models predict brain responses from quantitative features the stimuli that elicited them [1]. In recent years, fitting encoding models to data from functional magnetic resonance imaging (fMRI) experiments has become powerful approach for understanding information processing the brain. While many studies have shown that encoding models achieve good performance when trained and tested on brain responses to a single stimulus modality, such as language [2-9] or vision [10-15], the human brain is remarkable in its ability integrate information across multiple modalities. There is growing evidence that this capacity for multimodal processing is supported by aligned cortical representations of the same concepts in different modalities-fol for instance, hearing "a dog is chasing a cat" and seeing a dog chasing cat may elicit similar patterns of brain activity [16-21].
In this work, investigated the alignment between language and visual representations in the brain by training encoding models on fMRI responses to one modality and testing them on fMRI responses the other modality. Encoding models that successfully transfer across modalities can provide insights into how the two modalities are related [20]. Although previous work has compared language and vision encoding models, human annotations were required map language and visual stimuli into a shared semantic space [20]. To our knowledge, cross-modality transfer has yet to be
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

a
"i went class"
b
"It was hot day"
1.6
+.2
1.3
X
Transfomer representations
Language encading model
Brain IMRI responses
Transformer reprosintatiooss
Vision croding modaa
Predictions
Brain (1MR)
respories
Transtorme representattions
Vision encoding model
Brain responsees
Transtormer regrrssieiaaiions
Language encoding model
Brain (MR)
Predictions
resporiees
X
Figure 1: Cross-modality encoding model transfer. (a) Multimodal transformers were used to extract features of story and movie stimuli. Language encoding models were fit (using L2- L2-regularized linear regression) predict story fMRI responses from story stimuli; vision encoding models were fit to predict movie fMRI responses from movie stimuli. (b) As the language and vision encoding models share representational space, language encoding models could be used to predict IMRI responses to movie stimuli, and vision encoding models to predict fMRI responses to story stimuli. Encoding model performance was quantified by the linear correlation between the predicted and the actual response time-courses in each voxel.
demonstrated using encoding models trained on stimulus-computable features that capture the rich connections between language and vision.
One way to extract aligned features of language and visual stimuli is using transformer models trained on multimodal objectives like image-text matching [22-28]. Recent studies have shown that multimodal transformers can model brain responses to language and visual stimuli [29], often outper- forming unimodal transformers [30-32], suggesting that multimodal training enables transformers learn like representations. However, these studies do not assess whether representations from multimodal transformers can be used to train encoding models that transfer across modalities. Since multimodal transformers are trained to process paired language and visual inputs, the representations learned for concept in language could be correlated with the representations learned for that concept in vision. This alignment between language and visual representations in multimodal transformers could facilitate the transfer encoding models across modalities, which relies on the alignment between language and visual representations in the brain.
To test this, we used the BridgeTower [28] multimodal transformer to model fMRI responses to naturalistic stories [9] and movies [12]. We separately obtained quantitative features of story and movie stimuli by extracting latent representations from BridgeTower. We estimated language encoding models using story features and story fMRI responses, and vision encoding models using movie features and movie fMRI responses (Figure la). We evaluated how well the language encoding models can predict movie fMRI responses from movie features (slory - movie) and how well vision encoding models can predict story fMRI responses from story features (movie - slory) (Figure b). We compared this to how well the language encoding models can predict story fMRI responses from story fcatures (story + story) and how well the vision encoding models can predict movie fMRI responses from movie features (movie movie).
We found that encoding models trained on brain responses one modality could accurately predict brain responses to the other modality. In many brain regions outside of sensory and motor cortex, story - movie performance approached movie - movie performance, suggesting that these regions encode highly similar representations of concepts in language and vision. To assess these representations, we performed principal components analysis on the encoding model weights and identified semantic dimensions that are shared between concept representations in language and vision. Finally, we found that cross-modality performance was higher for features extracted from multimodal transformers than for linearly aligned features extracted from unimodal transformers.

---

Our results characterize how concepts in language and vision are aligned in the brain and demonstrate that multimodal transformers can learn representations that reflect this alignment.
2 Multimodal transformers
Multimodal transformers are trained on paired language and visual data to perform self-supervised tasks such as image-text matching. Typically, these models are used to extract representations of paired language and visual input for downstream tasks such as visual question answering. However, since multimodal training objectives may impose some degree of alignment between language and visual tokens for the same concept, these models could also be used to extract aligned representations of language and visual input in isolation. For instance, latent representations extracted from the sentence "a dog is chasing a cat" may be correlated with latent representations extracted from a picture of dog chasing a cat.
2.1 Feature extraction
In this study, we extracted stimulus features using a pretrained BridgeTooerr model [28]. BridgeTTower is vision-language transformer trained on image-caption pairs from the Conceptual Captions [33], SBU Captions [34], MS COCO Captions [35], and Visual Genome [36] datasets. For each image-caption pair, the caption is processed using language encoder initialized with pretrained RoBERTa parameters [37] while the image is processed using a vision encoder initialized with pretrained ViT parameters [38]. The early layers of BridgeTTower process language and visual tokens independently, while the later layers of Bridge'Tower are cross-modal layers that process language and visual tokens together. Results are shown for the BridgeTower Base model; corresponding results for the
model are shown in Appendix F.
We used Bridge Tower extract features from the story (Section 3.1) and movie (Section 3.2) stimuli that were used in the fMRI experiments. Each story and movie stimulus was separately processed using BridgeTower by running forward passes with input from the corresponding modality. Hidden representations were extracted from each layer of BridgeTower as the inputs were processed.
For stories, segments of transcripts were provided to the model without accompanying visual inputs. feature vector was obtained for every word by padding the target word with context of 20 words both before and after. For movies, single frames were provided to the model without accompanying language inputs. Movies were presented at 15 frames per second, and feature vector was obtained for every 2-second segment by averaging latent representations across the 30 corresponding frames. Feature extraction was done on a node with 10 Intel Xeon Platinum 81 CPUs and an Nvidia Quadro RTX 6000 GPU.
2.2 Feature alignment
Transformers compute representations for each layer by attending to different combinations of the input tokens. While multimodal training tasks may require models to align language and visual tokens for the same concept, the nature of this alignment depends on the type of attention mechanism used to combine language and visual tokens [39]. BridgeTowwe uses a co-attention mechanism wherein language and visual tokens are passed through different projection matrices, and query vectors from each modality are only scored against key vectors from the other modality. As consequence, the language and visual feature spaces extracted from each layer of Bridge may differ up to a linear transformation.
To correct for these potential transformations, we used the Flickr30K dataset wi0j-which consists of paired captions and images-to estimate linear transformation matrices that explicitly align the language and visual feature spaces extracted from BridgeTower We first used Bridge to separately extract language features of each caption and visual features of each image. We then used L2-regularized linear regression to estimate image + coption matrices that predict each language feature from the visual features, and caplion - image matrices that predict each visual feature from the language features. We evaluated the matrices by measuring how well they align language and visual features for held out samples from the Flickr30K dataset, and found that the matrices achieved significant alignment of language and visual features (Appendix C).
3

---

Before using the language encoding model to predict movie fMRI responses. we first used the image - caption matrix to project the movie features into the language feature space. Similarly, before using the vision encoding model to predict story fMRI responses, we first used the caption. image matrix to project the story features into the visual feature space.
To show that this explicit feature alignment is only necessary for certain attention mechanisms, we repeated our analyses using a KD-VLP [24]] vision-language transformer (Appendix G). KD-VLP uses merged attention mechanism wherein language and visual tokens are passed through the same projection matrices and query vectors are scored against all key vectors, making explicit feature alignment unnecessary.
3 fMRI experiments
We analyzed publicly available fMRI data from five subjects (2 female, 3 male) who participated in a story listening experiment and movie watching experiment [20]. Previous studies have modeled the story fMRI data as a function of the story stimuli [9, 4, 41] and the movie fMRI data as a function of the movie stimuli [12]. Blood-oxygen level dependent (BOLD) brain signals were recorded using gradient-echo EPI on a a 3T Siemens TIM Trio scanner at the UC Berkeley Brain Imaging Center with 32-channel volume coil, TR = 2.0045 seconds, TE = 31 ms, flip angle 70 degrees, voxel size 2.24 x 2.24 x 4.1 mm (slice thickness 3.5 mm with 18 percent slice gap), matrix size 100 x 100, and 30 axial slices. All experiments and subject compensation were approved by the UC Berkeley Committee for the Protection of Human Subjects. Further details about the data are provided in Appendix A.
3.1
Story experiment
Stimuli for the story experiment consisted of 10 naturally spoken narrative stories from The Moth Radio Hour totaling just over 2 hours [9]. The stories were presented over Sensimetrics $14 headphones. Subjects were instructed listen the stories with their eyes closed. Each story was played during single fMRI scan.
3.2
Movie experiment
Stimuli for the movie experiment consisted of 12 videos totaling 2 hours [12, 14). Each video was made by concatenating a sequence 10-20 s clips from movies drawn from the Apple Quick HD gallery and YouTube. The videos were presented 15 frames per second. The videos originally contained dialogue and music, but were presented silently to the subjects. Subjects were instructed to fixate on a dot at the center of the screen. Each video was played during single IMRI scan.
4 Voxelwise encoding models
Our study builds upon the voxelwise modeling framework used in many previous fMRI studies []]. Voxelwise encoding models learn a mapping from stimuli to the brain responses that they elicit in each individual subject [9]. Brain images recorded at times t= 1...T are given by y(t) E IR" where m is the number of voxels in the cerebral cortex. Responses for one subject are represented by the response matrices Ystory E R Tatory xm and Ymovie IR Tmuvic Km.
The story and movie features
resampled to the fMRI acquisition times using Lanezos filter. To account for the hemodynamic response, finite impulse response model with 4 delays (2, 4, 6, and 8 seconds) was applied the downsampled features. This resulted in the delayed stimulus matrices X Xetory E IR atory x4k and X movie E I movie x4k where k = 768 is the dimensionality of the Bridge features.
We modeled the mapping between stimulus features and brain responses with a linear model Y = =XB. Each column of represents voxel's 's linear weights the 4k delayed features. The weights were estimated using L-2reeularized linear regression. Regularization parameters were independently selected for each voxel using 50 iterations of cross-validation procedure.
Voxelwise modeling was done on a workstation with an Intel Core i9-7900X CPU. Further details about voxelwise modeling are provided in Appendix B.

---

4.1 Evaluation
Voxelwise encoding models were evaluated by predicting the response matrices Yest from the stimulus matrices X test for stimuli that were excluded from model estimation. Prediction performance for each voxel was quantified by the linear correlation between the predicted and the actual response time-courses [9, 4, 6, 2].
To quantify source - target performance from a source modality to a target modality, we estimated encoding models using all source scans and evaluated prediction performance each target scan. We averaged linear correlations across the target seans to obtain a score soogree-targe for each voxel. A high I ssource-targge score indicates that voxel's tuning for concepts in the target modality is aligned with its tuning for concepts in the source modality.
We compared this cross- moos--mddlltyy performance against the within-modality performance of an en- coding model trained on the target modality. To quantify target -> target performance, we held out each target scan, estimated encoding models using the remaining target scans, and evaluated prediction performance on the held out target scan. We averaged linear correlations across the held out target scans to obtain a score Itarget->torge for each voxel.
Many previous studies evaluated encoding models on averaged responses to single test stimulus [9, 42, 4]. Here we evaluated encoding models on single repetition responses to many test stimuli, as our stimuli vary in semantic content. While our evaluation produces less biased estimates of cross-modality and within-modality performance, the correlation values will be lower than previously reported results due to the lower signal-to-nn toniiee ratio of single repetition response data.
We separately identified voxels with statistically significant story - story and movie + movie within- midhin-modalitt scores. We computed null distributions of the within-modality scores using blockwise permutation test to account for autocorrelation in the voxel responses (Appendix C).
4.2 Layer selection
Separate encoding models were trained using stimulus features extracted from each layer of BridgeTTower. We summarized performance across layers by estimating the best layer for each voxel using a bootstrap procedure. For each test scan, we estimated the best layer for each voxel based mean prediction performance across the remaining test scans. We then used the selected layer for each voxel to compute prediction performance for that voxel on the held out test scan. We used this procedure for all analyses unless noted otherwise.
5 Results
We separately estimated language and vision encoding models for each subject. We used these models to compute mmoor,- "mmovi--story Tstory-story. and ("movie-movie scores for each voxel.
5.1 Cross-modality performance
Cross-modakee performance was visualized by projecting Istoryy-movie and I'movic-sstrr scores for each voxel in one subject onto a flattened cortical surface (Figure 2a; see Appendix E for other subjects; see Appendix for BridgeTower-Laagg model results and Appendix for KD-VLP model results). We found positive "story-movii and mmooic-sstory scores in many parietal, temporal, and frontal regions, which have previously been shown to represent the mcaning of concepts in language [9] and vision [12]. The high story + movie scores and positive (albeit lower) movie - story scores suggest that these voxels have similar tuning for the same concepts across modalities [20].
Conversely, we found negative Issory--mert and mmovie-sstory scores in visual cortex. Previous studies have reported that the tuning to perceptual information visual cortex may be inverted during conceptual processing in the absence of perception [43, 44]. If there is systematically inverted tuning between language and vision, it should be possible to first estimate which voxels would have negative cross- coos-itoaaltyy scores using separate validation data, and then multiply their weights by -1 before computing I'sory--monnt and r Emonic-sstory on test data. We performed this correction using bootstrap procedure across the test scans. For each test scan, we estimated which voxels have inverted
5

---

stoy more
story
b
Uncorrected
Corrected
Unconacted
Corrected
0.02
0.02
0.00
0.00
Highost
Highest
Layer depth
Layer depth
Figure Cross-modality prediction performance. Encoding models estimated brain sponses to one modality were evaluated on brain responses to the other modality. Cross-modality performance is measured by the linear correlation (r) between predicted and actual responses. (a) -'tory--movie and Y mmovie-story scores for each voxel one subject are displayed on the subject's s cortical surface. voxel appears red if its score is positive, blue its score negative, and white if its score is zero. White outlines show regions of interest (ROIs) identified using separate localizer data. AC and VC denote auditory cortex and visual cortex. "sterr--manne and 'movie-storyy scores were positive in regions that have previously been found to represent the meaning of concepts in language and vision, but negative in visual cortex. (b) Prediction performance for each layer of Bridge/Tower. Negative scores were corrected by using held out data to one-parameter model for each voxel that predicts whether the encoding model weights should be negated before computing transfer performance. Scores were averaged across voxels and then across subjects to provide unbiased albeit conservatively low summaries of model performance. Error bars indicate standard eтrоr of the mean across subjects.
tuning based on the mean prediction performance across the remaining test scans. We then multiplied the weights of these voxels by- -1 before computing prediction performance on the held out test scan.
Figure 2b shows story movic and mouie story performance across cortex before and after this correction. We summarized performance for each layer of BridgeTTowa by averaging the linear correlations across all cortical voxels and subjects. Averaging across cortical voxels is unbiased way to compare different encoding models [4], but it produces conservatively low correlation values since many cortical voxels are not involved in processing language or vision. Across layers, the correction significantly improved story + movie performance (one-sided paired t-test; < 0.05, 0.0053) and movie -> story performance
t(4) 7.5295, Fcorrected 0.0230. Funcorrected (one-sided paired t-test; p < 0.05, t(4) = 6.6356. Foorrected = 0.0097, F Funcorrected = 0.0031), providing evidence for systematically inverted tuning for the same concepts across modalities.
5.2 Comparing cross-modality and within modiiny performance
While the previous analysis identified voxels with similar tuning for concepts in language and vision, it did not characterize the extent of this cross-modal similarity. To do this, next compared cross-modality performance to within-modality performance for each voxel (Figure 3a).
To quantify the amount of information that the language encoding model learns about tuning for movies, we divided Istory-smorie by mmovie--moese for each voxel. If the movie responses in a voxel are well-predicted by the vision encoding model but poorly predicted by the language encoding model, this value should low. Conversely, the movie responses are predicted about as well using both the vision and language encoding models, then this value should be close to 1. In visual cortex, which represents structural features of visual stimuli [14], the language encoding model performed much worse than the vision encoding model predicting movie responses. In significantly

---

entry 20000
move staxy /aayy stay
b
Distance from V1 (rnm)
100
AC (mmm)
100
0.8
0.8
Focose stay
6.4
0.4
Eeory
0.2
0.0
0.2
0.6
Density
0.0
0.2
0.4
0.6
Density
Fmovic
'stay
Figure 3: Comparing cross-modality and within-modalitt prediction performance. Cross- compared against within coodiitmo scores in voxels with statistically
modality scores significant within within-modalnn modality scores. (a) Cross- modss-t scores and muthin-mddality scores for each voxel in subject are projected onto the subject's flattened cortical surface. A voxel appears dark if its cross-modality score is much lower than its within-modality score, and approaches its within-modality
bright if its moss-modalityy voxels under the within-modality model (q(FRR) < 0.05, one-sided permutation test) are shown. (b) Histograms compare modality scores within-modality scores. For movie responses, - scores were much lower than mmmovi-- mouse scores near visual cortex but approached -movii--movie scores in other regions. For story responses, I'moviesstory scores were generally much lower than Tstory-tstory scores across cortex.
Only well-predicted
predicted voxels outside of visual cortex, which represent the meaning of visual stimuli [45, 12, 20], the language encoding model often approached the performance of the vision encoding model (Figure 3b).
Similarly, to quantify the amount of information that the vision encoding model learns about tuning for stories, we divided mmmoii-+story by T Tstory-+story for each voxel. In auditory cortex, which represents acoustic and articulatory features of language stimuli [42]. the vision encoding model performed much worse than the language encoding model at predicting story responses, In some significantly predicted voxels outside of auditory cortex, which have been shown to represent the meaning of language stimuli [42, 9]. the vision encoding model performed relatively better, but still did not approach the performance of the language encoding model (Figure 3b).
These results suggest that visual tuning can often be estimated solely based how voxel responds to stories, while is much harder to estimate language tuning solely based on how a voxel responds to movies [46]. One potential confound that could contribute this asymmetry is that the story stimuli contain both concrete concepts (such as places) and abstract concepts (such as emotions) while the movie stimuli mostly contain concrete concepts. Another potential confound that the story stimuli contain information at longer timescale than the movie stimuli, which consist only of 10-20 second clips. To isolate whether the asymmetry Figure driven by differences between language and visual representations in the brain, future work could use story and movie stimuli that matched in terms of semantics and timescale coverage.
5.3 Encoding model principal components
The earlier analyses showed that encoding models can readily transfer across modalities, at least in the direction from language vision. But what kind of information is that these cross-modal models capturing? To understand the semantic dimensions that underlie the shared tuning for concepts in

---

a
Language PC highest captions
Language PC highest images
Mean
Chance
Muntinddl voxsl conseeation
"tis wife is telinn hin be acnds eur leaktiier and be acgatin that all the aeaity food tastes gross"
0.4
young siring her
due proded what siee looking
Language PC 1 lowest captione
Language PC loweet images
"fuur zan with display tible set up top dhouldeer and
mokk dlspiiyingg soee plane and other soupynes"
-0.2
people colerfal clobhing fabii tablos"
6
9
10
Deility
Language
PC
d
Language projection (anguage model)
Language PC projection (vision mode))
Mean
Chance
Unimodal voxal comelstion
0.4
5
10
Language PC
Figure 4: Encoding model principal components. Principal components analysis identified the first 10 principal components (PCs) of language encoding model weights. (a) Each caption in Flickr30k was projected onto language PC 1. This PC distinguishes captions that refer to people and social cnteractions-wiieee cortex, and frontal cortex-from places and objects- which are represented in superior parietal cortex and middle frontal cortex. (b) Each image in Flickr30k was projected onto language PC 1. Here language PC 1 distinguishes images of people-which are represented in EBA, OFA, FFA, inferior parietal cortex, precuneus, temporal cortex, and frontal cortex- from images of places-which are represented in superior parietal cortex and middle frontal cortex. (c) In voxels that were well predicted by both the language and the vision encoding models (red on inset flatmap), projections of language and vision encoding model weights were significantly correlated (*) for several language PCs, indicating semantic dimensions that are shared between language and visual representations. (d) In voxels that were well predicted by only the language or the vision encoding models (blue on the inset flatmap), projections of language and vision encoding model weights were not significantly correlated for any language PCs. For all results, error bars indicate standard error of the mean across subjects.
are represented in inferior parietal cortex, precuneus, temporal
language and vision, we next examined the principal components of the language encoding model weights. We first determined the 10,000 best predicted voxels in each subject using a bootstrap procedure (Appendix C). We next averaged the encoding model weights across the 4 delays for each feature to remove temporal information. We finally applied principal components analysis to the averaged encoding model weights, producing 768 orthogonal principal components (PCs) that are ordered by the amount of variance they explain across the voxels. We projected stimulus features onto each PC to interpret the semantic dimension that the PC captures, and we projected encoding model weights onto each PC to assess how the corresponding semantic dimension is represented across cortex. We estimated encoding models using layer 8 of Bridge' Tower, which has the highest average performance across cortex, and did not correct for negative cross-modalits scores.
Projecting Flickr30k caption features onto the first PC of the language encoding model weights (language PC 1), we found that phrases with positive language PC projections tend to refer to people and social interactions, while phrases with negative language PC 1 projections tend to refer to places and objects (Figure 4a). Projecting the language encoding model weights onto language PC 1, we found that voxels with positive projections were mostly located in inferior parietal cortex, precuneus, temporal cortex, and regions of frontal cortex; voxels with negative projections were mostly located in superior parietal cortex and middle frontal cortex. These findings are consistent with previous studies that mapped how different concepts in language are represented across cortex 41].
Since our previous results show that many voxels have shared tuning for concepts in language and vision, the semantic dimensions captured by the language PCs may also underlie the space of visual representations. Projecting Flickr30k image features onto language PC 1, we found that images with positive language PC projections tend to contain people, while images with negative language PC I projections tend to contain places such as natural scenes (Figure 4b). Projecting the vision
8

---

encoding model weights onto language PC 1, we found similar patterns to the language encoding model projections outside of visual cortex. However, we additionally found voxels with positive projections in visual cortex regions known to represent faces (OFA, FFA) and body parts (EBA) in vision. These results suggest that the semantic dimension captured by language PC partially shared between language and visual representations.
We next quantified the degree which each of the top 10 language PCs is shared between lan- guage and visual representations. For each PC, we spatially correlated the projections of the language and the vision encoding model weights. We separately computed spatial correlations across multimodal voxels that were well predicted by both the language and vision encoding mod- els als-operationalizeed as the 10,000 voxels with the highest min( mam((ssory-rstory I'movie-movie (omooeee-aa as well as across unimodal voxels that were well predicted by either the language or vision en- coding model but not both oothaooperaionaalizeed as the 10,000 remaining voxels with the highest max(Fstory (ttoryytryystt Toooic- "mmoii--movve)) The spatial correlations quantify how similarly each popula- tion of voxels represents each semantic dimension in language and vision.
We tested the significance of these correlations using a blockwise permutation test (Appendix C). For multimodal voxels (Figure 4c), the projections of the language and the vision encoding model weights were significantly correlated for language PCs 1, 3, and 5 (q(FDR) < 0.05; see Appendix D for further analyses). For unimodal voxels (Figure 4d), the projections of the language and the vision encoding model weights were not significantly correlated for any of the language PCs.
5.4 Comparing transfer performance using multimodal and unimodal transformers
Finally, we isolated the effects of multimodal training on cross-modality performance. To provide a unimodal baseline, we estimated language encoding models using RoBERTa [37] and vision encoding models using ViT [38]. Since these unimodal transformers were used to initialize BridgeTower, they provide baseline for how well language and visual features are aligned prior to multimodal training.
To perform cross-modal transfer with features from the unimodal transformers, we first estimated linear alignment matrices (Section 2.2) on the Flickr30k dataset. We estimated image - caption matrices that predict each RoBERTa language feature from the ViT visual features, and caption -> image matrices that predict each VIT visual feature from the RoBERTa language features. We then used these alignment matrices to evaluate how well RoBERTa language encoding model can predict movie fMRI responses using VIT movie features, and how well VIT vision encoding model can predict story fMRI responses using RoBERTa story features.
Across cortex, we found that multimodal features led to significantly higher story + movie performance (one-sided paired t-test: p 0.05, t(4) 22.3377, 7 maatimooda 0.0230. Funémodal = 0.0219) and movie + story performance (one-sided paired t-test; < 0.05, t(4) = 5.3746, mmaltimodal 0.0097, Funimodal = 0.0074) than unimodal features (Figure 5a). In particular, multimodal features led to higher story + movie performance outside of visual cortex, and higher movie + story performance outside of auditory cortex (Figure 5b). These results suggest that multimodal training objectives induce the BridgeTower model to learn more complex connections between language and visual representations than a simple linear alignment between modalities.
To isolate the impact of any kind of training on transfer performance, we compared cross-modality performance for a trained BridgeTowwe model cross-modality performance for a randomly initial- ized BridgeTTower model. We found that the trained model substantially outperformed the randomly initialized model (Appendix H).
Discussion
Our study demonstrates that encoding models trained on brain responses to language or visual stimuli can be used to predict brain responses to stimuli in the other modality, indicating similar conceptual representations of language and visual stimuli in the brain [21, 20]. Our analyses identified the regions in which these representations are aligned, as well as the semantic dimensions underlying this alignment. Notably, however, while tuning for concepts in language and vision is positively correlated in most regions outside of visual cortex, it is negatively correlated in visual cortex. Understanding the nature of this inverted tuning is an important direction for future work that could provide deeper insights into the relationship between language and vision [43, 44].

---

a
Unimodal better
Muttimodal better
"mat" fuv
0.1
maxi muti uay)
story mowie
move story
b
Distance from V1 (mon)
Distance (mm)
CA
Couttrode
muuthodd
0.2
02
0.0
0.2
0.4
Deneity
0.0
0.2
Density
funimodal
lunimodal
Figure 5: Transfer performance using features from multimodal and unimodal transformers. Cross-modality scores were compared between multimodal encoding models that extract features using BridgeToowrr and unimodal encoding models that extract features using RoBERTa and ViT. (a) The difference between the multimodal score and the unimodal score each voxel in subject is projected the subject's 's flattened cortical surface. A voxel appears red if it is better predicted by multimodal features, blue if it better predicted by unimodal features, white if it is well predieted by both, and black if it is well predicted by neither. (b) Histograms compare multimodal scores to unimodal scores. For story - movie transfer, multimodal features outperform unimodal features in regions outside of visual cortex.
To estimate the cross- modal encoding models, we used the BridgeTower multimodal transformer to extract features of the story and movie stimuli. The successful transfer performance demonstrates that multimodal transformers learn aligned representations of language and visual input. Moreover, stimulus features extracted from multimodal transformers led to better cross-mes cooss-modlltyy performance than linearly aligned stimulus features extracted from unimodal transformers, suggesting that multi- modal training tasks enable Bridge' Tower to learn connections between language and visual concepts that go beyond a simple linear alignment between unimodal representations.
One limitation of our study is the size of the fMRI dataset. Many naturalistic neuroimaging ex- periments collect large amount of brain data from a small number of subjects, which allows for the replication of the effects in each individual subject [12, 8, 9, 47, 48]. While our results are consistent across subjects (Appendix E), would be ideal replicate these effects in larger and more diverse population. A second limitation of our study is that the story and movie stimuli are not matched terms of semantics and timescale coverage, which could lead to asymmetries between story movie and morie - story performance. Finally, a third limitation of our study is that there may be relevant stimulus features that are not captured by current multimodal transformers. This could lead us to underestimate the degree of multimodality in the brain. As future multimodal trans- formers learn to extract increasingly relevant features from story and movie stimuli, new experiments similar to our study might reveal additional brain regions with shared tuning across modalities.
Acknowledgments and Disclosure of Funding
This work was supported by the National Institute Deafness and Other Communication Disorders under award number 1R01DC020088 (A.G.H.), the Whitehall Foundation (A.G.H.), the Alfred P. Sloan Foundation (A.G.H.) and the Burroughs Wellcome Fund (A.G.H.).
10

---

References
[1] Thomas Naselaris, Kendrick Kay, Shinji Nishimoto, and Jack Gallant. Encoding and decoding in IMRI. Neuroimage, 56:400-410, 2011.
[2] Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications Biology, 5, 2022.
[3]
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Rose Price. Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner K. Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Y. Matias, Kenneth A. Norman, Orrin Devinsky, and Uni Hasson. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25:369-380, 2022.
[4] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fMRI. In Advances in Neural Information Processing Systems, 2018
[5] Amanda LeBel, Shailee Jain, and Alexander Huth. Voxelwise encoding models show that cerebellar language representations are highly conceptual. Journal of Neuroscience, 41:10341- 10355, 2021.
[6] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy G. Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture language: Integrative modeling converges predictive processing. Proceedings of the National Academy of Sciences, 118:c2105646) 18, 2021.
[7] Mariya Toneva and Leila Wehbe. Interpreting and improving natural- cetural-languaee processing (in machines) with natural language-pan -rroossssgo (in the brain). In Advances in Neural Information Processing Systems, 2019.
[8] Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PloS one, 9:e1 12575, 2014.
[9] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532:453-458, 2016.
[10] Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neuroimage, 152:184-194, 2017.
[11] Umut Güęlü and Marcel AJ van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35:10005-10014, 2015.
[12] Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron,
1224, 2012.
[13] Thomas Naselaris, Ryan J Prenger, Kendrick Kay, Michael Oliver, and Jack Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63:902-915, 2009.
[14] Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21:1641-1646, 2011.
[15] Kendrick Kay, Thomas Naselaris, Ryan J Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452:352-355, 2008.

---

[16] Jeffrey R Binder and Rutvik H Desai. The neurobiology of semantic memory. Trends in Cognitive Sciences, 15:527-536, 2011.
[17] Barry J Devereux, Alex Clarke, Andreas Marouchos, and Lorraine K Tyler. Representational similarity analysis reveals commonalities and differences in the semantic processing of words and objects. Journal of Neuroscience, 33:1 1e:18890-18111 2013.
[18] Scott Fairhall and Alfonso Caramazza. Brain regions that represent amodal conceptual knowledge. Journal of Neuroscience, 33:10552-10558, 2013.
[19] Alex Martin. SAAPES-groudding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain. Psychonomic Bulletin & Review, 23:979 990, 2016.
[20] Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James S Gao, Anwar O Nunez-Elizalde, and Jack L Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience, 24:1628-1636, 2021.
[21] Jerry Tang, Amanda LeBel, Shailee Jain, and Alcxander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 24:1-9, 2023,
[22] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjin Liu. UNITER: Universal image-text representation learning. In European Confer- ence on Computer Vision, 2019.
[23] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: aaiio--nnd-anguaae transformer without convolution or region supervision. In International Conference on Machine Learning, 2021.
[24] Yongfei Liu, Chenfei Wu, Shao-yen Tseng, Vasudev Lal, Xuming He, and Nan Duan. KD-VLP: Improving end-to-end vision-and-language pretraining with object knowledge distillation. arXiv preprint arXiv:2 10504, 2021.
[25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for aioon-and-language tasks. Advances in Neural Information Processing Systems, 2019.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[27] Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 077490 2019.
[28] Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, and Nan Duan. BridgeTower: Build- ing bridges between encoders in vision-language representation learning. arXiv preprint arXiv:2206.0866 08657, 2022.
[29] Dota Tianai Dong and Mariya Toneva. Interpreting multimodal video transformers using brain recordings. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
[30] Haoyu Lu, Qiongyi Zhou, Nanyi Fei, Zhiwu Lu, Mingyu Ding, Jingyuan Wen, Changde Du. Xin Zhao, Hao Sun, Huiguang He, et al. Multimodal foundation models are better simulators of the human brain. arXiv preprint arXiv:2208. 08263,, 2022.
[31] Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Incor- porating natural language into vision models improves prediction and understanding of higher visual cortex. bioRxiv, 2022.
[32] Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S. Bapi. Visio- linguistic brain encoding. In International Conference on Computational Linguistics, 2022.
12

---

[33] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Association for Compulational Linguistics, 2018.
[34] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems, 2011. 1.
[35] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014.
[36] Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2016.
[37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 1rXiv:190..1169. 2019.
[38] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 199999 2020.
[39] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean- Baatitat Alayrac, and Aida Ne- matzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. Association for Computational Linguistics, 2021.
[40] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Association for Computational Linguistics, 2014.
[41] Fatma Deniz, Anwar O Nunez-Elizalde, Alexander G Huth, and Jack L Gallant. The representa- tion of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39:7722-7736, 2019.
[42] Wendy A de Heer, Alexander G Huth, Thomas Griffiths, Jack L Gallant, and Frédéric E Theunissen. The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37:6539 6557,, 2017.
[43] Bradley R Buchsbaum, Sabrina Lemire-Rodget Candice Fang, and Hervé Abdi. The neural basis of vivid memory is patterned on perception. Journal of Cognitive Neuroscience, 24:1867- 1883, 2012.
[44] Martin Szinte and Tomas Knapen. Visual organization of the default network. Cerebral Cortex, 30:3518-3523 2020.
[45] Anwar O Elizald, Alexander G Huth, and Jack L Gallant. Voxelwise encoding models with non-spherical multivariate normal priors. Neuroimage, 197:482-492. 2019.
[46] Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. In Advances in Neural Information Processing Systems, 2022.
[47] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity. PLoS computational biology, 15:e1006633, 2019.
[48] Emily J Allen, Ghislain St- Yve,, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 15:116-126, 2022.
[49] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing, 2014.
13


Reviews:
[{"review_id": "Isx4vV87Tc", "reviewer": "Reviewer_C9Qg", "summary": "This paper studies whether we can use multi-modal transformers to predict brain activities. It specifically uses neural nets’ encoding for languages to predict brains’ activities for seeing vidieos, and neural nets’ encoding for images to predict brains’ activities. Their finding is that these prediction models are statistically significant so multi-modal transformers may resemble the underlying mechanism of brains. \n", "questions": "\n\n1.\tHow can one make sense of the very weak correlation results? Like there are some statistical evidence that transformers resemble the brain but transformers are only < 1% like brains? \n2.\tDo those t(4) mean t-stat/z-score? Do we need any kind of serial adjustments when one calculate these scores because I imagine input and/or response could potentially be correlated. I guess some bootstrapping machinaries are designed to address/alleviate this issue? \n3.\tWhy the correlation numbers in Fig 3 & 5 are much higher than those in Fig. 2? I guess I completely missed a fraction of the paper… \n4.\tDo it still make sense to do PCA analysis when a model’s r2 is already that low? Some t-stat seems to drop to 2-ish (so shall we interpret them as significant or not?).\n", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\nUnderstanding how neural nets are related to real brains has been a central question in deep learning. So this paper makes an important addition to the literature. While prior works already used similar frameworks (using encoding for transformers to predict MRI data/images), this work still seems quite impressive because it needs to circumvent quite a few kinks, such as aligning the visual and text encoding and carefully designing the statistical experiments to confirm models significance because the r2/r are fairly small. \n", "weaknesses": "I feel the paper has a very high bar on the audience, i.e., requiring one to have sufficient knowledge on transformers, MRI, and statistical analysis (multiple hypothesis test). I don’t know what to suggest to fix the problem as the paper is already very carefully written. Maybe having Appendices A and B to give more background materials on MRI and the statistical tools they use. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and making these suggestions. We have addressed your concerns in order below:\n\n>I feel the paper has a very high bar on the audience, i.e., requiring one to have sufficient knowledge on transformers, MRI, and statistical analysis (multiple hypothesis test). I don’t know what to suggest to fix the problem as the paper is already very carefully written. Maybe having Appendices A and B to give more background materials on MRI and the statistical tools they use.\n\nWe used an fMRI dataset and an encoding model framework that have been described in several previous studies [1, 2, 3]. As a result, we focused our methods sections on explaining the aspects of our study that differ from previous studies. In the final paper, we will emphasize how our study builds on previous studies. We will also describe the fMRI dataset and the encoding model framework in more detail in Appendices A and B.\n\n>How can one make sense of the very weak correlation results? Like there are some statistical evidence that transformers resemble the brain but transformers are only < 1% like brains?\n\nFigure 2b quantifies encoding model performance by taking the average correlation across all cortical voxels. Many fMRI studies select a subpopulation of voxels with an independent localizer task that isolates the cognitive effect of interest. However, we do not believe there is an effective multimodal localizer that would select all voxels that would represent the semantic dimensions of interest. Averaging across cortex is therefore an unbiased way to compare the different encoding models. However, averaging across cortex does yield conservatively low correlation values, since there are many voxels (e.g. those in motor cortex) which are not involved in story or movie perception. Figure 2a shows that many regions involved in story or movie perception have correlations greater than 0.1, and Figure 3b shows that these correlations can reach 0.5 for $story→movie$ transfer and 0.2 for $movie→story$ transfer. In the final paper, we will emphasize that Figure 2b reports average correlation across cortex, which is a conservative but unbiased metric.\n\nWe will also note that we evaluated encoding models on single-trial fMRI responses, whereas many previous studies evaluated encoding models on fMRI responses averaged across multiple repeats of the same stimulus. Evaluating on single-trial responses to a large number of stimuli provides high semantic coverage. However, averaging across repeats will increase the signal-to-noise ratio of the data and lead to higher correlations than those reported in our study. \n\nThank you for bringing up this point, and we hope that clarifying the correlation results will strengthen our paper.\n\n>Do those t(4) mean t-stat/z-score? Do we need any kind of serial adjustments when one calculate these scores because I imagine input and/or response could potentially be correlated. I guess some bootstrapping machinaries are designed to address/alleviate this issue?\n\nYes, $t$(4) indicates a t-test with 4 degrees of freedom. These tests were conducted across 5 individuals, so there is no autocorrelation that needs to be accounted for.\n\nIn analyses where we computed significance within each voxel (e.g. to obtain the voxel masks used in Figure 3) we used a blockwise permutation test to account for the autocorrelation in the responses (Appendix A.1). We will describe this process in more detail (line 158) in the final paper.\n\n>Why the correlation numbers in Fig 3 & 5 are much higher than those in Fig. 2? I guess I completely missed a fraction of the paper…\n\nAs mentioned above, Figure 2b reports average correlation across cortex, while the histograms in Figures 3b and 5b report correlations for individual cortical locations. Further, the histograms in Figures 3b and 5b are restricted to voxels with statistically significant $target→target$ scores. In the final paper, we will emphasize that Figure 2b reports average correlation across cortex.\n\n>Do it still make sense to do PCA analysis when a model’s r2 is already that low? Some t-stat seems to drop to 2-ish (so shall we interpret them as significant or not?).\n\nWe performed PCA using the top 10,000 voxels for each subject, which have an average correlation of 0.045. This approach is consistent with what has been done in previous studies [1, 2, 4, 5]. Furthermore, we found principal components that are consistent with those found in these previous studies.\n\nAll of the reported t-tests were statistically significant after correcting for multiple comparisons. The different t-tests correspond to different analyses (the first test compares multimodal transfer performance before and after correcting for negative correlations, whereas the second test compares multimodal transfer performance to unimodal transfer performance), so the t-statistics are not comparable across tests."}, {"review_id": "8JhXBPmTed", "reviewer": "Reviewer_V1MV", "summary": "In this paper, the authors used a pre-trained multimodal transformer, the BridgeTower model, to obtain feature representations of language and visual stimuli in the fMRI experiments. They trained encoding models on these representations and brain responses, and demonstrated that models trained on one modality could predict brain responses to another modality. Brain regions with aligned representations and semantic dimensions were discovered. The authors also defined a correlation score for evaluation and performed comparison with unimodal transformers. ", "questions": "Besides the score defined from correlation, what are some other possible measurements? What are the inter-subject differences?", "limitations": "The limitations were not adequately addressed in the paper. One possible discussion could be about the data limitation, and future work should include more subjects, maybe from different cultural backgrounds speaking different languages.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Investigating the multimodal processing of brain activity is an interesting problem. This paper presented a novel idea to develop brain encoding models from representations of different stimulus modality learned from a multimodal transformer and fMRI responses. It provides a new approach for analyzing the alignment of language and visual representations in the brain. ", "weaknesses": "The authors used a publicly available dataset containing fMRI responses from only five subjects, which is quite limited. Some of the discoveries, including brain regions for representation alignment, semantic dimensions, and the differences in cross-modality process, need to be tested or validated with experiment data. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and making these suggestions. We have addressed your concerns in order below:\n\n>The authors used a publicly available dataset containing fMRI responses from only five subjects, which is quite limited. Some of the discoveries, including brain regions for representation alignment, semantic dimensions, and the differences in cross-modality process, need to be tested or validated with experiment data.\n\nAlthough five subjects is low relative to some other human neuroimaging studies, this number is comparable to previous studies that model brain responses to naturalistic language [2, 6] and vision [1, 7, 8]. In many experiments involving naturalistic stimuli, a small number of subjects is scanned, but a large amount of brain data is collected for each subject. This allows for the replication of the effects in each individual subject (Appendix C), which can be more powerful than looking for average effects across a group [9]. However, we agree that having a larger and more diverse dataset is preferable, and we will discuss this limitation in the final paper.\n\nAll of the results of the study were based on data from fMRI experiments, so we would appreciate it if the reviewer could further clarify what it means to “test or validate them with experiment data”.\n\n>Besides the score defined from correlation, what are some other possible measurements [for encoding model performance]? \n\nSome studies quantify encoding model prediction performance using a classification task, in which encoding model predictions are used to predict which of two stimulus segments occurred at a given time [6, 10, 11]. However, linear correlation is the most commonly used metric used to quantify encoding model prediction performance [2, 12, 13, 14]. We will reference the studies that use this metric in the final paper.\n\n>What are the inter-subject differences [for encoding model performance]?\n\nOur flatmaps suggest that the effects that we observe are consistent across subjects (Appendix C). In Figures 2 and 4, we summarize variability across subjects using error bars. We will note how our results are similar and different across subjects in the final paper.\n\n>The limitations were not adequately addressed in the paper. One possible discussion could be about the data limitation, and future work should include more subjects, maybe from different cultural backgrounds speaking different languages.\n\nThank you for raising this point. In the final paper, we will add a third discussion paragraph summarizing limitations, including the stimulus limitations discussed in Section 5.2 (line 217) and the dataset limitations discussed above."}, {"review_id": "mUeYcgvlRE", "reviewer": "Reviewer_LQBp", "summary": "This work seeks to identify multi-modal processing in the brain. The proposed technique can be summarized as follows: a language encoding model is trained to predict neural story responses from story representations, and a vision language encoding model is used to predict movie responses from movie representations. Then, the language encoding model is used to try to predict movie responses from movie representations. The quality of predictions is used to measure the amount of cross-modal information being processed in a given brain region. An analogous procedure is carried out for the vision encoding model.", "questions": "- line 129: is it possible to be more specific about what the movie stimuli were like? Did the movies have dialogue? Music? \n- Figure 2: The white lines denoting the ROIs are hard to see. How exactly are these ROIs obtained?\n- Figure 2: It would also be helpful to label the visual area and the auditory areas on the inset 3D brains as well.\n- Figure 3: For $r_{story\\rightarrow movie}/r_{movie\\rightarrow movie}$, is the upper bound of this quantity 1 by default? Or did it just so happen that none of the cross-modal scores were better than the within-modal scores\n- Line 215-223 discuss reasons why the $r_{movie\\rightarrow story}$ were lower than $r_{story \\rightarrow movie}$. Is there a way to rule out the possibility that the alignment of feature spaces (discussed in 2.2) is simply worse in one direction? \n- Line 229: By what metric are the \"top\" voxels selected? \n- Line 231: How can stimulus features be projected onto each PC? Don't the stimulus features and the encoding weights occupy completely different spaces?", "limitations": "Limitations and negative societal impact are not discussed, but are probably not relevant here.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- As far as I can tell, the technique is novel. There are other works that look for multi-modality in the brain by trying to map multi-modal model activation to neural activation, but this is the first work to look for shared conceptual representation in the brain directly, by training two separate vision and language encoders and comparing the prediction performances after swapping the modalities.\n- This method seems mostly sound, with some remaining areas that could still be clarified (see weaknesses)", "weaknesses": "- The experiment described in section 5.1 and figure 2 is difficult to interpret on its own. The ability to predict visual response from a model trained on language stimuli is not enough to conclude that an area is multimodal. There is some uncertainty due to the fact that we don't know the quality of the alignment between feature spaces (discussed in section 2.2). In the limit, if the alignment between features is perfect, then the representation of an image of a cat can be perfectly aligned to the representation of the word \"cat\", and from there it becomes unclear whether a good $r_{movie\\rightarrow story}$ is due to a good alignment or true multi-modality in the brain.\n- Fortunately, this problem is partially addressed by normalizing the scores by within-modality performance, as is done in 5.2, but even then, a poor alignment could result in poor correlations across all regions. For both cases, it would be helpful to see the quality of the alignment, perhaps in the appendix.\n- I had trouble understanding the methods discussed in section 5.3 \"Encoding model principal components\" (see questions below). \n- My biggest concern with this work is significance to the NeurIPS community. The most interesting results, the identification of cross-modal areas in the brain, seem to be neuroscience related. The most relevant ML takeaways from this work are already well known, namely the benefits of multi-modal training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and making these suggestions. We have addressed your concerns in order below:\n\n>The experiment described in section 5.1 and figure 2 is difficult to interpret on its own…\n\nWe respectfully disagree with this assessment. Take the suggested example where the BridgeTower representation for cats in vision is identical to the BridgeTower representation for the word \"cat\" in language, and we are evaluating if a voxel that responds to cats in vision is multimodal (i.e. if it responds similarly to the word “cat” in language). During training, the encoding model learns that the voxel has a response of $β_{cat}$ when cats appear in movies. During $movie→story$ transfer, the encoding model predicts a response of $β_{cat}$ when the word “cat” occurs in stories. Transfer performance will only be high if the voxel has an actual response of $β_{cat}$ to the word “cat” in language.\n\nIn other words, a region with good transfer performance must respond similarly to concepts in language and vision, which is how we operationalize multimodality in the brain (line 22). In the final paper, we will emphasize how our transfer metric $r_{source→target}$ (line 150) relates to our definition of multimodality (line 22).\n\n>Fortunately, this problem is partially addressed by normalizing the scores by within-modality performance…\n\nThank you for raising this point. We will include our measure of alignment quality in the final paper.\n\nWe estimated alignment matrices using the Flickr30k dataset (line 103). To test alignment quality, we estimated alignment matrices on 80% of the image-caption pairs and evaluated them on the remaining 20%. We scored alignment quality by correlating the predicted and actual values for each feature across test pairs, and averaging the correlations across features. Finally, we normalized alignment scores by reporting the number of standard deviations above the mean of a null distribution, which we obtained by randomly shuffling the test captions and images.\n\nThese results are shown in Figure R1. The $caption→image$ matrices had a normalized score of 221.30 σ, while the $image→caption$ matrices had a normalized score of 197.28 σ. Thus in both directions the alignment was roughly 200 standard deviations better than random.\n\n>I had trouble understanding the methods discussed in section 5.3…\n\nThe methods used in Section 5.3 were based on previous studies that used PCA to characterize semantic tuning in the brain [1, 2, 4, 5]. We will clarify these methods in Appendix A.2 of the final paper.\n\n>My biggest concern with this work is significance to the NeurIPS community…\n\nMany papers recently published in NeurIPS [11, 12, 15] have shown that understanding the relationships between human and artificial information processing systems can benefit both fields of study. We believe that our results relating multimodal transformers and multimodal representations in the brain will be of interest to neuroscientists, computer scientists, and the growing number of researchers who work in the intersection of the fields.\n\n>line 129: is it possible to be more specific about what the movie stimuli were like…\n\nThe movie stimuli (https://gin.g-node.org/gallantlab/shortclips/src/master/stimuli) originally had dialogue and music, but the clips were presented silently to the subjects. We will clarify this in the final paper.\n\n>Figure 2: The white lines denoting the ROIs are hard to see…\n\nROIs were identified using functional localizers described in [1]. The relevant ROIs for this paper are auditory cortex (AC; identified using segments of music, speech, and natural sounds) and primary visual cortex (V1; identified using rotating wedges and expanding / contracting rings). In the final paper we will make the ROI lines more visible and include details about ROI localization in the Appendix.\n\n>Figure 2: It would also be helpful to label the visual area and the auditory areas on the inset 3D brains as well.\n\nWe will label the 3D brains in the final paper.\n\n>Figure 3: For r_story→movie/r_movie→movie, is the upper bound of this quantity 1 by default…\n\n$r_{s→m}/r_{m→m}$ does not have a strict upper bound at 1. For instance, $r_{s→m}$ may be higher than $r_{m→m}$ if the story and movie datasets differ in semantic coverage (line 217) or signal-to-noise ratio. Nonetheless, $r_{s→m}/r_{m→m}$ rarely exceeded 1 in voxels that were well-predicted by the vision encoding model (Figure 3).\n\n>Line 215-223 discuss reasons why the r_movie→story were lower than r_story→movie…\n\nAs mentioned above, we tested the quality of the alignment matrices by holding out test pairs from the Flickr30k dataset. Alignment was much higher than chance level and fairly symmetric, so it is unlikely that the asymmetry observed in Figure 3 was due to differences in alignment quality.\n\nPrevious studies have observed a similar asymmetry [15], and we agree that characterizing the sources of this asymmetry is an important direction for future work.\n\n>Line 229: By what metric are the \"top\" voxels selected?\n\nWe selected voxels using a cross-validation procedure (line 143). In each iteration, 20% of the timepoints were removed from the training dataset and reserved for validation. Models were estimated on the remaining 80% of the timepoints and used to predict responses for the reserved timepoints. The 10,000 voxels with the highest linear correlations were used for PCA. We will clarify this in the final paper.\n\n>Line 231: How can stimulus features be projected onto each PC…\n\nEncoding model weights predict how a voxel responds to stimulus features. For instance, a voxel that responds to animal words will have weights that are aligned with the BridgeTower features of animal words. The only difference is that encoding model weights have 4 delays for each feature (line 137). We remove this temporal information from the weights before PCA by averaging across the four delays for each feature [2]. We will clarify this in the final paper."}, {"review_id": "MLW5kH7wmU", "reviewer": "Reviewer_1ukF", "summary": "This paper aims to fit fMRI data using transformer models. Of note, the paper aims to extrapolate from models that fit fMRI responses to visual stimuli to be able to account for fMRI responses to what may presumably be language stimuli. The results throughout the paper are extremely weak but are presented as if the models can provide insights into the fMRI signals.", "questions": "This paper would need an enormous amount of work to bring it to a level where it has a minimal scientific description of findings, starting with an open discussion of all the steps involved, documenting whether there is any reproducibility in the data, showing that the fMRI signals relate to language or auditory or visual stimuli, and finally potentially a role for neural network models. The current results leave it open as to whether the fMRI data relates to the stimuli and compellingly demonstrate that the neural networks cannot capture the fMRI data. \n\nOther points. \nFigure 4 includes enigmatic color scales that go from “-“ to “+”. Color scales should have numbers.\n", "limitations": "The authors failed to describe all the multiple limitations and to adequately interpret their results.", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The idea of comparing representations to visual stimuli and auditory stimuli is certainly very interesting. \n\nThe possibility of elucidating “conceptual” representations that transcend any one input modality is also quite interesting. \n", "weaknesses": "It would be great to start by demonstrating that the fMRI data actually relates to the visual and story stimuli. This should be shown by rigorous description of the fMRI data, with actual units, error bars, and statistics. There is nothing in the current paper that suggests that there is any explainable variance in the fMRI data.\n\nEven though the paper does not show that the fMRI data contain any information, assuming that there is some degree of correlation between the fMRI data and the stimuli, and after all the undisclosed maneuvering, the results indicate that the models simply cannot capture the fMRI data. Take Fig. 1b. The uncorrected correlation coefficient hovers around 0. After more maneuvering and correction, the authors manage to get to 0.01. In other words, there is no correlation between the model outputs and the fMRI data. \n\nFigure 3 is more challenging to interpret because the authors stop reporting r values and report normalized r values. This is a poor practice. For example, it could be that r_{movie > movie}=0.03 and r_{storie>movie}=0.01 and the ratio is 0.33 which might confuse the reader into thinking that there is something there. But the bottom line is that all of the r values are negligibly small.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and making these suggestions. We have addressed your concerns in order below:\n\n>It would be great to start by demonstrating that the fMRI data actually relates to the visual and story stimuli…\n\nWe analyzed a publicly released dataset that has been used in previous studies relating fMRI responses to naturalistic stories and movies [1, 2, 3]. Furthermore, a large body of literature in the past two decades has demonstrated that fMRI responses to naturalistic stories and movies similar or identical to those used here contain meaningful information [2, 16, 17, 18], so a complete description of the fMRI dataset is out of the scope of this paper. In the final revision, we will emphasize that the fMRI data has been rigorously studied and modeled in prior peer-reviewed work.\n\nThe fMRI response for a cortical voxel is given by the time-course of the blood-oxygen level dependent (BOLD) signal. The information in the fMRI response is contained not in the absolute value of the BOLD signal, but in its time-course. The objective of an encoding model is to predict the response time-course from stimulus features [19]. As described in Section 4.1, encoding models are evaluated by predicting brain responses to novel stimuli and computing the linear correlation between the predicted and the actual response time-courses. If we can successfully predict responses in a voxel from stimulus features, then our model has successfully explained some variance in the fMRI data.\n\nWe report prediction performance in Figures 2, 3, and 5. Figure 2a shows that many voxels have cross-modality correlations greater than 0.1. Figure 3b shows that cross-modality correlations can reach 0.5 for $story→movie$ and 0.2 for $movie→story$, while within-modality correlations can reach 0.6 for $movie→movie$ and 0.4 for $story→story$. This demonstrates that there are many cortical regions where the encoding models explain a substantial fraction of the data variance.\n\n>Even though the paper does not show that the fMRI data contain any information…\n\nFigure 2b quantifies encoding model performance by taking the average correlation across all cortical voxels. As described in our reply to Reviewer C9Qg, averaging across cortex is an unbiased way to compare different encoding models. However, averaging across cortex does yield conservatively low correlation values, since there are many voxels (e.g. those in motor cortex) which are not involved in story or movie perception. Figure 2a shows that many regions involved in story or movie perception have correlations greater than 0.1, and Figure 3b shows that these correlations can reach up to 0.5 for $story→movie$ transfer, and up to 0.2 for $movie→story$ transfer. In the final paper, we will emphasize that the average correlation across cortex reported in Figure 2b is a conservatively low metric.\n\n>Figure 3 is more challenging to interpret because the authors stop reporting r values and report normalized r values…\n\nPlease note that Figure 3a only shows $r_{source→target}/r_{target→target}$ for voxels with statistically significant (see Appendix A.1) values of $r_{target→target}$, so the ratios will not be inflated by negligibly small denominators. Furthermore, $r_{source→target}$ is directly compared to $r_{target→target}$ in the Figure 3b histograms, which show that many regions have high cross-modality and within-modality performance. Finally, Figure 3 is not meant to be a normalization of the results in Figure 2, but a way to compare cross-modality performance to within-modality performance. Many previous studies have demonstrated that language and vision encoding models achieve good within-modality performance (line 20), so within-modality performance provides a meaningful ceiling for cross-modality performance [4].\n\nIn the final paper, we will emphasize that previous language and vision encoding models achieve good within-modality performance (line 20).\n\n>This paper would need an enormous amount of work to bring it to a level where it has a minimal scientific description of findings…\n\nOur study builds upon a commonly used framework for modeling brain responses to naturalistic stimuli using features extracted from neural networks [20]. There is little debate that fMRI signals relate to language and visual stimuli, as demonstrated by studies that read information about the stimulus out of fMRI data [10, 17, 18, 21, 22], even using the same dataset as that used here ([23]: “Decoding the semantic content…”). There is also little debate that neural network encoding models can successfully predict brain responses to a single stimulus modality after being trained on responses to that stimulus modality [11, 12, 13, 24, 25]. In the final paper, we will emphasize how our study builds on previous studies. We will also describe the fMRI dataset and the encoding model framework in more detail in the Appendices.\n\nThe goal of our paper was to compare cross-modality performance to within-modality performance. We believe that our results convincingly demonstrate this, and that our analysis framework and the conclusions drawn from our results are comparable to previous fMRI modeling papers published in NeurIPS [11, 12].\n\n>Other points. Figure 4 includes enigmatic color scales that go from “-“ to “+”. Color scales should have numbers.\n\nFigure 4 shows the projection of encoding model weights onto principal components. We used “-” and “+” labels instead of numbers because the numerical scale of this projection is not meaningful."}, {"review_id": "SRiwJAquFD", "reviewer": "Reviewer_Ebp3", "summary": "There is a fastly growing literature analyzing how deep learning models' representations have predictive power over fMRI brain measurements. These papers typically train a regressor model (typically a linear regressor) to predict the fMRI measurements from the neural models' representations. Current encoding models are typically trained and tested on fMRI brain activity to each modality (i.e. language or visual or auditory)  in isolation. They then evaluate predictive power by measuring the correlation between predictions and actual measurements.\n\nThis paper contributes to that literature by using representations from the multimodal Transformer (BridgeTower) model that can extract aligned representations of concepts in language and vision before using it in the above pipeline. Since language and vision rely on similar concept representations, authors investigated how well the multimodal Transformer representations can transfer across fMRI responses to stories and movies. By comparing the correlation between actual fMRI and predictive fMRI from multimodal and unimodal representations, the authors draw conclusions that multimodal transformers learn more aligned representations of concepts in language and vision. Finally, the authors perform a cross-modal experiment in which how well the language encoding models can predict movie-fMRI responses from narrative story features (story → movie) and how well the vision encoding models can predict narrative story-fMRI responses from movie features (movie → story). ", "questions": "* Did the authors fine-tune the BridgeTown multimodal Transformer on the current dataset?\n* Is there any effect on encoding model performance in which adding noise to one modality features and predicting the other modality fMRI response?\n* What if we initialize the model with random weights to perform neural encoding?\n* It could be interesting, if authors can test encoding performance for both single-stream and dual-stream multimodal Transformer models?", "limitations": "The authors have discussed several limitations and made future directions for the research community.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper contains the following key contributions:\n* The novelty of this work: using a multimodal Transformer as the encoder architecture to extract the aligned concept representations for narrative stories and movies to model fMRI responses to naturalistic stories and movies, respectively.\n* Encoding models trained on brain responses to one modality could accurately predict brain responses to the other modality.\n* Cross-modality performance was higher for features extracted from multimodal transformers than for linearly aligned features extracted from unimodal transformers.\n\nOriginality:\n* The idea of how concepts in language and vision are aligned in the brain and demonstrated that alignment using multimodal transformers representations.\n* Mapping story features -> movie fMRI and movie -> story fMRI is interesting.\n\nClarity: The paper is written well. The information provided in the submission is sufficient to reproduce the results.\n\nSignificance: The idea of using a multimodal Transformer encoder model to extract aligned language-visual concept representations to predict fMRI responses is exciting. The paper has novel cognitive insights about language and visual ROIs; clear experimental evaluation, and adequate related work. \n\nQuality: The paper supports its claims with enough details. The paper is well-written and easy to follow. ", "weaknesses": "* Since the narrative stories used in the story experiment are completely different from short movie clips, how much is the overlap percentage of concepts in both modalities?\n* This problem is more interesting if authors can use the same information for both listening and watching a movie (i.e. the same subject is watching a movie and listening to the same story). Similar to the following work: The same subject was reading and listening same narrative story.  Deniz F., Nunez-Elizalde A.O., Huth A. G., Gallant J. L., The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality . Journal of Neuroscience\n* It could be interesting if authors could use captions for the videos and compare the performance of story-movie and captions-movie. Similarly, stories-captions and captions-stories.\n* Which regions have a higher estimated noise ceiling in listening to and watching a movie on the same subject?\n* What is the estimated noise ceiling when using the subject's story voxel to predict the subject movie voxels?\n* The authors missed some recent works that studied multimodal Transformers for brain encoding.\nAuthors missed the following papers:\nINTERPRETING MULTIMODAL VIDEO TRANSFORMERS USING BRAIN RECORDINGS, Dota et al. 2023, https://openreview.net/pdf?id=p-vL3rmYoqh\nVisio-Linguistic Brain Encoding, Oota et al. 2022, https://aclanthology.org/2022.coling-1.11.pdf\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and making these suggestions. We have addressed your concerns in order below:\n\n>Since the narrative stories used in the story experiment are completely different from short movie clips…\n\nThank you for raising this point. We will describe the concepts contained in the story and movie stimuli in the Appendix of the final paper.\n\nTo approximate the overlap in concepts, we clustered GloVe vectors into 100 semantic categories. We then created 100-dimensional story and movie category vectors. The story category vector indicates how frequently each of the 100 categories occurs in the stories. The movie category vector indicates how frequently each of the 100 categories occurs in the movies, based on WordNet labels assigned to each movie frame [1]. The linear correlation between the story and movie category vectors was 0.5, indicating a reasonable degree of overlap. \n\nWe plotted the category vectors in Figure R2, which will be included in the Appendix of the final paper. Categories like location proper nouns (e.g. “Brooklyn”, “Ithaca”) occurred more in the stories. Categories like nature words (e.g. “bloom”, “soil”) occurred more in the movies. To plot the results on a log scale, we filtered out categories which did not occur in at least one of the stimulus modalities. However, such categories typically occurred infrequently in the other modality as well.\n\n>This problem is more interesting if authors can use the same information for both listening and watching a movie…\n\nWe agree that using stimuli that are more closely matched in terms of semantics and timescales is an important direction for future work (line 217). We will emphasize this future direction in the discussion of the final paper.\n\n>It could be interesting if authors could use captions for the videos and compare the performance of story-movie and captions-movie…\n\nSince the movies were constructed by concatenating a sequence of 10-20 s video clips, there are unfortunately no high-quality captions readily available for this comparison. However, we agree that this is an interesting direction for future work. Previous studies have found that encoding models trained on caption features could be used to predict brain responses to images, albeit not as accurately as encoding models trained on image features (see [5]).\n\n>Which regions have a higher estimated noise ceiling in listening to and watching a movie…\n\nWe evaluated our encoding models on single-trial responses, rather than responses to multiple repeats of the same stimulus. While evaluating on single-trial responses to a large number of stimuli provides high semantic coverage, there is no way to estimate a noise ceiling using single-trial responses.\n\nThe dataset that we used [3] also includes responses to multiple repeats of a test stimulus. However, the dataset only provides the averaged responses across repeats, whereas noise ceiling estimation requires having the responses to each individual repeat.  \n\n>What is the estimated noise ceiling when using the subject's story voxel to predict the subject movie voxels?\n\nThe noise ceiling is a function of the test data rather than the encoding model, so the $source→target$ noise ceiling will be the same as the $target→target$ noise ceiling. As we expect cross-modality performance to be lower than within-modality performance, $r_{target→target}$ provides an approximate lower bound on the noise ceiling for $r_{source→target}$ (Figure 3).\n\n>The authors missed some recent works that studied multimodal Transformers for brain encoding…\n\nThank you for linking to these papers. We agree that these studies are highly relevant, and we will reference them in the final paper.\n\n>Did the authors fine-tune the BridgeTown multimodal Transformer on the current dataset?\n\nNo, we did not. We used BridgeTower models that were pretrained on separate datasets (please refer to Section 2.1 for details).\n\n>Is there any effect on encoding model performance in which adding noise to one modality features and predicting the other modality fMRI response?\n\nTo test this, we added Gaussian random noise to the stimulus features. For $story→movie$ transfer, prediction performance was robust to noise in the $story$ features. For $movie→story$ transfer, prediction performance slightly increased when noise was added to the movie features, which may result from the noise affecting negative as well as positive correlations. These results are shown in Figure R3.\n\n>What if we initialize the model with random weights to perform neural encoding?\n\nPrevious studies have found that within-modality performance for randomly initialized neural networks is above chance, but below that of trained neural networks [13]. After fitting alignment matrices, we expect cross-modality performance to be above chance, but below that of trained unimodal neural networks (Section 5.4) and trained multimodal neural networks. We are currently waiting on results of this experiment, which we will include in the final paper. That said, we believe that the trained unimodal neural networks provide the most direct control for evaluating the effects of multimodal training.\n\n>It could be interesting, if authors can test encoding performance for both single-stream and dual-stream multimodal Transformer models?\n\nTo test this, we performed many of the same analyses using KD-VLP [26] which is a single-stream multimodal transformer. These results are shown in Figure R4. We found very similar patterns of transfer performance across cortex. Across voxels and subjects, KD-VLP performance ($r_{story→movie} = 0.17$, $r_{movie→story} = 0.08$) was slightly worse than BridgeTower performance ($r_{story→movie} = 0.23$, $r_{movie→story} = 0.10$). However, these differences could be due to training data and training tasks, as well as model architectures. We will include this analysis in the Appendix of the final paper."}]

Author rebuttal:
We thank the reviewers for taking the time to carefully read our paper, and for providing detailed feedback. We are excited to see that many reviewers agree that our approach is novel and yields interesting insights, and we sincerely appreciate the concerns and suggestions raised by all reviewers. In response, we have addressed these points in the subsequent sections for each individual reviewer, and will incorporate changes accordingly in the final paper. We hope these will serve to amplify the clarity and robustness of our research.

We have attached a list of figures, which we reference in the rebuttals as R1, R2, R3, and R4.

We reference the following papers in the rebuttals:

[1] Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012).

[2] Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).

[3] Popham, S. F. et al. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nat. Neurosci. 24, 1628–1636 (2021).

[4] Deniz, F., Nunez-Elizalde, A. O., Huth, A. G. & Gallant, J. L. The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality. J. Neurosci. 39, 7722–7736 (2019).

[5] Wang, A. Y., Kay, K., Naselaris, T., Tarr, M. J. & Wehbe, L. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. bioRxiv 2022.09.27.508760 (2022) doi:10.1101/2022.09.27.508760.

[6] Wehbe, L. et al. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS One 9, e112575 (2014).

[7] Shen, G., Horikawa, T., Majima, K. & Kamitani, Y. Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15, e1006633 (2019).

[8] Allen, E. J. et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci. 25, 116–126 (2022).

[9] Braga, R. M. & Buckner, R. L. Parallel Interdigitated Distributed Networks within the Individual Estimated by Intrinsic Functional Connectivity. Neuron 95, 457–471.e5 (2017).

[10] Mitchell, T. M. et al. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195 (2008).

[11] Toneva, M. & Wehbe, L. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). in Advances in Neural Information Processing Systems (2019).

[12] Jain, S. & Huth, A. Incorporating Context into Language Encoding Models for fMRI. in Advances in Neural Information Processing Systems (2018).

[13] Schrimpf, M. et al. The neural architecture of language: Integrative modeling converges on predictive processing. Proc. Natl. Acad. Sci. U. S. A. 118, e2105646118 (2021).

[14] Caucheteux, C. & King, J.-R. Brains and algorithms partially converge in natural language processing. Commun. Biol. 5, 134 (2022).

[15] Lin, S., Sprague, T., & Singh, A. K. Mind reader: Reconstructing complex images from brain activities. in Advances in Neural Information Processing Systems (2022).

[16] Hasson, U., Malach, R. & Heeger, D. J. Reliability of cortical activity during natural stimulation. Trends Cogn. Sci. 14, 40–48 (2010).

[17] Nishimoto, S. et al. Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1641–1646 (2011).

[18] Tang, J., LeBel, A., Jain, S. & Huth, A. G. Semantic reconstruction of continuous language from non-invasive brain recordings. Nat. Neurosci. 26, 858–866 (2023).

[19] Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI. Neuroimage 56, 400–410 (2011).

[20] Jain, S., Vo, V. A., Wehbe, L. & Huth, A. G. Computational language modeling and the promise of in silico experimentation. Neurobiol. Lang. (Camb.) doi:10.1162/nol_a_00101/114613.

[21] Haxby, J. V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).

[22] Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–355 (2008).

[23] Huth, A. G. et al. Decoding the Semantic Content of Natural Movies from Human Brain Activity. Front. Syst. Neurosci. 10, 81 (2016).

[24] Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl. Acad. Sci. U. S. A. 111, 8619–8624 (2014).

[25] Güçlü, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience 35, 10005–10014 (2015).

[26] Liu, Y. et al. KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation. arXiv [cs.CV] (2021).

Meta-review: The manuscript demonstrate cross-modality decoding in fMRI where a language model predicts visual responses and a vision model predicts language responses. Authors find aligned visual-language representations in many areas of the brain. Reviewers, bar one who had fundamental concerns about fMRI itself, were unanimous in their recommendation that the manuscript makes an important contribution.

Reviewers appreciated the fairly simple method whose results are easy to interpret. While the impact on both neuroscience and ML is fairly limited at present, finding multimodal transformer architectures that are more brain-like could have high impact in the future, as could understanding how multimodal information is processed in the brain. This work is a prerequisite for both of these directions.

Reviewers repeatedly asked for a thorough discussion about limitations. I would have liked to read this before acceptance, but this year manuscripts could not be updated and unfortunately reviewers were very limited in the response they could provide to authors. Authors promised such a section and I encourage them to follow through, as the reviewers mentioned multiple possible limitations which are necessary for contextualizing this work and for enabling followup research.
Decision: Accept (poster)