Multi-modal brain encoding models for multi-modal stimuli

Authors: SUBBA REDDY OOTA, Khushbu Pahwa, mounika marreddy, Maneesh Kumar Singh, Manish Gupta, Bapi Raju Surampudi
Primary area: applications to neuroscience & cognitive science
Keywords: brain encoding, fMRI, multi-modal models, multi-modal stimuli, Transformers, videos, speech, language

Abstract:
Despite participants engaging in unimodal stimuli, such as watching images or silent videos, recent work has demonstrated that multi-modal Transformer models can predict visual brain activity impressively well, even with incongruent modality representations. This raises the question of how accurately these multi-modal models can predict brain activity when participants are engaged in multi-modal stimuli. As these models grow increasingly popular, their use in studying neural activity provides insights into how our brains respond to such multi-modal naturalistic stimuli, i.e., where it separates and integrates information across modalities through a hierarchy of early sensory regions to higher cognition (language regions). We investigate this question by using multiple unimodal and two types of multi-modal models—cross-modal and jointly pretrained—to determine which type of models is more relevant to fMRI brain activity when participants are engaged in watching movies (videos with audio). We observe that both types of multi-modal models show improved alignment in several language and visual regions. This study also helps in identifying which brain regions process unimodal versus multi-modal information. We further investigate the contribution of each modality to multi-modal alignment by carefully removing unimodal features one by one from multi-modal representations, and find that there is additional information beyond the unimodal embeddings that is processed in the visual and language regions. Based on this investigation, we find that while for cross-modal models, their brain alignment is partially attributed to the video modality; for jointly pretrained models, it is partially attributed to both the video and audio modalities. These findings serve as strong motivation for the neuro-science community to investigate the interpretability of these models for deepening our understanding of multi-modal information processing in brain.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729798182822

thread_type:
official_review

<thread_object>
creator: reviewer_8uj4

<content>
summary:
The authors aim to address whether there is a difference in brain-alignment based on whether multimodal models had cross-modality pretraining (separate encoders for two modalities) or joint pretraining (shared encoder across modalities). They compare one model of each type to video and speech models and evaluate the prediction in a large-scale open access movie dataset. Using residual analyses, they investigate whether there are multi-modal representations in visual and language regions of the brain.

strengths:
For the most part, the authors use standard and well validated methods to answer their question. I particularly like the approach of evaluating the performance on entire held-out videos and using the residual analysis to investing multi-modal effects above and beyond unimodal effects.

weaknesses:
The authors only evaluate two multi-modal models and a small number of unimodal models. However, the chosen models differ on many factors (e.g. architecture, training data) in addition to the input modality and as a result, it is premature to draw conclusions about semantic representations in the brain that may be attributable to any of these factors. To my mind, there are two ways to mitigate this concern: 1) controlled model training and evaluation so that only one factor varies at a time, or 2) testing many different models of a given class such that even across significant model variations there is a robust effect of modality regardless of particular model factors. I think that this is a serious concern because not  all prior work has found that multi-modal models are more brain aligned. In controlled comparisons between visual models with the same architecture and training data, there was no performance increase as a result of contrastive image-language training (Conwell et al., 2023). These authors suggest that the higher alignment of CLIP relative to unimodal models in other work may be training set size. 

Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2023). What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? (p. 2022.03.28.485868). bioRxiv. https://doi.org/10.1101/2022.03.28.485868

A minor weakness of the paper is that the authors use custom, non-standard acronyms and names for brain regions (e.g., scene visual area, SV, and object visual processing region, OV). It is confusing as a reader, but more critically, it difficult to understand what has been found for particular regions across the field, making the status of the literature more tenuous. I would suggest that the authors adopt standard acronyms throughout (e.g., PPA instead of SV). 

Although the paper overall is fairly clear, section 6.3 and the corresponding figures (4, 9, and 10) are difficult to follow. I welcome clarification because, outside of a few lines in the discussion, I am having a hard time understanding which regions do show a multi-modal effect. Additionally, I think that the authors should emphasize whether they uncover expected unimodal effects in primary sensory cortices. In particular, we would expect that EVC would be predicted by visual models with no additional multi-modal contribution and similarly in AC but for auditory models. I am having trouble determining whether that is the case from the figures, but if it is not the case, it would lend more weight to my major concern about differences between models beyond modality.

questions:
Why did the authors choose to use parametric statistics? To my knowledge non-parametric statistics are more common in NeuroAI to estimate a baseline performance rather than assuming one. 

Are the brains on the bottom in Figure 5 the medial view? It is difficult to see why there are four brains in each box. Outside of labeling, showing the sulci on the inflated brains would help to orient the reader to what is being shown.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730370221570

thread_type:
official_review

<thread_object>
creator: reviewer_cwb1

<content>
summary:
This paper compares different multimodal AI models to human brain responses while participants view audiovisual movies. They compare two different multimodal models, one cross-modal model that learns separate visual/audio embeddings and projects them into a shared representational space, and one jointly pretrained multimodal model, and three unimodal (vision or speech) models. The results show that in most brain regions multimodal training improves encoding model performance of voxel activity, particularly compared to unimodal speech models. The authors do additional residual analysis to understand the unique contribution of multimodal models (over unimodal models) to brain predictivity.

Overall, this is an interesting approach and the paper has many strengths. The link between results and overall conclusions was not always clear. In particular, the small number of highly varied models makes it difficult to draw strong conclusions about parallels between multimodal processing in the models and brains. Finally, there were several clarification/presentation questions.

strengths:
There are many interesting and novel aspects to this paper. First, while there have been extensive encoding model studies on visual or audio stimuli, few have looked at model comparison to multimodal movies. The comparison of audiovisual models to this data is particularly novel. Further the comparison of different types of multimodal models is interesting (though it is difficult to draw strong conclusions about what their comparison tells you about multimodal processing in the brain, see below), and particularly the attempts to quantify what additional explanatory power a multimodal model has over unimodal models. The encoding analyses were also robust, particularly the cross-movie train/test splits.

weaknesses:
The biggest issues stem from the comparison of the performance of a relatively small number of models that differ along many factors. This limitation makes it difficult to attribute any model differences to multimodality or different cross-modal training schemes, as the model architecture and training sets vary from model to model. 

The fMRI dataset uses a small-n, data-rich design, which is good, but given this, it would help to see the results at the individual subjects’ level (in the appendix). On bar plots, it would be nice to plot each of the six subjects as a point overlaid on the average bar (rather than error bars which can obscure differences across the small number of subjects).

The residual analyses and results are somewhat confusing. The residual correlation with unimodal features was 0.56, which is still quite high. Given this, it is not clear that unimodal information was removed from multimodal models. Alternatively, the authors could do the residual analysis on the brain instead of the models (i.e., fit a model with both unimodal and cross modal and predict with just cross modal). Relatedly they could calculate the product measure or unique variance explained by multimodal models above unimodal models from this joint model. 

Overall, the language and visual region responses look largely the same in most analyses. There are some quantitative/statistical differences, but the pattern is extremely similar. The authors should address this.

Perhaps related to the above point, all regions of interest are defined anatomically, but there is a fair amount of subject-wise variability in the anatomy of high-level functional regions, such as language. The authors should address this limitation.

Acronyms and plots were difficult to follow. Would help to spell out throughout vision versus lang regions for example, and clarify the legend in figure 4 (e.g., it was not clear on first read that “-“ indicates “minus”).

Figure 5 is difficult to read and interpret. In terms of clarification, the authors should specify hemispheres/views (it looks like a lateral view on top, medial on bottom, but I’m not certain). The results look extremely noisy and seem to show random patterns, with as much red as blue. Blue are areas the unimodal models perform better? How should that be interpreted? The legend says the colorbar indicates “percentage increase/decrease. Does this mean 0.5% or 50%? If the former, these are very tiny differences, which perhaps explains the above confusion, but I believe makes it difficult to draw any strong conclusions about these results. 

I had questions about two of the conclusions listed in the discussion. I was unsure what the second part of conclusion 2 (“This variance can be partially attributed to the removal of video features alone, rather than auditory features.”) meant. I am also unconvinced of conclusion #4 given the overall similarity between language and visual brain regions described above.

Typos:
-	Line 224: “audo” --> “audio”
-	Line 276: “wmploy” --> “employ”

questions:
It would help to have additional methodological details in some sections:
-	Was cross-subject prediction generated by using all of the predictor subjects voxels, to predict the target subject’s voxel-wise responses?
-	What are the six different modalities in the image-bind modality? I thought it was an audio-visual model (which I would consider two modalities)
-	The layer-wise predictions for models are shown in the appendix, but do the main text figures use all layers or best layer? If the latter, how is this layer selected.
-	Are the whole brain results averaged across all cortical voxels? Or only those with above-threshold cross-subject prediction? Or some other criteria?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703198616

thread_type:
official_review

<thread_object>
creator: reviewer_gd8j

<content>
summary:
- Existing work uses unimodal models to identify language and vision processing pathways in the brain. This paper studies multimodal processing in the brain. Multimodal networks are aligned to the brain, and regions with better alignment are identified as the sites of multimodal processing. To verify that the alignment is actually due to multimodality, unimodal influence is removed from multimodal representations by subtracting out the multimodal target, as predicted by the unimodal input, from the multimodal features.

strengths:
- This work is novel in that it is the first to use fMRI. But other works also take similar approaches to identifying multimodal processing (see weaknesses).
- Findings of the difference between cross-modal and jointly-trained models with respect to regions is novel
- These findings have neuroscientific significance
- Appropriate random baselines are used to give context to alignment numbers is given

weaknesses:
- Relationship with previous work [1] and similar concurrent work [2,3] that uses multimodal models to probe for multimodal processing in the brain should be discussed. [1] studies cross-modal and jointly trained networks and uses a naturalistic multi-modal stimuli.
- The random baseline described in 6.1 is good to make sure that the trained model weights matter. To better get a sense of whether the alignment actually reflects processing in the brain, another good sanity check would be to run a permutation test in which you keep the trained weights, but give the movie stimulus inputs to the model in scrambled order. This would give a floor for the scale of alignments that we see in subsequent results.

## Small things
- Line 276 typo: wmploy -> employ

## References
[1] Subramaniam, V., et al. "Revealing Vision-Language Integration in the Brain with Multimodal Networks." International Conference on Machine Learning. International Conference on Machine Learning (ICML), 2024.

[2] Kewenig, Viktor, et al. "Evidence of Human-Like Visual-Linguistic Integration in Multimodal Large Language Models During Predictive Language Processing." arXiv preprint arXiv:2308.06035 (2023).

[3] Dong, Dota Tianai, and Mariya Toneva. "Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain." arXiv preprint arXiv:2311.07766 (2023).

questions:
- For the procedure described in the Figure 1b caption on removing unimodal influence: why subtract out the unimodal contribution? Why not learn a regression directly from the unimodal contribution itself, i.e., the predictions of $r$?
- Can it be said that the IB-audio and IB-video unimodal representations described on lines 223-224 are not truly unimodal, since they are extracted from a model that was trained with multimodal inputs? Then, they reflect correspondences between language and vision.
- Figure 3 second row, middle two columns: Why does the green bar not have $\wedge *$ for SV? It seems significantly higher than both light green bars. Why does the green bar have $\wedge *$ for MT? It only seems significantly higher than on light green bar.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732107553298

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Responses to Reviewers cWB1 and 8Uj4:

comment:
*We are grateful to all reviewers for their strong positive feedback, time and their constructive suggestions, which will further strengthen the impact of our work.*

**CQ1: Challenges in comparing relatively small number of models (Reviewers: cWB1, 8Uj4)**

Thank you for this interesting question. We would like to provide a clear discussion as follows.

* We appreciate the concern raised by the reviewer about performance comparison across a small cohort of models. 
* It is to be noted that there exist a relatively large number of vision-language or vision-alone or language-alone or speech-alone models as compared to video-audio models that is the primary focus of the current investigation. Therefore the current effort needs to take into consideration the sparsity of video-audio model availability. 
* Despite this limitation, the current paper reports comparison across three video-only, two speech-only, one jointly-pretrained video-audio, and one cross-modal multi-modal model – to the best of our knowledge, this is by far the largest cohort of comparative analysis of multi-modal models and their alignment with brain representations resulting from multi-modal stimuli. 
* Now, let us discuss the state-of-the-art practices in brain alignment experiments in CQ2.

We have added this discussion in **Appendix L** of the revised paper. 

**CQ2. Impact of diverse model architectures on performance comparison (Reviewers: cWB1, 8Uj4)**

Several prior brain encoding studies in the literature compared a variety of language/speech models (differing in their size, architecture, training dataset, etc.) and their brain alignment.

* *[Schrimpf et al. 2021]* investigated 43 language models ranging from distributed word embeddings models like Word2Vec, GloVe, FastText, to Sequence models such as RNN, LSTM, Contextualized models like ELMo, Transformer models like BERT, GPT-2 and Transformer-XL with its variations such as base, small, and larger models. Although all these models have different architectures, training datasets, *[Schrimpf et al. 2021]* considered each model as a subject and computed normalized brain predictivity, i.e., what percentage the model explains the variance given a ceiling value for each voxel.
* Similarly, *[Toneva et al. 2019]* used four different language models such as ELMo, BERT, USE and Transformer-XL and compared the explained variance of each model while doing brain alignment. 
* Further, *[Aw et al. 2023]* use four longer-context language models such as BART, Big-Bird, Longformer and Long-T5 and verify the deeper understanding of language models and brain alignment by the amount of variance explained by each model.
* Similarly, *[Antonello et al. 2022]* used 101 language models including both pretrained and task-based language models and compared the amount of explained variance in the brain by extracting the semantic representations and whether these representations are closer to brain-level semantics. 
* Recently, *[Oota et al. 2023] [Oota et al. 2024]* used four text-based language models such as BERT, GPT-2, FLAN-T5 and BART, speech-based language models such as Wav2Vec2.0 and Whisper and verified the amount of explained variance in the brain at different language regions.

It is important to observe that all the above studies utilize a number of language models that are different in training architecture and training datasets, however the primary goal of all these studies is to investigate how close the semantic representations captured by each model aligns with brain-relevant semantics.

The extensive precedent in the literature, from studies comparing 43 models *[Schrimpf et al. 2021]* to those examining 101 models *[Antonello et al. 2022]*, demonstrates that this approach is both valid and valuable for understanding the relationship between artificial and biological language processing.

We have added this discussion in **Appendix M** of the revised paper.

*[Schrimpf et al. 2021], The neural architecture of language: Integrative modeling converges on predictive processing. PNAS 2021*

*[Toneva et al. 2019], Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). NeurIPS 2019*

*[Antonello et al. 2022], Low-dimensional structure in the space of language representations is reflected in brain responses, NeurIPS 2022*

*[Aw et al. 2023], Training language models to summarize narratives improves brain alignment, ICLR 2023*

*[Oota et al. 2024], Speech language models lack important brain relevant semantics, ACL 2024*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732119801870

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our responses and revision

comment:
**We are grateful to all reviewers for their strong positive feedback, time and their constructive suggestions, which will further strengthen the impact of our work.**

**Summary of Reviewer Strengths:**

1. Novelty of Approach: This work is the first to use fMRI to investigate differences in cross-modal and jointly-trained multimodal models with respect to brain regions, a novel contribution with neuroscientific significance. **(reviewers: gD8J, cWB1)**
2. Significance of Findings: This study provides novel insights into the differences between cross-modal and jointly-trained models and their explanatory power for multimodal processing in the brain. **(reviewers: gD8J, cWB1)**
3. Well designed experiments: **(reviewers: gD8J, cWB1, 8Uj4)**
   - Authors use standard and well-validated methods were employed effectively, providing confidence in the study's findings.
   - The use of random baselines contextualizes alignment numbers, improving interpretability.
   - Cross-movie train/test splits and residual analyses add robustness to the encoding analyses, offering a more comprehensive evaluation of multimodal effects beyond unimodal contributions.
4. Innovative Comparisons: **(reviewers: gD8J, cWB1)**
	- The comparison between different types of multimodal models (e.g., audiovisual and unimodal) is interesting

**Additional changes to the draft during the rebuttal process**

We have updated the main manuscript and the appendix to address these following comments. The changes made in the manuscript are highlighted in blue color. The major additional changes are listed below.

1. **Extended Related Works** (Reviewer gD8j, 8Uj4):  We discuss how our current study is different from previous studies [Subramaniam et al. (2024)], [Kewenig et al. (2024) ], [Dong et al. (2023)] in the following aspects:
   - Subramaniam et al. (2024) employed vision-language models based on image-text pairs, potentially overlooking the temporal dynamics of continuous movies, and used SEEG, which is limited in spatial resolution and coverage. Their study focused on cross-modal integration without exploring jointly pretrained models, with each participant viewing different movie stimuli, leading to varied inputs across participants.
   - Kewenig et al. (2024) focused on behavioral evidence, demonstrating that multimodal attention models can leverage contextual information to predict upcoming words in a manner aligned with human behavior. Notably, the study did not involve brain recordings; instead, the authors collected behavioral ratings and focused on human attention as indexed through eye-tracking data.
   - Dong et al. (2023) study compared brain alignment performance before and after fine-tuning but did not explore jointly-pretrained mutlimodal models vs. multiple unimodal models, leaving open questions about which multimodal approaches best predict brain activity.
   - Conwell et al. (2022) and Wang et al. (2022): These studies are discussed in the context of their findings that contrastive image-language training does not always lead to performance improvements, particularly in tightly controlled experiments.
We have added results of these experiments in **Appendix J** of the revised paper.

2. **Baseline Analysis: Scrambling Inputs to Multimodal Models** (Reviewer gD8j): 
   - we conducted an additional baseline experiment where we kept the trained weights unchanged and shuffled the movie stimuli into a scrambled order as input to the two multimodal models: cross-modal and jointly-pretrained models.  
   - The results demonstrate that embeddings from multimodal models exhibit significantly better brain alignment compared to both randomly initialized models and when passing scrambled clips as input.

3. **Whole Brain, Language and Visual ROIs analysis: Shared and Unique variance between Multimodal and Unimodal models** (Reviewer cWB1): To empirically verify residual analysis, we now build two voxelwise joint encoding models: (1) that includes both unimodal and cross-modal features, (2)  that includes both unimodal and jointly-pretrained features. 
   - Using these joint encoding models and prior encoding models, we compute the unique variance explained by each model i.e. unimodal and cross-modal, unimodal and jointly-pretrained models. 
   - The results, presented in Appendix N, Figures 12, 13 and 14, for the whole brain and the language network, reveal that the shared variance between the jointly pretrained (TVLT) model and the unimodal models is significantly lower than that observed between the cross-modal (IBConcat) model and the unimodal models.

4. **Updation of Brainmap Plots and Figures in Appendix** (Reviewer cWB1): The revised appendix includes updates to Brainmap plots and figures (Figs. 7, 8, 9 and 10), featuring standard ROI names. Additionally, subject values are now represented as individual points on bar plots instead of using error bars, ensuring clearer visualization of subject-specific data.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper considers the problem of finding alignments between active regions of the brain from fMRI signals when a person watches multimodal content (audiovisual movies) and multimodal features extracted from state-of-the-art deep learning models. Correlation results using the Movie-10 dataset presents many interesting conclusions, especially regarding better alignment of multimodal models against unimodal ones, including better alignment of video models than speech/audio. 

The paper received favorable reviews, with one accept and two borderline accepts. The reviewers appreciated the novelty of finding correlations between fMRI data and multimodal/unimodal features and the extensive experiments.