Video decoding from human fMRI data with a multi-stream sensory approach

Authors: matteo  ferrante
matteo  ciferri
nicola  toschi
marco  fumero
clementine  domine
zorah  lähner
donato  crisostomi
luca  moschella
kimberly  stachenfeld
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-20 10:57:20

Abstract: In this study, we present a novel multi-stream sensory approach for decoding video stimuli from human fMRI data. Leveraging a dataset of 1,000 short video clips and associated fMRI data, we explore the integration of visual, textual, and audio modalities to enhance the accuracy of brain decoding models. We develop subject-specific encoding models that predict brain activity from modality-specific embeddings and apply functional alignment across subjects to improve model generalization. Our decoding framework employs Ridge regression within identified regions of interest ) for each modality, followed by a retrieval process based on Euclidean search. The results demonstrate that integrating multiple sensory streams significantly enhances the performance of decoding models, with the combined Video+Text+Audio modality achieving the highest identification and retrieval accuracy.

Main paper: 
 introduction: 
 Vision is one of the primary modalities through which we interpret the external world, involving complex dynamics related to movement, object recognition, tracking, and multisensory integration. Understanding how the brain processes this information is a heavily researched yet still not fully understood area. Recent advancements in brain encoding and decoding using non-invasive techniques like EEG, MEG, and fMRI have led to the development of computational models that map external stimuli to brain representations and vice versa, and the availability of large public fMRI datasets and multimodal foundation models has facilitated these advancements [18] . Typically, encoding involves using a pretrained model to generate modality-specific embeddings , which are then projected onto brain activity via linear mapping techniques. For decoding, linear or non-linear models project brain activity measured in concomitance with the stimuli into an embedding space representing the latter. These embeddings can be used for tasks like retrieval-identifying the stimulus linked to a specific brain activity pattern -or reconstruction of the stimuli themselves using generative models. Recent literature has shown significant progress in fMRI-based image decoding [21, 3, 10, 12, 23, 19] , Figure 1 : Decoding pipeline that translates brain activity into modality-specific embeddings to retrieve the corresponding video stimuli. Brain activity data is processed through modality-specific ROIs for text, video, and audio. Decoding models, trained with Ridge regression, estimate embeddings for each modality (z t , z v , z a ). These embeddings are then combined in a retrieval module to identify and output the candidate videos that best match the estimated embeddings, effectively decoding the original video stimulus.particularly for image retrieval and reconstruction. Most methods are variations of the following concept: the brain creates a representation of external stimuli, while we can obtain external stimulus representation (i.e. image embeddings) using a computational model, then learning a mapping between these two representations completes the decoding pipeline. These methods differ e.g. in the generative models used, in how the models are conditioned, and in the techniques used to compute the mapping between brain activity and embeddings-ranging from linear layers to neural networks. However, they all revolve around the central idea that the brain computes something analogous to image embeddings (ideally a representation in a manifold homomorphic the the model's one) which can be captured via fMRI measurements, and that with sufficient data, a mapping between brain activity and computational model-derived embeddings can effectively link brain activity to external stimuli. This concept also underpins language encoding and decoding [1, 24, 15] , where large language model embeddings serve as surrogates to approximate, through linear layers, the language processing which occurs in the brain during listening and comprehension. Similar approaches have shown promising results in decoding music from brain activity [8, 13] . The closest related work in this domain is represented by [4, 22] , which directly addresses the problem of video reconstruction from fMRI data using a different dataset from the on employed here [26] . This latter dataset comprises 18 training videos (each 8 minutes long) and 5 test videos of the same length, collected from 3 subjects. The approach in [4, 22] is based on subject-specific semantic mapping between fMRI data and Contrastive Language Image Pretraining (CLIP) embeddings, along with attention based modules to condition generative models for generating temporally coherent images to reconstruct videos. In this work, we approach video decoding from a different perspective. We use a rich dataset [6] of 1000 short video stimuli and concomitant fMRI data to build cross-subject models [11] for decoding through video retrieval from fMRI data. We hypothesize that video processing in the brain can be decomposed into three distinct streams: a visual stream (recognizing shapes, patterns, and objects in images), a semantic stream (understanding what is happening in the video), and an audio stream, which provides multisensory integration that aids in video comprehension. Based on this hypothesis, our pipeline consists of three main components: First, we construct subject-specific encoding models that predict brain activity from modality-specific embeddings (video, text, and audio) to identify responsive brain regions (regions of interest, ROIs) for each modality. Next, we perform functional alignment [14] to create a robust training and testing set across subjects. Finally, we develop a set of modality-specific decoding models that estimate embeddings from brain activity, which can be used for video retrieval. We demonstrate how multistream integration enhances decoding performance and provide examples at https://mind2music.my.canva.site/decoding-video-nips-sito.  
material and methods: 
 We analyzed a public available fMRI dataset acquired while ten subjects watched a video, originally released as part of the Algonauts 2021 challenge [6, 16] . In the main experiment, participants viewed 1,000 training videos and 102 testing videos multiple times, all presented without audio. MRI data were collected using a 3T Siemens Trio scanner, and preprocessing included standard fMRI procedures such as slice time correction and normalization. The data were part of the Algonauts 2021 challenge, focusing on 1,000 fMRI-video pairs. For more detailed information about the data collection and preprocessing, please refer to the appendix and the original article [6] . Our primary objective was to identify brain regions responsive to identify potentially distinct brain regions responsive to each specific modality. We extracted captions from videos using a video captioning model [25] , along with video frames and audio. These stimuli representations (semantic, visual, and audio) were processed using pretrained computational models to obtain modality-specific embeddings. Transformer-based models, such as XCLIP [17, 20] , were used to extract video and text embeddings, while CLAP [9] was employed to obtain audio embeddings. We then modeled the mapping between brain activity and embeddings using Ridge regression. For each modality, the encoding models took as input embeddings with a dimensionality of 512 and projected them onto estimated brain activity, which had a dimensionality corresponding to the number of voxels (ranging from 10,836 to 21,573, depending on the subject). All models were subject-specific and trained using nested 5-fold cross-validation to prevent any potential circularity with the decoding models. The inner loop was used for hyperparameter optimization, while the outer loop predicted held-out data from the training set. Once the full training set was predicted, Pearson correlation was computed along the samples dimension, resulting in a voxel-wise map of correlations between predicted and actual brain activity for each modality. A threshold of 0.15, determined empirically, was then applied to create a ROI for each modality, which was used in subsequent analyses.We found "activations" in visual, auditory, language, and multimodal integration brain areas corresponds to video, audio, and text inputs, indicating distributed processing across both unimodal and multimodal regions. To account for variability in brain structure and function across individuals, we used a modality-specific functional alignment strategy within a 5-fold nested cross-validation framework. This approach aligns brain activity patterns across subjects for each modality (semantic, visual, and audio), maximizing the similarity of functional responses while retaining modality-specific information. Following recent literature [11, 7, 2] , we employed ridge regression as a regularization technique to improve robustness and generalizability by addressing multicollinearity in the high-dimensional fMRI data. This alignment method aimed to enhance the performance of our decoding models across multiple subjects. The above procedures resulted, for each modality, in a set of 9,000 training video-fMRI pairs (900 pairs per subject) and 1,000 testing pairs (100 pairs per subject). We then developed modality-specific decoding models by training cross-validated Ridge regression models within each identified ROI. These models were used to estimate the modality embeddings from brain activity (e.g., estimating text embeddings from regions responsive to text stimuli, and similarly for visual and audio modalities). For video retrieval, we employed a Euclidean search strategy, selecting the top-N closest test videos based on the L2 distance between the estimated and true embeddings for each modality. Additionally, we implemented a modality integration-based search by concatenating embeddings from different modalities, thereby allowing for a more comprehensive retrieval process that leverages the combined information from multiple sensory streams. As evaluation we report three metric useful to identity quality of retrieval and decoding. The first two are Top-1 and Top-5 accuracy, which simply count how many times the first retrieved videos is exactly the stimulus or when the the stimulus is correctly retrieved among the first 5 five retrieved videos. To complete the analysis and evaluate the quality of the decoded embeddings, we also report the identification accuracy, originally defined in [23] . This metric is a pairwise measure based on correlation, where a value of 0.5 indicates random predictions and a value of 1 signifies perfect predictions. The identification accuracy counts the number of times the estimated embeddings correlate more strongly with the true embedding than with other embeddings. This metric provides a robust evaluation of the quality of the estimated embeddings, which is crucial for more complex tasks such as generation. 
results: 
 The results, summarized in Fig. 3 and Tables 2, 1 , suggest that multimodal integration significantly enhances decoding and retrieval performance. On average, the Video+Text+Audio combination achieved the highest identification accuracy, around 0.94, consistently outperforming other combinations. Video+Text and Video+Audio combinations also performed well, with average accuracies of 0.92 and 0.91, respectively. All combinations performed above chance level, highlighting the effectiveness of multimodal integration. In terms of video retrieval, the Video+Text+Audio combination again provided the highest average Top-1 and Top-5 accuracies, confirming that combining multiple modalities provides best results for decoding and retrieval tasks. 
discussion and conclusions: 
 In this study, we explored the advantages of a multi-stream approach for decoding brain activity, focusing on integrating Video, Text, and Audio modalities. Combining multiple modalities enhances decoding and retrieval performance compared to using individual modalities. The Video+Text+Audio combination consistently outperformed other combinations, indicating that this multi-stream integration captures more comprehensive and complementary information.To implement this approach, we leveraged state-of-the-art multimodal models like XCLIP and CLAP. Growing evidence suggests that multimodal models more accurately capture brain patterns than unimodal models, due to their ability to process richer and more informative embeddings [5, 18, 1] . These embeddings reflect the multimodal sensory integration mechanisms that occur in the brain, aligning with the brain's tendency to synthesize data from various senses to build cohesive representations. This integration is particularly advantageous in neuroimaging contexts, where understanding brain patterns relies on diverse information from multiple sensory domains.Interestingly, the audio modality, despite the fact that videos were presented without sound, still performed above chance level in both identification and retrieval tasks. This suggests that the brain may integrate audio-visual information even in the absence of one modality, aligning with findings from hearing and optical illusions where the brain uses available sensory data to construct a complete perceptual experience. This observation underlines the complex and interconnected nature of sensory processing in the brain, where information from one modality can influence the processing of another, even when it is not directly presented. This study is not without limitations. The reliance on pre-trained models for generating embeddings may not fully capture the specific neural representations relevant to each participant, potentially introducing model biases. As these models are trained on vast, diverse datasets, their internal representations may inadvertently reflect biases or inaccuracies that could misinterpret individual brain activity. This issue emphasizes the need to consider the ethical implications of decoding models, particularly when dealing with sensitive or personal neural data. Implementing safeguards and ethical guidelines, including transparent and interpretable models, can mitigate the risks of misinterpretation or misuse of brain decoding technology.Looking ahead, a promising direction for future work involves the development of generative models that can not only decode brain activity into embeddings but also reconstruct the original stimuli. One potential pathway could involve a two-step approach: first, a multimodal encoding model could be developed, as we have done in this study. Then, building on prior work [10] , brain activity could be decoded into a textual representation of the videos, which could be used alongside a generative model in combination with a Bayesian approach to generate videos that match the brain activity measured. This advancement would enable a more direct and intuitive understanding of how the brain encodes and processes complex sensory information, providing deeper insights into the neural mechanisms of perception.As we advance in the field of brain decoding, the creation of generative models that reconstruct stimuli from brain data could bring transformative applications across various fields, but also underscores the importance of ethical considerations. Ensuring neural privacy and responsible use of decoding models is paramount, especially as this technology continues to mature. 
 
 Appendix: The study involved ten participants, each undergoing five scanning sessions. The first session was a localizer experiment where participants passively viewed short videos (unrelated to the main experiment) to define visual ROIs. The remaining four sessions constituted the main experiment, where participants focused on a fixation cross while viewing 3-second training and testing videos without audio. Videos were presented in 13 runs per session, each lasting about 7 minutes. By the end, participants had viewed 1,000 training videos three times and another set of 102 videos ten times. Using data from the localizer experiment, nine non-overlapping ROIs were defined for each participant, covering regions from early visual cortex (V1, V2, V3, V4) to higher-level areas responding to objects and categories (EBA, FFA, STS, LOC, PPA). MRI data were collected on a 3T Siemens Trio scanner with consistent acquisition parameters across all sessions (TR = 1750 ms, resolution = 2.5 mm³, 54 slices, multi-band factor = 2). Preprocessing was done using fMRIprep, including slice time correction, realignment, co-registration, and normalization to MNI space. Data were interpolated from TR = 1750 ms to 1000 ms using the pchip method. FIR basis functions modeled the BOLD signal for each voxel, extracting beta values from 5 to 9 seconds post-video onset. These were averaged across time and used in subsequent analyses. Our study focused on data from the Algonauts 2021 challenge, specifically 1,000 fMRI-video pairs, with the first 900 used for training and the last 100 for testing.  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Innovative Multi-Modal Approach to Video Decoding from fMRI Data: A Comprehensive Evaluation
main_review: main_review: The manuscript presents a high-quality and thorough methodology for decoding video stimuli from human fMRI data using a novel multi-stream sensory approach. The authors effectively utilize a substantial dataset consisting of 1,000 short video clips paired with corresponding fMRI data collected from ten subjects. Their approach incorporates advanced computational techniques, including Ridge regression, which serves as a regularization method to mitigate multicollinearity issues that are often prevalent in high-dimensional neuroimaging data. The study notably employs functional alignment methods across subjects, allowing for the normalization of brain activity patterns to maximize the robustness of the derived models. By identifying regions of interest (ROI) specific to each modality—video, audio, and text—the authors enhance the predictive accuracy of their subject-specific encoding models. The results convincingly indicate that the integration of multiple sensory modalities significantly improves decoding performance, particularly with the Video+Text+Audio combination achieving a top identification accuracy of approximately 0.94, outpacing other combinations such as Video+Text and Video+Audio.

The clarity of the writing is generally strong, with a systematic structure that guides readers from a foundational introduction to complex methodology, results, and conclusions. However, more precise elaboration on the technical aspects of the methods would improve the manuscript's accessibility. For instance, the authors employ state-of-the-art transformer architectures, such as XCLIP for generating embeddings from video and text, and CLAP for audio embeddings. A deeper explanation of how these models are fine-tuned to capture modality-specific features would be beneficial. Additionally, the nested 5-fold cross-validation technique is a robust choice for hyperparameter optimization and validation, yet a detailed discussion on the process of aligning functional responses across subjects—possibly through techniques like non-linear transformations or the use of technique-specific distance metrics—would enhance understanding. 

Overall, while the manuscript is well articulated, addressing these specific technical aspects would ensure that it is engaging to a wider audience, including those less familiar with advanced computational techniques.In terms of originality, this study makes a significant contribution to the field of brain decoding by pioneering the multi-sensory integration approach to enhance decoding performance from fMRI data. The hypothesis positing that video processing can be decomposed into distinct streams—visual, semantic, and auditory—is supported by the findings that demonstrate nuanced performance across different modality combinations. Notably, the manuscript reveals how the auditory modality, despite being absent during video presentation, still provides above-chance levels of identification and retrieval accuracy, suggesting that the brain engages in predictive processing even when certain sensory information is not directly available. Furthermore, while the ethical implications of neural privacy are briefly addressed, a thorough exploration of how biases inherent in pre-trained models, such as those used for generating video, text, and audio embeddings, might skew individual neural representation would strengthen the discussion. It is crucial to consider how algorithmic biases can lead to misinterpretations, especially as decoding technologies continue to evolve, necessitating a framework for ethical use and transparency in the application of these advanced techniques.
title: title: The paper tackles an interesting problem by proposing a multi-modal approach for decoding video stimuli from fMRI data, integrating visual, text, and audio modalities. The framework shows potential, but the experimental setup and evaluation need more precision and depth to validate the robustness of the method. Improvements are needed in figure clarity, label accuracy, metric justification, and experimental design. Additionally, the lack of comparison with existing methods limits the assessment of the approach’s novelty. Addressing these concerns would strengthen the paper’s contribution to the field of brain decoding.
main_review: main_review: The paper addresses an interesting problem by proposing a multi-modal approach to decode video stimuli from fMRI data, integrating visual, text, and audio modalities. The authors attempt to demonstrate the effectiveness of their framework, using Ridge regression and functional alignment to improve identification and retrieval accuracy. While the results are promising, the experimental setup and evaluation require more precision and depth to fully confirm the robustness of the approach. A clearer experimental design and more thorough evaluation metrics would strengthen the claims and provide a more concrete validation of the method. There are some suggestions for possible improvements:

1.	The labels in Figure 1 are too small and not clearly visible, and some x-axis labels are cropped, making the figure difficult to interpret.

2.	The labels in Figures 4, 5, and 6 are incorrect and need to be revised.

3.	The visualization and coloring in Figure 3 are confusing, and some labels are not fully shown. This figure needs better clarity in terms of coloring and label placement.

4.	Figure 2 lacks a colorbar, which is crucial for understanding the range of correlations, and only cortical voxels are shown without discussion of subcortical voxel activities.

5.	The identification accuracy metric may not fully capture model performance. Although the embeddings may correlate more strongly with the true embeddings than others, the overall correlation could still be low, as seen in the top-1 and top-5 accuracies in Figure 3.

6.	The terms Video+Audio+Text are confusing because video inherently contains visual and audio elements. More appropriate terms would be Visual, Text (Semantic), and Audio modalities to clearly distinguish the different streams.

7.	The encoding step requires learning a large number of parameters, yet the paper does not discuss the number of samples used for learning these parameters, which is an important consideration for model robustness.

8.	There is a misplaced ")" in line 8 of the abstract, and a typo exists in line 42 that should be corrected.

9.	The paper does not present significance measures for the accuracy distributions, making it difficult to determine whether there are statistically significant differences in performance when recovering videos using different combinations of modalities (e.g., comparison between the first four categories from the left in Figure 3).

10.	There is no comparison with existing multi-modal brain decoding methods or simpler baselines, making it difficult to assess the true novelty and improvement of the proposed approach. Including comparisons with previous methods would strengthen the claim of novelty and demonstrate the advantage of the current approach.
title: title: Multimodal Decoding of fMRI Data: A Peer Review of Video Decoding from Human fMRI Using a Multi-stream Sensory Approach
main_review: main_review: ## Strengths:

### 1. Innovative Multimodal Approach
The paper presents a novel approach to decoding brain activity by integrating multiple sensory streams—visual, textual, and audio modalities—using fMRI data. This multimodal framework allows for a more comprehensive understanding of how different stimuli are processed by the brain, significantly improving the performance of decoding models. The combination of these modalities (Video+Text+Audio) achieves the highest identification and retrieval accuracy, which is a major advancement in this field.

### 2. Subject-Specific Encoding Models
The study uses subject-specific models to predict brain activity based on modality-specific embeddings, enhancing accuracy and generalizability. The functional alignment technique across subjects further strengthens the robustness of these models, allowing for better cross-subject comparisons and consistency.

### 3. Thorough Evaluation Metrics
The use of Top-1 and Top-5 accuracy, alongside identification accuracy metrics, provides a clear and comprehensive evaluation of the decoding framework. The high performance of the multimodal integration models (Video+Text+Audio) across all metrics demonstrates the effectiveness of the proposed approach.

### 4. Data Collection and Processing
The paper utilizes a well-structured dataset from the Algonauts 2021 challenge, including detailed fMRI data from multiple subjects. Preprocessing steps such as slice time correction, normalization, and functional alignment were well executed, which adds to the credibility of the results.

## Weaknesses:

### 1. Over-reliance on Pretrained Models
The approach heavily depends on pretrained models (XCLIP for video and text embeddings, CLAP for audio), which might not fully capture the neural representations specific to each participant. While this provides a good starting point, it limits the study's ability to dive deeper into individual differences in brain activity and how specific features of stimuli are processed.

### 2. Audio Modality Ambiguity
The paper reports that even though the videos were presented without sound, the audio modality still performed above chance level. While this observation is intriguing, the explanation for this phenomenon remains speculative and lacks sufficient theoretical discussion. A more in-depth exploration of why the audio modality contributes to decoding without sound stimuli would strengthen this part of the paper.

### 3. Generative Model Limitations
The study hints at the potential future application of generative models to reconstruct stimuli from brain activity. However, no significant steps are taken towards implementing or testing such models in the current work. While the focus is on retrieval and identification, the paper could have explored initial steps towards reconstruction, given that this is a key interest in brain decoding research.

### 4. Limited Dataset
Although the dataset of 1,000 videos is substantial, a broader range of stimuli, particularly in terms of diversity in content (beyond short clips), would make the findings more generalizable. Additionally, while the authors use functional alignment across subjects, the dataset still comes from a relatively small number of subjects (10), which may limit the conclusions regarding cross-subject generalizability.

## Suggestions for Improvement:

- **Address the Pretrained Model Limitation**: A deeper exploration of the limitations of using pretrained models for extracting embeddings, as well as potential alternatives or custom approaches, would enhance the study’s depth. Integrating models trained specifically on brain data could yield better alignment with neural representations.

- **Explore the Audio Phenomenon Further**: The unexpected performance of the audio modality without sound input is fascinating. A more detailed investigation into this finding, potentially supported by additional experiments, could add an important dimension to the paper’s contributions.

- **Progress Toward Generative Models**: While retrieval and identification are well-handled, the paper would benefit from an initial exploration of generative models that can reconstruct the original stimuli from brain activity, as mentioned in the conclusion. Even preliminary experiments in this area would provide a roadmap for future work.

- **Expand Dataset and Modality Range**: Incorporating a more diverse set of video clips or additional sensory modalities could further strengthen the findings. This would also offer more insights into how different types of stimuli are processed in the brain, beyond the specific setup used in this study.

## Conclusion:
This paper provides a novel and well-executed contribution to the field of brain decoding by integrating multimodal sensory streams and leveraging subject-specific encoding models. The approach shows strong potential for improving decoding accuracy, particularly with the combination of video, text, and audio modalities. However, there are some limitations, such as the reliance on pretrained models and the unexplained performance of the audio modality without sound. Further work could explore generative models and larger, more diverse datasets to expand on these promising results.

---

The paper is a solid contribution, with a clear path forward for improvements and future research directions.
title: title: While this paper presents a solid multi-stream sensory approach for decoding video stimuli from fMRI data by combining visual, textual, and audio modalities, the conclusion that fusing all three improves performance is somewhat predictable. It is intuitive that more information from additional modalities would improve decoding accuracy.
main_review: main_review: ### Quality
The paper is of relatively high quality in terms of methodology and implementation. It presents a robust pipeline for decoding video stimuli from fMRI data using a multi-modal approach incorporating video, text, and audio embeddings.

### Clarity
The paper is clear and well-structured. The authors provide a thorough explanation of their decoding pipeline, including visual aids that enhance understanding. The step-by-step breakdown of the encoding and retrieval processes is well-presented.

### Originality
While the multi-modal integration of video, text, and audio in fMRI decoding is interesting, it is not a groundbreaking innovation. The conclusion that combining more sensory streams improves decoding performance is intuitive.

### Pros:
I liked two things about this paper:

1. They proposed a novel approach by integrating multiple modalities (video, text, audio) for fMRI-based video decoding. The inclusion of these different sensory streams helps improve decoding accuracy, which is a step forward from single-modality approaches.
2. The cross-subject functional alignment strategy enhances model generalization, which is crucial for building robust brain decoding models across individuals.

### Cons:
I have a few significant concerns as well:

1. They rely on pre-trained models (XCLIP, CLAP) for generating embeddings, which may not fully capture the specific neural representations relevant to individual participants. Instead of using pre-trained models as-is, they can be fine-tuned with fMRI data specific to each individual.  
2. Although the paper mentions the potential of generative models for future work, it doesn't explore them in this study. Given the current trend toward using diffusion models or GANs in fMRI decoding, this could be seen as a missing aspect of the current work.
3. Although the results demonstrate that fusing three modalities improves performance, this finding is somewhat predictable and not novel.