Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity
Authors: Zijiao Chen
Jiaxin Qing
Juan Helen Zhou
Venue: NeurIPS 2023 oral (2023)
Keywords: Video Reconstruction from Brain Activities
Diffusion Model
Contrastive Learning
TL;DR: We present Mind-Video for reconstructing continuous video from fMRI data via multimodal contrastive learning and inflated Stable Diffusion model.

Abstract: Reconstructing human vision from brain activities has been an appealing task that helps to understand our cognitive process. Even though recent research has seen great success in reconstructing static images from non-invasive brain recordings, work on recovering continuous visual experiences in the form of videos is limited. In this work, we propose Mind-Video that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. 
We show that high-quality videos of arbitrary frame rates can be reconstructed with Mind-Video using adversarial guidance. The recovered videos were evaluated with various semantic and pixel-level metrics. We achieved an average accuracy of 85% in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by 45%. We also show that our model is biologically plausible and interpretable, reflecting established physiological processes.

Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity
Zijiao Chen"
Jiaxin Qing"
National University of Singapore zi zijiao. meenuunnus.co
The Chinese University of Hong Kong jqing0ie. cuhk. edu. hk
Juan Helen Zhou'
National University of Singapore
helen, zwou@nus edu.
https:/mmind-video.ccom
Abstract
Reconstructing human vision from brain activities has been an appealing task that helps understand cognitive process. Even though recent research has seen great success in reconstructing static images from non-invasive brain recordings, work on recovering continuous visual experiences in the form of videos is limited. In this work, we propose MinD-Video that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. We show that high-quality videos of arbitrary frame rates can be reconstructed with HinD-Video using adversarial guidance. The recovered videos were evaluated with various semantic and pixel-level metrics. We achieved an average accuracy of 85% in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by 45%. We also show that our model is biologically plausible and interpretable, reflecting established physiological processes.
Introduction
Life unfolds like a film reel, each moment seamlessly transitioning into the next, forming a "perpetual theater" of experiences. This dynamic narrative forms perception, explored through the naturalistic paradigm, painting the brain as a moviegoer engrossed in the relentless film of experience. Understanding the information hidden within of recreating human vision from brain recordings, especially using non-invasive tools like functional Mag- netic Resonance Imaging (fMRI, is exciting but difficult task. Non-invasive methods, while less intrusive, capture limited information, susceptible to various interferences like noise [I]. Furthermore, the acquisition of neuroimaging data is complex, costly process. Despite these complexities, progress has been made, notably in learning valuable fMRI features with limited IMRI-annotation pairs. Deep learning and represen- and static image reconstruc- tion 5 6 7 8), advancing our understanding of the vibrant, ever-changing spectacle of human perception.
complex brain activities is big puzzle in cognitive neuroscience. The task
tation learning have achieved significant results in visual class detections
Unlike still images, vision is continuous, diverse flow of scenes, motions, and objects. To recover dynamic visual experience, the challenge lies in the nature of fMRI, which measures blood oxygenation level dependent (BOLD) signals and captures snapshots of brain activity every few seconds. Fach fMRI scan
"Equal contributions
i Corresponding author
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Visual Stimuli
Video watching during MRI scan
Brain bacoding
Our Reconstructed Video
Docoding
Figure 1. Brain decoding & video reconstruction. We propose progressive learning approach to recover continuous visual experience from fMRI. High-quality vidcos with accurate semanties and motions are reconstructed.
essentially represents an "average" of brain activity during the snapshot. In contrast, a typical video has about 30 frames per second (FPS). If an fMRI frame takes seconds, during that time, 60 video frames- potentially containing various objects, motions, and scene changes- are presented as visual stimuli. Thus, decoding fMRI and recovering videos at an FPS much higher than the fMRI's s temporal resolution a complex task.
Hemodynamic response (HR) 9 refers to the lags between neuronal events and activation in BOLD signals. When visual stimulus is presented, the recorded BOLD signal will have certain delays with respect to the stimulus event. Moreover, the HR varies across subjects and brain regions [10] Thus, the common practice that shifts the fMRI by a fixed number in time to compensate for the HR would be sub-optimal.
In this work, we present MinD-Vidie a two-module pipeline designed to bridge the gap between image and video brain decoding. Our model progressively leams from brain signals, gaining deeper understanding of the semantic space through multiple stages in the first module. Initially, we leverage large-scale unsupervised leaming with masked brain modeling learn general visual fMRI features. We then distill semantic-related features using the multimodality of the annotated dataset, training the fMRI encoder in the Contrastive Language-Imaggee Pre- rre-rrinnng (CLIP) space with contrastive learning. In the second module, the learned features are fine-tuned through co-training with an augmented stable diffusion model, which is specifically tailored for video generation under fMRI guidance. Our contributions are summarized follows:
We introduced a flexible and adaptable brain decoding pipeline decoupled into two modules: an fMRI encoder and an augmented stable diffusion model, trained separately and finetuned together.
We designed a progressive learning scheme where the encoder learns brain features through multiple stages, including multimodal contrastive learning with spatiotemporal attention for windowed fMRL.
We augmented the stable diffusion model for scene-dynamic video generation with near-frame attention. We also designed adversarial guidance for distinguishable fMRI conditioning.
We recovered high-quality videos with accurate semantics, e.g., motions and scene dynamies. Results are evaluated with semantic and pixel metrics at video and frame levels. An accuracy of 85% is achieved in semantic metrics and 0.19 in SSIM, outperforming the previous state-of-the-arr approuches by 45%.
The attention analysis revealed mapping to the visual cortex and higher cognitive networks, suggesting our model is biologically plausible and interpretable.
2 Background
Image Reconstruction Image reconstruction from fMRI was first explored in (2), which showed that hierar- chical image features and semantic classes could be decoded from the fMRI data collected when the partici- pants were looking static visual stimulus. Authors in [5 6 designed separable autoencoder that enables self-supervised learning in fMRI and images to increase training data. Based on a similar philosophy, [ proposed to perform self-supervised learning on large-scale IMRI dataset using masked data modeling as a pretext task. Using stable diffusion model as a generative prior and the pre-trained IMRI features as condi- tions, 7 reconstructed high-fidelin images with high semantic correspondence the groundtruth stimuli.
Video Reconstruction The conventional method formulated the video reconstruction as multiple image reconstructions [T], leading to low frame rates and frame inconsistency. Nonetheless, [ showed that low-level image features and classes can also be decoded from iMRI collected with video stimulus. Using

---

SC-Aen
(MRI Attention
(AMRI, Video, Caption) Latent Alignment
Video Generation with Stable Diffusion
(MR)
Add Noise Stops
SC-MEM Encoder
IMRI Token Embedding
MMS Tokores
HiMRI
Astivated (MRI Vosels
Imagle Latent
GT Video
Latent
De-noise T Stops U-Net
SMRI Spatial Aftentiinn
Insage
Himage
HHH
*
Video Frames
Contrastive
Caption
Video
CLIP-baned
Encoder
Teat
Mttext
Latent Space
Generated
Adversarial Guidance for IMRI
SC-Ain Cross-Am T-Aiin
Block
INF Temporal Altertans
Chr
Averaged fMR]
Frozen
Figure 2. MinD- Video Overview. Our method has two modules that are trained separately and then finetuned together. The fMRI encoder progressively learns fMRI features through multiple stages, including MBM pre-training and multimodal contrastive learning. spatiotemporal attention is designed to process multiple fMRI in sliding window. The augmented Stable Diffusion trained with videos and then tuned with the IMRI encoder using annotated data.
fMRI representations encoded with linear layer as conditions, [2 generated higher quality and frame rate videos with a conditional video GAN. However, the results are limited by data scarcity, especially for GAN training, which generally requires a large amount of data. [3] took a similar approach as (5), which relied on the same separable autoencoder that enables self-supe learning. Even though better results were achieved than 12), the generated videos were of low visual fidelity and semantic meanings. In addition, recent work by [14] utilizes contrastive learning to derive latent embeddings from mice neural data, enabling the generation of videos based these learned representations.
MBM Masked brain modeling (MBM) is a pre-text task that enables self-supervised learning in a large IMRI dataset proposed in 77. aiming to build a brain foundation model. It learns general features of IMRI by trying to recover masked data from the remainings, similar to the GPT [55 and MAE [16,, after which knowledge can be distilled and transferred to downstream task with limited data and few-step
tuning 17 15 16 Z
CLIP Contrastive Language-Iman Pre- Training (CLIP) a pre-training technique that builds a shared latent space for images and natural languages by large-scale contrastive learning (88). The training aims to minimize the cosine distance of paired image and text latent while maximizing permutations pairs within a batch. The shared latent space (CLIP space) contains rich semantic information on both images and texts.
Stable Diffusion Diffusion models are emerging probabilistic generative models defined by a reversible Markov chain [19 20) As variant, stable diffusion generates compressed version of the data (data latent) instead of generating the data directly. As works in the data latent space, the computational requirement is significantly reduced, and higher-quality images with more details can generated in the latent space (21]
Due to its high generative quality, it has been studied in various brain decoding tasks 7 g
3 Methodology
3.1 Motivation and Overview
Aiming for flexible design, MinD-Video is decoupled into two modules: an IMRI encoder and a video generative model, as illustrated in Fig. [2] These two modules are trained separately and then finetuned together, which allows for easy adaption of new models if better architectures of either one are available. As representation learning model, the encoder in the first module transfers the pre- re-processed fMRI into embeddings, which are used as condition for video generations. For this purpose, the embedding should have the following traits: 1) It should contain rich and compact information about the visual stimulus presented during the scan. 2) should close to the embedding domain with which the generative model is trained. When designing the generative model, it is essential that the model produces not only diverse, high-quality videos with high computational efficiency but also handles potential scene transitions, mirroring the dynamic visual stimuli experienced during scans.
3

---

3.2 The fMRI Pre- Pre-processing
The IMRI captures whole-brain activity with BOLD signals (voxels). Each voxel is assigned to a region of interest (ROI) for focused analysis. Here, we concentrate on voxels activated during visual stimuli. There are two ways to define the ROIs: one uses a pre-defined parcellation such as [22] and [23] to obtain the visual cortex; the other relies on statistical tests identify activated voxels during stimuli. Our large-scale pre-training is based on the parcellation in [22], while statistical tests are employed for the target dataset, i.e., the Wen (2018) [1] dataset containing fMRI-video pairs. To determine activated regions, we calculate intra-subject reproducibilityy of each voxel, correlating fMRI data across multiple viewings. The correlation coefficients are then converted to z-scores and averaged. We compute statistical signifieance using one-sample t-test (P<0.01, DOF=17, Bonferroni conrection). We selected the top 50% of the most significant voxels after the statistical test. Notice that most of the identified voxels are from the visual cortex.
3.3 Progressive Learning and fMRI Encoder
Progressive learning is used an efficient training scheme where general knowledge is learned first, and then more task-specific knowledge is distilled through finetuning [24] 25 26). To generate meaningful embeddings specific for visual decoding, we design a progressive learning pipeline, which learns IMRI features through multiple stages, starting from general features to more specific and semantic-related features. We will show that the progressive learnigg process is reflected biologically in the evolution of fMRI attention maps.
Large-scale Pre-training with MBM Similar to 77.. a large-scale pre-training with masked brain modeling (MBM) is used to learn general features of the visual cortex. With the same setup, an asymmetric vision- branse autoencoder (27 [6] is trained on the Human Connectome Project 28 with the visual cortex (VI to V4) defined by (22 Specifically, fMRI data of the visual cortex is rearranged from 3D into 1D space in the order of visual processing hierarchy, which then divided into patches of the same size. The patches will be transformed into tokens, and a large portion (~75%) of the tokens will be randomly masked in the encoder during training. With the autoencoder architecture, a simple decoder aims to recover the masked tokens from the unmasked token embeddings generated by the encoder. The main idea behind the MBM is that the training objective can be achieved with high accuracy using a simple decoder, the token embeddings generated by the encoder will be a rich and compact description of the original IMRI data. We refer readers to for detailed descriptions and reasonings of the MBM.
Spatiotemporal Attention for Windowed fMRI For the purpose of image reconstruction, the original IMRI encoder in 77 shifts the fMRI data by 6s, which is then averaged every 9 seconds and processed individually. This process only considers the spatial information in its attention layers. In the video reconstruction, if we directly map one fMRI to the video frames presented (e.g.. frames), the video reconstruction task can be formulated as a one-to-one decoding task, where each set of IMRI data corresponds to 6 frames. We call each (IMRI-frames) pair fMRI frame window. However, this direct mapping activity and the associated BOLD signals in fMRI data due to the nature of hemodynamic response. Thus, when a visual stimulus (i.e., a video frame) is presented at time f, the fMRI data obtained at f may not contain complete information about this frame. Namely, a lag occurs between the presented visual stimulus and the underlying information recorded by fMRI. This phenomenon is depicted in Fig. 3
sub-optimal because of the time delay between brain
lag
BOLD signals at
BOLD signals
visual stimulus T
time
The hemodynamic response function (HRF) is usu- ally used to model the relationship between neural activity and BOLD signals 20) an LTI system, the signal y(f is represented as the convolution of a stimulus function s(t) and the HR k(t), i.e., (s*)= The h(t) is often modeled with a linear combination of some basis functions, which can be collated into matrix form: Y=X,++e where Y represents the observed data, ,9 is a vector of regression coefficients, and e is a vector of unexplained emorr values. However, e varies significantly across individuals and sessions due to age, cognitive state, and specific visual stimuli, which influence the firing rate, onset latency, and neuronal activity duration. These variations impact the estimation of and ultimately affect the accuracy of the MMRl-based analysis. Gener-
Figure 3. Due to hemodynamic response, the BOLD signal (blue) lags few seconds behind the visual stimulus (red), causing a discrepancy between IMRI and the stimulus,

---

ally, there are two ways to address individual variations: using personalized HRF models or developing algorithms that adapt to each participant. We choose the latter due to its superior flexibility and robustness.
Aiming to obtain sufficient information to decode each scan window and account for the HR, we propose a spatiotemporal attention layer process multiple fMRI frames in sliding window. Consider sliding window defined as De = [11,22++1""= -. where E RRnxpxb are token embeddings of the IMRI and nt, p, are the batch size, patch size, and embedding dimension, respectively. So we have Fe E @R"xxxxx,, where w is the window size. Recall that the attention is given by attn=softmax vd
[QK")
To calculate spatial attention, we use the network inflation trick [30], where we merge the first two dimensions of ze and obtain 2ppt ERnuxxxb, Then the query and key are calculated in Eq. 1 as
Q=
K K=
W spot
spots
Likewise, we merge the first and the third dimension of ae to calculate the temporal attention, obtaining temp @lmnpxuxb Again, the query and key are calculated in Eq. (2) with
Et
Q= temp W temps
K K=#+ K=mtemp temp, W. temp'
(2)
The spatial attention learns correlations among the fMRI tokens, describing the spatial correlations of the fMRI patches. Then the temporal attention learns the correlations of fMRI from the sliding window, including sufficient information to cover the lag due to HR, as illustrated in the "IMRI Attention" in Fig. 2
Multimodal Contrastive Learning Recall that the fMRI encoder is pre-trained to learn general features of the visual cortex, and then it is augmented with temporal attention heads to process a sliding window of IMRL. In this step, we further train the augmented encoder with (IMRI image, caption] } triplets and pull the IMRI embeddings closer to a shared CLIP space containing rich semantic information. Additionally, the generative model is usually pre- praitrained with text conditioning. Thus, pulling the fMRI embeddings closer to the text-image shared space ensures its understandability by the generative model during conditioning,
Firstly, videos in the training are downsampled to a smaller frame rate (3FPS). Each frame is then captioned with BLIP [31]. which creates the (image, text) pairs. With the CLIP text encoder and image encoder being fixed, the CLIP loss (18) is calculated for fMRI-image and fMRi-text, respectively. Denote the pooled text embedding, image embedding, and fMRI embedding by emb,, emb; emby E RRnx The contrastive language-image-IMER loss is given by
L= (=(Cuup(embb, embi) + CCLI emb,))/2.
where (a b) = Crossrrrttoooooe (ntospprrrpy [0.1,....nll). with being a scaling factor. Extra care is needed to reduce similar frames in a batch for better contrastive pairs. From Eq. (3. we see that the loss largely depends on the batch size n.. Thus, a large n with data augmentation on all modalities appreciated.
3.4 Video Generative Module
The Stable Diffusion model 22 is used as the base generative model considering its excellence in generation quality, computational requirements, and weights availability. However, as the stable diffusion model is an image-generative model, temporal constraints need to be applied in order for video generation.
Geene-Dynamic Sparse Causal (SC) Attention Authors in 30 use a network inflation trick with sparse temporal attention to adapt the stable diffusion to video generative model. Specifically, the sparse temporal attention effectively conditions each frame on its previous frame and the first frame, which ensures frame consistency and also keeps the scene unchanged. However, the human vision consists of possible scene changes, so the video generation should also be scene-dynamic. Thus, we relax the constraint in (0) and condition each frame on its previous two frames, ensuring the video smoothness while allowing scene dynamics. Using notations from [30 the SC attention is calculated with the query, key, and value given by
Q=Wº.2o Zozz
K K-W . 21 20, 1l, V T-W .[2u, 21 Zo, sb
where Ze;, denotes the latent of the i-th frame during the generation.
Adversarial Guidance for fMRI Classifier-fre free guidance is widely used in the conditional sampling of diffusion models for its flexibility and generation diversity, where the noise update function is given by
éa (2, c) =0)=) +s(e0(2z, c) -c((2)))
(5)
5

---

where e is the condition, s is the guidance scale and c0(-) is a score estimator implemented with UNet [2] Interestingly. Eq. (5) can be changed to an adversarial guidance version [2
Co(zt, ?) = (2,, 2) + ((00(2, c) c0(=s, 2));
where ē is the negative guidance. In effect, generated contents can be controlled through "what to generate" (positive guidance) and "what not to generate" (negative guidance). When č null condition, the noise update function falls back to Eq. (5), the regular classifier-free guidance. In order to generate diverse videos for different IMRI, guaranteeing the distinguishability of the inputs, we average all fMRI in the testing set and use the averaged one the negative condition. Specifically, for each fMRI input, the fMRI encoder will generate an unpooled token embedding 2 E R where I is the latent channel number. Denote the averaged fMRI as T. We have the noise update function (0(2). I, co +)+ s(o (2t, x) co-co(2+, Z)).
Divide and Refine With a decoupled structure, two modules are trained separately: the IMRI encoder is trained in large-scale dataset and then tuned in the target dataset with contrastive learning; the generative module is trained with videos and captions from the target dataset. In the second phase, two modules are tuned together with fMRI-video pairs, where the encoder and part of the generative model are trained. Different from (30,, we tune the whole self-attention, cross- atts-ntttn,ion, and comporal-attentio heads instead of only the query projectors, as different modality used for conditioning. The second phase is also the last stage of encoder progressive learning, after which the encoder finally generates token embeddings that contain rich semantic information and are easy to understand by the generative model.
3.5 Learning from the Brain Interpretability
Our objectives extend beyond brain decoding and reconstruction. We also aim to understand the biological principles of the decoding process. To this end, we visualize average attention maps from the first, middle, and last layers of the fMRI encoder across all testing samples. This approach allows us to observe the transition from capturing local relations in early layers recognizing more global, abstract features in deeper layers [33]. Additionally, attention maps are visualized for models in different learning stages: large-scale pre-training contrastive learning, and co-training. By projecting attention back to brain surface maps, we can easily visualize each brain region's contributions and the learning progress through each stage.
4 Experiments
4.1 Datasets
Pre-training dataset Human Connectome Project (HCP) 1200 Subject Release (2 For our upstream
pre-training dataset, we employed resting-state and task-evoked fMRI data from the HCP, Building upon we obtained 600,000 fMRI segments from a substantial amount of fMRI scan data.
77,
Paired EMRI-Vides dataset A publicly available benchmark fMRI-vidco datasct [T] was used, comprising fMRT and video clips. The fMRI were collected using a 3T MRT scanner TR of 2 seconds with three subjects. The training data included 18 segments of 8-minute video clips, totaling 2.4 video hours and yielding 4,320 paired training examples. The test data comprised 5 segments of 8-minute video clips, resulting in 40 minutes of test video and 1,200 test IMRIs The video stimuli were diverse, covering animals, humans, and natural scenery, and featured varying lengths at temporal resolution of 30 IPSS.
4.2
Implementation Details
The original videos are downsampled from 30 FPS to 3 FPS for efficient training and testing. leading to frames per fMRI frame. In our implementation, we reconstruct video of 2 seconds (6 frames) from sliding window of fMRI frames. However, thanks to the spatiotemporal attention head design that encodes multiple IMRI at once, our method can reconstruct longer videos from more IMRI frames if more GPU memory is available.
A baseb fMRI encoder with a patch size of 16, depth of 24, and an embedding dimension of 1024 is used. After pre-training with a mask ratio of 0.75, the encoder will augmented with a projection head that projects the token embedding into the dimension of 77 768. The Stable Diffusion V1-5 (1] trained at the resolution of 512 is used. But we tune the augmented Stable Diffusion for video generations at the resolution of 256 256 with 3 FPS. Notice that the FPS and image are downsampled for efficcintt

---

experiments, and our method can also work with full temporal and spatial resolution. All parameters in the IMRI encoder pre- veetrrining are the same as 7 with eight RTX3090, while other stages are trained with one RTX3090.
The fMRT encoder pre-training on large-scale dataset is the most resource- consuniig part, but only one pre-training is needed for different subjects. This is in contrast to the other downstream stages. Multimodal contrastive learning and co-training are performed for each subject individually due to individual variability. The inference is performed with 200 DDIM M steps. See Supplementary for more details.
4.3 Evaluation Metrics
Following prior studies, we utilize both frame-based and video-based metries. Frame-based metrics evaluate each frame individually, providing a snapshot evaluation, whereas video-based metrics assess sequences of frames, encapsulating the dynamics across frames. Both are used for comprehensive analysis. Unless stated otherwise, all test set videos are used for evaluating the three subjects.
Framte-based Metries Our frame-based metries are divided into two classes, pixel-level metrics and semantics-level metrics. We use the structural similarity index measure (SSIM) [35] as the pixel-level metric and the N-way top-K accuracy classification test as the semantics-level metric. Specifically, for each frame in a scan window, we calculate the SSIM and classification test accuracy with respect to the groundtruth frame. To perform the classification test, we basically compare the classification results of the groundtruth (GT) and the predicted frame (PF) using ImageNet classifier. If the GT clas is within the top-K probability of the PF classification results from randomly picked classes, including the GT class, we declare a successful trial. The test is repeated for 100 times, and the average success rate is reported.
Video-based Metric The video-based metric measures the video semantics using the classification test well, except that video classifier is used. The video classifier based on VideoMAF (36) is trained on Kinetics-400 37 an annotated video dataset with 400 classes, including motions, human interactions, etc.
5 Results
We compare our method against three IMRI-video baselines: Wen et al. (2018) (1]. Wang et al. (2022) 12
and Kupershmidt et al. (2022) [ Visual comparisons are shown in Fig. 4 and quantitative comparisons are shown in Fig. 6 where publicly available data and samples are used for comparison. As shown, we generate high-quality videos with more semantically meaningful content. Following the literature, we evaluate the SSIM of Subject 1 with the first testing video, achieving a score of 0.186, outperforming the state-of-the-ee by 45%. When comparing with Kupershmidt et al. (2022), we evaluate all test videos for different subjects, and our method outperforms by 35% on average, as shown in Fig. 6 Using semantic- level metrics, our method achieves a success rate of 0.849 and 0.2 in 2-way and 50-way top-1 accuracy classification tests, respectively, with the video classifier. The image classifier gives a success rate of 0.795 and 0.19, respectively, with the same tests, which significantly surpasses the chance level of these two tests (2-way: 0.5, 50-way: 0.02). Full results are shown in Tab. [ We also compare our results with an image-IMRI model by Chen et al. (2023) [7]. An image is produced for each IMRI, samples of which shown in Fig. 4 with the "walking man" as the groundtruth. Even though the results and groundtruth are semantically matching, frame consistency and image quality are not satisfying. A lag due to the hemodynamic response is also observed. The first frame actually corresponds the previous groundtruth.
Different Subjects After 1MRI pre- pre-poocessing each subject varies in the size of ROIs, where Subject 1, and have 6016, 6224, and 3744 voxels, respectively. As Subject 3 has only half the voxels of the others, a larger batch size can be used during contrastive learning, which may lead to better results as shown in Tab. I Nonetheless, all subjects are consistent both numeric and visual evaluations (See Supplementary).
Recovered Semantics In the video reconstruction, we define the semantics as the objects, animals, persons, and scenes in the vidcos, as well as the motions and scene dynamics, c.g., people rumning, fast-moving scenes, close-up scenes, long-shot scenes, etc. We show that even though the fMRI has low temporal resolution, contains enough information to recover the mentioned semantics. Fig. 5/ooss a few examples of reconstructed frames using our method. Firstly, we can see that the basic objects, animals, persons, and soene types can be well recovered. More importantly, the motions, such as running, dancing, and singing,
"Since video frames usually cannot be well described with single class, we use the top-3 classification results as the GT class, and successful event is declared if the test succeeds with any of the GT class.
7

---

Ours
Kupershmit, 2022
Wang. 2022
2018
MinD-Vis Chen, 2023
Figure 4, Compare with Benchmarks, We compare our results with the samples provided in the previous literature. Our method generates samples that are more semantically meaningful and match with the groundtruth.
GT
Ours
Figure 5. Reconstruction Diversity. Various motions, scenes, persons, and animals can be correctly recovered, A sample with transition is shown the bottom right.
Table 1. Ablation study window sizes, multimodal contrastive learming, and adversarial guidance (AG). Evaluations on different subjects are also shown. Full Model: win=2, Sub 1. Colors reflect statistical significance (two-sample t-test) compared to the Full Model. 0.0001 (purple); 0.01 (pink), 0.05 (yellow); 0.05 (green)
Video based
based
Semantic-lever
Semantic-leve
Pixel-level
2-wayt
50-wayf
2-way?
50-way?
SSIM
Full Model
0.853=0.08
0.202+0.02
0.792+*008
0.172=0.01
0.171
Window Size
0.851 =0.08
0,195+0.02
0.759
0.165=0.01
0.169
Window Size 3
0.826=0.08
0.161
0.765=20.08
0.137
0.161
w/o Contrastive
0.844=0.08
0.157
0.750
0.088
0.135
Text-fMRI Contra
=039
0.185
0.
0.154=0.01
0.164
Img- -MRI Contra
0.845 10.03
0.189 10.01
0.783
1.101
0.164
w/o AG
=0.99
0.19840.02
0.775
0.1
0.152
Subject Subject
0.841 1.000
0.103
0.784
0.158
0.171
0.846=0.08
0.21642002
0.81 0.812200.08
0.1
0.187
and the scene dynamics, such the close-up of person, the fast-motion scenes, and the long-shot scene of a city view, can also be reconstructed correctly. This result is also reflected our numerical metrics, which consider both frame semantics and video semantics, including various categories of motions and scenes.
We test our method using different window sizes, starting from window size of up to 3. When the window size one, the fMRI encoder falls back to normal MBM encoder 77. Tab. [ shows that when all other parameters are fixed, window size of 2 gives the best performance in general, which is reasonable as the hemodynamic response usually will not longer than two scan windows. Additionally, also test the effectiveness of multimodal contrastive learning. As shown in Tab. I without contrastive
Abalations

---

Ours
Kupershmidi
0:186
0.182
0.15
0.128
0,118
0.135
0.136
0.121
0.10
0.10
0.05
0.00
Ours
Kupenhmidt Method
Wing
Sub1
Sub2 Subject
Suh3
Figure SSIM Comparision. Left: comparison with Kupershmidt (2022) [3] Wang (2022) on
commonly available test samples (Subject I, test video 1). Right: comparison with Kupershmidt et al. (2022) on all subjects, all test videos.
learning, the generation quality degrades significantly. When two modalities are used, either text-fMRI or image-fMRI, the performance is inferior the full modalities used in contrastive learning. Actually, the reconstructed videos are visually worse than the full model (See Supplementary). Thus, shows that the full progressive learning pipeline is crucial for the fMRI encoder to learn useful representations for this task. We also assess the reconstruction results without adversarial guidance. visual evaluations decrease substantially. In fact, the generated videos can be highly similar sometimes, indicating that the negative guidance critical in increasing the distinguishability of fMRI embeddings. See the Supplementary for more ablation studies and visual results.
result, both numeric and
5.1 Interpretation Results
We summarize the attention analysis in Fig. We present the sum of the normalized attention within brain flat map, where
Yco17 networks (23) in the bar charts. Voxel-wise attention value is displayed we see comprchensive structural attention throughout the whole region. The average attention across all testing samples and attention heads is computed, revealing three insights into how transformers decode fMRI data.
Dominance of visual cortex: The visual cortex emerges as the most influential region. This region, encompassing both the central (VisCent) and peripheral visual (VisPeri) fields, consistently attracts the highest attention across different layers and training stages (shown in Fig. B). ). In all cases, the visual cortex always the top predictor, which aligns with prior research, emphasizing in processing visual spatiotemporal information However, the visual cortex is not the sole determinant of vision. Higher cognitive networks, such as the dorsal attention network (DorsAttn) involved in voluntary visuospatial attention control [39], and the default mode network (Default) associated with thoughts and recollations [40], also contribute to visual perceptions process 41 as shown in Fig
vital role of the visual cortex
Leyen-dependent hierarchy: The layers of the fMRI encoder function in hierarchical manner, shown Fig. [7C, D and In the early layers of the network (panel C), we observe focus on the structural information of the input data, marked by clear segmentation of different brain regions by attention values, aligning with the visual processing hierarchy 442. As the network dives into deeper layers (panel D & E), the learned information becomes more dispersed. The distinction between regions diminishes, indicating shift toward learning more holistic and abstract visual features in deeper layers.
Learning semantics progressively: To illustrate the learning progress of the fMRI encoder, we analyze the first-layer attention after all learning stages, as shown in panel before contrastive learning, after contrastive learning, and after co-training with the video generation model. We observe an increase in attention in higher cognitive networks and a decrease in the visual cortex as learning progresses. This indicates the encoder assimilates more semantic information as it evolves through each learning stage, improving the learning of cognitive-related features in the early layers.
Conclusion and Discussion
Conclusion We propose MinD Video, which reconstructs high-quality videos with arbitrary frame rates from brain activities represented by fMRI signals. Starting from large-scale pre-training to multimodal

---

A
B
Altandion Valuee
Layer First Layer
Layer
Last Layer
Attention Value
Leatning Stages
Vishert
VirPari
Contrastive
After Contrastive
DonsArtin
Defauit
Delaui
VisCent
DorsÂttnA
VisPeri
DorsAttnö
DefaurtA
Others
Brain Network
C
D
-0.5
First Layer
Middle Layer
Last Layer
Figure Attention visualization. We visualized attention maps for different transformer layers (C, D, E) and learning stages (B) with bar charts and brain flat maps. Brain networks are marked on brain surface map (A). Normalized attention values shown.
contrastive learning with augmented spatiotemporal attention, our fMRI encoder learss features from a sliding time window of fMRI progressively. Then, finetune augmented stable diffusion for vidco generations, which is then co-trained with the fMRI encoder. Finally, we show that with fMRI adversarial guidance, MinD-Vide recovers videos with accurate semantics, motions, and scene dynamics compared with the groundtruth, establishing a new state-oftthe-ee in this domain. We also show from attention maps that the trained model decodes fMRI with reliable biological principles.
Even though our method reconstructs videos with matching semantics and scenes, the pixel- level accuracy is not satisfactory, as shown in visual samples (more in the Supplementary). We believe that the results can be improved by introducing pixel-level information to control the diffusion denoising process, addition the visual semantic information decoded from the brain. Multi-scale information from the brain recordings can be fused to provide a more detailed visual description for generations.
Limitations
Another limitation lies in the inter-subject generalization ability. Our method is still within the intra- subject level, and the inter-subject generalization ability remains unexplored due to individual variations. Additionally, our method only less than 10% of the voxels from the cortex for reconstructions, while using the whole brain data remains unexploited.
Broader Impacts We believe that brain decoding has promising applications brain-computer interfaces large models develop, and it will be non-negligible component of the study to understand our cognitive process. However, governmental regulations and efforts from research communities are required to ensure the privacy of one's biological data and avoid any malicious usage of this technology.
7 Acknowledgements
This work was supported in part by National Medical Research Council, Singapore (NMRC/OFLCCI9May- 0035 to J-H Zhou), RIE2020 AME Programmatic Grant from A (to J-H Zhou), Ministry of Education Tier 2 Grant and Yong Loo Lin School of Medicine Research Core Funding (to J-H Zhou), National University Singapore, Singapore.
10

---

References
[1] T. T. Liu, "Noise contributions to the finr signal: An overview," Neurolmage, vol. 143, pp. 141-151, 2016.
[2] T. Horikawa and Y. Kamitani, "Generic decoding of seen and imagined objects using hierarchical visual features," Nature communications, vol. 8, no. 1, pp. 1-15, 2017.
[3] P. R. Roelfsema, D. Denys, and P. C. Klink, "Mind reading and writing: The future of neurotechnology." 2018.
Trends in cognitive sciences, vol. 22, no. 7. pp.
[4] G. Shen, T. Horikawa, K. Majima, and Y. Kamitani, "Deep image reconstruction from human brain activity," PLoS computational biology, vol. 15, no. 1, e1006633, 2019.
[5] R. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, and M. Irani, "From voxels to pixels and back: Self-supervision natural-image reconstruction from fmni,"" Advances in Neural Information Processing Systems, vol. 32, 2019.
[6] G. Gaziv, R. Beliy, N. Granot, A. Hoogi, F. Strappini, T. Golan, and M. Irani, "Self-supervised natural image reconstruction and large-scale semantic classification from brain activity." Neurolmage, vol. 254, p. 119121, 2022.
[7] Z. Chen, J. Qing, T. Xiang, W. L. Yue, and J. H. Zhou, "Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding," " Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), 2023.
[8] Y. Takagi and S. Nishimoto, "High-resolution image reconstruction with latent diffusion models from human brain activity," Proc. CVPR, 2023.
[9] R. L. Buckner, "Event-related fmri and the hemodynamic response." " Human brain mapping, vol. 6, no. 5-6, pp. 373-377, 1998.
[10] J. Dubois and R. Adolphs, "Building science of individual differences from fmri,"" Trends in cognitive sciences, vol. 20, no. 6, pp. 425 443, 2016.
[11] H. Wen, J. Shi, Y. Zhang, K.-H. Lu, J. Cao, and Z. Liu, "Neural encoding and decoding with deep learning for dynamic natural vision," Cerebral cortex, vol. 28, no. 12, pp. 4136 4160, 2018.
[12] C. Wang, H. Yan, W. Huang, J. Li, Y. Wang, Y.S.. Fan, W. Sheng, T. Liu, R. Li, and H. Chen, "Reconstructing rapid natural vision with fini-conditional video generative adversarial network." Cerebral Cortex, vol. 32, no. 20, pp. 4502-4511 2022.
[13] G. Kupershmidr, R. Beliy, G. Gaziv, and M. Irani, "A penny for your (visual) thoughts: Self-supervised reconstruction of natural movies from brain activity," arXiv preprint arXiv:2206. 035444, 2022.
[14] S. Schneider, J. H. Lee, and M. W. Mathis, "Learnable latent embeddings for joint behavioural and neural analysis," Nature, pp. 1-9, 2023.
[15] A. Radford, J. WU, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAl blog, vol. I, no. 8, p. 9, 2019.
[16] K. He, X. Chen, S. Xie, Y. Li, Dollár, and R. Girshick, "Masked autoencoders are scalable vision learerr," in Proc. CVPR 22, 2022, pp. 16000-16009.
[17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint 18000:181,.04405, 2018.
[18] Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark et al., "Learning transferable visual models from natural language supervision," in Interna- tional conference on machine learning. PMLR, 2021, pp. 8748-8763.
[19] J. Ho, A. Jain, and Abbeel, "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems 33: Annual Conf. on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
[20] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," Advances in Neural Information Processing Systems, vol. 32, 2019.
[21] R. Rombach, A. Blattmann, D. Lorenz, Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in Proc. CVPR'22, 22, 2022, pp. 10684-10695
11

---

[22] M. F. Glasser, T. S. Coalson, E. C. Robbnson, C. D. Hacker, J. Harwell, E. Yacoub. K. Ugurbil, J. Andersson, C. F. Beckmann, M. Jenkinson et al., "A multi- parcellation of human cerebral cortex," Nature, vol. 536, no. 7615, pp. 171-178, 2016.
[23] B. T. Yeo. F. M. Krienen, J. Sepulcre, M. R. Sabuncu, D. Lashkari, M. Hollinshead, J. L. Roffman, J. W. Smoller, I.. 7öllei, J. R. Polimeni et al., "The organization of the human cerebral cortex estimated by intrinsic functional connectivity." Journal of neurophysollgyy. 2011.
[24] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, "Progressive neural networks," arXiv preprint arXiv: 1606.04671, 2016.
[25] M. Tan and Q. Le. "Efficiietnetv2: Smaller models and faster training," in International conference on machine learning. PMLR, 2021, pp. 10096-10 106.
[26] C. Li, B. Zhuang, G. Wang. X. Liang, X. Chang, and Y. Yang. "Automated progressive learning for efficient training of vision transformers," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12486-124499
[27] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly el al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010. 1929, 2020.
[28] D. C. Van Essen, M. Smith, D. M. Barch, E. Behrens, E. Yacoub, K. Uggrbii, W.-M. H. Consortium el al., "The wu-minn human connectome project: an overview," Neuroinage, vol. 80, pp. 62-79, 2013.
[29] M. A. Lindquist, J. M. Loh, L. Y. Atlas, and T. D. Wager, "Modeling the hemodynamic response function in finri: efficiency, bias and mis-modeling." Neuroimage. vol. 45, no. I, pp. S187-S198, 2009.
[30] J. Z. Wu, Y. Ge, X. Wang. W. Lei, Y. Gu, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou, "Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation," arXiv preprint arXiv:2212.1 11,5552 2022.
[31] J. Li, D. Li, C. Xiong, and S. Hoi. "Blip: Bootstrapping language-image pre-training for unified vision-lanen Iision-language understanding and generation." in International Conference on Machine Learning. PMLR, 2022, pp. 12 12888812900 900.
[32] J. Ho and T. Salimans, "Clussifier-frve diffusion guidance," in NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2022.
[33] A. Ghiasi, H. Kazemi, E. Borgnia, S. Reich, M. Shu. M. Goldblum, A. G. Wilson, and T. Goldstein, "What do vision transformers learn? a visual exploration." arXiv preprint arXiv:2112.06727, 2022.
[34] J. Song. C. Meng, and S. Ermon, "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502, 2020.
[35] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE transactions on image processing, vol. 13, no. 4, pp. 600-612, 2004.
[36] Z. Tong, Y. Song, J. Wang, and L. Wang, "Videomae: Masked autoencoders are data iata-efficieet leamers for self-supervised video pre-training." arXiv preprint arXiv:2203. /2602, 2022.
[37] W. Kay, J. Carreira, K. Simonyan, Zhang, C. Hillier, S. Vijayanarasimhan. Viola, T. Green, T. Back, P. Natsev et al., "The kinetics human action video dataset," arXiv preprint arXiv: 2017.
[38] J. H. van Hateren and D. L. Ruderman, "Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex," Proceedings of the Royal Society of London. Series B: Biological Sciences, vol. 265, no. 1412, pp. 2315-2320, 1998.
[39] J. M. Kincade, R. A. Abrams, S. V. Astafiev, G. L. Shulman, and M. Corbetta, "An event-related functional magnetic resonance imaging study of voluntary and stimulus-driven orienting of attention," Journal of Neuroscience, vol. 25, no. 18, pp. 4593-4604 2005.
[40] J. R. Andrews- "The brain's default network and its adaptive role in internal mentation," The Neuroscientist, vol. 18, no. 3. pp. 251-270, 2012.
[41] U. Hasson, O. Furman, D. Clark, Y. Dudai, and L. Davachi, "Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding." Neuron, vol. 57, no. 3, pp. 452-462, 2008.
12

---

[42] D. C. Van Essen and J. H. Maunsell, "Hierarchical organization and functional streams in the visual
cortex," Trends in neurosciences, vol. 6, pp. 370-375, 1983.
13

---

Supplementary Material
A
Q&A Session
Where does the performance gain come from? Is it driven by the diffusion model only? The performance gain our model is not solely driven by the diffusion model. It is, in fact, a result of the combined efforts of two major stages: the fMRI encoder and the stable diffusion model.
In the first stage, the fMRI encoder plays erucial role in learning representations from the brain. It effectively captures the complex spatiotemporal information embedded the fMRI data, allowing the model to understand and interpret the underlying neural actívities. This step is particularly important it forms the foundation of our model and significandly influences the subsequent steps.
In the second stage, the stable diffusion model steps in to generate videos. One of the key advantages of our stable diffusion model over other generative models, such as GANs, lies in its ability to produce higher-quality videos. It leverages the representations learned by the fMRI encoder and utilizes its unique diffusion process to generate videos that are not only of superior quality but also better align with the original neural activities.
What does the IMRI encoder learn? Why don't we train an IMRI to object classifier, followed by generation model? The fMRI encoder is designed to learn intricate representations from brain activity, These representations go beyond simple categorical information to encompass more nuanced semantic details that can't 't be adequately captured by discrete class labels (e.g. image texture, depth, etc.). Due to the richness and diversity of human thought and perception, a model that can handle continuous semantics, rather than discrete ones, necessary.
The proposal to train an fMRI-to-object classifier followed by a generation model does not align with our goal of comprehensive brain decoding. This is largely due crucial trade-off between classification complexity and solution space:
Classification Complexity: Classifying fMRI data into a large number of classes (c.g., 1000 classes) is non-trivial. As reported in (2). reasonable performance can only be achieved in a smaller classification task (less than 50-way), due to the limited data per category and the complexity of the task.
Limited Solution Space: The solution space of discrete classes is significantly more restricted than that of continuous semantics. Thus, a classifier may not capture the complex, multi-faceted nature of brain activities.
This trade-off illustrates why classifier might not be the best approach for this task. In contrast, proposed method focuses on learning continuous semantic representations from the brain, which better reflects the complexity and diversity of neural processes. This approach not only improves the quality of the generated videos but also provides more meaningful and interpretable insights into brain decoding.
the fMRI-video generation pipeline simply imputing missing frames in a sequence of static images based on the IMRI-image generation pipeline? No, the fMRI-to-vided generation process is not merely an imputation on the IMRIIto-image generation pipeline. While both involve generating visual content based on fMRI data, the tasks and their complexities are fundamentally different.
The fMRI-to-image generation involves mapping brain activity static image. This task primarily focuses on spatial information, that is, which brain regions are active and how they relate to elements in an image.
In contrast, the IMRIto-video generation task involves mapping brain activity to dynamic vidcos. This task is considerably more complex as it requires the model to capture both spatial information and temporal dynamics. It's I's not just about predicting which brain regions are active, but also about understanding how these activations change
time and how they relate to moving elements in a video.
Adding to the complexity the hemodynamic response inherent fMRI data, which introduces a delay and blur in the timing of neural activity. This necessitates careful handling of the temporal aspects in the data. Furthermore, the temporal resolution of fMRI is quite low, making it challenging to capture fast-paced changes in neural activity.
14

---

We also use a stable diffusion process as our generative model, which is probabilistic model. This means that the generation process involves a degree of randomness, leading to slight differences during each generation for video frames. Additionally, in video generation, we need to ensure consistency across video frames, which adds another layer of complexity.
What's the major difference between Mind- Video and Mind-Vis ? Our work extends beyond Mind- Vis by tackling the fMRI-based video reconstruction problem. Unlike image reconstruction, this adds another level of complexity. The key differences can be summarized in the following.
Problem formulation: In Mind- dynamic fMRI recordings are averaged to create "snap- shot". While in this work, dynamic fMRI time series is directly used to recover a video, which requires considering the spatial features and the temporal features of fMRI. Additionally, the hemodynamic response is significant challenge in our work, making the one-to-one mapping between IMRI and video even more difficult.
Architecture: To address the unique challenges, we made two key improvements. First, we enhanced the fMRI encoder to handle sliding time window of fMRI, capturing spatial and temporal information with distinct attention heads. Second, we employed multimodal contrastive learning to align fMRI with the semantic space of text and images before the co- co- training. This contrasts Mind- Vis in which co-training was performed directly without contrastive leaming.
Do all trainable models require re-training for each subject? How computationally demanding is it? How is the training process repeated for each subject independently? Not all models necessitate re-training for every individual. The most resource- inntrce-intensive phase is the large-scale pretraining of the MBM encoder, which remains consistent across subjects and does not demand re-training. Nevertheless, we do finetune both the MBM Encoder and the generative model using the fMRI data specific to each subject. This fine-tuning step, however, is relatively lightweight in terms of computational demands.
B More Implementation Details
Large-Scale Pre-training The large-scale pre-traiming uses the same setup as the MBM described in 7 A ViT-large-basee model with 1 I-dimensional patchifier is trained with hyperparameters shown in Tab. B.1 The training takes around 3 days using 8RTX3090 GPUs. The training is performed on the 600,000 fMRI from HCP. Same as the literature, after the large-scale pre-training, the autoencoder is tuned with fMRI data from the target dataset, namely, Wen (2018), using MBM as well. The tuning is performed using a small learning rate and epochs.
parameter
value
parameter
value
parameter
value
parameter
value
patch size
16
encoder depth
24
decoder embed dim
512
clip gradient
0.8
embedding dim
1024
encoder heads
16
learning rate
2.5c-4
weight decay
0.05
mask ratio
0.75
decoder depth
8
warm-up epochs
40
batch size
500
mip ratio
1.0
decoder heuds
16
epochs
500
optimizer
AdamW
Table I.I. Hyperparameterss used in the large-scale pre-training
Multimodal Contrastive Learning In this step, will take the pre-trained IMRI encoder from the previous step and augment it with temporal attention heads to accommodate multiple fMRL. Then contrastive learning is performed with MMR-iimaee-terr triplets. The image is randomly-picked frame from an fMRI scan window. As mentioned, there are two important factors in the contrastive: batch size and data augmentations. Therefore, data augmentations for fMRI, where 20% of voxels are randomly set to zeros each time. The random crop is applied to videos with probability of 0.5. To augment the frame captions, we apply synonym augmentation and random word swapping. Due to a small dataset size (~~4000), we use a dropout rate of 0.6 to avoid overfitting. For Subject and 2, the training is performed with a batch size of 20, while a batch size of 32 used due to fewer IMRI voxels with Subject 3. Training for all subjects is performed for 1,2000 steps with a learning rate of 2 10 The training takes around 10 hours using one RTX3090.
applied for all modalities. Random sparsification is used
Training of Augmented Stable Diffusion The stable diffusion model is augmented with temporal attention heads for video generation. We train the augmented stable diffusion model with videos from the target datascts. The videos are downsampled from 30 FPS to 3 FPS at resolution of 256 256 due to limited
15

---

GPU memory, even though our model can work with the full time resolution. This step is important for two reasons: 1) the augmented temporal heads are untrained; the stable diffusion pre-trained at resolution of 512x so we need to adapt it to lower resolution.
During the training, update the self-attention heads ("attnl"), cross-attention heads ("attn2"), and lemporal attention heads ("attn_temmp The training is performed with text conditioning for 800 steps. We a learning rate 2 10 and batch size of 14. The training takes around 2 hours using one RTX3090. Visual results show that videos of high quality can be generated with text conditioning after this step.
Co-training The fMRI encoder produces embeddings of dimensions 768, which are used to condition the augmented stable model during co-training. The whole fMRI encoder is updated, and only part of the stable diffusion updated (same as the last step). The training is performed with batch size 9 and learning rate of 3 10-5 for 1,5000 steps. The training takes around 16 hours using one RTX3090.
Inference All samples are generated with 200 diffusion steps using fMRI adversarial guidance. The fMRI adversarial guidance uses average fMRI as the negative guidance with guidance scale of 12.5.
C Analysis of Visual Results
We test on all three subjects Wen (2018) dataset. Around 6000 voxels are identified as ROI for Subject and 2, while around 3000 voxels are identified for Subject 3. Thus, larger batch size can be used when training with Subject 3, which may be the reason for its better numeric evaluation results. Nonetheless, all three subjects show consistent generation results. Some samples are shown in Fig. C.I
GT
Sub1
Sub2
Sub3
Figure C.I. Samples from different subjects.
Visual results the ablation studies are shown Fig. The Full model is trained with our full pipeline and inference with adversarial guidance. In contrastive learning ablation, tested with incomplete modality, namely, image-fMRI and text-fMRI, respectively. Similar to the numeric evaluations, using incomplete contrastive gave unsatisfactory visual result compared to using all three modalities. However, incomplete modality still outperformed inferencing without adversarial guidance significantly, which generated visually meaningless results.
Some fail cases are shown in Fig. C It is observed that even though some fail cases generated different animals and objects compared to the groundtruth, other semantics like the motions, color, and scene dynamics can still be correctly reconstructed. For example, although the airplane and flying bird are not reconstructed, similar fast-motion scenes recovered in Fig. C33
D More Ablation Studies
Other than the ablation studies shown in Tab. we include extra ablation studies in Tab, D22 to demonstrate the effectivenees of masked brain modeling (MBM) in our framework. Two experiments performed.
16

---

GT
Full
Image Contrastive
Text Contrastive
w/o Adversarial
Guidance
Figure C.2. Reconstruction samples for ablation studies. The Full model full modality contrastive learning with adversarial guidance.
The first experiment involves excluding the MBM pre-training. while the second experiment removes both the MBM pre-training and the contrastive training from the proposed method.
The results show the importance of MBM and contrastive training for all metrics. The performance drops as much as 55% without both components and 30% without only MBM. The visual quality of the generated videos also follows similar trend as in Fig.
50 Kay, Top-1 Accuracy
50 Way, Top-1 Accuracy
Image Identification Tests
Video Identification Tests
SSIM
Full Model
0.172-10.01 0.002
0.202++0.02
0.171
w/o MBM, Contrastive
0..015
0.143
w/o MBM, w/o Contrastive
0.076
0.108
0.123
Table D.2. Extra ablation studies masked brain modeling.
E More Evaluation Results
Limited by the publicly available video samples and codes for reproduction in the literature, we had some difficulties in comparing numerically with other methods. In the previous literature, Kupershmidt, 2022
K and Wen, 2018 [T]] have only released part their reconstructed videos, while Wang, 2022 [12] have not released their reconstructed videos. Also, because their released videos are based different frame rates from ours (1Hz in [33, 0.5Hz []] 3Hz in ours), we did not perform the identification tests those videos, as it might not be a fair comparison. Instead, relied commonly available

---

GT
Ours
GT
Ours
Figure C.3. Fail cases.
metric reported in their papers, SSIM. This metric calculated quantitatively across all testing samples according to their papers.
Nevertheless, in this section, we perform the identification test those incomplete samples only (released by previous work) and compare the results across different methods (including ours). Please see Tab. Our method outperformed the two previous methods in identification test accuracy.
50-Way, Top-1 Accuracy
50-Way, Top-1 Accuracy
Image Identification Tests
Video Identification Tests
Ours
0. 0.90-
0.265-1-.00
Kupershmidt, 2022
0.019
0.238 + 0.02
Wen, 2018
0.07++0.01
0.166++00016
Table E.3. Comparision of identification Tests with previous works.
Source of Errors
short, the failure cases can be attributed by two factors:
Lack of pixel-level controllability Due to the probabilistic nature of the diffusion model and the current conditioning method, the generation process lacks strong control from the fMRI latent to generate strictly matching low-level features, such as shapes, color, and geometric information. We believe this would be an important perspective for future research on this task.
Uncontrollable factors during the scan. Mind wandering and imagination of the subject are usually inevitable during the scan. It has been shown that imagination involved and can be decoded to some extent from the visual cortex, which can lead to mismatching between the ground truth and the generation results.
18


Reviews:
[{"review_id": "c6jdjuBmb5", "reviewer": "Reviewer_19wY", "summary": "The authors have developed an fMRI to video model trained using contrastive learning and stable diffusion. The generted videos are evaluated based on the semantics of their content and pixel level metrics at video and frame level, some of which utilize pretrained classifiers trained on ImageNet and VideoMAE. The work builds on top of and shares attributes with the MinD-Vis model including the MBM approach. It constitues of two modules: fMRI encoder and a video generative model, trained separately, and finetuned together.", "questions": "In the \"Multimodal Contrastive Learning\" subsection, the idea of {fMRI, video, caption} triplet needs further explanation. If embeddings of fMRI, video and caption encoded via three separate encoders, what does the \"augmented endcoder\" refer to in this context?\n\nIn the same subsection, the idea of concatenating captions with \"Then\" in-between is discussed. How impactful was the addition of \"Then\" keyword in-between the two captions? Did you run an ablation study that demonstrates positive effect?\n\nIn the \"Scene-Dynamic Sparse Causal (SC) Attention\" subsection, with regards to the attention maps, we wonder if the authors monitored attention-maps across transition frames; i.e., did they observe the attention to previous frames (i-2 and i-1) to disipate during transitions?\n\nDo all trainable models (e.g. SC-MBM Encoder, etc.) require re-training for each subject?\n", "limitations": "Yes. \nThe intra-subject limitation of the method is highlighted; however, it is not clear whether the training process was repeated for each subject independently. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper proposes a sound architecture constitued of Spatial and Temporal attention, multimodal contrastive learning, adversarial guidance, and diffusion models. The end2end pre-processing and training process, including usage of pretrained models such as BLIP for video captioning have a few introguing novelties. \n", "weaknesses": "Eventhough reference [6] is limited to generating images, the differentiating factors between the current work and reference [6] is better be highlighted in more details.\n\nIn the \"Adversarial Guidanc for fMRI\", there is a claim about using the \"average all fMRI in the testing set as the negative guidance\". Usage of the \"testing\" set in the training process, even in its aggregated (averaged) sense is not the best design. \n\nGiven the complexity of the architecture, the ablation study could have been improved to highlight the impact of more components.\n\nAssuming the trainable modules require re-training per subject, the proposed design raises concerns around practicality of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your recognition of our contributions and novelty.  Our point-by-point responses to the comments are as follows.  \n\n> 1. The differentiating factors between the current work and [6] is better be highlighted.\n\n**Response:** We thank the reviewer for this important point. Our work extends beyond [6] by tackling the fMRI-based video reconstruction problem. Unlike image reconstruction, this adds another level of complexity. The key differences can be summarized in the following. We will revise our paper to highlight this point.\n\n- **Problem formulation**. In [6], dynamic fMRI recordings are averaged to create a “snapshot”. While in this work, **dynamic fMRI time-series** is directly used to recover a video, which requires considering the **spatial features** and the **temporal features** of fMRI. Additionally, the **hemodynamic response** is a significant challenge in our work, making the one-to-one mapping between fMRI and video even more difficult. \n\n- **Architecture**. To address the unique challenges, we made two key improvements. First, we enhanced the fMRI encoder to handle a sliding time window of fMRI, capturing spatial and temporal information with distinct attention heads. Second, we employed **multimodal contrastive learning** to align fMRI with the semantic space of text and images before the co-training. This contrasts [6] in which co-training was performed directly without contrastive learning. \n\n\n> 2. Usage of the \"testing\" set in the training process is not the best design.\n\n**Response:** To clarify, the adversarial guidance was only used in the **inference stage (testing)** to create a stronger condition and to increase the signal-to-noise ratio of the testing fMRI dataset. The averaged testing fMRI data was not involved in any part of the training process. \n\n> 3. Given the complexity of the architecture, the ablation study could have been improved.\n\n**Response:** As recommended, we have now conducted additional ablation studies to examine the impact of two crucial components in our approach: masked brain modeling (MBM) pre-training and contrastive training.\n\nWe performed two new experiments on top of the existing ablation experiments in the table below. The first experiment involves **excluding the MBM pre-training**, while the second experiment **removes both the MBM pre-training and the contrastive training** from the proposed method.\n\nThe results show the importance of MBM and contrastive training for all metrics. The performance drops as much as 55% without both components and 30% without only MBM. The visual quality of the generated videos also follows a similar trend, which will be detailed in our revised manuscript.\n\nAgain, we sincerely appreciate your comment. These additional experiments strengthen the empirical evidence supporting our proposed approach.  \n\n| | 50-Way, Top-1 Accuracy| 50-Way, Top-1 Accuracy| |\n|-|-|-|-|\n| | Image Identification Tests| Video Identification Tests | SSIM |\n| Full Model| **0.172 +- 0.01**| **0.202 +- 0.02**| **0.171**|\n| w/o MBM, w/ Contrastive   | 0.122 +- 0.012| 0.169 +- 0.015| 0.143 |\n| w/o MBM, w/o Contrastive | 0.076 +- 0.008| 0.138 +- 0.013| 0.123 |\n\n> 4.  Assuming the trainable modules require re-training per subject, the proposed design raises concerns around practicality.\n\n**Response:** We acknowledge the practical concern regarding per-subject re-training process. But it is worth noting that almost all the existing methods for this specific neuroscience application, i.e., “brain decoding”, rely on per-subject training due to high inter-subject variability and limited datasets. Cross-subject model generalization remains an **open problem** and an exciting **future work direction** of our group and others.\n\n> 5. What does the \"augmented encoder\" refer to in this context?\n\n**Response:** The fMRI encoder in [6] can only process a single fMRI frame without considering the temporal dynamics of the fMRI recordings (i.e., multiple frames). We changed its architecture to **encode multiple fMRI frames in a sliding time window manner**, considering **spatial and temporal correlations** during feature learning. Thus, it is called an “augmented encoder”.\n\n> 6. How impactful was the addition of \"Then\" keyword in-between the two captions? \n\n**Response**:  We apologize for the confusion. To clarify, \"Then\" concatenation was only used in the augmented stable diffusion training (not the contrastive learning). This stage does not impact the fMRI feature learning process significantly, thus having **minimal influence on the final results**. To improve clarity, we will relocate the captions of the concatenation part to the stable diffusion training subsection in our revision. \n\n> 7. We wonder if the authors monitored attention-maps across transition frames?\n\n**Response:** Yes, we did monitor attention maps across transition frames. However, we **did not observe** that the attention to the previous frames noticeably dissipated during the transitions, which led to frames containing parts of the content from the two scenes. \n\nThis can be attributed to the nature of our generated videos: they span only 2 seconds, mirroring the 2-second TR of the fMRI. Given this brief duration, distinguishing between transition and non-transition frames becomes inherently challenging. However, the scene-changing decoding is an exciting question and an unexplored field worth further research. \n\n> 8. Do all trainable models require re-training for each subject?\n> 9. it is not clear whether the training process was repeated for each subject independently.\n\n**Response:** No. The most resource-consuming part is the large-scale pretraining of the MBM encoder, which **does not require re-training** for each subject. But we do **need to finetune** both the MBM Encoder and the generative model using each subject's fMRI. However, this process is **not computationally expensive**. We will make this part clearer in the revision."}, {"review_id": "fzW92y4zoq", "reviewer": "Reviewer_4NZX", "summary": "This research proposes a method called Mind-Video to reconstruct videos from brain activity. By utilizing continuous fMRI data and advanced techniques, Mind-Video can generate high-quality videos with arbitrary frame rates. The model outperforms previous methods in accuracy and structural similarity. \n", "questions": "None", "limitations": "The limitations are well explained by the authors.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The present work contributes with an innovative approach for reconstructing continuous visual experiences from brain activities. By utilizing masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model, their method, called Mind-Video, surpasses previous techniques in reconstructing high-quality videos.\n\nThis paper presents a meticulous study of previous work, which is important in the development of the present work. Also, the technical aspects are clearly explained and have also been evaluated using the correct metrics.\n\nThe experimental evaluation of the proposed model demonstrates its superior performance compared with other works.\n\nThe developed methodology provides interpretability, which is a very important factor in medical applications.\n\nThe interpretation of the results is good, which strengthens the results and the paper in general.\n\nIn summary, the work is an interesting application of deep learning in the medical area, and it also has a remarkable novelty.", "weaknesses": "The work is very interesting, the resulting video resembles the ground true in terms of activity, but in terms of scene there is still a considerable difference.\n\nIn the \"Paired fMRI-Video dataset\" part, only three subjects are used, this is a limitation in terms of generalization. It is a limitation but it would be interesting to use more subjects and have more generalizable results. Nevertheless, it is ahead of several previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the strong support and the positive comments on our work. Your inspiring questions and comments are valuable for our future work. Our point-by-point responses are as follows. \n\n> 1. The work is very interesting, the resulting video resembles the ground true in terms of activity, but in terms of scene there is still a considerable difference.\n\n**Response:**  We appreciate the useful feedback on the generated videos. We agree that while our model captures activity patterns well, scene reconstruction still presents challenges. This stems from the significantly **lower signal-to-noise ratio and increased complexities in the spatial-temporal dynamics** in the paired fMRI-video data compared to the paired fMRI-image data. Furthermore, there is higher inherent variability in **individuals' imaginations due to the dynamic nature of videos (in contrast to static images)**. In the proposed model, we focused more on the video semantic recovery than the low-level visual features. We are aligned with the reviewer that this limitation should be addressed in future research. \n\n> 2. In the \"Paired fMRI-Video dataset\" part, only three subjects are used, this is a limitation in terms of generalization. It is a limitation, but it would be interesting to use more subjects and have more generalizable results. Nevertheless, it is ahead of several previous works.\n\n**Response:** We appreciate the reviewer sharing his/her valuable perspective on this work.  **We totally agree that utilizing more subjects can help evaluate and potentially enhance the generalizability of our method.** Unfortunately, this is subject to the availability of suitable fMRI-video datasets currently. This is a common problem for this type of neuroscience application. We will acknowledge cross-subject model generalization as a crucial **future work direction** in the revised manuscript. More work is needed to perform well-designed brain recording experiments and methodological development to increase the generalizability and interpretability.  "}, {"review_id": "VgLKYxaNLR", "reviewer": "Reviewer_ZvgZ", "summary": "The paper presents a pipeline to decode videos from fMRI brain activity data. The pipeline is divided into several consecutive steps. First, a Transformer-based autoencoder is trained on an unsupervised Masked Brain Modelling task to learn fMRI representations on a large corpus of fMRI data. The fMRI encoder is then augmented with a spatial and temporal attention modules to enable the processing of temporal frames. Second, the fMRI encoder is further trained on a contrastive alignment task where fMRI representations are pulled closer to CLIP-based image and text representations of the corresponding video frames. Finally, a Stable Diffusion UNet model is augmented with temporal attention to allow its conditioning on the latents of the two previous frames and is finetuned end-to-end with the fMRI encoder. Experiments on a public dataset of participants watching videos inside an fMRI scanner show better performance (SSIM) as compared to existing baselines. Ablation studies are used to evaluate the impact of windowing hyperparameters and of different design choices. Attention weights from different Transformer layers are projected on a cortical map and visualized to highlight correspondence with different brain networks.", "questions": "1. What is the impact of the masked modelling pretraining on downstream performance? Given the training budget required for this step of the pipeline, it might be important to know the order of performance improvement it can bring.\n2. I do not fully understand the idea behind the \"network inflation trick\" mentioned at line 157. Am I correct in thinking that what is called a batch here is built by taking consecutive fMRI frames, rather than by randomly sampling fMRI frames across e.g. a recording?\n3. At the contrastive learning stage, what are the inputs to the fMRI encoder? Is it multiple fMRI frames, in which case there is a sequence of latent predictions per video segment, or is it still single fMRI frames matched to a single image frame? (The answer to this question might influence the relevance of the next question.)\n4. I understand from line 231 that the decoded videos are generated from a single fMRI frame (yielding 6 video frames at 3 fps). In this particular case, the diffusion model is used in a similar setting as an fMRI-to-image model, i.e. it is conditioned by a unique brain-derived latent collected at time t. The resulting video could therefore be seen as an \"hallucination\" of the diffusion model based on an initial frame, rather than a truly dynamic decoded video, i.e. where temporal changes in brain activity drive changes in video frames. I assume that this might change if longer videos were generated e.g. based on 2 consecutive TRs. Is that something you have tried and if so, do the resulting videos remain temporally consistent when the conditioning is updated to a new fMRI frame?\n5. Results of Table 1 show that removing adversarial guidance has a very mild effect on performance, especially for the video-based semantic metric. However corresponding results in Figure C2 suggests removing adversarial guidance dramatically impacts decoding performance. Does this just happen to be an unrepresentative example?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: The combination of self-supervised learning, contrastive alignment in image/text latent space and conditional video generation for fMRI-to-video decoding is novel. \n\nQuality: Most claims are supported (see below for possible exception to this). Qualitative and quantitative results are convincing. \n\nClarity: The manuscript is overall clearly written and well organized. \n\nSignificance: As one of the first fMRI-to-video approaches, this work is likely to inspire other work in the brain decoding literature. The qualitative and quantitative results suggest this is a clear improvement over existing baselines. ", "weaknesses": "Quality: The underlying core claim of the submission, i.e. that the dynamic information contained in brain activity data can be used to reconstruct videos, might not be fully supported in the experimental settings presented in the paper (see Question 4). Essentially, it is not clear to me that the presented results prove temporally evolving brain activity can be decoded into videos; rather, it seems to suggest existing video diffusion models can be conditioned on a temporally-fixed (i.e. coming from time t) brain-derived latent representation.\n\nClarity: I found the description of the inputs at the contrastive learning stage unclear (Q3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your appreciation of the novelty and potential impact of our work and your important suggestions.  Our point-by-point responses are as follows.\n\n> 1. What is the impact of the masked modelling pretraining? \n\n**Response:** We thank the reviewer for raising this important point. We have now conducted additional ablation studies to thoroughly examine the impact of Masked Brain Modeling (MBM) in our approach. Specifically, we performed two experiments:\n\n1. We conducted an experiment where we **excluded the MBM pre-training** from our proposed model.\n2. We performed another experiment where **both the MBM pre-training and contrastive training were excluded** from our proposed model.\n\nFor evaluation, we employed 50-way, top-1 identification tests, as detailed in the paper, along with Structural Similarity Index (SSIM) metrics. The results are tabulated below.   \n\n| | 50-Way, Top-1 Accuracy| 50-Way, Top-1 Accuracy   |  |\n|-|-|-|-|\n|  | Image Identification Tests    | Video Identification Tests | SSIM |\n| Full Model | **0.172 +- 0.01**                  | **0.202 +- 0.02** | **0.171**|\n| w/o MBM, w/ Contrastive   | 0.122 +- 0.012| 0.169 +- 0.015 | 0.143 |\n| w/o MBM, w/o Contrastive | 0.076 +- 0.008| 0.138 +- 0.013 | 0.123 |\n\nAs you can see from the Table, we observed a significant drop in performance when excluding MBM from our model, and this drop further intensified when both MBM and contrastive learning were excluded. Moreover, the visual quality of the generated results aligned well with these quantitative metrics. We will include both in our revised submission. \n\n> 2. I do not fully understand the idea behind the \"network inflation trick\" mentioned at line 157. \n\n**Response:** The batch here consists of **consecutive fMRI frames from a sliding time window**. The network inflation trick is a technique that enables the model to process an extra dimension by rearranging the input data shapes. For example, the transformer designed in previous work [6] for image reconstructions can only take input with 3 dimensions: n, p, and b, where n is the batch size and the last two dimensions will be used in the attention calculation in the transformer. Now we want to add an extra time dimension without changing the structure of the transformer such that the pre-trained weights provided in [6] can still be used. So to handle an input of shape (n, w, p, b), where w is the extra time dimension (sliding time window size), we can just merge two dimensions of the input into either (nw, p, b) or (np, w, b). When the dimension becomes (nw, p, b), the attention layer in the transformer is learning spatial correlations of the input, thus called spatial attention. When the dimension becomes (np, w, b), it learns the temporal correlations, thus called temporal attention. Combining spatial attention and temporal attention enables our model to learn spatial and temporal information from consecutive fMRI frames.  \n\n\n> 3. I found the description of the inputs at the contrastive learning stage unclear (Q3).\n\n**Response:** We apologize for the confusion. The input to the fMRI encoder at the contrastive learning stage is **a sliding time window of fMRI, i.e., multiple fMRI frames, which are matched to a video clip**. We will provide more details in our response to the next question. The output of the fMRI encoder is an embedding that is learned by considering both the spatial and the temporal features of the fMRI frames. We will revise our manuscript to make this point clearer.  \n\n\n> 4. Question 4\n\n**Response:** We apologize again for the confusion. In line 231 of our paper, we are meant to say each fMRI frame corresponds to 2 seconds of video, as the TR of fMRI scanning is 2 seconds. However, considering the nature of hemodynamic response function of the human brain, a 2-second video may be encoded by fMRI frames at both t and t+1. Therefore, we create a sliding time window that groups multiple consecutive fMRI frames, which is used as input to the fMRI encoder. For example, consider a consecutive fMRI and videos represented by [f1, f2, f3, f4] and [v1, v2, v3, v4]. Assuming a window size of 2, we will use [(f1, f2, v1), (f2, f3, v2), (f3, f4, v3) … ] as the (fMRIs, video) pairs for training and testing. As we can see, the input to our pipeline is actually consecutive fMRI frames from a sliding time window. For consecutive fMRI-based sliding windows, the decoding results will be highly similar if they are matched to a similar groundtruth. However, there could be some style inconsistency between consecutive fMRI windows due to the probabilistic nature of the diffusion model, which we believe could be better improved in future research.   \n\nAnother clarification is that we only reconstructed a 2-second video each time because of memory limitation of our GPU. Longer video generation requires larger GPU memory. Nevertheless, thanks to the sliding time window design, we can theoretically expand our approach and reconstruct longer videos with a larger fMRI window size with enough GPU memory. \n\n> 5. Results of Table 1 show that removing adversarial guidance has a very mild effect.\n\n**Response:** To clarify, without adversarial guidance, most generated videos are visually worse than our full method. In Table 1, it does drop from 0.172 to 0.117 in frame-based semantic level metrics and from 0.171 to 0.152  in the SSIM. However, even though the results are visually worse, **some low-level features are still generated**, such as animal-like moving objects, blue burry water-like objects, etc, with matching motions, which can still be classified into a correct semantic category in the metrics. Overall, adversarial guidance is still helpful. "}, {"review_id": "s5FfHr41HJ", "reviewer": "Reviewer_tnAn", "summary": "This paper focuses on the task of reconstructing human vision from brain activities. The authors propose MinD-Video that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. And the results are evaluated with semantic and pixel metrics at video and frame levels.", "questions": "1. The proposed method is of limited novelty, the authors need to rethink their main contributions.\n2. The writing needs to be improved, and the central point needs to be emphasized.", "limitations": "See weakness part.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The authors provide both quantitative and qualitative results, and also provide some interpretable visualization results for demonstration.\n2. The proposed method seems to perform better than previous non-diffusion methods.", "weaknesses": "1. The pre-training of fMRI encoder is composed of generative and contrastive objectives, which is very similar to previous works[1] that use masked contrastive pre-training in diffusion models. And the overall framework is a combination of existing methods or tricks (masked contrastive pretraining[1]+spatio-temporal attention (ST-Attn) mechanism[2]+stable diffusion[3]). Thus the novelty of this work is limited with respect to diffusion model field.\n\n2. The proposed method contains two many tricks and modules, and causes no central point with respect to its technical contributions, the overall presentation is hard to follow.\n\n3. I have reproduced the method, the results seem not as good as the demonstration in the paper and there sometimes exists a style inconsistency between GTs and generated results. The authors seem to cherry-pick the generated videos.\n\n[1] Yang L, Huang Z, Song Y, et al. Diffusion-based scene graph to image generation with masked contrastive pre-training[J]. arXiv preprint arXiv:2211.11138, 2022.\n\n[2] Wu J Z, Ge Y, Wang X, et al. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation[J]. arXiv preprint arXiv:2212.11565, 2022.\n\n[3] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and effort in reviewing our work. We also appreciate your interest in our work and the useful suggestions. Our point-by-point responses are as follows. \n\n> 1. the novelty of this work is limited with respect to diffusion model field.\n> 2. The proposed method is of limited novelty.\n\n**Response:** We would like to take this opportunity to emphasize that though our method builds upon the established techniques, the novelty of our work is not in implementing these techniques per se. Instead, it lies in applying these techniques in a new and challenging domain: learning dynamic brain activities and reconstructing visual stimuli from the brain. Our work is at the intersection of neuroscience and CV, where the focus is not solely on inventing new tricks or structures for the diffusion model. Instead, **the main aim is to tackle the unique challenges of fMRI** and make proper methodological adjustments to adapt state-of-the-art generative models to our specific task. This has been recognized by the other four reviewers. \n\nThanks for providing the additional reference. However, [1] provided a learning method for **scene graphs**, which is **an entirely different modality** from fMRI or any other brain recordings. Even though our method shares a similar philosophy with [1], this does not negate the novelty of our work. **The specific way of problem formulation, data modeling, and problem-solving is also an essential part of research novelty. Besides, the masking, contrastive, and generative objectives are standard techniques in representation learning.**\n\nFurthermore, we would like to highlight a few key differences. \n- We learn features of **dynamic fMRI with masking**, while [1] learns a **static scene graph without masking**. \n- Our work focuses on learning the **biological features** of fMRI, whereas [1] aims to learn the **geometric information** in the scene graph. \n- We must consider the **hemodynamic response**, a unique challenge when matching dynamic fMRI to videos. In contrast, the scene graph in [1] is static and matched to a static image. \n- We used a **3-modality** contrast among fMRI, video, and text to create a shared multi-modality space. While [1] used a **2-modality** contrast between scene graphs and images to discriminate a binary objective. \n\nIn summary, while our method shares high-level similarities with [1] in terms of using generative and contrastive objectives, the comparison should not overlook the **different modalities and unique challenges** each work addresses. [1] deals with scene graphs, a static graphical structure, using an off-the-shelf graph encoder. Our work handles fMRI, a dynamic and biologically complex modality. Our problems involve understanding dynamic data subject to hemodynamic responses and the biological interplay of various brain regions - a quite different beast. \n\nWe hope our work can encourage more research in this field and lead to more novel development in methods and applications. We are confident that with further reflection, the value of our approach will become more apparent.\n\n> 3. two many tricks. no central point \n\n**Response:** We apologize for the confusion. To clarify, all our modules and tricks are **summarized and depicted graphically** in Figure 2. Critical modules and methods are **color-coded and represented by different shapes** in Figure 2. Our technical contributions are summarized on Page 2 of our paper with **5 bullet points**, each containing two short sentences. As suggested, we will revise the paper to make our contributions and method descriptions clearer to improve readability.\n\n> 4. I have reproduced the method, the results seem not as good\n\n**Response:** We thank the reviewer for strong interest in our work and his/her effort to try to reproduce our findings. We are thrilled to see the level of engagement our work has elicited. To clarify, we assure you that we have presented a representative range of qualitative results of the generated videos in our paper. More importantly, we also evaluated the performance based on quantitative metrics and compared with other methods as presented in the original paper. Given this doubt from the reviewer, we have now included full generation samples in the ”Official Comment”.\n\nIt would be great to know if such quantitative evaluations have been reproduced using the same data following the same process. It is worth noting that differences in outcomes could be due to various factors, including whether we are working with full samples or subsamples, data processing approaches, parameter settings, or the specific pretrain checkpoints used. \nOur codes will be made publicly available upon publication to ensure a fair reproduction for the community. \n\nRegarding the style inconsistency, we suspect the reviewer's concern is related to the visually distinct outcomes of generated results for sample inputs at each sampling. We acknowledge that this is a typical characteristic of the probabilistic nature inherent in the diffusion model, which represents one of the limitations of the diffusion-based decoding approach. However, despite these variations, **the semantic contents and evaluation metrics of the generated results remain consistent, as demonstrated in our paper**."}, {"review_id": "fERhuFSWBe", "reviewer": "Reviewer_1EQ5", "summary": "This paper aims to reconstruct high-quality video from brain activity. A novel model called MinD-Video is proposed, which could learn spatiotemporal information from continuous fMRI data.", "questions": "1. Please address my above concerns.\n2. Do you think the Video Generation part should have controllability? I mean the current Video Generation part has no ability to reconstruct a video that is the same as source video, which would be the upper bound of the overall performance.", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is quite interesting to reconstruct videos according to human brain regions' actions, Although the task is hard.\n2. I believe all the newest methods are worth being tried to achieve fMRI data reconstruction, including Stable Diffusion and its video version.\n3. The latent alignment and spatiotemporal attention are well-designed in this paper.", "weaknesses": "1. In Figure 4, we can see that Wen (2018) could reconstruct the shape and Kupershmit (2022) could reconstruct the texture. While videos generated by MinD-Video lack shape and texture. So I think previous works could affect more on the Video Generation part of MinD-Video.\n2. The quantitative results are not sufficient. The metric values are not compared with other methods, and the gap between some experimental settings is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the recognition of our contribution and the invaluable and constructive comments. Our point-by-point response is provided as follows. \n\n> 1. In Figure 4, we can see that Wen (2018) could reconstruct the shape and Kupershmit (2022) could reconstruct the texture. While videos generated by MinD-Video lack shape and texture. So I think previous works could affect more on the Video Generation part of MinD-Video.\n\n**Response:** Thank you for this insightful comment. Pixel level and semantic level decodings recover visual stimuli from two different perspectives, where the trade-off between fidelity and meaningfulness needs to be considered. In this work, **we prioritize the recovery of visual semantics** in fMRI, which is crucial for understanding the complex mechanism of human perception. \n\n**We recognize the significance of pixel and texture level information**, which contributes to generating visually closer results to the ground truth. For instance, leveraging results from previous methods could serve as a valuable guidance or initial starting point for our generative process.\n\nNonetheless, our aim is to establish a foundation for future research that integrates both pixel-level features and visual semantics in this particular task. By doing so, we anticipate that future studies can further refine and enhance the decoding process, leading to even more comprehensive and accurate outcomes.\n\n\n> 2. The quantitative results are not sufficient. The metric values are not compared with other methods, and the gap between some experimental settings is limited.\n\n**Response:** We thank the reviewer for the constructive suggestion. **Limited by the publicly available video samples and codes** for reproduction in the literature, we had some difficulties in comparing numerically with other methods. In the previous literature, Kupershmidt, 2022 [12] and Wen, 2018 [10] **have only released part of their reconstructed videos**. Also, because their released videos are **basead on different frame rates from ours** (1Hz in [12], 0.5Hz in [10], 3Hz in ours), we did not perform the identification tests on those videos, which might not be a fair comparison. Instead, we relied on **a commonly available metric reported in their papers**, SSIM.  This metric was calculated quantitatively **across all testing samples** according to their papers. We did the same for our approach in the original paper. \n\nNevertheless, as recommended, we have now decided to perform the identification test on those **incomplete samples** only (released by previous work) and compare the results across different methods (including ours). Please see the table below. Our method outperformed the two previous methods in identification test accuracy. We will include these results and  **possible limitations in the revised manuscript**. \n\n|                                            | 50-Way, Top-1 Accuracy     | 50-Way, Top-1 Accuracy   |          \n|------------------------------------ |-------------------------------------|------------------------------------|\n|                                            | Image Identification Tests    | Video Identification Tests | \n| Ours                                   | **0.195 +- 0.016**                | **0.265 +- 0.02**              | \n| Kupershmidt, 2022             | 0.179 +- 0.017                     | 0.238 +- 0.02                    | \n| Wen, 2018                          | 0.07 +- 0.01                         | 0.166 +- 0.016                  | \n\nIn order to strengthen our experimental settings, we performed extra ablation studies on different components of our designs on top of the existing ablation experiments. The first experiment involves **excluding the MBM pre-training**, while the second experiment **removes both the MBM pre-training and the contrastive training** from the proposed method. As shown in the table below, both the MBM pre-training and the contrastive training are critical to getting the best results, and removing any of them will incur a significant drop in performance.\n\n\n\n|                                            | 50-Way, Top-1 Accuracy     | 50-Way, Top-1 Accuracy   |          |\n|-------------------------------------|-------------------------------------|------------------------------------|---------|\n|                                            | Image Identification Tests    | Video Identification Tests | SSIM |\n| Full Model                          | **0.172 +- 0.01**                  | **0.202 +- 0.02**              | **0.171** |\n| w/o MBM, w/ Contrastive   | 0.122 +- 0.012                     | 0.169 +- 0.015                  | 0.143 |\n| w/o MBM, w/o Contrastive | 0.076 +- 0.008                     | 0.138 +- 0.013                  | 0.123 |\n\n> 3. Do you think the Video Generation part should have controllability? I mean the current Video Generation part has no ability to reconstruct a video that is the same as source video, which would be the upper bound of the overall performance.\n\n**Response:**  Thank you for your useful comment. Indeed, we agree that incorporating some form of controllability in the Video Generation part of our model could enhance the reconstruction of low-level image features like shape, texture, and location, bringing us closer to the source video. In fact, we believe that **controllability is a critical feature** for this kind of research, which is worth further research. We will include this part as future work in our revision."}]

Author rebuttal:
We are grateful to all five reviewers and AC/SACs for their valuable time, insightful comments, and useful suggestions. We will carefully revise our paper according to the comments. Our point-by-point response to the reviewers’ comments has been added to the individual chat box for each reviewer. We believe that the revised manuscript has been enhanced and the concerns have been well addressed. 

Due to the character limitation, citations mentioned in the rebuttal are included below. 

[1] Yang L, Huang Z, Song Y, et al. Diffusion-based scene graph to image generation with masked contrastive pre-training[J]. arXiv preprint arXiv:2211.11138, 2022.

[2] Wu J Z, Ge Y, Wang X, et al. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation[J]. arXiv preprint arXiv:2212.11565, 2022.

[3] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[6] Z. Chen, J. Qing, T. Xiang, W. L. Yue, and J. H. Zhou, “Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.


Meta-review: The paper has received positive feedback from the reviewers. There were some concerns that were addressed by the rebuttal. this paper presents a great contribution to the field of computational neuroscience. The authors are encouraged to address the comments in their final paper.
Decision: Accept (oral)