SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Paired fMRI-Video dataset A publicly available benchmark fMRI-video dataset [11] was used, comprising fMRI and video clips. The fMRI were collected using a 3T MRI scanner at a TR of 2 seconds with three subjects. The training data included 18 segments of 8-minute video clips, totaling 2.4 video hours and yielding 4,320 paired training examples. The test data comprised 5 segments of 8-minute video clips, resulting in 40 minutes of test video and 1,200 test fMRIs. The video stimuli were diverse, covering animals, humans, and natural scenery, and featured varying lengths at a temporal resolution of 30 FPS.
The original videos are downsampled from 30 FPS to 3 FPS for efficient training and testing, leading to 6 frames per fMRI frame. In our implementation, we reconstruct a video of 2 seconds (6 frames) from a sliding window of fMRI frames. However, thanks to the spatiotemporal attention head design that encodes multiple fMRI at once, our method can reconstruct longer videos from more fMRI frames if more GPU memory is available.
A ViT-based fMRI encoder with a patch size of 16, a depth of 24, and an embedding dimension of 1024 is used. After pre-training with a mask ratio of 0.75, the encoder will be augmented with a projection head that projects the token embedding into the dimension of  $77 \times 768$ . The Stable Diffusion V1-5 [21] trained at the resolution of  $512 \times 512$  is used. But we tune the augmented Stable Diffusion for video generations at the resolution of  $256 \times 256$  with 3 FPS. Notice that the FPS and image are downsampled for efficient
experiments, and our method can also work with full temporal and spatial resolution. All parameters in the fMRI encoder pre-training are the same as [7] with eight RTX3090, while other stages are trained with one RTX3090.
The fMRI encoder pre-training on a large-scale dataset is the most resource-consuming part, but only one pre-training is needed for different subjects. This is in contrast to the other downstream stages. Multimodal contrastive learning and co-training are performed for each subject individually due to individual variability. The inference is performed with 200 DDIM [34] steps. See Supplementary for more details.
Following prior studies, we utilize both frame-based and video-based metrics. Frame-based metrics evaluate each frame individually, providing a snapshot evaluation, whereas video-based metrics assess sequences of frames, encapsulating the dynamics across frames. Both are used for a comprehensive analysis. Unless stated otherwise, all test set videos are used for evaluating the three subjects.
Frame-based Metrics Our frame-based metrics are divided into two classes, pixel-level metrics and semantics-level metrics. We use the structural similarity index measure (SSIM) [35] as the pixel-level metric and the N-way top-K accuracy classification test as the semantics-level metric. Specifically, for each frame in a scan window, we calculate the SSIM and classification test accuracy with respect to the groundtruth frame. To perform the classification test, we basically compare the classification results of the groundtruth (GT) and the predicted frame (PF) using an ImageNet classifier. If the GT class<sup>3</sup> is within the top-K probability of the PF classification results from N randomly picked classes, including the GT class, we declare a successful trial. The test is repeated for 100 times, and the average success rate is reported.
Video-based Metric The video-based metric measures the video semantics using the classification test as well, except that a video classifier is used. The video classifier based on VideoMAE [36] is trained on Kinetics-400 [37], an annotated video dataset with 400 classes, including motions, human interactions, etc.
We compare our method against three fMRI-video baselines: Wen et al. (2018) [11], Wang et al. (2022) [12] and Kupershmidt et al. (2022) [13]. Visual comparisons are shown in Fig. 4, and quantitative comparisons are shown in Fig. 6, where publicly available data and samples are used for comparison. As shown, we generate high-quality videos with more semantically meaningful content. Following the literature, we evaluate the SSIM of Subject 1 with the first testing video, achieving a score of 0.186, outperforming the state-of-the-art by  $45\%$ . When comparing with Kupershmidt et al. (2022), we evaluate all test videos for different subjects, and our method outperforms by  $35\%$  on average, as shown in Fig. 6. Using semantic-level metrics, our method achieves a success rate of 0.849 and 0.2 in 2-way and 50-way top-1 accuracy classification tests, respectively, with the video classifier. The image classifier gives a success rate of 0.795 and 0.19, respectively, with the same tests, which significantly surpasses the chance level of these two tests (2-way: 0.5, 50-way: 0.02). Full results are shown in Tab. 1. We also compare our results with an image-fMRI model by Chen et al. (2023) [7]. An image is produced for each fMRI, samples of which are shown in Fig. 4 with the "walking man" as the groundtruth. Even though the results and groundtruth are semantically matching, frame consistency and image quality are not satisfying. A lag due to the hemodynamic response is also observed. The first frame actually corresponds to the previous groundtruth.
Different Subjects After fMRI pre-processing, each subject varies in the size of ROIs, where Subject 1, 2 and 3 have 6016, 6224, and 3744 voxels, respectively. As Subject 3 has only half the voxels of the others, a larger batch size can be used during contrastive learning, which may lead to better results as shown in Tab. 1. Nonetheless, all subjects are consistent in both numeric and visual evaluations (See Supplementary).
Recovered Semantics In the video reconstruction, we define the semantics as the objects, animals, persons, and scenes in the videos, as well as the motions and scene dynamics, e.g., people running, fast-moving scenes, close-up scenes, long-shot scenes, etc. We show that even though the fMRI has a low temporal resolution, it contains enough information to recover the mentioned semantics. Fig. 5 shows a few examples of reconstructed frames using our method. Firstly, we can see that the basic objects, animals, persons, and scene types can be well recovered. More importantly, the motions, such as running, dancing, and singing,
<sup>3</sup>Since video frames usually cannot be well described with a single class, we use the top-3 classification results as the GT class, and a successful event is declared if the test succeeds with any of the GT class.
Figure 4. Compare with Benchmarks. We compare our results with the samples provided in the previous literature. Our method generates samples that are more semantically meaningful and match with the groundtruth.
Figure 5. Reconstruction Diversity. Various motions, scenes, persons, and animals can be correctly recovered. A sample with a scene transition is shown on the bottom right.
Table 1. Ablation study on window sizes, multimodal contrastive learning, and adversarial guidance (AG). Evaluations on different subjects are also shown. Full Model: win=2, Sub 1. Colors reflect statistical significance (two-sample t-test) compared to the Full Model.  $p < {0.0001}$  (purple);  $p < {0.01}$  (pink);  $p < {0.05}$  (yellow);  $p > {0.05}$  (green)
Video-based	Frame-based
Semantic-level	Semantic-level	Pixel-level
2-way↑	50-way↑	2-way↑	50-way↑	SSIM↑
Full Model	0.853±0.03	0.202±0.02	0.792±0.03	0.172±0.01	0.171
Window Size = 1	0.851±0.03	0.195±0.02	0.759±0.03	0.165±0.01	0.169
Window Size = 3	0.826±0.03	0.161±0.01	0.765±0.03	0.137±0.01	0.161
w/o Contrastive	0.844±0.03	0.157±0.02	0.750±0.03	0.088±0.07	0.135
Text-fMRI Contra	0.839±0.03	0.185±0.01	0.78±0.03	0.154±0.01	0.164
Img-fMRI Contra	0.845±0.03	0.189±0.01	0.783±0.03	0.151±0.01	0.164
w/o AG	0.859±0.03	0.198±0.02	0.775±0.03	0.117±0.01	0.152
Subject 2	0.841±0.03	0.173±0.02	0.784±0.03	0.158±0.13	0.171
Subject 3	0.846±0.03	0.216±0.02	0.812±0.03	0.193±0.01	0.187
and the scene dynamics, such as the close-up of a person, the fast-motion scenes, and the long-shot scene of a city view, can also be reconstructed correctly. This result is also reflected in our numerical metrics, which consider both frame semantics and video semantics, including various categories of motions and scenes.
Abalations We test our method using different window sizes, starting from a window size of 1 up to 3. When the window size is one, the fMRI encoder falls back to a normal MBM encoder in [7]. Tab. 1 shows that when all other parameters are fixed, a window size of 2 gives the best performance in general, which is reasonable as the hemodynamic response usually will not be longer than two scan windows. Additionally, we also test the effectiveness of multimodal contrastive learning. As shown in Tab. 1, without contrastive
Figure 6. SSIM Comparison. Left: comparison with Kupershmidt et al. (2022) [13] and Wang et al. (2022) [12] on commonly available test samples (Subject 1, test video 1). Right: comparison with Kupershmidt et al. (2022) on all subjects, all test videos.
learning, the generation quality degrades significantly. When two modalities are used, either text-fMRI or image-fMRI, the performance is inferior to the full modalities used in contrastive learning. Actually, the reconstructed videos are visually worse than the full model (See Supplementary). Thus, it shows that the full progressive learning pipeline is crucial for the fMRI encoder to learn useful representations for this task. We also assess the reconstruction results without adversarial guidance. As a result, both numeric and visual evaluations decrease substantially. In fact, the generated videos can be highly similar sometimes, indicating that the negative guidance is critical in increasing the distinguishability of fMRI embeddings. See the Supplementary for more ablation studies and visual results.
5.1 Interpretation Results
We summarize the attention analysis in Fig. 7. We present the sum of the normalized attention within Yeo17 networks [23] in the bar charts. Voxel-wise attention value is displayed on a brain flat map, where we see comprehensive structural attention throughout the whole region. The average attention across all testing samples and attention heads is computed, revealing three insights into how transformers decode fMRI data.
Dominance of visual cortex: The visual cortex emerges as the most influential region. This region, encompassing both the central (VisCent) and peripheral visual (VisPeri) fields, consistently attracts the highest attention across different layers and training stages (shown in Fig. 7B). In all cases, the visual cortex is always the top predictor, which aligns with prior research, emphasizing the vital role of the visual cortex in processing visual spatiotemporal information[38]. However, the visual cortex is not the sole determinant of vision. Higher cognitive networks, such as the dorsal attention network (DorsAttn) involved in voluntary visuospatial attention control [39], and the default mode network (Default) associated with thoughts and recollations [40], also contribute to visual perceptions process [41] as shown in Fig. 7.
Layer-dependent hierarchy: The layers of the fMRI encoder function in a hierarchical manner, as shown in Fig. 7C, D and E. In the early layers of the network (panel A & C), we observe a focus on the structural information of the input data, marked by a clear segmentation of different brain regions by attention values, aligning with the visual processing hierarchy [42]. As the network dives into deeper layers (panel D & E), the learned information becomes more dispersed. The distinction between regions diminishes, indicating a shift toward learning more holistic and abstract visual features in deeper layers.
Learning semantics progressively: To illustrate the learning progress of the fMRI encoder, we analyze the first-layer attention after all learning stages, as shown in panel B: before contrastive learning, after contrastive learning, and after co-training with the video generation model. We observe an increase in attention in higher cognitive networks and a decrease in the visual cortex as learning progresses. This indicates the encoder assimilates more semantic information as it evolves through each learning stage, improving the learning of cognitive-related features in the early layers.
Conclusion We propose MinD-Video, which reconstructs high-quality videos with arbitrary frame rates from brain activities represented by fMRI signals. Starting from large-scale pre-training to multimodal
Figure 7. Attention visualization. We visualized the attention maps for different transformer layers (C, D, E) and learning stages (B) with bar charts and brain flat maps. Brain networks are marked on a brain surface map (A). Normalized attention values are shown.
contrastive learning with augmented spatiotemporal attention, our fMRI encoder learns features from a sliding time window of fMRI progressively. Then, we finetune an augmented stable diffusion for video generations, which is then co-trained with the fMRI encoder. Finally, we show that with fMRI adversarial guidance, MinD-Video recovers videos with accurate semantics, motions, and scene dynamics compared with the groundtruth, establishing a new state-of-the-art in this domain. We also show from attention maps that the trained model decodes fMRI with reliable biological principles.
Limitations Even though our method reconstructs videos with matching semantics and scenes, the pixel-level accuracy is not satisfactory, as shown in visual samples (more in the Supplementary). We believe that the results can be improved by introducing pixel-level information to control the diffusion denoising process, in addition to the visual semantic information decoded from the brain. Multi-scale information from the brain recordings can be fused to provide a more detailed visual description for generations.
Another limitation lies in the inter-subject generalization ability. Our method is still within the intra-subject level, and the inter-subject generalization ability remains unexplored due to individual variations. Additionally, our method only uses less than  $10\%$  of the voxels from the cortex for reconstructions, while using the whole brain data remains unexploited.
Broader Impacts We believe that brain decoding has promising applications in brain-computer interfaces as large models develop, and it will be a non-negligible component of the study to understand our cognitive process. However, governmental regulations and efforts from research communities are required to ensure the privacy of one's biological data and avoid any malicious usage of this technology.
This work was supported in part by National Medical Research Council, Singapore (NMRC/OFLCG19May0035 to J-H Zhou), RIE2020 AME Programmatic Grant from A\*STAR (to J-H Zhou), Ministry of Education Tier 2 Grant and Yong Loo Lin School of Medicine Research Core Funding (to J-H Zhou), National University of Singapore, Singapore.
[42] D. C. Van Essen and J. H. Maunsell, "Hierarchical organization and functional streams in the visual cortex," Trends in neurosciences, vol. 6, pp. 370-375, 1983.
A Q&A Session
Where does the performance gain come from? Is it driven by the diffusion model only? The performance gain in our model is not solely driven by the diffusion model. It is, in fact, a result of the combined efforts of two major stages: the fMRI encoder and the stable diffusion model.
In the first stage, the fMRI encoder plays a crucial role in learning representations from the brain. It effectively captures the complex spatiotemporal information embedded in the fMRI data, allowing the model to understand and interpret the underlying neural activities. This step is particularly important as it forms the foundation of our model and significantly influences the subsequent steps.
In the second stage, the stable diffusion model steps in to generate videos. One of the key advantages of our stable diffusion model over other generative models, such as GANs, lies in its ability to produce higher-quality videos. It leverages the representations learned by the fMRI encoder and utilizes its unique diffusion process to generate videos that are not only of superior quality but also better align with the original neural activities.
What does the fMRI encoder learn? Why don't we train an fMRI to object classifier, followed by a generation model? The fMRI encoder is designed to learn intricate representations from brain activity. These representations go beyond simple categorical information to encompass more nuanced semantic details that can't be adequately captured by discrete class labels (e.g. image texture, depth, etc.). Due to the richness and diversity of human thought and perception, a model that can handle continuous semantics, rather than discrete ones, is necessary.
The proposal to train an fMRI-to-object classifier followed by a generation model does not align with our goal of comprehensive brain decoding. This is largely due to a crucial trade-off between classification complexity and solution space:
- Classification Complexity: Classifying fMRI data into a large number of classes (e.g., 1000 classes) is non-trivial. As reported in [2], reasonable performance can only be achieved in a smaller classification task (less than 50-way), due to the limited data per category and the complexity of the task.
- Limited Solution Space: The solution space of discrete classes is significantly more restricted than that of continuous semantics. Thus, a classifier may not capture the complex, multi-faceted nature of brain activities.
This trade-off illustrates why a classifier might not be the best approach for this task. In contrast, our proposed method focuses on learning continuous semantic representations from the brain, which better reflects the complexity and diversity of neural processes. This approach not only improves the quality of the generated videos but also provides more meaningful and interpretable insights into brain decoding.
Is the fMRI-video generation pipeline simply imputing missing frames in a sequence of static images based on the fMRI-image generation pipeline? No, the fMRI-to-video generation process is not merely an imputation on the fMRI-to-image generation pipeline. While both involve generating visual content based on fMRI data, the tasks and their complexities are fundamentally different.
The fMRI-to-image generation involves mapping brain activity to a static image. This task primarily focuses on spatial information, that is, which brain regions are active and how they relate to elements in an image.
In contrast, the fMRI-to-video generation task involves mapping brain activity to dynamic videos. This task is considerably more complex as it requires the model to capture both spatial information and temporal dynamics. It's not just about predicting which brain regions are active, but also about understanding how these activations change over time and how they relate to moving elements in a video.
Adding to the complexity is the hemodynamic response inherent in fMRI data, which introduces a delay and blur in the timing of neural activity. This necessitates careful handling of the temporal aspects in the data. Furthermore, the temporal resolution of fMRI is quite low, making it challenging to capture fast-paced changes in neural activity.
We also use a stable diffusion process as our generative model, which is a probabilistic model. This means that the generation process involves a degree of randomness, leading to slight differences during each generation for video frames. Additionally, in video generation, we need to ensure consistency across video frames, which adds another layer of complexity.
What's the major difference between Mind-Video and Mind-Vis [7]? Our work extends beyond Mind-Vis by tackling the fMRI-based video reconstruction problem. Unlike image reconstruction, this adds another level of complexity. The key differences can be summarized in the following.
- Problem formulation: In Mind-Vis, dynamic fMRI recordings are averaged to create a "snapshot". While in this work, dynamic fMRI time series is directly used to recover a video, which requires considering the spatial features and the temporal features of fMRI. Additionally, the hemodynamic response is a significant challenge in our work, making the one-to-one mapping between fMRI and video even more difficult.
- Architecture: To address the unique challenges, we made two key improvements. First, we enhanced the fMRI encoder to handle a sliding time window of fMRI, capturing spatial and temporal information with distinct attention heads. Second, we employed multimodal contrastive learning to align fMRI with the semantic space of text and images before the co-training. This contrasts Mind-Vis in which co-training was performed directly without contrastive learning.
Do all trainable models require re-training for each subject? How computationally demanding is it? How is the training process repeated for each subject independently? Not all models necessitate re-training for every individual. The most resource-intensive phase is the large-scale pretraining of the MBM encoder, which remains consistent across subjects and does not demand re-training. Nevertheless, we do finetune both the MBM Encoder and the generative model using the fMRI data specific to each subject. This fine-tuning step, however, is relatively lightweight in terms of computational demands.
Large-Scale Pre-training The large-scale pre-training uses the same setup as the MBM described in [7]. A ViT-large-based model with a 1-dimensional patchifier is trained with hyperparameters shown in Tab. B.1. The training takes around 3 days using 8RTX3090 GPUs. The training is performed on the 600,000 fMRI from HCP. Same as the literature, after the large-scale pre-training, the autoencoder is tuned with fMRI data from the target dataset, namely, Wen (2018), using MBM as well. The tuning is performed using a small learning rate and epochs.
parameter	value	parameter	value	parameter	value	parameter	value
patch size	16	encoder depth	24	decoder embed dim	512	clip gradient	0.8
embedding dim	1024	encoder heads	16	learning rate	2.5e-4	weight decay	0.05
mask ratio	0.75	decoder depth	8	warm-up epochs	40	batch size	500
mlp ratio	1.0	decoder heads	16	epochs	500	optimizer	AdamW
Table B.1. Hyperparameters used in the large-scale pre-training
Multimodal Contrastive Learning In this step, we will take the pre-trained fMRI encoder from the previous step and augment it with temporal attention heads to accommodate multiple fMRI. Then contrastive learning is performed with fMRI-image-text triplets. The image is a randomly-picked frame from an fMRI scan window. As mentioned, there are two important factors in the contrastive: batch size and data augmentations. Therefore, data augmentations are applied for all modalities. Random sparsification is used for fMRI, where  $20\%$  of voxels are randomly set to zeros each time. The random crop is applied to videos with a probability of 0.5. To augment the frame captions, we apply synonym augmentation and random word swapping. Due to a small dataset size ( $\sim 4000$ ), we use a dropout rate of 0.6 to avoid overfitting. For Subject 1 and 2, the training is performed with a batch size of 20, while a batch size of 32 is used due to fewer fMRI voxels with Subject 3. Training for all subjects is performed for 1,2000 steps with a learning rate of  $2 \times 10^{-5}$ . The training takes around 10 hours using one RTX3090.
Training of Augmented Stable Diffusion The stable diffusion model is augmented with temporal attention heads for video generation. We train the augmented stable diffusion model with videos from the target datasets. The videos are downsampled from 30 FPS to 3 FPS at a resolution of  $256 \times 256$  due to limited
GPU memory, even though our model can work with the full time resolution. This step is important for two reasons: 1) the augmented temporal heads are untrained; 2) the stable diffusion is pre-trained at a resolution of  $512 \times 512$ , so we need to adapt it to a lower resolution.