SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zijiao Chen*
zijiao.chen@u.nus.edu
Jiaxin Qing*
jqing@ie.cuhk.edu.hk
Juan Helen Zhou†
helen.zhou@nus.edu.sg
https://mind-video.com
Reconstructing human vision from brain activities has been an appealing task that helps to understand our cognitive process. Even though recent research has seen great success in reconstructing static images from non-invasive brain recordings, work on recovering continuous visual experiences in the form of videos is limited. In this work, we propose MinD-Video that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. We show that high-quality videos of arbitrary frame rates can be reconstructed with MinD-Video using adversarial guidance. The recovered videos were evaluated with various semantic and pixel-level metrics. We achieved an average accuracy of  $85\%$  in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by  $45\%$ . We also show that our model is biologically plausible and interpretable, reflecting established physiological processes.
Life unfolds like a film reel, each moment seamlessly transitioning into the next, forming a "perpetual theater" of experiences. This dynamic narrative forms our perception, explored through the naturalistic paradigm, painting the brain as a moviegoer engrossed in the relentless film of experience. Understanding the information hidden within our complex brain activities is a big puzzle in cognitive neuroscience. The task of recreating human vision from brain recordings, especially using non-invasive tools like functional Magnetic Resonance Imaging (fMRI), is an exciting but difficult task. Non-invasive methods, while less intrusive, capture limited information, susceptible to various interferences like noise [1]. Furthermore, the acquisition of neuroimaging data is a complex, costly process. Despite these complexities, progress has been made, notably in learning valuable fMRI features with limited fMRI-annotation pairs. Deep learning and representation learning have achieved significant results in visual class detections [2, 3] and static image reconstruction [4, 5, 6, 7, 8], advancing our understanding of the vibrant, ever-changing spectacle of human perception.
Unlike still images, our vision is a continuous, diverse flow of scenes, motions, and objects. To recover dynamic visual experience, the challenge lies in the nature of fMRI, which measures blood oxygenation level dependent (BOLD) signals and captures snapshots of brain activity every few seconds. Each fMRI scan
Figure 1. Brain decoding & video reconstruction. We propose a progressive learning approach to recover continuous visual experience from fMRI. High-quality videos with accurate semantics and motions are reconstructed.
essentially represents an "average" of brain activity during the snapshot. In contrast, a typical video has about 30 frames per second (FPS). If an fMRI frame takes 2 seconds, during that time, 60 video frames - potentially containing various objects, motions, and scene changes - are presented as visual stimuli. Thus, decoding fMRI and recovering videos at an FPS much higher than the fMRI's temporal resolution is a complex task.
Hemodynamic response (HR) [9] refers to the lags between neuronal events and activation in BOLD signals. When a visual stimulus is presented, the recorded BOLD signal will have certain delays with respect to the stimulus event. Moreover, the HR varies across subjects and brain regions [10]. Thus, the common practice that shifts the fMRI by a fixed number in time to compensate for the HR would be sub-optimal.
In this work, we present MinD-Video, a two-module pipeline designed to bridge the gap between image and video brain decoding. Our model progressively learns from brain signals, gaining a deeper understanding of the semantic space through multiple stages in the first module. Initially, we leverage large-scale unsupervised learning with masked brain modeling to learn general visual fMRI features. We then distill semantic-related features using the multimodality of the annotated dataset, training the fMRI encoder in the Contrastive Language-Image Pre-Training (CLIP) space with contrastive learning. In the second module, the learned features are fine-tuned through co-training with an augmented stable diffusion model, which is specifically tailored for video generation under fMRI guidance. Our contributions are summarized as follows:
- We introduced a flexible and adaptable brain decoding pipeline decoupled into two modules: an fMRI encoder and an augmented stable diffusion model, trained separately and finetuned together.
- We designed a progressive learning scheme where the encoder learns brain features through multiple stages, including multimodal contrastive learning with spatiotemporal attention for windowed fMRI.
- We augmented the stable diffusion model for scene-dynamic video generation with near-frame attention. We also designed adversarial guidance for distinguishable fMRI conditioning.
- We recovered high-quality videos with accurate semantics, e.g., motions and scene dynamics. Results are evaluated with semantic and pixel metrics at video and frame levels. An accuracy of  $85\%$  is achieved in semantic metrics and 0.19 in SSIM, outperforming the previous state-of-the-art approaches by  $45\%$ .
- The attention analysis revealed mapping to the visual cortex and higher cognitive networks, suggesting our model is biologically plausible and interpretable.
Image Reconstruction Image reconstruction from fMRI was first explored in [2], which showed that hierarchical image features and semantic classes could be decoded from the fMRI data collected when the participants were looking at a static visual stimulus. Authors in [5, 6] designed a separable autoencoder that enables self-supervised learning in fMRI and images to increase training data. Based on a similar philosophy, [7] proposed to perform self-supervised learning on a large-scale fMRI dataset using masked data modeling as a pretext task. Using a stable diffusion model as a generative prior and the pre-trained fMRI features as conditions, [7] reconstructed high-fidelity images with high semantic correspondence to the groundtruth stimuli.
Video Reconstruction The conventional method formulated the video reconstruction as multiple image reconstructions [11], leading to low frame rates and frame inconsistency. Nonetheless, [11] showed that low-level image features and classes can also be decoded from fMRI collected with video stimulus. Using
Figure 2. MinD-Video Overview. Our method has two modules that are trained separately and then finetuned together. The fMRI encoder progressively learns fMRI features through multiple stages, including MBM pre-training and multimodal contrastive learning. A spatiotemporal attention is designed to process multiple fMRI in a sliding window. The augmented Stable Diffusion is trained with videos and then tuned with the fMRI encoder using annotated data.
fMRI representations encoded with a linear layer as conditions, [12] generated higher quality and frame rate videos with a conditional video GAN. However, the results are limited by data scarcity, especially for GAN training, which generally requires a large amount of data. [13] took a similar approach as [5], which relied on the same separable autoencoder that enables self-supervised learning. Even though better results were achieved than [12], the generated videos were of low visual fidelity and semantic meanings. In addition, recent work by [14] utilizes contrastive learning to derive latent embeddings from mice neural data, enabling the generation of videos based on these learned representations.
MBM Masked brain modeling (MBM) is a pre-text task that enables self-supervised learning in a large fMRI dataset proposed in [7], aiming to build a brain foundation model. It learns general features of fMRI by trying to recover masked data from the remainings, similar to the GPT [15] and MAE [16], after which knowledge can be distilled and transferred to a downstream task with limited data and a few-step tuning [17, 15, 16, 7].
CLIP Contrastive Language-Image Pre-Training (CLIP) is a pre-training technique that builds a shared latent space for images and natural languages by large-scale contrastive learning [18]. The training aims to minimize the cosine distance of paired image and text latent while maximizing permutations of pairs within a batch. The shared latent space (CLIP space) contains rich semantic information on both images and texts.
Stable Diffusion Diffusion models are emerging probabilistic generative models defined by a reversible Markov chain [19, 20]. As a variant, stable diffusion generates a compressed version of the data (data latent) instead of generating the data directly. As it works in the data latent space, the computational requirement is significantly reduced, and higher-quality images with more details can be generated in the latent space [21]. Due to its high generative quality, it has been studied in various brain decoding tasks [7, 8].
Aiming for a flexible design, MinD-Video is decoupled into two modules: an fMRI encoder and a video generative model, as illustrated in Fig. 2. These two modules are trained separately and then finetuned together, which allows for easy adaption of new models if better architectures of either one are available. As a representation learning model, the encoder in the first module transfers the pre-processed fMRI into embeddings, which are used as a condition for video generations. For this purpose, the embedding should have the following traits: 1) It should contain rich and compact information about the visual stimulus presented during the scan. 2) It should be close to the embedding domain with which the generative model is trained. When designing the generative model, it is essential that the model produces not only diverse, high-quality videos with high computational efficiency but also handles potential scene transitions, mirroring the dynamic visual stimuli experienced during scans.
3.2 The fMRI Pre-processing
The fMRI captures whole-brain activity with BOLD signals (voxels). Each voxel is assigned to a region of interest (ROI) for focused analysis. Here, we concentrate on voxels activated during visual stimuli. There are two ways to define the ROIs: one uses a pre-defined parcellation such as [22] and [23] to obtain the visual cortex; the other relies on statistical tests to identify activated voxels during stimuli. Our large-scale pre-training is based on the parcellation in [22], while statistical tests are employed for the target dataset, i.e., the Wen (2018) [11] dataset containing fMRI-video pairs. To determine activated regions, we calculate intra-subject reproducibility of each voxel, correlating fMRI data across multiple viewings. The correlation coefficients are then converted to z-scores and averaged. We compute statistical significance using a one-sample t-test  $(\mathrm{P} < 0.01$ ,  $\mathrm{DOF} = 17$ , Bonferroni correction). We selected the top  $50\%$  of the most significant voxels after the statistical test. Notice that most of the identified voxels are from the visual cortex.
3.3 Progressive Learning and fMRI Encoder
Progressive learning is used as an efficient training scheme where general knowledge is learned first, and then more task-specific knowledge is distilled through finetuning [24, 25, 26]. To generate meaningful embeddings specific for visual decoding, we design a progressive learning pipeline, which learns fMRI features through multiple stages, starting from general features to more specific and semantic-related features. We will show that the progressive learning process is reflected biologically in the evolution of fMRI attention maps.
Large-scale Pre-training with MBM Similar to [7], a large-scale pre-training with masked brain modeling (MBM) is used to learn general features of the visual cortex. With the same setup, an asymmetric vision-transformer-based autoencoder [27, 16] is trained on the Human Connectome Project [28] with the visual cortex (V1 to V4) defined by [22]. Specifically, fMRI data of the visual cortex is rearranged from 3D into 1D space in the order of visual processing hierarchy, which is then divided into patches of the same size. The patches will be transformed into tokens, and a large portion ( $\sim 75\%$ ) of the tokens will be randomly masked in the encoder during training. With the autoencoder architecture, a simple decoder aims to recover the masked tokens from the unmasked token embeddings generated by the encoder. The main idea behind the MBM is that if the training objective can be achieved with high accuracy using a simple decoder, the token embeddings generated by the encoder will be a rich and compact description of the original fMRI data. We refer readers to [7] for detailed descriptions and reasonings of the MBM.
Spatiotemporal Attention for Windowed fMRI For the purpose of image reconstruction, the original fMRI encoder in [7] shifts the fMRI data by 6s, which is then averaged every 9 seconds and processed individually. This process only considers the spatial information in its attention layers. In the video reconstruction, if we directly map one fMRI to the video frames presented (e.g., 6 frames), the video reconstruction task can be formulated as a one-to-one decoding task, where each set of fMRI data corresponds to 6 frames. We call each {fMRI-frames} pair a fMRI frame window. However, this direct mapping is sub-optimal because of the time delay between brain activity and the associated BOLD signals in fMRI data due to the nature of hemodynamic response.
Thus, when a visual stimulus (i.e., a video frame) is presented at time  $t$ , the fMRI data obtained at  $t$  may not contain complete information about this frame. Namely, a lag occurs between the presented visual stimulus and the underlying information recorded by fMRI. This phenomenon is depicted in Fig. 3.
The hemodynamic response function (HRF) is usually used to model the relationship between neural activity and BOLD signals [29]. In an LTI system, the signal  $y(t)$  is represented as the convolution of
a stimulus function  $s(t)$  and the HR  $h(t)$ , i.e.,  $y(t) = (s*h)(t)$ . The  $h(t)$  is often modeled with a linear combination of some basis functions, which can be collated into a matrix form:  $Y = X\beta + e$ , where  $Y$  represents the observed data,  $\beta$  is a vector of regression coefficients, and  $e$  is a vector of unexplained error values. However,  $e$  varies significantly across individuals and sessions due to age, cognitive state, and specific visual stimuli, which influence the firing rate, onset latency, and neuronal activity duration. These variations impact the estimation of  $e$  and ultimately affect the accuracy of the fMRI-based analysis. Gener-
Figure 3. Due to hemodynamic response, the BOLD signal (blue) lags a few seconds behind the visual stimulus (red), causing a discrepancy between fMRI and the stimulus.
ally, there are two ways to address individual variations: using personalized HRF models or developing algorithms that adapt to each participant. We choose the latter due to its superior flexibility and robustness.
Aiming to obtain sufficient information to decode each scan window and account for the HR, we propose a spatiotemporal attention layer to process multiple fMRI frames in a sliding window. Consider a sliding window defined as  $\boldsymbol{x}_t = \{x_t, x_{t+1}, \dots, x_{t+w-1}\}$ , where  $x_t \in \mathbb{R}^{n \times p \times b}$  are token embeddings of the fMRI at  $t$  and  $n, p, b$  are the batch size, patch size, and embedding dimension, respectively. So we have  $\boldsymbol{x}_t \in \mathbb{R}^{n \times w \times p \times b}$ , where  $w$  is the window size. Recall that the attention is given by  $\mathrm{attn} = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)$ . To calculate spatial attention, we use the network inflation trick [30], where we merge the first two dimensions of  $\boldsymbol{x}_t$  and obtain  $\boldsymbol{x}_t^{spat} \in \mathbb{R}^{nw \times p \times b}$ . Then the query and key are calculated in Eq. (1) as
$$
Q = \boldsymbol {x} _ {t} ^ {\text {s p a t}} \cdot W _ {\text {s p a t}} ^ {Q}, \quad K = \boldsymbol {x} _ {t} ^ {\text {s p a t}} \cdot W _ {\text {s p a t}} ^ {K}. \tag {1}
$$
Likewise, we merge the first and the third dimension of  $\pmb{x}_t$  to calculate the temporal attention, obtaining  $\pmb{x}_t^{\text{temp}} \in \mathbb{R}^{np \times w \times b}$ . Again, the query and key are calculated in Eq. (2) with
$$
Q = \boldsymbol {x} _ {t} ^ {\text {t e m p}} \cdot W _ {\text {t e m p}} ^ {Q}, \quad K = \boldsymbol {x} _ {t} ^ {\text {t e m p}} \cdot W _ {\text {t e m p}} ^ {K}. \tag {2}
$$
The spatial attention learns correlations among the fMRI tokens, describing the spatial correlations of the fMRI patches. Then the temporal attention learns the correlations of fMRI from the sliding window, including sufficient information to cover the lag due to HR, as illustrated in the "fMRI Attention" in Fig. 2.
Multimodal Contrastive Learning Recall that the fMRI encoder is pre-trained to learn general features of the visual cortex, and then it is augmented with temporal attention heads to process a sliding window of fMRI. In this step, we further train the augmented encoder with {fMRI, image, caption} triplets and pull the fMRI embeddings closer to a shared CLIP space containing rich semantic information. Additionally, the generative model is usually pre-trained with text conditioning. Thus, pulling the fMRI embeddings closer to the text-image shared space ensures its understandability by the generative model during conditioning.
Firstly, videos in the training set are downsampled to a smaller frame rate (3FPS). Each frame is then captioned with BLIP [31], which creates the {image, text} pairs. With the CLIP text encoder and image encoder being fixed, the CLIP loss [18] is calculated for fMRI-image and fMRI-text, respectively. Denote the pooled text embedding, image embedding, and fMRI embedding by  $emb_t$ ,  $emb_i$ ,  $emb_f \in \mathbb{R}^{n \times b}$ . The contrastive language-image-fMRI loss is given by
$$
\mathcal {L} = \left(\mathcal {L} _ {\mathrm {C L I P}} \left(e m b _ {f}, e m b _ {t}\right) + \mathcal {L} _ {\mathrm {C L I P}} \left(e m b _ {f}, e m b _ {i}\right)\right) / 2, \tag {3}
$$
where  $\mathcal{L}_{\mathrm{CLIP}}(a,b) = \mathrm{CrossEntropy}(\epsilon a\cdot b^{\mathrm{T}},[0,1,\dots,n])$  with  $\epsilon$  being a scaling factor. Extra care is needed to reduce similar frames in a batch for better contrastive pairs. From Eq. (3), we see that the loss largely depends on the batch size  $n$ . Thus, a large  $n$  with data augmentation on all modalities is appreciated.
3.4 Video Generative Module
The Stable Diffusion model [21] is used as the base generative model considering its excellence in generation quality, computational requirements, and weights availability. However, as the stable diffusion model is an image-generative model, temporal constraints need to be applied in order for video generation.
Scene-Dynamic Sparse Causal (SC) Attention Authors in [30] use a network inflation trick with sparse temporal attention to adapt the stable diffusion to a video generative model. Specifically, the sparse temporal attention effectively conditions each frame on its previous frame and the first frame, which ensures frame consistency and also keeps the scene unchanged. However, the human vision consists of possible scene changes, so the video generation should also be scene-dynamic. Thus, we relax the constraint in [30] and condition each frame on its previous two frames, ensuring the video smoothness while allowing scene dynamics. Using notations from [30], the SC attention is calculated with the query, key, and value given by
$$
Q = W ^ {Q} \cdot z _ {v _ {i}}, \quad K = W ^ {K} \cdot \left[ z _ {v _ {i - 2}}, z _ {v _ {i - 1}} \right], \quad V = W ^ {V} \cdot \left[ z _ {v _ {i - 2}}, z _ {v _ {i - 1}} \right], \tag {4}
$$
where  $z_{v_i}$  denotes the latent of the  $i$ -th frame during the generation.
Adversarial Guidance for fMRI Classifier-free guidance is widely used in the conditional sampling of diffusion models for its flexibility and generation diversity, where the noise update function is given by
$$
\hat {\epsilon} _ {\theta} \left(z _ {t}, c\right) = \epsilon_ {\theta} \left(z _ {t}\right) + s \left(\epsilon_ {\theta} \left(z _ {t}, c\right) - \epsilon_ {\theta} \left(z _ {t}\right)\right), \tag {5}
$$
where  $c$  is the condition,  $s$  is the guidance scale and  $\epsilon_{\theta}(\cdot)$  is a score estimator implemented with UNet [32]. Interestingly, Eq. (5) can be changed to an adversarial guidance version [21]
$$
\hat {\epsilon} _ {\theta} \left(z _ {t}, c, \bar {c}\right) = \epsilon_ {\theta} \left(z _ {t}, \bar {c}\right) + s \left(\epsilon_ {\theta} \left(z _ {t}, c\right) - \epsilon_ {\theta} \left(z _ {t}, \bar {c}\right)\right), \tag {6}
$$
where  $\bar{c}$  is the negative guidance. In effect, generated contents can be controlled through "what to generate" (positive guidance) and "what not to generate" (negative guidance). When  $\bar{c}$  is a null condition, the noise update function falls back to Eq. (5), the regular classifier-free guidance. In order to generate diverse videos for different fMRI, guaranteeing the distinguishability of the inputs, we average all fMRI in the testing set and use the averaged one as the negative condition. Specifically, for each fMRI input, the fMRI encoder will generate an unpooled token embedding  $x\in \mathbb{R}^{l\times b}$ , where  $l$  is the latent channel number. Denote the averaged fMRI as  $\bar{x}$ . We have the noise update function  $\hat{\epsilon}_{\theta}(z_t,x,\bar{x}) = \epsilon_{\theta}(z_t,\bar{x}) + s(\epsilon_{\theta}(z_t,x) - \epsilon_{\theta}(z_t,\bar{x}))$ .
Divide and Refine With a decoupled structure, two modules are trained separately: the fMRI encoder is trained in a large-scale dataset and then tuned in the target dataset with contrastive learning; the generative module is trained with videos and captions from the target dataset. In the second phase, two modules are tuned together with fMRI-video pairs, where the encoder and part of the generative model are trained. Different from [30], we tune the whole self-attention, cross-attention, and temporal-attention heads instead of only the query projectors, as a different modality is used for conditioning. The second phase is also the last stage of encoder progressive learning, after which the encoder finally generates token embeddings that contain rich semantic information and are easy to understand by the generative model.
3.5 Learning from the Brain - Interpretability
Our objectives extend beyond brain decoding and reconstruction. We also aim to understand the biological principles of the decoding process. To this end, we visualize average attention maps from the first, middle, and last layers of the fMRI encoder across all testing samples. This approach allows us to observe the transition from capturing local relations in early layers to recognizing more global, abstract features in deeper layers [33]. Additionally, attention maps are visualized for models in different learning stages: large-scale pre-training, contrastive learning, and co-training. By projecting attention back to brain surface maps, we can easily visualize each brain region's contributions and the learning progress through each stage.
Pre-training dataset Human Connectome Project (HCP) 1200 Subject Release [28]: For our upstream pre-training dataset, we employed resting-state and task-evoked fMRI data from the HCP. Building upon [7], we obtained 600,000 fMRI segments from a substantial amount of fMRI scan data.