SlowGuess/NeurIPS_2023_OCR huggingface dataset:
During the training, we update the self-attention heads ("attn1"), cross-attention heads ("attn2"), and temporal attention heads ("attn_temp"). The training is performed with text conditioning for 800 steps. We use a learning rate  $2 \times 10^{-5}$  and a batch size of 14. The training takes around 2 hours using one RTX3090. Visual results show that videos of high quality can be generated with text conditioning after this step.
Co-training The fMRI encoder produces embeddings of dimensions  $77 \times 768$ , which are used to condition the augmented stable model during co-training. The whole fMRI encoder is updated, and only part of the stable diffusion is updated (same as the last step). The training is performed with a batch size of 9 and a learning rate of  $3 \times 10^{-5}$  for 1,5000 steps. The training takes around 16 hours using one RTX3090.
Inference All samples are generated with 200 diffusion steps using fMRI adversarial guidance. The fMRI adversarial guidance uses an average fMRI as the negative guidance with a guidance scale of 12.5.
C Analysis of Visual Results
We test on all three subjects in Wen (2018) dataset. Around 6000 voxels are identified as ROI for Subject 1 and 2, while around 3000 voxels are identified for Subject 3. Thus, a larger batch size can be used when training with Subject 3, which may be the reason for its better numeric evaluation results. Nonetheless, all three subjects show consistent generation results. Some samples are shown in Fig. C.1.
Figure C.1. Samples from different subjects.
Visual results of the ablation studies are shown in Fig. C.2. The Full model is trained with our full pipeline and inference with adversarial guidance. In contrastive learning ablation, we tested with incomplete modality, namely, image-fMRI and text-fMRI, respectively. Similar to the numeric evaluations, using incomplete contrastive gave an unsatisfactory visual result compared to using all three modalities. However, incomplete modality still outperformed inferencing without adversarial guidance significantly, which generated visually meaningless results.
Some fail cases are shown in Fig. C.3. It is observed that even though some fail cases generated different animals and objects compared to the groundtruth, other semantics like the motions, color, and scene dynamics can still be correctly reconstructed. For example, although the airplane and flying bird are not reconstructed, similar fast-motion scenes are recovered in Fig. C.3.
D More Ablation Studies
Other than the ablation studies shown in Tab. 1, we include extra ablation studies in Tab. D.2 to demonstrate the effectiveness of masked brain modeling (MBM) in our framework. Two experiments are performed.
Figure C.2. Reconstruction samples for ablation studies. The Full model uses full modality contrastive learning with adversarial guidance.
The first experiment involves excluding the MBM pre-training, while the second experiment removes both the MBM pre-training and the contrastive training from the proposed method.
The results show the importance of MBM and contrastive training for all metrics. The performance drops as much as  $55\%$  without both components and  $30\%$  without only MBM. The visual quality of the generated videos also follows a similar trend as in Fig. C.2.
50-Way, Top-1 Accuracy	50-Way, Top-1 Accuracy
Image Identification Tests	Video Identification Tests	SSIM
Full Model	0.172±0.01	0.202±0.02	0.171
w/o MBM, w/ Contrastive	0.122±0.012	0.169±0.015	0.143
w/o MBM, w/o Contrastive	0.076±0.008	0.138±0.013	0.123
Table D.2. Extra ablation studies on masked brain modeling.
E More Evaluation Results
Limited by the publicly available video samples and codes for reproduction in the literature, we had some difficulties in comparing numerically with other methods. In the previous literature, Kupershmidt, 2022 [13] and Wen, 2018 [11] have only released part of their reconstructed videos, while Wang, 2022 [12] have not released their reconstructed videos. Also, because their released videos are based on different frame rates from ours (1Hz in [13], 0.5Hz in [11], 3Hz in ours), we did not perform the identification tests on those videos, as it might not be a fair comparison. Instead, we relied on a commonly available
Figure C.3. Fail cases.
metric reported in their papers, SSIM. This metric was calculated quantitatively across all testing samples according to their papers.
Nevertheless, in this section, we perform the identification test on those incomplete samples only (released by previous work) and compare the results across different methods (including ours). Please see Tab. E.3. Our method outperformed the two previous methods in identification test accuracy.
50-Way, Top-1 Accuracy	50-Way, Top-1 Accuracy
Image Identification Tests	Video Identification Tests
Ours	0.195±0.016	0.265±0.02
Kupershmidt, 2022	0.179±0.017	0.238±0.02
Wen, 2018	0.07±0.01	0.166±0.016
Table E.3. Comparison of identification Tests with previous works.
F Source of Errors
In short, the failure cases can be attributed by two factors:
- Lack of pixel-level controllability. Due to the probabilistic nature of the diffusion model and the current conditioning method, the generation process lacks strong control from the fMRI latent to generate strictly matching low-level features, such as shapes, color, and geometric information. We believe this would be an important perspective for future research on this task.
- Uncontrollable factors during the scan. Mind wandering and imagination of the subject are usually inevitable during the scan. It has been shown that imagination is involved and can be decoded to some extent from the visual cortex, which can lead to mismatching between the ground truth and the generation results.