Animate Your Thoughts: Reconstruction of Dynamic Natural Vision from Human Brain Activity
Authors: Yizhuo Lu
Changde Du
Chong Wang
Xuanliu Zhu
Liuyun Jiang
Xujin Li
Huiguang He
Venue: ICLR 2025 Poster (2025)
Keywords: Video reconstruction
Brain-computer Interface (BCI).
Primary area: applications to neuroscience & cognitive science
TL;DR: We introduce a video reconstruction model that decouples semantic, structural, and motion information from fMRI. We mitigate the interference of external video data on motion information decoding through a rational experimental design.

Abstract: Reconstructing human dynamic vision from brain activity is a challenging task with great scientific significance.  Although prior video reconstruction methods have made substantial progress, they still suffer from several limitations, including: (1) difficulty in simultaneously reconciling semantic (e.g. categorical descriptions), structure (e.g. size and color), and consistent motion information (e.g. order of frames); (2) low temporal resolution of fMRI, which poses a challenge in decoding multiple frames of video dynamics from a single fMRI frame; (3) reliance on video generation models, which introduces ambiguity regarding whether the dynamics observed in the reconstructed videos are genuinely derived from fMRI data or are hallucinations from generative model. To overcome these limitations,  we propose a two-stage model named Mind-Animator. During the fMRI-to-feature stage, we decouple semantic, structure, and motion features from fMRI. Specifically, we employ fMRI-vision-language tri-modal contrastive learning to decode semantic feature from fMRI and design a sparse causal attention mechanism for decoding multi-frame video motion features through a next-frame-prediction task. In the feature-to-video stage, these features are integrated into videos using an inflated Stable Diffusion, effectively eliminating external video data interference.  Extensive experiments on multiple video-fMRI datasets demonstrate that our model achieves state-of-the-art performance. Comprehensive visualization analyses further elucidate the interpretability of our model from a neurobiological perspective.  Project page: https://mind-animator-design.github.io/.

Published as a conference paper at ICLR 2025
ANIMATE YOUR THOUGHTS: RECONSTRUCTION OF DYNAMIC NATURAL VISION FROM HUMAN BRAIN AC- TIVITY
Yizhuo Lu Changde Du'st, Chong Wang', Xuanliu Zhu Liuyum Jiang
Xujin Li and
Huiguang He Hel2.3 1,2,3*
Key Laboratory of Brain Cogmition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences
School of Future Technology, University of Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences School of Computer and Artificial Intelligence, Zhengzhou University Beijing University of Posts and Telecommunications
Equaa Contribution
Corresponding Author
(luyizhuo2023,
changde du, huiguang. he @ia...a. acc.ccc..
ABSTRACT
Reconstructing human dynamic vision from brain activity is challenging task with great scientific significance. Although prior video reconstruction methods have made substantial progress, they still suffer from several limitations, including: (1) difficulty in simultaneously reconciling semantic (e.g. categorical descriptions), structure (e.g. size and color), and consistent motion information (e.g. order of frames); (2) low temporal resolution of fMRI, which poses a challenge in decoding multiple frames of video dynamics from single fMRI frame; (3) reliance video generation models, which introduces ambiguity regarding whether the dynamics observed the reconstructed videos are genuinely derived from fMRI data or are hallucinations from generative model. To overcome these limitations, we propose a two-stage model named Mind-Animator, During the iMRI-to-feature stage, we decouple semantic, structure, and motion features from IMRI. Specifically, we employ IMRI- MuRl-viiion-llanuaae tri-modal contrastive learning to decode semantic feature from fMRI and design sparse causal attention mechanism for decoding multi-frame video motion features through a next-frame ceenee-rediction task. the caature-to-vides stage, these features are integrated into videos using inflated Stable Diffusion, effectively eliminating external video data interference, Extensive experiments on multiple video-fMRI datasets demonstrate that model achieves state-of-the-ee performance. Comprehensive visualization analyses further eluci- date the interpretability of our model from neurobiological perspective. Project page: https: htt.///iinn/////////i////// animat or des ing. gn. g tttthhib.... io/.
INTRODUCTION
Advances in sensory neuroscience offer new perspectives brain function and could enhance artificial intelligence development (Palazzo al. (2021); Yargholi & Hossein-Zadeh (2016). One of the critical aspects to the research is neural decoding, which links visual stimuli to corresponding functional magnetic resonance imaging (IMRI) brain recordings. Neural decoding methods include classification, identification, and reconstruction, with this study focusing the most challenging aspect: reconstruction.
Prior methods have made significant strides in the classification (Yargholi & Hossein-Zadeh (2016); Horikawa & Kamitani (2017); Fujiwara et al. (2013)) and identification (Kay et al. (2008); Wildgruber al. (2005) of static stimulus images. Remarkably, some researchers have advanced to the point

---

Published as a conference paper at ICLR 2025
where they can reconstruct (Naselaris al. (2009); Van Gerven et al. (2010); Chen et al. (2023); Takagi & Nishimoto (2023); Ozcelik et al. (2022); Beliy et al. (2019)) images from brain signals that closely resemble the original stimulus images. In reality, the majority of visual stimuli we encounter in daily life are continuous and dynamic, hence there is a growing interest in reconstructing video from brain signals. Building on previous work that decoupled semantic and structural information from IMRI to reconstruct images (Scotti et al. (2024); Lu et al. (2023); Fang et al. (2020)), we argue that when the visual stimulus shifts from static images dynamic videos, as shown in Figure is crucial to account for three dimensions: semantic, structural, and motion, considering the brain's s processing of dynamic visual information.
Due to the inherent nature of fMRI, which relies on the slow blood oxygenation level dependent (BOLD) signal (Logothetis (2002); Kim Ogawa (2012)), neural activity integrated over a period exceeding 10 seconds ( 300 video frames). This integration delay poses a fundamental challenge in eapturing rapid motion dynamies. Consequently, the task of reconstructing videos from fMRI signals becomes exceedingly challenging.
To address this challenge, Nishimoto et al. (2011) transforms the video re- construction task into an identification task, employing the -anioy model (Adelson & Bergen (1985)) and Bayesian inference videos from a predefined video library. Subsequently, Han al. (2019), Wen et al. (2018) and Wang et al. (2022) map brain responses to the feature spaces of deep neural network (DNN) to reconstruct video stimuli. To miti- gate the scarcity of video-fMRI data, Kupershmidt et al. (2022) utilizes self-supervised learning (Kingma & Welling (2014)) to incorporate a large amount of unpaired video data. While these efforts have confirmed the feasibility of video reconstruc- tion from fMRI, the results are notably deficient in explicit semantic information. Chen et al. (2024) utilizes contrastive learning map fMRI to the Contrastive Language-Iman (Radford et al. (2021)) representation space and co-training with a video generation model, success- fully reconstructing coherent videos with clear semantic information for the first time. However, this work does not consider structure information such as color and position, and it is uncertain whether the motion information the reconstructed videos originate from the fMRI or the external video data used in training the video generation model.
Where the object in the scerario, what its size, color,shapes shape? (Lowilevel Structure Information)
A black
the edge of yellow background.
How the objects the scene move? (Motion Information)
Thee figure moves right left the movie.
retrieve
What is this scenario7 (ligh-level Semantic Information
soldier walking desert.
Figure 1: The human brain's 's comprehension of dynamic visual scenes. When receiving dynamic visual information, human brain gradually comprehends low-level structural details such as position, shape and color in the primary vi- sual cortex, discerns motion information, and ultimately con- structs high-level semantic information in the higher visual cortex, such as an overall description of the scene.
Pre-training (CLIP)
In summary, current video reconstruction models face two challenges:
(1) They fail to simultancously capture semantic, structure, and motion information within the reconstructed videos.
(2) The reliance external video datasets and video generation models introduces ambiguity regarding whether the dynamics observed in the reconstructed videos are genuinely derived from IMRI data are hallucinations from video generative model.
To address the issues, we introduce Mind-Animator, a video reconstruction model that decouples semantic, structure, and motion information from fMRI, as illustrated in Figure 3. Specifically, we map fMRI to the CLIP representation space and the Vector Quantized- -iiiizedd-aaiiiinnalll Autoencoder (VQ-VAE) (Van Den Oord et al. (2017)) latent space to capture semantic and structure information. We design a Transformer-batt based (Vaswani et al. (2017)) motion decoder to extract motion information frame by frame from fMRI through a next- next-frame- frame--redicttoon structure, and motion information is fed into an inflated Stable Diffusion (Rombach et al. (2022); Wu et al. (2023)) without any fine-tuning with video data to generate each frame of the video.
task. Finally, the decoded semantic,
Our contributions are summarized follows:
(1) Method: We propose Animaann, which enables video reconstruction by decoupling semantic, structural, and motion information from fMRI data for the first time.
2

---

Published as a conference paper at ICLR 2025
To address the mismatch in timescales between fMRI and video data, we propose Consistency Motion Generator with Sparse Causal Attention. This model decodes subtle yet significant motion patterns through a next-frame token prediction task despite the limitations imposed by the slow BOLD signal integration fMRI.
(2) Interpretability: We use voxel-wise and ROI-wise visualization techniques to elucidate the interpretability of our proposed model from a neurobiological perspective.
(3) Comprehensive evaluation: We introduce eight evaluation metrics that comprehensively assess the reconstruction results of our model and all previous models across three dimensions--sematii structure, and spatiotemporal consistency-on three publicly available video-fMRI datasets. This establishes our work as the first unified benchmark for subsequent researchers. We will release all data and code facilitate future research.
2
RELATED WORK
2.1
RECONSTRUCTING VIDEOS FROM BRAIN ACTIVITIES
DMN
Viceo
Noise
Video gane model
Noise
Infatad image gananation model
Video
PMRI
Figure 2: Overview of the video reconstruction paradigms.
The video reconstruction task involves recreating the video frames subject was viewing based on their brain responses (e.g., fMRI). The challenge in video reconstruction lies the significant discrepancy between the temporal resolution of fMRI (0.5 Hz) and the frame rate of the stimulus video (30 Hz), making it difficult to model the mapping between fMRI signals and video content.
To tackle this challenge, Nishimoto al. (2011) reframes vidco reconstruction as identification task, using the MitionnEEnerye model (Adelson & Bergen (1985)) and Bayesian inference to retrieve videos from predefined library. With the advancement of deep learning, early works by Han et al. (2019), Wen et al. (2018) and Wang et al. (2022), as shown in Figure 2 (a), mapped brain responses to the feature spaces of deep neural networks (DNNs) for end-to-end video reconstruction. To address the scarcity of paired video-fMRI data, Kupershmidt et al. (2022) further advanced this approach by leveraging self-supervised learning to incorporate a large volume of unpaired video data. Although these studies demonstrated the feasibility of reconstructing videos from fMRI signals, the results notably lacked explicit semantic information. As shown in Figure 2 (b), with advancements in multimodal and generative models, Chen et al. (2024), Sun et al. (2024) used contrastive learning to map fMRI signals to the CL.P latent space for semantic decoding, followed by input into video generation model for reconstruction. This approach produces semantically coherent and smooth videos, but it remains unclear whether the motion information in the reconstructions originates from the fMRI from the external video data used to train the video generation model.
To address the above issues, we propose Mind-Anin Animator. By independently decoding semantic, structural, and motion information from fMRI signals and inputting them into an inflated image generation model, we ensure that the motion in the reconstructed videos originates solely from the fMRI data.
2.2 DIFFUSION MODELS FOR VIDRO GENERATION
After significant progress in text-to-image (T2I) generation, diffusion models have drawn interest for text-to-video (T2V) tasks. Ho et al. (2022b) made a breakthrough by introducing 3D diffusion U-Net for video generation, followed by advancements like cascaded sampling frameworks and super-resolution methods (Ho et al. (2022a)), and the integration of temporal attention mechanisms (Singer et al. (2022); Zhou et al. (2022); He et al. (2022b)). However, due to limited paired text-video datasets and high memory demands of 3D U-Nets, alternative approaches have emerged, refining pre-trane mreitrained T2I models for T2V tasks. Khachatryan et al. (2023) and Wu et al. (2023) introduced techniques like tross-frama attention and inter-frann correlation consideration to adapt T2I models for video generation.
3

---

Published as a conference paper at ICLR 2025
In work on video reconstruction from fMRI, we avoided pre-trained T2V models prevent external video data from interfering with motion information decoding from fMRI. As shown in Figure (c), we adapted an inflated T2I model to generate each frame. This ensured that the motion information in the reconstructed videos was derived solely from fMRI decoding, as the generative model had never been exposed to video data.
METHODOLOGGY
Inference pipeline of MilmaAnimator
Vidersiizask
DER speal
nsie
Recanstracte video
Stracure
Combbkane Moiiee Gassr aive
Deceder
Dicader
Sensaric Beadkrr
MLP &
Stage I: EMRtt-n-fattrrr
Stage 2: featar-tt--rido
Training pipeline of Stage
Zio
IMRI siganl
Aepojcction
#MSE(C MLPUDD
Semsanic fvaturee
Combieces Maiiee Gencrater
Structure feature
Sereeaar e
VO-VAR &
Feet carten
Temperal Modak
Motisn festure
CLii
Maskkng
Trainable
Viloo vitudl
Lemmosti Fa
Lit
Lyo
Lecrurture
Provz
Figure 3: The overall architecture of on fMRI. Three decoders are trained during the IMMII-t-feature stage to disentangle semantic, structural, and motion feature from IMRI, respectively. In the feature-to- stage, the decoded information imput into an inflated Text-to- Io-Ima (T2I) model for video reconstruction.
two-stage video reconstruction model based
3.1 PROBLEM STATEMENT
We aim to decode videos from brain activity recorded with fMRI when healthy participants watch sequence of natural videos. Let and Y denote the voxel space and pixel space, respectively, Let X; R be the fMRI signal when video Vs,j R! 1xxx6112x122 is presented to the participant, where n is the number of fMRI voxels, j is the frame ID of video and i E [1, E [1, 8), with N the total number of videos. Let (k) denotes the feature space, {semanlic, structure, motion]. The goal of TMRl-to-feature stage is to train decoders D(k): : X > [k),, and the goal of feature-to- video stage is to construct a generation model G Zsemanttic) Z (structure) x (motion) + Y. without introducing motion information from external video data.
3.2 FMRI-TO-FEATURE STAGE
Semantic Decoder. Due the low signal-to-non -o-isss ratio of the fMRI signal x, and the substantial dimension discrepancy with the text condition c; 1x of Stable Diffusion (SD), learning mapping between them directly is prone to overfitting. Considering both the lower dimensionality (t; V; R 1Ix512 and the robust semantic information embedded in the latent space of CLIP (Gao et al. (2020)), and given that CLIP has been shown to outperform various single-modal DNNs employ bidirectional InfoNCE
in explaining cortical activity (Wang al.; Zhou et al. (2024)), loss to align the fMRI embedding f, with the latent space of CLIP (Vit-B/32) 5212, followed by a layer MLP to map it to text condition c, In this context, f, D Semantic (X;), where D Semantic is a three-layer MLP.
+ log (2pl:(2i,2.)/1) B log Σド erp(s(2,, 2;)/T)
crp(s((2,
LIAISONNE
(I)
exp(( E 2:)/T)
where s(..)) is the cosine similarity, z and & are the latent representation from two modalities, B the batch size, and is learned temperature parameter. Then, given f Bx512 Bx512
4

---

Published as a conference paper at ICLR 2025
and t R Rx522 as the respective embeddings of fMRI, video, and text, the IMRI-vision-Langee language tri-modal loss is:
Semantic [=Bu + -aannt//)-ot--oaa-aaaaa-aa--rr [BBuffoNC(((.V)
For stable training, we have also designed the following loss function to bring the fMRI and text embeddings closer together:
1 f
Prooeectionl
Σ ;=1
t;
B
Subsequently, to map the IMRI embedding f; to the text condition C₁ for the purpose of conditioning generative image models, projection loss is utilized,
L Projecttinn
MMLP(f) c,112-
Finally, we combine the semantic and projection losses using tuned hyperparameters λ1, X2.
LCombined Proojection] + X1 LSemantic + X2 Projection2-
Structure Decoder. For a short video clip, it can be assumed that the low-level feature (e.g. size, shape, and color) contained each frame remains largely consistent with that of the first frame. Consequently, we utilize the token extracted from the first frame by TQ-VAE as structural feature and train the structural decoder (a two-layer MLP) using the standard mean squared error (MSE) loss function. Let & denote the encoder of VQVAE, the structure loss is defined as:
B
L Structure
Σ B
Distructure (f.) Dvv..1)
(6)
Consistency Motion Generator. Drawing inspiration from natural language processing, we treat each video frame token as a word embedding and develop an L-layer Transformer-based Consistency Motion Generator (CMG) to implicitly decode dynamic information between consecutive frames.
Marking
Mask
Vaible frame tokens
Erenadingg:
laper
Eardding
Moduli
Spatial
Com staaee
Reshage
VQ-TAF
dgeal
Traheeding Iaya
Figure 4: The architecture of CMG with Temporal Module and fMRI guided Spatial Module.
Handling raw pixels for a video clip v, xf in the CMG can be computationally intensive. To overcome this, we follow Stable Diffusion's 's approach (Rombach et al. (2022)) by projecting the video into latent space using a pre-trained VAE tokenizer. After compressing the video clip to & (vv:) R/x3x 4 x It
divided into patches, which are then converted into frame tokens Liok E Pxdetken through embedding layer.
In the Temporal Module, visible frame tokens (rok(Vi) gmxddssken and positional encoding Epos &Rmxxdokke are jointly input into a Sparse Causal Self-Attention layer to learn inter-frame temporal information. This attention layer incorporates a specially designed Sparse Causal Mask to ensure sparsity between frames. As illustrated in Figure the mask is divided into fixed and random components. The fixed mask ensures that each frame cannot access information from subsequent frames, while the random mask maintains sparsity among visible frames, preventing the model from
5

---

Published as a conference paper at ICLR 2025
taking shortcuts (Tong et al. (2022)). During inference, eliminate the random mask. For other variants of the spatial-tempores module, please refer to Appendix D.2.
In the Spatial Module, to extract spatial information for subsequent frames from fMRL, the embedding of the visible frames 2, serves as the Query, while the fMRI signal f. after passing through an embedding layer, serves the Key and Value in the cross-attention block, shown in Eq. (7). Following residual connections and layer normalization, 2, is input into the Feed Forward Network (FFN) to predict the subsequent unseen frame tokens (V:.)). E [m++ 1,n]:
21 CCros Allention Q2 V); 1= 1,1,2.....
Q = =Wo' 2,, K= WK Emb(f), = Wv Emb(f),
2 =F)N(LL()) z-1).. = 1.2,....
(8)
Then, the final motion consistency loss is defined as:
1 B n
LConsistency
Σ
Σ
(Poo)(Ves)) - Look (..
(9)
B
im j j=m++1
3.3 TEATURE-TO-VIDEO STAGE
Inflated Stable Diffusion for Video Reconstruction. Despite the rapid development of video generation models capable of producing vivid videos from text conditions, is crucial to emphasize that the objective of our work is disentangle semantic, structural, and motion information from fMRI to reconstruct the stimulus video. Utilizing pre- vee-traieed video generation models could obscure whether the motion information in the reconstructed video originates from the fMRI external video data.
To address this issue, employ the network inflation (Carreira & Zisserman (2017); Khachatryan et al. (2023); Wu et al. (2023)) technique to implement an inflated Stable Diffusion, which is used to reconstruct each frame of the video without introducing additional motion information. Specifically, after the motion features ((v:) E Bx333x y are decoded, they are reshaped (B, 8 W) + (B f,3, W. and input into the U-Net of Stable Diffusion for reverse denoising. The result is then mapped back to pixel space through the VQ-VAE decoder and reshaped ((B f. 3, H, W) > (B, f. H, W)) to yield the final video V; E Bx33x In this context, B denotes the batch dimension, with B = during inference.
4 EXPERIMENT
4.1 DATASETS
In this study, utilize three publicly available video-fMRI datasets, which encompass paired stimulus videos and their corresponding fMRI responses. As depicted in Table 1, these datasets collectively comprise brain signals recorded from multiple healthy subjects while they are viewing the videos. The video stimuli are diverse, covering animals, humans, and natural scenery. For detailed information on the datasets and preprocessing steps, please refer Appendix B.
Table I: Characteristics of the video-fMin MMRI datasets used in our experiments.
Dataset
Adopted subjects
TR
Train samples
Test samples
CC2017 (Wen et al. (2018))
3
2s
4320
1200
HCP (Mareus al. (2011))
1s
2736
304
Algonauts2021 (Cichy et al. (2021))
1.75s
900
100
4.2 EVALUATION METRICS
To comprehensively evaluate the performance of model, the following evaluation metrics.

---

Published as a conference paper at ICLR 2025
Semantic-level metrics. Following prior studies (Chen et al. (2023; 2024)),
the N-way top-K accuracy classification test and VIFI-score as the semantic-level metrics. For the classification test, implement two modes: image-based (2 (2-way-I) and video-based (2-way-V). We describe this evaluation method in Algorithm 2. For the VIFI-score, we utilize a CLIP model fine-tuned on the video dataset (VIFICLIP) (Rasheed al. (2023)) to extract features from both the ground truth and predicted vidcos, followed by the calculation of cosine similarity.
Pixel-level metrics. We employ the structural similarity index measure (SSIM), peak signal-to- noise ratio (PSNR), and based Pearson correlation coefficient (Swain & Ballard (1991)) (Hue-pec) as pixel-level metrics.
Spatiotemporal (ST) -level metrics. We adopt CLIP-pec, a widely used metric in video editing research (Wu et al. (2023)), to evaluate the smoothness and consistency between consecutive video frames. This metric computes the CLIP image embeddings for each frame in the predicted videos and reports the average cosine similarity between all pairs of adjacent frames. Considering the input to the video reconstruction task contains substantial noise, there are instances where every pixel of each reconstructed video frame is either zero or noise, which would artificially inflate the CLIP score if used directly. Therefore, we calculate the CLIP score only when the VIFI-CLIP value exceeds the average level (0.6); otherwise, we assign a score of O.
To measure the similarity of motion trajectories, we introduce End-Point Error (EPE) (Barron et al. (1994)), which calculates the Fuclidean distance between the predicted and ground truth endpoints for each frame.
RESULTS
5.1
COMPARATIVE EXPERIMENTAL RESULTS
GT
Chen
INeurPs
2003
2822
(Cerebrol Corter 2002)
Cortee 20071 and
2011
Figure 5: Reconstruction results on CC2017 dataset. Our reconstructed results are highlighted with red box, while those of Wen and Nishimoto are delincated by blue and green boxes, respectively.
Table 2: Quantitative comparison of reconstruction results on the CC2017 dataset. All metries are averaged across all samples and three subjects, with the best results highlighted bold and the second-best results underlined. EV refers to the External Video Dataset. The symbol i refers to using Stable Diffusion fine-tuned on video data. 100 sample sets were constructed using the bootstrap method for hypothesis testing. Colors reflect statistical significance (Wilcoxon test for paired samples) compared to our model. < 0.0001 (purple); p < 0.01 (pink): < 0.05 (yellow): p > 0.05 (green).
Semantic-level 1
Pixel-level
ST-llvel
Models
Training data
2-way-1 2-way-V
VIFI
SSIM
PSNR
Hue-pee CLIPT EPE.
Nishimoto (Nishimoto et al. (201
CC2017
0.742
0.119
8.383
0.737
Wen (Wen al. (2018)
CC2017
0.771
0.130
8.031
0.637
Kupershmidt (Kupershmidi al. (2022))
CC2017+EV
0.769
0.591
0.140
10.637
0.616
0.382
f-CVGAN (Wang et al. (2022))
CC2017
0.721
0.777
0.592
0.108
11.043
0.583
0.399
6.344
Mind-vided (Chen et al. (2024))
7CCP
0.797
0.848
0.593
0.177
8.868
0.768
0.409
6.125
Ours
CC2017
0.805
0.830
0.608
0.321
9.220
0.786
0.425
5,422
7

---

Published as a conference paper at ICLR 2025
We compare our model with all previous video reconstruction models the aforementioned datasets. In the computation of quantitative metrics, the results of Wen et al. (2018) pertain to the first segment derived from the whole test set. Visual of the test set, whereas the results of other researchers comparisons on CC2017 dataset are presented in Figure 5, while quantitative comparisons are detailed in Table 2, which indicates that our model achieves SOTA performance in six out of eight metrics. Specifically, our model outperforms the previous SOTA model by 83% and 13% in terms of SSIM and EPE respectively, which underscores the benefits of incorporating structural and motion information. The results in Tables 3 and 4 demonstrate that our model maintains strong performance on other datasets as well. For instance, our model outperforms Mind- video by 196% 21% 4% on the HCP and 275% 27% 5% on the Algonauts 2021 dataset across the three pixel-level metrics.
Table 3: Quantitative comparison of reconstruc- tion results on the HCP dataset. The full table can be found in Appendix E.7.
Table 4: Quantitative comparison of reconstruc- tion results on the Algonauts 2021 dataset. The full table can be found in Appendix E.8.
Models
Semantic-level
2-way-I
SSIM
Pixel-level 1
PSNR
Hue-pee
Models
Semantic-level
2-way-1
SSIM
Pixel-level
PSNR
Hue-pos
1
Nishimoto
0.658
0.321
11.316
0.645
Nishimoto
0.687
0,443
9.578
0.666
Wen
0.702
0.058
10.197
0.727
FCVGAN
0.159
13,033
Wen
0.625
0.172
8.822
0.627
Mind-video
0.779
0.116
9.275
0.793
Mind-video
0.681
0.124
8.673
0.796
Ours
0.786
0.344
11.233
0.829
Ours
0.701
0.465
10.989
0.833
5.2 ABLATION STUDY
Table 5: Ablation study about our proposed decoders on subject 1 of CC2017 dataset. More results
on subject and 3 can be found in Appendix 11 and 12. 100 sample sets were constructed using the bootstrap method for hypothesis testing. Colors reflect statistical significance (Wilcoxon test for
paired samples) compared to the Full Model. < 0.0001 (purple); p < 0.01 (pink); < 0.05 (yellow): > 0.05 (green).
Semantic- f
Pixel-level f
ST-level
Models
2-way-1
2-way-V
VIF-I-score
SSIM
PSNR
Hue-pec
CLIP-pecT
EPEL
w/o Semantic
0.679
0.766
0.523
0.097
8.005
0.737
0.123
8.719
w/o Structure
0.789
0.814
0.555
0.184
8.712
0.791
0.260
7.683
w/o Motion
0.674
0.789
0.585
0.136
8.611
0.715
0.376
6.374
Full Model 0.812
0.839
0.604
0.319
9.116
0.778
0.413
5.572
In this subsection, we conduct detailed ablation study to assess the effectiveness of the three decoders we proposed and evaluate the impact of various hyperparameters on video reconstruction. First, we present the results obtained using the full model. Then, based on the full model, we eliminate the semantic decoder (w/o Semantic) and the structure decoder (w/o Structure) separately, replacing their outputs with random Gaussian noise. For the consistency motion generator, we replace it with simple MLPs to model each frame individually (w/o Motion). Table 5 demonstrates that the removal of any decoder results in a significant decline in the model's performance across nearly all metries, which shows the efficacy of our proposed decoders. Notably, the Hue-pee significantly increased after removing the structure decoder. We hypothesize that while fMRI contains structure and motion information, its low signal-to-noise ratio introduces noise that affects the generation quality of Sable Diffusion.
6
INTERPRETABILITY ANALYSIS
6.1 HAVE WE TRULY DECODED MOTION INFORMATION FROM FMRI?
Following the work of Wang et al. (2022), we conduct shuffle tests on three subjects from the CC2017 dataset to evaluate whether the CMG accurately decodes motion information from fMRI, focusing on reconstructed videos with clear semantic decoding. Specifically, for each 8-frame reconstructed video clip from each subject, we shuffle the frame order 100 times randomly and compute spatiotemporal- level metrics on the original and shuffled frames. Subsequently, we estimate the P-value by the following formula: P = 100 8,/ /000, where 6, = if the i-th shuffle outperforms the reconstruction result in the original order based on the metrics; otherwise, d; 0. A lower P-value signifies closer
8

---

Published as a conference paper at ICLR 2025
alignment between the sequential order of the reconstructed video and the ground truth. We repeat the shuffle test times under conditions with and without the CMG, as illustrated in Figure
AAA
1.--
1.0
P-ellr of shuftfe tost
Pryalies of stuftte
pro
p=0.15
0.6
WD cons pency Sctton GaBerRtY
Cerdisenty Metion Generatr
0.6
Werrsiieayy Medon Gererator wit Cersioonnyy Seekon Desevtir
0.6
wis Consinency Matton Ge Genersorr
Cane thency Motton Generater
celling
calling
02-
0.-
CLiP-pzc
EPC
CUP-ppc
CPC
CLI--pcc
EPE
(a)
(b)
(c)
Figure 6: The results of shuffle test the CC2017 dataset. The experiment is repeated times subjects, with the mean and std presented in subplots (a), (b), and (c), respectively. Paired t-tests with Bonferroni correction are performed, with significance denoted as p < 0.001 ) p 0.01(*+). < (+05(+), and p > (.05(NN) for non-significant results.
It can be observed that the P-value of EPE is significantly lower than 0.05 when CMG is applied. However, although the P-value of CLIP-pec is significantly smaller with CMG compared without CMG, the P-value remains significantly greater than 0.05. To explain this, further repeated the shuffle test on the reconstruction results' noise ceiling (videos generated directly using the test features). The results show that even for the noise ceiling, the P-value of CLIP-pec remains significantly greater than 0,05. This indicates that: (1) we have indeed decoded motion information from fMRI, and (2) EPE is to decode motion information.
effective metric than CLIP-pec for evaluating the model's ability
To further validate whether the decoded motion information originates from fMRI guidance or the CMG's autoregressive training, we removed the IMRI guidance during the CMG module's s training (w/o fMRI guidance) by replacing the cross-attention in the Spatial Module with self- attention, while keeping the rest of the architec- ture and hyperparameters unchanged. As shown in Figure 7, removing the fMRI guidance led to significant deterioration in EPE, confirming that the proposed CMG effectively decodes mo- tion information from fMRI Additionally, com- paring the removal of the entire CMG module (w/o Motion) with the removal of fMRI guid- ance (w/o fMRI guidance), we find that the latter accounts the majority of the impact on EPE (i.e., 90% of the decrease in EPE can be attributed to the absence of fMRI guidance). This further underscores the critical role of fMRI guidance in accurately decoding motion information from brain signals.
12
wio Motion
w/o IMRI guidance
Full Model
End-Poirt
Figure 7: Ablation experiment results of IMRI guidance on the test sets subject 1 and subject from the CC2017 dataset.
6.2 WHICH BRAIN REGIONS ARE RESPONSIBLE FOR DECODING DIFFERENT FEATURES, RESPECTIVELY?
To investigate voxels in which brain regions are responsible for decoding different features (semantic, structure, motion) during the fMRl-to-feature stage, we compute the voxel-wise importance maps in the visual cortex. Specifically, for a trained decoder, we multiply the weight matrix of the linear layers, then average the results across the feature dimension, and normalize them to estimate the importance weights for each voxel. A higher weight indicates that the voxel plays more significant role in feature decoding. We project the importance maps of subject I's voxels from the CC2017 dataset onto the visual cortex, depicted in Figure 8. To obtain ROI-wise importance maps, calculate the average of the importance weights of voxels contained within each Region Interest (ROI), with the results presented in Figure 9. The results from other subjects are presented in Appendix F.

---

Published as a conference paper at ICLR 2025
Doveal
Denal
Darsal
Anterker
Anteriee
Anturter
Veatral
Ventral
Narmalined weights for decadinn aemmrttic Seatur
Nomodired weights deaoding atructum fesstre
Norma
attention
(a) Semantic
(b) Structure
(c) Mation
Figure 8: Voxel-wise importance maps projected onto the visual cortex of subject 1. The lighter the color, the greater the weight of the voxel the interpretation of feature.
1061
11000
K00
47000
110
100
030
000
KJ50
000
5098
1900
020
0d3
0.10
wegghtpoppatiion
0.15
0.30
0.25
0.30
000
0.5
weigh proportion
0.35
0.20
8625 0:50
veigtt groportion
6.159 8200
(a) Semantic
(b) Siructure
(c) Molion
Figure 9: ROI-wise importance maps in the visual cortex of subject 1.
Figure 8 (a) indicates that high-level visual cortical areas (HVC, such as MT) contribute more significantly to the decoding of semantic feature, accounting for 60.5% of the total, as shown in Figure (a). Figure (c) and 9 (c) indicates that both LVC and HVC contribute to the decoding of motion information, with significant weight attributed to MT and TPOJ. This observation is consistent with previous work (Born & Bradley (2005)), which validates the function of MT and TPOJ in visual motion perception and processing.
We also identify the following findings in Figure 9: (1) MT shows significant activation for semantic decoding. This observation aligns with the functional segregation and interaction between the dorsal and ventral pathways during dynamic visual input processing (Ingle et al. (1982)). Specifically, the dorsal-dorsal pathway is associated with action control, whereas the central-dorsal pathway is involved in action understanding and recognition (Rizzolatti & Matelli (2003)). This finding aligns with the latter. (2) VI predominantly activated when decoding motion features, reflecting the visual system's 's parallel processing capability. Motion information in the dorsal pathway does not strictly follow hierarchical processing (Zeki & Shipp (1988)). As noted by Nassi et al. (Nassi & Callaway (2009)), VI directly projects motion-related information, such direction and speed, to MT for further processing. For detailed neurobiological explanations, please refer to the Appendix G.
7 CONCLUSION
We introduce video reconstruction model (Mind-Animatorr) that decouples semantic, structural, and motion information from IMRL, achieving state-of-ther performance across public datasets. We mitigate the interference of external video data motion information decoding through a rational ex- perimental design. The results of the shuffle test demonstrate that the motion information we decoded indeed originates from fMRL, rather than being spontaneity from generative model. Additionally, the visualization of voxel-wiss and ROI-wise importance maps substantiate the neurobiological interpretability of our model.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENT
This work was supported in part by the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB093000); in part by Beijing Natural Science Foundation under Grant L.230166; and in part by the National Natural Science Foundation of China under Grant 62206284 and 62020106015. Thanks to Prof. Wei Wang for the discussions and suggestions provided for our revisions in Section 6.2. We are grateful to Prof.Juan Helen Zhou and Dr.Zijjao Chen for their patient answers to our questions and for making the results of the Mind-video test set public. We also extend our thanks to Prof.Michal Irani, Gr.Ganit Kupershmidt, and Do.Roman Beliy for providing us with all the reconstruction results of their models on the test set. We would like to express our appreciation to Prof.Zhongming Liu and Dr. Haiguang Wen for their open-sourced high-quality video-fMRI dataset and the preprocessing procedures.
ETHICS STATEMENT
The pursuit of unraveling and emulating the brain's 's intricate visual processing systems has been comnerttone endeavor for researchers in computational neuroscience and artificial intelligence. Recent advancements in neural decoding and the reconstruction of visual stimuli from brain activity have opened up numerous possibilities, fueling concerns about the potential harmful use cases mind reading.
We argue that these concerns can be alleviated for two main reasons: (1) Mind reading requires brain activity recording devices with very high spatial resolution, and data acquisition systems like IMRI, which possess high spatial resolution, are not easily portable; (2) Although there are now several portable brain activity recording devices, achieving mind reading would require the subject to maintain intense focus and cooperate with the data collection process.
REFERENCES
Edward H Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a, 2(2):284-29) 1985.
Richard A Andersen, C Asanuma, G Essick, and RM Siegel. Corticocortical connections of anatom- ically and physiologically defined subdivisions within the inferior parictal lobule. Journal of Comparative Neurology, 296(()::65-)))
1990.
John Barron, David Fleet, and Steven S Beauchemin. Performance of optical flow techniques. International journal of computer vision, 12:43-77, 1994.
Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini. Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural -imare reconstruction from fMRI. Advances in Neural Information Processing Systems, 32, 2019.
Yohann Benchetrit, Hubert Banville, and Jéan-Rémi King. Brain decoding: toward real-time reconstruction of visual perception. International Conference on Learning Representations, 2024.
Gene J Blatt, Richard A Andersen, and Gene R Stoner. Visual receptive field organization and cortico-cortical connections of the lateral intraparietal area (area lip) in the macaque. Journal of Comparative Neurology, 299(4):421 145,, 1990.
Richard T Born and David C Bradley. Structure and Function of Visual Area MT. Annu. Rev. Neurosci, 18:157-189, 2005.
Edward M Callaway. Structure and function parallel pathways in the primate early visual system. The Journal of physiology, 566(1 13:13-19, 2005.
Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308. 2017.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-super learning (chapelle, o. et al., eds.; (book reviews). IEEE Transactions Neural Networks, 20(3):542-54, 2009.

---

Published as a conference paper at ICLR 2025
Zijiao Chen, Jiaxin Qing, Tiange Xiang. Wan Lin Yue, and Juan Helen Zhou. Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
22710-22720, 2023.
Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video recon- struction from brain activity. Advances in Neural Information Processing Systems, 36, 2024.
Radoslaw Martin Cichy, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Polina lamshchinina, Monika Graumann, Alex Andonian, NAR Murty, K Kay, Gemma Roig, al. The Algonauts Project 2021 Challenge: How the human brain makes sense of a world in motion. arXiv preprint MrXiv:2104.13711 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Tao Fang, Yu Qi, and Gang Pan. Reconstructing perceptive images from brain activity by shape- semantic GAN. Advances in Neural Information Processing Systems, 33:13038-113448 2020.
Yusuke Fujiwara, Yoichi Miyawaki, and Yukiyasu Kamitani. Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural compulation, 25(4):979-1005, 2013.
Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. SketchyCoco: Image generation from freehand scene sketches. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5174-5183, 2020.
Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. Self-supervised natural image reconstruction and large-scale semantic classification from brain activity. Neurolmage, 254: 19121, 2022.
Charles D Gilbert and Wu Li. Top-down influences on visual processing. Nature reviews neuroscience, 14(5):350-363, 2013.
Matthew F Glasser, Timothy Coalson, Emma Robinson, Carl Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):111-178, 2016.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. Communications of the ACM, 63(1 ((((11)::3-144:: 144, 2020.
Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuneu. Decoding natural image stimuli from fMRI data with a surface-basee convolutional network. arXiv preprint arXiv:2212.02409, 2022.
Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational Autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex. Neurolmage, 198:125 198:125-136, 136.136,220 2019.
James V Haxby, M Ida Gobbini, Maura I. Furey, Alumit Ishai, Jennifer I. Schouten, and Pietro Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539):2425 2430, 2001.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autocncoders are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022a.
Yingging Hc, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.132221 2022b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in neural information processing systems, 33:6840-6851, 2020.
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen Video; High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video Diffusion Models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022b.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.
David J Ingle, Melvyn A Goodale, Richard JW Mansfield, et al. Analysis of visual behavior. Mit Press Cambridge, MA, 1982.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(71855:352-335 2008.
KN Kay. Naselaris T, Prenger RJ, Gallant JL. Identifying natural images from human brain activity. nature, 452:352-355, 2008
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954-1555 15964, 2023.
Seong-Gi Gin Kim and Seiji Ogawa. Biophysical and physiological origins of blood oxygenation level- dependent fMRI signals. Journal of Cerebral Blood Flow & Metabolism, 32(7):1 188-1206, 2012.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. International Conference on Learning Representations, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Ganit Kupershmidt, Roman Beliy, Guy Gaziv, and Michal Irani. A penny for your (visual) thoughts: Self-supervised reconstruction of natural movies from brain activity. arXiv preprint 2rXiv:2206.035444, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 11772, PMLR, 2023.
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind Reader: Reconstructing Complex Images from Brain Activities. Advances in Neural Information Processing Systems, 35:29624-29636. 2022.
Nikos K Logothetis. The neural basis of the blood-oryyga--eeel---eeeeeedd functional magnetic resonance imaging signal. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 357(1444):0033-033,, 2002.
Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. Minddiffuser: Con- trolled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the 3 31st ACM International Conference on Multimedia, pp. 5899-5908, 2023.
Daniel S Marcus, John Harwell, Timothy Olsen, Michael Hodge, Matthew F Glasser, Fred Prior, Mark Jenkinson, Timothy Laumann, Sandra W Curtiss, and David C Van Essen. Informaties and data mining tools and strategies for the human connectome project. Frontiers in neuroinformatiic, 5:4, 2011.
Nikola T Markov, Julien Vezoli, Pascal Chameau, Amaud Falchier, René Quilodran, Cyril Huissoud, Camille Lamy, Pierre Misery, Pascale Giroud, Shimon Ullman, et al. Anatomy of hierarchy: feedforward and feedback pathways in macaque visual cortex. Journal of comparative neurology, 522(():225-255 2014.
13

---

Published as a conference paper at ICLR 2025
JH Maunsell and DAVID C van Essen. The connections of the middle temporal visual area (mt) and their relationship to a cortical hierarchy in the macaque monkey. Journal of Neuroscience, 3(12): 2563-2586 1983.
Thomas Naselaris, Ryan Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant, Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6):902-95) 915, 2009.
Jonathan J Nassi and Edward Callaway. Parallel processing strategies of the primate visual system. Nature reviews neuroscience. 1065):360-372. 2009.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):1641-1666 2011.
Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of perceived images from fMRI patterns and semantic brain exploration using instance-conditioned GANs. In 2022 International Joint Conference Neural Networks (ICNN), pp. 1-8. IEEE, 2022.
Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, Joseph Schmidt, and Mubarak Shah. Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):383333849 2021. doi: 10.1 (L19///PAMI.202022955555
Carlos R Ponce, Stephen Lomber, and Richard Born. Integrating motion and depth via parallel pathways. Nature neuroscience, 11(2)221622) 223, 2008.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned CLIP Models are Efficient Video Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6545 6554, 2023.
Giacomo Rizzolatti and Massimo Matelli. Two different streams form the dorsal visual system: anatomy and functions. Experimental brain research, 153:146-1577 2003.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211 1:::2112255 1252, 2015.
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg. Kenneth Norman, et al. Reconstructing the Mind's Eye: fMRI-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.
Stewart Shipp and Semir Zeki. The organization of connections between v5 and v1 in macaque monkey visual cortex. European Journal of Neuroscience, 1(4):309-33,, 1989.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.1441 2022.
14

---

Published as a conference paper at ICLR 2025
Jingyuan Sun, Mingxiao Li, Zijiao Chen, and Marie-Francine Moens. Neurocine: Decoding vivid video sequences from human brain activties. arXiv preprint arXiv:2402.01590 2024.
Michael J Swain and Dana H Ballard. Color indexing. International journal of computer vision, 7(1): 11-32, 1991.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 14453-14463, 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked Autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 10078-10009,, 2022.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Marcel AJ Van Gerven, Botond Cseke, Floris P De Lange, and Tom Heskes. Efficient Bayesian multivariate fMRI analysis using a sparsifying spatio-temporal prior. Neurolmage, 50(1): 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances neural information processing systems, 30, 2017.
Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe, Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nat Mach Intell 5, 1415-1426 (2023). http://kooorr///0000342255556220
Chong Wang, Hongmei Yan, Wei Huang, Jiyi Li, Yuting Wang, Yun-Shuang Fan, Wei Sheng, Tao Liu, Rong Li, and Huafu Chen. Reconstructing rapid natural vision with IMRIIcconditoonll video generative adversarial network. Cerebral Cortex, 32(20):4502-44 -5101, 2022.
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12): 4136 4160, 2018.
Johannes G Wijmans and Richard W Baker. The solution-diffusion model: a review. Journal of membrane science, 107(1-2):1 21, 1995.
Dirk Wildgruber, Axel Riecker, Ingo Hertrich, Michael Erb, Wolfgang Grodd, Thomas Ethofer, and Hermann Ackermann. Identification emotional intonation evaluated by IMRL. Neuroimage, 24 (4):1233-1241, 2005.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-vided One-shot tuning of image diffusion models for text-to-video generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623-7633 2023.
Takao Yamasaki and Shozo Tobimatsu. Motion perception in healthy humans and cognitive disorders. In Early detection and rehabilitation technologies for dementia: Neuroscience and biomedical applications, pp. 156-161. IGI Global, 2011.
Elahe' Yargholi and Gholam-Ali Hossein-Zadee Brain from fMRI data employing Bayesian networks. Frontiers in human neuroscience, 10:351, 2016.
of hand written digits
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition, pp. 2528 2535. IEEF, 2010.
Semir Zeki and Stewart Shipp. The functional logic of cortical connections. Nature, 335(6188): 31 311-317, 1988.
15

---

Published as a conference paper at ICLR 2025
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11015, 2022.
Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. CLIP-MUSED: CLIP-Guided Multi- Subject Visual Neural Information Semantic Decoding. International Conference on Learning Representations, 2024.
16

---

Published as a conference paper at ICLR 2025
Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity
Appendix
CONTENTS
1
Introduction
2
Related work
2.1
Reconstructing Videos from Brain Activities
2.2
Diffusion Models for Video Generation
3
Methodology
3.1
Problem Statement
3.2
FMRI-to-feature Stage
3.3
Feature-to-video Stage
4
Experiment
4.1
Datasets
4.2
Evaluation Metrics
5
Results
5.1
Comparative Experimental Results
5.2
Ablation Study
Interpretability Analysis
6.1
Have we truly decoded motion information from fMRI?
6.2
Which brain regions are responsible for decoding different features, respectively?
7
Conclusion
10
8 ACKNOWLEDGEMENT
11
9 Ethics Statement
11
Related Work
A.1
Reconstructing Human Vision from Brain Activities
19
A.2
Diffusion Models
19
A.3
Diffusion Models for Video Generation
20
Data Preprocessing
20
B.1
Video Captioning with BLIP2
20
B.2
CC2017
21
B.3
HCP
22
B.4
Algonauts2021
22
B.5 Data Acquisition
22
C Implementation Details
22
C.I Hyperparameter Settings
22
C.2 Evaluation Metric Implementation
23
Model Architecture
24
D.I Definition of Frequently Used Symbols
24
D.2 Consistency Motion Generator
24
D.3 Text-to-Image Network Inflation
24
17

---

Published as a conference paper at ICLR 2025
Additional Experimental Results
25
E.1
The impact of different spatial-temporal architecture designs.
25
E.2
A Detailed Parameter Sensitivity Analysis Patch size.
27
E,3
A Detailed Parameter Sensitivity Analysis on X and A2.
27
E,4
A Detailed Parameter sensitivity analysis
and mask ratio.
28
E.5
Supplementary Ablation Studies on Subjects 2 and 3 of the CC2017 Dataset
28
E.6
Further Results on the CC2017 Dataset..
30
E.6.1
THE FINE-GRANNED RETRIEVAL EXPERIMENTAL RESULTS.
30
E.6.2
Comprehensive quantitative comparison results on the CC2017 Dataset.
30
E.6.3
More Reconstruction Results on Multiple Subjects
30
E.7
Further Results on the HCP Dataset
E.8
Further Results on the Algonauts2021 Dataset
32
F Further Results on Interpretability Analysis
35
F.I
Why Utilize Contrastive Learning? Which brain regions are responsible for decoding different features, respectively?
35
F2
36
G Supplementary Knowledge on the Mechanisms of Dynamic Visual Information Process- ing in the Visual Cortex.
39
18

---

Published as a conference paper at ICLR 2025
A RELATED WORK
A.1 RECONSTRUCTING HUMAN VISION FROM BRAIN ACTIVITIES
RECONSTRUCTING IMAGES FROM BRAIN ACTIVITIES
Building on Haxby et al. (2001)'s seminal work, the field of neural decoding has seen a proliferation of tasks with significant implications for guiding research. These tasks can be broadly classified into three categories: stimulus classification, identification, and reconstruction, with the latter being the most challenging and the focus of our study.
Traditional image reconstruction techniques rely on linear regression models to correlate fMRI with manually defined image features (Kay (2008); Naselaris et al. (2009); Fujiwara et al. (2013)). yielding blurry results and a heavy reliance on manual feature selection. However, the advent of deep learning has revolutionized this domain. Deep neural networks (DNNs) have become increasingly prevalent for their ability to address the scarcity of stimulus-IMRI pairs through semi-supervised learning (Chapelle et al. (2009)), as demonstrated by Beliy et al. (2019) and Gaziv et al. (2022). Yet, these models often fail to capture discernible semantic information. Chen et al. (2023) employed pre-training and fine-tuning approach on fMRI data, leveraging methods akin to Masked Autoencoder(MAIF (He et al. (2022a)) and Latent Diffusion Models([.DM) (Rombach et al. (2022)) to improve reconstruction quality. Ozcelik et al. (2022) and Gu et al. (2022) utilized self-supervvied models for feature extraction, followed by iterative optimization to refine the reconstruction process. The integration of semantic information from text, facilitated by Contrastive Language-Image (2021)), has been instrumental in reconstructing complex natural images. Lin et al. (2022) and Takagi & Nishimoto (2023) demonstrated the potential of aligning fMRI with CLIP representations and mapping fMRI to text and image features for high-fidelity reconstruction.
Pre-Training (CLIP) (Radford et al.
While rapid advancements have been made in stimulus reconstruction, with some researchers achiev- ing reconstructions from brain signals that closely approximate the original stimuli, the majority of prior work has focused on static image reconstruction. This study, however, shifts the focus to the more challenging task of video reconstruction.
RECONSTRUCTING VIDEOS FROM BRAIN ACTIVITIES
Compared to image reconstruction, the challenge in video reconstruction lies in the significant discrepancy between the temporal resolution of fMRI (0.5Hz) and the frame rate of the stimulus video (30Hz), which presents a substantial challenge in modeling the mapping between fMRI signals and video content. To overcome the challenge, Nishimoto et al. (2011) transformed the video reconstruction task into a identification problem, employing the Motion-Energy model (Adelson & Bergen (1985) and Bayesian inference to reconstruct videos from predefined video library. Subsequently, Han et al. (2019) and Wen et al. (2018) mapped brain responses to the feature spaces of DNN reconstruct down-sampled (with the frame rate reduced to 1Hz) video stimuli. Specifically, Han et al. mapped IMRI data to a VAEKingma & Welling (2014) pretrained on the ImageNet ILSVRC2012 Russakovsky et al. (2015) dataset to reconstruct a single frame, while Wen al. mapped fMRI data to the feature space of AlexNetKrizhevsky et al. (2012) and used a deconvolutional neural networkZeiler et al. (2010) to reconstruct a single frame. The aforementioned studies have preliminarily validated the feasibility of reconstructing video frames from fMRI. Wang et al. (2022) developed an f-CVGAN that learns temporal and spatial information in iMRI through separate discriminators (Goodfellow et al. (2020)). To mitigate the scarcity of IMRI-video data, Kupershmidt et al. (2022) utilized self-supervised learning (Kingma & Welling (2014)) to incorporate a large amount of unpaired video data. These efforts have validated the feasibility of video reconstruction from fMRI, albeit with a lack of explicit semantic information in the results. Chen et al. (2024) utilized contrastive learning map fMRI to the CLIP representation space and fine-tuned inflated Stable Diffusion (Rombach et al. (2022); Wu et al. (2023)) on a video-text dataset as a video generation model, successfully reconstructing coherent videos with clear semantic information for the first time. However, Chen did not consider structure information such as color and position, and it was uncertain whether the motion information in the reconstructed videos originated from the fMRI or the video generation model.
A.2 DIFFUSION MODELS
Diffusion models (Wijmans & Baker (1995); Ho et al. (2020)), a class of probabilistic generative models, have increasingly rivaled or surpassed the performance of Generative Adversarial Networks (GAN) (Goodfellow et al. (2020)) in specific tasks within the field of computer vision. Diffusion models encompass forward diffusion process and reverse denoising process, each exhibiting
19

---

Published as a conference paper at ICLR 2025
Markovian behavior. The forward process incrementally introduces Gaussian noise into the original image, culminating in a transition to standard Gaussian noise. The forward diffusion process can be represented as g(2,|_,-11 addition. The reverse denoising process employs the U-Net (Ronneberger et al. (2015)) architecture to accurately model the noise distribution each timestep t. The image synthesis is achieved through a sequential denoising and sampling procedure, initiated from standard Gaussian noise.
Naxi 1 (1 - az ))), where denotes the time step of cach noise
In the context of image generation tasks, the conventional diffusion model executes two Markov processes in large pixel space, resulting in substantial computational resource utilization. To VAE Den Oord et al. ()) encoder to transform the pixel space into a low-dimensional latent space. Subsequently, the diffusion model's 's training and generation are performed in the latent space, with the final generated image obtained by utilizing the VQ-VAE decoder. This approach significantly reduces computational resource requirements and inference time while preserving the quality of generated images.
address this issue, Latent Diffusion Models (LDM) (Rombach al. (2022)) employs VQ-VAE (Van
A.3 DIFFUSION MODELS FOR VIDEO GENERATION
After achieving significant progress in rext-to-image (T2I) generation tasks, diffusion models have piqued the interest of researchers in exploring their potential for text-to-video (T2V) generation. The pioneering work by Ho et al. (2022b), introducing the 3D diffusion U-Net, marked significant progress in applying Diffusion Models to video generation. This was followed by further advancements by Ho et al. (2022a), , who utilized a cascaded sampling framework and super-resolution method to generate high-resolution videos. Subsequent contributions have expanded upon this work, notably with the incorporation of a temporal attention mechanism over frames by Singer et al. (2022) in Make-A- MagicVideo, Video, and He et al. (2022b) with LVDM, have integrated this mechanism into latent Diffusion Models, significantly enhancing video generation capabilities. However, due to the scarcity of paired text-video datasets and the high memory requirements for training 3D U-Nets, alternative approaches are being explored. These involve refining pre-trained T2I models to directly undertake T2V tasks. Khachatryan al. (2023) introduced two enhancements to enable zero-shot adaptation of T2I models to T2V tasks: (1) the implementation of cross- fros-frame attention, ensuring that the generation of each current frame in a video considers information from preceding frames; and (2) the consideration of inter-frame correlations during noise sampling, rather than random sampling for each frame independently. Wu et al. (2023) also employed cross-frame attention and achieved one-shot video editing by fine-tuning partial model parameters on individual vidcos.
Video. Zhou et al. (2022) with Magic
In this work. tasked with vidco reconstruction from fMRI, we eschewed the use of pre-trained T2V models to mitigate the interference of external video data with the decoding of motion information from fMRI. Inspired by the fross-frame attention mechanism, network inflation techniques, enabling it to generate multi-frame videos. Consequently, the generative model employed in our study has never been exposed video data, ensuring that the motion information in the reconstructed videos is solely derived from the fMRI decoding process.
adapted T2I model through
B DATA PREPROCESSING
For the stimulus videos of the three datasets described below, we segmented them into 2-second clips, down-sampled the frame rate to 4Hz (i.e. evenly extracting 8 frames), then centrally cropped each frame, and resized each to a shape of 512x512. Following the approach of Chen et al. (2024), we employed BLIP2 (Li et al. (2023)) to obtain textual descriptions for each video clip, with lengths not exceeding 20 words.
B. VIDEO CAPTIONING WITH BLIP2
Due to the absence of open-source video captioning models at the time of experimentation, we utilize the image captioning model BLIP2 to obtain text descriptions corresponding to each video clip. Two considerations are paramount in the design of the video captioning process: (1) the length of the text descriptions should not be excessively long, and (2) the text descriptions must reflect the scene transitions within the video segments. To achieve the first objective, we employ the following prompt: 'Question: What does this image describe? Answer in 20 words or less. Answer:'. Regarding the second point, extracte 1st and 6th frames from every set of 8 frames, inputting them into BLIP2 to obtain their text descriptions. Subsequently, we calculate the CLIP similarity between each of the two text descriptions and 3rd frame. If the difference is no more than 0.05, it indicates minimal
20

---

Published as a conference paper at ICLR 2025
scene change, leading the random selection of one text description from either 1st 6th frame to represent the video clip. Otherwise, indicating a scene transition, we concatenate the two text descriptions with then' to provide comprehensive textual description of the video clip. PyTordl code for the video captioning process depicted in Algorithm 1.
Algorithm Porch code video captioning process
import torch
import clip
3
import random
import numpy up
import Image
6 from lavis.models import Load_model_ane Laciiee_and___eepeee preprocess
device torch. device c'uda:5)
clip model, preprocesss = clwp.loadd"Vii-B///
devicerdevice)
10
model, vis pro__ssoesssrs = Load model and preprocesa (name = "blip2_ts" 25", model
model_type type
pretrain fiaraiin_faanxxxx"" is_eval True, device device)
11
prompt "Question: What does this image describe? Answer in 20 words or less. Answer:"
12
def Baaoo__Cptloonnggtme
Train Crein_Captions (Train saddeo_path_oooe path_root);
15
for i in (1d)((aange
(gd[(aagge
17
frames root Train_vided Sias___idee__pth_rooo
'seg() (1_1((('ormmm"" .ormat 1, 3 1)
()/'
18
framel Image.open (frames_root + 000000000000 jpg') convert ( ("RGB")
framel via_processors ("eval") (framel) unsqueeze
to (davice)
Image. ofaggaee root 000000000000000000000000000000 joo convert ("RGB")
21
frame2 vis processorsss "rocessors["eval"" ["eval"] (frame2) unsqueezee 0000002. jpg') convert ("RGB")
to (device)
22
frame_mis mid mmage.open.eo (frames_ sooe
clip imagem clip pespppprerooesssss (frame mid) unsqueeze (0) to (device)
24
25
caption1 generate (( (("image": framel, "prompt"; prompt))
caption2 model. redel..eneerae ("image": frame2, "prompt" prompt])
28
text clip. tokenize (captionl) to (device) clip. tokenizeeeeeeeeee (captiinn)) (device)
31
with torch.no. _o_drad())
image_ features3 text featuresl = clip model. encode encode text (text1) text text_featuresz
image (clip_image image)
34
clip_model, encode encode_eeere text (text2)
cos siml torch.cosing similarity (image_ featuress, text featuresl)
37
cos sim2 torch.cosine
wetphhooooeeeeiilleeee saineeesimllitttyeee (image (image_ features3, text text_ featuresa
abs (cos (cos_sinl) siml cos cos_sim2)
0.05:
40
number = handom.random. if number >= 0.5:
caption caption1
43
else:
caption caption2[0]
46
caption caption1 + and then caption2[0)
47
Train Captions aeaptiioon..ppennccaap....
49
Train_Capii Casii_Captionsst np. array (Train_Captiinng
Train Captions
52
B.2 CC2017
CC2017 (Wen et al. (2018)) dataset was first used in the work of Wen et al. (2018) This dataset include fMRI data from subjects who view variety of movie clips (23 x23") with central fixation cross x0.8%)0.8% x0.8). Clips are divided into 18 training movies and 5 testing movies, each eight minutes long, presented and 10 times to each subject, respectively. MRI (TI and T2-weighted) and fMRI data (2-second temporal resolution) are acquired using a 3-T system. The fMRI volumes are processed for artifact removal, motion correction (6 DOF), registered MNI space, and projected onto cortical surfaces coregistered to template.
To extract voxels in the activated visual areas, we calculate the correlation of the time series for each voxel's 's activation across 2 trials within the training set. Subsequently, we apply Fisher's z-transform to the computed correlations, average the results across 18 sessions, and identify the most significant 4500 voxels using paired t-test form mask. This mask is computed separately for each of the 3 subjects on their respective training sets and then applied to both training and test set data, with the selected voxels averaged across trials. Following the work of Nishimoto et al. (201 and Han
21

---

Published as a conference paper at ICLR 2025
al. (2019), we utilize BOLD signals with 4-second lag to represent the movie stimulus responses, thereby accounting for the hemodynamic response delay.
B.3 HCP
This dataset is part of the Human Connectome Project (HCP) (Marcus et al. (2011)), encompassing BOLD (blood-oxygen-leve dependent) responses from 158 subjects. For the subsequent experiments, three subjects (100610, 1028 16, and 104416) are randomly selected from this dataset. Data acquisition is performed using a 7T MRI scanner with spatial resolution of 1.6 millimeters and repetition time (TR) of second. The utilized BOLD signals undergo standard HCP preprocessing procedures, which include correction for head motion and distortion, high-pase pass filtering, and removal of temporal artifacts via independent component analysis (ICA). The preprocessed BOLD responses are then registered to the MNI standard space.
Due to the difficulty in directly acquiring IMRI data from multiple trials, we directly utilize the parcellation of the human cerebral cortex proposed by Glasser et al. (2016) extract voxels within the activated visual cortex. The resulting ROIS we extract include: VI, V2, V3, hV4, PPA, FFA, LO, PHC, MT, MST, and TPOJ, totaling 5820 voxels. Following the work of Nishimoto al. (201 I) and Han et al. (2019), we utilize BOLD signals with a 4-second lag to represent the movie stimulus responses, thereby accounting for the hemodynamic response delay. Given that no prior work has conducted video reconstruction experiments on these three subjects from the HCP dataset, except for Wang et al. (2022), we randomly shuffle all video segments and allocate 90% for the training set, with the remaining 10% reserved for the test set.
B.4 ALGONNAUT20021
This dataset is publicly released for the 2021 Algonauts Challenge (Cichy et al. (2021)). During data acquisition, 10 participants passively view 1100 silent videos of everyday events, each approximately 3 seconds in duration, presented three times. The participants' fMRI is recorded using 3T Trio Siemens scanner with a spatial resolution of 2.5 millimeters and repetition time (TR) of 1.75 seconds. The fMRI preprocessing involves steps such as slice-timing correction, realignment, coregistration, and normalization to the MNI space. Additionally, the iMRI data are interpolated to a TR of seconds.
The dataset has been officially preprocessed, allowing us to extract brain responses from nine regions of interest (ROIs) within the visual cortex, including four primary and intermediate visual cortical areas V1, V2, V3, and V4, as well as five higher visual cortical areas: the Extrastriate Body Area (EBA), Fusiform Face Area (FFA), Superior Temporal Sulcus (STS), Lateral Occipital Cortex (LOC), and Parahippocampal Place Area (PPA). These areas selectively respond to body, face, biological motion and facial information, objects, and scene information, respectively. In the experiment, the average neural response across three stimulus repetitions is taken for brain activity. As the test set data are not yet public, we utilize the first 900 sets of data for training and the 900- 1000 sets for testing.
B.5 DATA ACQUISITION
The open-souree datasets used in this paper can be accessed via the following links:
(I) CC2017:
edu
(2) HCP: htttpp://www.bummcccooooooe .rg.
(3) Algonauts2021:
html
C IMPLEMENTATION DETAILS
C.1 HYPERPARAMETER SETTINGS
For all three datasets employed in the experiments, during the training of the Semantic Decoder, we set to 0.5, A to 0.01, and X2 to 0.5. The batch size is set to 64, and the learning rate set to 2e-4, with training conducted 100 epochs. Given the critical role of data volume and augmentation methods in the training of contrastive learning models, and the scarcity video-IMRI paired data, we implement specific data augmentation techniques to prevent overfitting. For IMRI data, we randomly select 20% of the voxels during each iteration, zero out 50% of their values. For image data, we randomly crop one frame from eight video frames to size of 400x400 pixels, and then resize it to 224x224. When extracting the CLIP image features v for each video, we input each frame into the CLIP visual encoder and then compute the average across all frames. For text data, due to the
22

---

Published as a conference paper at ICLR 2025
presence of similar video clips in the training set (derived from complete video segment), BLIP2 often provide identical textual descriptions, which leads to overfitting. To mitigate this, we apply aggressive augmentation techniques. For input sentence, perform Synonym Replacement with a 50% probability and Random Insertion, Random Swap, and Random Deletion with a 20% probability each.
During the training of the Structural Decoder, we set the batch size to 64 and the learning rate to le-6. To stabilize the learning process, we conduct the training process for 100 epochs with 50-step warmup.
When training the Consistency Motion Generator, we the patch size to 64 and the mask ratio of the Sparse Causal mask to 0.6 during the training phase, with batch size of 64 and learning rate of 4e-5. Similarly, for stability, we implement 50-step warmup followed by 300 epochs training. Taking three subjects from the CC2017 dataset as example, we utilize the first 4000 data points as the training set and the subsequent 320 data points as the validation set. Following this, we retrain the model on the entire training set, which comprises 4320 data points. The loss curve is depicted in Figure 10.
Loss
Sementic Decoder
Structure Decoder
Consisienccy Motion Generator
2.5
Vol loss
3:10
Ve loss Twin
Train Joss
Thain Joaa
2.0
3.20
0.30
0.85
0.5
0.75
Epoch
Figure 10: The training and validation loss curves for subject 1 in the CC2017 dataset.
According to Figure 10, in the inference phase, utilize model parameters saved at the 30th, 75th, and 70th epochs for the Semantic Decoder, Structural Decoder, and Consistency Motion Generator, respectively. For the generative model, we employ inflated Stable Diffusion V1-5. Given that Stable Diffusion operates on Gaussian-distributed latent space inputs (i.e., ZT) for Text-to-lmage task, and the distribution of decoded structural information does not align with this, apply 250 steps of Gaussian smoothing to it, which includes 50 steps of ddim inversion. All experiments are conducted on an A100 80G GPU, with the training phase taking 8 hours and the inference phase taking 12 hours for each dataset.
C.2 EVALUATION METRIC IMPLEMENTATION
SEMANTIC-LEVEL
This algorithm performs the N-trial n-way top-1 1 classification test. We describe our evaluation method in Algorithm 2. For the image-based scenario, utilize Vision Transformer (ViT) (Dosovitskiy et al. (2021)) pre-trained on ImageNet as the classifier. For video-based scenario, a pre-trained VideoMAE (Tong al. (2022)) is employed the classifier.
SPATIOTEMPORAL-LENV
Considering the input the video reconstruction task contains substantial noise, there are instances where every pixel of each reconstructed video frame is either zero or noise, which would artificially inflate the CLIP score used directly. Therefore, we calculate the CLIP score only when the VIFI-CLIP value exceeds the average level; otherwise, we assign score of O.
SHUFFFE TEST
Since not every test sample is accurately reconstructed, some fail to be reconstructed in terms of semantics, structure, or motion information, and some reconstruction results are entirely noise (failed samples detailed in Appendix E.6.3). Therefore, it is meaningless to perform the shuffle test the entire test set; we only conduct the shuffle test on the samples that are successfully reconstructed. For details, please refer to the open-source code.
23

---

Published as a conference paper at ICLR 2025
Algorithm 2 N-trial n-way top-1 classification test
1: Input pre-trained classifiers Cimage (+), Cuideo (-). video pair (Generated Video æ, Corresponding GT Video ž), mode (video-basee based or image-based)
2: Output success rate [0, 1]
3: if mode=-video-bbsed then
4: for N trials do
5:
I - Cuideo get the ground-truth class
6:
{Po,..., P...) < Cuideo get the output probabilities
7:
{Pg. Pyn: Pan-1 -pick n-1 random classes
8:
success if arg maxx(ps Pyn: . Pyn-1
y
9:
end for
10: r number of success / N
else
12:
for 8 frames do
13:
for N trials do
14:
Vi < (T; get the ground-truth class
15:
{Po. po9et f- Cimage get the output probabilities
16:
{ps. P ..... Pyr.n- pick n-1 random classes
17:
success if arg m (Pg. Py.. - = 9;
y1
18:
end for number of success / N end for
19:
20:
21:
Σ T//8
22:
end if
D MODEL ARCHITECTURE
D.1 DEFINITION OF FREQUENTLY USED SYMBOLS
The symbols frequently used in this work are defined in Table 6.
D.2 CONSISTENCY MOTION GENERATOR
Figure 4 illustrates the Consistency Motion Generator, which is primarily composed two modules: the Temporal Module and the Spatial Module.
The Temporal Module is tasked with learning the temporal dynamics from the visible frames. Given the severe information redundancy between video frame tokens, specifically design Sparse Causal mask. As shown in Figure on the top, during training, the mask is divided into fixed and random components. The fixed mask ensures that each frame cannot access information from subsequent frames, while the random mask maintains sparsity among visible frames, preventing the model from taking shortcuts (Tong al. (2022)) and accelerating training. During inference, we eliminate the random mask to allow full utilization of information from all preceding frames for predicting future frames.
Since single fMRI frame captures information from several video frames, we design a cross attention mechanism within the Spatial Module to extract the necessary temporal and spatial information for predicting the next frame token from the fMRI data.
D.3 TEXT-TO-IMACE NETWORK INFLATION
To leverage pre-trained weights from large-scale image datasets, such as ImageNet, for the pre-training of massive video understanding models, Carreira & Zisserman (2017) pioneered the expansion of filters and pooling kernels in 2D ConvNets into the third dimension to create 3D filters and pooling kernels. This process transforms NxN filters used for images into NxNxN 3D filters, providing beneficial starting point for 3D video understanding models by utilizing spatial features learned from large-scale image datasets.
In the field of generative model, several attempts have been made to extend generative image models to video models. A key technique employed in this work involves augmenting the Query, Key, and Value of the attention module, illustrated below:
Q W Z0z3
K W.K
[Zng Zec Zec-il: .):
V WV (Exo Po;-1h
(10)
24

---

Published as a conference paper at ICLR 2025
Table 6: Definition of frequently used symbols
Symbol
Definition
X
Voxel space
Pixel space
Z(k)
Feature space, k semmanticsttuucuuree structure,
D(k)
Feature decoder, & {semantic, structure, motion
(p-))
Encoder of pretrained VQ-VAE
M.P(-)
Trainable multilayer perceptron
Emb(-)
Linear embedding layer
n
Number of voxels
i-th IMRI activity pattern, x, E R1
X;
Frames of i-th video, V.,j E R |1 .e [1,8]
Vi.j
i-th text condition of Stable Diffusion, C₁ R 1X200766
C2
f;
i-th fMRI embedding, f, E R I
i-th video embedding in CLIP space, V; E RIx512
Vi
t,
i-th text embedding in CLIP space, t, E R RIx512
W Q.
W K: Wv V
Projection matrices of attention mechanisms
Learned temperature parameter
a, X1, A2
Hyperparameters of semantic loss function
Number of CMG blocks
dtoken
Dimension of frame tokens
B
Batch size
-1
C2-norm operator
s(,,,)
Cosine similarity
where 2e, denotes the latent the i-th frame during the generation process.
E ADDITIONAL EXPERIMENTAL RESULTS
E. THE IMPACT OF DIFFERENT SPATIAL-TEMPORAL ARCHITECTURE DESIGNS.
Due computational resource constraints, adopted design in the CMG architecture that separates temporal and spatial attention modules. To capture motion information between video frames from IMRI, we designed distinct interactions between the temporal and spatial information of fMRI and video data.
For feature interaction, we primarily adopt two strategies:
(1) Cross-attention: As shown in Eq. (11), treat video representations Queries and fMRI representations as Keys and Values, using cross-attention to extract useful spatial and temporal information from fMRI.
V), 1= = Q Q=Wa' 2ı, K= Wx' Emb(f),
2
== Wv'-
(11)
(2) Adaptive layer normalization:
Inspired by the success of space-time self-attention in video modeling, modulate the spatial and temporal information of video representations using IMRI representations, as shown in Eq. (12).
ada (Z), f) Emb(() Norm(
Emb((fssiftt
(12)
As shown in Figure 11, we designed four different architectures for the Spatial-Temponte Temporal Fusion Layer (STFL). Under the same hyperparameters used in Section C, the training/validation loss curves for these architectures are presented in Figure 12.
As shown in Figure 12, regardless of whether cross-atee attention or adaptive layer normalization is used, any interaction between fMRI representations and temporal information in the video leads to extreme instability training and difficulty convergence. We hypothesize that this is due to the significantly lower temporal resolution of fMRI compared to video stimuli, making it difficult
25

---

Published as a conference paper at ICLR 2025
MMU signal
Vivible tokeas
Prodicted
Video stimales
VQ-VAE Encedder
Spatial-temporal Layer
Paickily
Temporal Medale
Spatial Medale
Overview the Consisteney Motion Generator
Medule
Sparial
Ville wnrrrdd
Spar Cand
Crese miterta
Cros attente
Loger
FEN
Lawer
Emsbelding
PIRI Eatbedding
(h) Spatial-temporal Fusion Layer-1
(c) Spatial-temporal Fusion Layer -2
Temporal Module
Spattal Medule
Temporal Module
Spatial Medule
iddreliadiie
Nee
Ldda LaverNini
Ld Laser
adi
Nat Laver
(MRR Exsbedding
[MRI Embeediing
(d) ppatial-temporal Fusion Layer-3
(e) Spattal-temporal Fusion Layer-4
Figure 11: Variants of the Spatial- Temporal Fusion Layer
Consistncty Mation Ginerater
consisteryy waa. Int
consiterxyy Motion
Comsistancy
1.0
05-
80
100
300
200
250
300
300
200
(9) Loss curve STF1-1
carvo for STFF-2
(<) Los curve for STFL-33
(4) curve STFL.-4
Figure 12: Loss curves for variants of the Spatial-Temporal Fusion Layer
26

---

Published as a conference paper at ICLR 2025
to explicitly extract useful temporal information from fMRI. Therefore, in the design of the CMG, we only allow interaction between fMRI representations and spatial information in the video. For temporal information, use sparse causal random mask to implicitly learn motion information between frames.
E.2 A DETAILED PARAMETER SENSITIVITY ANALYSIS ON PATCH SIZE.
Table 7: Parameter sensitivity analysis on Patch size.
Semantic-level 1
Pixel-Ievel
ST-level
Patch size
2-way-1
2-way-V
VIFI-score
SSIM
PSNR
Hue-pec
CLIP-pect
EPE]
4
0.704
0.792
0.495
0.214
10.257
0.756
0.072
8.473
8
0.698
0.794
0.477
0.148
9.744
0.720
0.043
9.935
16
0.675
0.778
0.481
0.139
9.773
0.761
0.049
7.625
32 0.718
0.798
0.503
0.176
9.330
0.751
0.094
7.671
64
0.812
0.841
0.602
0.321
9.124
0.774
0,425
5,580
We conducted detailed experiments to investigate the sensitivity video reconstruction to patch size. As shown in Table 7, setting a smaller patch size hinders the model's learning. This is because smaller patch size results a larger number of patches, forcing fine-grained patch-level interactions between iMRI and video representations. However, the low signal-to-nois ratio of fMRI does not support such fine- aene-grined interactions, and the excessive noise may even degrade the quality of the video representations.
E.3
DETAILED PARAMETER SENSITIVITY ANALYSIS ON X1 AND A2-
Tel Jmn
Tram Jon
0245
0023
0.90
0:25
0.44
6:23
(a) Ag =
(b) A:2 0
Figure 13: Loss curves for variants of a₁ and A2.
Table 8: Parameter sensitivity analysis for variants of ê₁ and X2 subject I of CC2017 dataset. 100 sample sets were constructed using the bootstrap method for hypothesis testing. Colors reflect statistical significance (Wilcoxon test for paired samples) compared to the Full Model. p < 0.0001 (purple) p 0.01 (oink): p 0.05 (yellow); p 0.05 (green).
Semantic-levelf
Pixel-levelt PSNR
ST-level
Model
2-way-1
2-way-V
VIFI-score
SSIM
Hue-pec
CLIP-pect
EPE[
0.01, Àg 0.25
0.765
0.825
0.581
0.318
10.199
0.780
0.396
6.025
A 0.005. A2- 0.5
0.786
0.824
0.591
0.320
9.109
0.776
0.407
5.898
\ = 0.01, A2 = 0.5 (Ours)
0.812
0.839
0.604
0.319
9.116
0.778
0.413
5,572
We conducted sensitivity analysis on the selection of \₁ and X2 using subl from the CC2017 dataset. As shown in Figure 13, when either X₃ or X is set to O, the semantic decider fails to converge, indicating that both the contrastive loss and projection loss play crucial roles in decoding semantic information. In Table 8, we set A = 0.01 and X₂ = 0.5 to ensure that both loss terms are balanced during optimization. When either X or X2 is adjusted, breaking this balance does not affect the
27

---

Published as a conference paper at ICLR 2025
structural level metrics of the reconstruction results, but it does impact the semantic and ST level metrics.
E.4 A DETAILED PARAMETER SENSITIVITY ANALYSIS ON AND MASK RATIO.
Table 9: Parameter sensitivity analysis on contrastive learning on subject 1 of CC2017 dataset. Colors reflect statistical signifi- cance (Wilcoxon test for paired samples) compared to Our Model. p 0.0001 (purple); < 0.01 (pink): < 0.05 (yellow) 0.05 (green).
Table 10: Parameter sensitivity analysis on sparse causal mask ratio on subject of CC2017 dataset. Colors reflect statistical significance (Wilcoxon test for paired samples) compared to Our Model. p 0.0001
(purplek: < 0.01 (pink): < 0.05 (yellow)p > 0.05 (green).
Semantic-
Pixel-level
ST-Ievel
2-way-1
2-way-V V VIFI-score -score
Models
SSIM
PSNR
Hue-pec
CLIP-poct
EPE]
a=)
(w/o fMRI-T)
0.794
0.833
0.594
Ratio=O
0.297
9.037
0.758
0.397
5,896
0=00.2
0.792
0.823
0.593
Ratio=0.2
0.276
8.847
0.767
0.382
5.311
a=0.5
0.812
0.839
0.604
Ratio=0.4
0.285
9.045
0.768
0.404
5.375
(Our Model)
Ratio=0.6
a=0.75
0.791
0.832
0.594
0.319
9.116
0.778
0.413
5,572
(Our Model)
4=1.0
(w/o (MRIIMR-V IMRI-V)
0.787
0,812
0.584
Ratio=0.8
0.296
9.057
0.767
0.409
5.733
During the training of Semantic Decoder, we control the weighting of the contrastive learning loss between IMRI-text ( (LBUsfoNCE(f. ) and IMRI-video L (BUnf ) through the hyperparameter a. We set different values for a (0, 0.25, 0.5, 0.75, 1.0), where a=0 signifies the exclusion of LBstafNNCE t), and a=l signifies the exclusion of LBiinfoNC (f, Table indicates that despite achieving optimal results for the three semantic- metrics when set to 0.5, variations in a do not significantly affect the results, except when a is 1, suggesting that the contrastive learning loss of IMRI-vides predominates. During the training of CMG, we set multiple values for the mask ratio (0, 0.2, 0.4, 0.6, 0.8) and calculate the results on pixel-level and ST-Ievel metrics, as shown Table 10. The results indicate that setting a moderate mask ratio (0.6) can prevent the model from taking shortcuts during training and effectively capture the temporal features between frames.
E.5 SUPPLEMENTARY ABLATION STUDIES ON SUBJECTS 2 AND 3 OF THE CC2017 DATASET
ITain_loss
0.8
Val_
Train Joss without niR__guddnnce
0.7
vel_ oss withou: fMRI guidance
0.6
0.5
0.4
0.3
100
150
200
300
Figure 14: Loss curves for CMG with or without fMRI guidance.
We also analyze the impact of different decoders on video reconstruction performance on subjects 2 and 3 of the CC2017 dataset, as shown in Tables 11 and 12. It is evident from the aforementioned tables that the removal of semantic decoder leads to significant decline in all metrics, whereas the removal of the other two decoders does not significantly affect the semantic metrics, thereby highlighting the crucial role of the semantic decoder in video reconstruction.
28

---

Published as a conference paper at ICLR 2025
Table 11: Ablation study subject of CC2017 dataset. 100 sample sets were constructed using
the bootstrap method for hypothesis testing. Colors reflect statistical significance (Wilcoxon for paired samples) compared to the Full Model. (green).
< 0.0001 (purplek: < 0.01 (pink): < 0.05 (yellow): p > 0.05
Cemantic-levelf
Pixel-level PSNR
ST-level
Model
2-way-1
2 2-way-V V
VIFI-score
SSIM
Huc-pec
CLIP-pect
EPE.
w/o Semantic
0.675
0.769
0.519
0.081
7.944
0.739
0.131
8.457
w/o Structure
0.806
0.826
0.566
0.167
8.700
0.801
0.302
8.011
w/o Motion
0.810
0.829
0.523
0.265
9.080
0.780
0.326
7.431
Full Model
0.811
0.827
0.609
0.292
9.250
0.790
0.423
5.329
Table Ablation study on subject CC2017 dataset. 100 sample sets were constructed using
the bootstrap method for hypothesis testing. Colors reflect statistical significance (Wilcoxon test for
paired samples) compared to the Full Model. 0.0001 (purple): < 0.01 (pink): 0.05 (yellow); 0.05
(green).
Semantic-
Pixel-level PSNR
ST-level
Model
2-way-1
2-way-V
VIFI-score
SSIM
Hue-pee
CLIP-pect
EPE,
w/o Semantic
0.673
0.778
0.523
0.084
8.109
0.738
0.136
7.891
w/o Structure
0.810
0.831
0.566
0.186
8.619
0.794
0.299
7.415
Motion
0.808
0.826
0.583
0.272
8.953
0.779
0.366
6.588
Full Model
0.792
0.823
0.607
0.348
9.287
0.791
0.419
5.356
Table 13: Retrieval accuracy (%) on CC2017 dataset. For the 'small test set', the chance-level accuracies for top-10 and top-100 accuracy are 0.83% and 8.3%, respectively. For the 'large test set', the chance-level accuracies for top- 10 and top-100 accuracy are 0.24% and 2.4%, respectively. Five random seeds were used during the training of the semantic decoder, and the results were tested for statistical significance. denotes our performance is significantly better than the compared method (Wilcoxon test, p=0.05).
CC2017
Datasct
Subjet
Subje12
Subjet3
Average
Model
Test set
top- 10
top-100 100
top-
100
top-10
t0p-100
top-10
top- 100
Wen (Wen et al, (2018))
Small
2.17.
19.50
3.33,
19.
2.75.
19.33,
Kupershmidr (Kupershmidr al. (2022)
Small
1.00.
8.57.
0.92.
8.24.
0.84.
8.24,
0.96.
8.35.
Mind-vidoo (Chen et al. (2024))
Small
3.22.
19.08.
2.76.
16.8.
3.58.
22.06.
3.18.
19.33.
Ours
Small
"5.88
22.68
4.75
26.90
4.50
24.67
"4.1"
24.72
Wen (Wen et (2018))
Large
1.11.
11.58.
2.08.
9.58,
1.75.
10.58.
Kupershmidt (Kupershmidt al. (2022)
Large
0.17.
2.94.
0.17.
2.77,
0.25.
2.18.
0.19.
2.63.
Mind-video (Chen et al. (2024))
Large
1.75.
7.17.
0.83.
5.17.
1.26.
0.00.
1.28.
7.11.
Ours
Lurge
2.17
12.50
2.25
17.00
2.75
16.42
2.39
15.31
29

---

Published as a conference paper at ICLR 2025
E.6 FURTHER RESULTS ON THE CC2017 DATASET
E.6.1 THE FINE- GRANNED RETRIEVAL EXPERIMENTAL RESULTS.
In addition the reconstruction task, we evaluate the retrieval task on the CC2017 dataset. We use top- 10 accuracy and top-100 accuracy as evaluation metrics. To assess the model's generalization capability, we perform retrieval not only on the CC2017 test set with 1,200 samples ( ("Small') but also on an extended stimulus set. Specifically, we augment the collection with 3,040 video clips from the HCP dataset, resulting in a total of 4,240 samples ("Large').
As shown in Table 13, our model achieves superior performance across all three subjects in the CC2017 dataset. Compared to Mind-Video, stimulus set is expanded to the 'Large' scale, demonstrating its generalization capability. Notably, Wen's 's results surpass those of Kupershmidt largely and are comparable to Mind-Video. This can be attributed their approach of reconstructing single frame from each video segment, simplifying the video reconstruction task into an image reconstruction task, thereby achieving superior performance on the retrieval metrics.
Video, our model exhibits a smaller performance drop when the
E.6.2 COMPREHENSIVE QUANTITATIVE COMPARISON RESULTS ON THE CC2017 DATASET.
Table 14: Quantitative comparison of reconstruction results across three subjects from the CC2017 dataset. For the 2- Z-way-I and 2- 2-way-V metrics, 100 repetitions were conducted, while other metrics were evaluated using 100 bootstrap trials. All metrics are averaged over the entire test set. The best results are highlighted in bold, and the second-bes are underlined. Colors indicate statistical significance (Wilcoxon test for paired samples) compared to our model. p 0.0001 (purple): p 0.01
(pink)p p 0.05 (yellow): p > 0.05 (green).
Semantic-level 1
Pixel-level
ST-level
Sub ID
Models
2-way-I
2-way-V
VIFI-score
SSIM
PSNR
Hue-poc
CLIPP-pocT
EPF]
Nishimoto (Nishameto et al. (2011))
0.727
0.116
8.012
0.753
Wen (Wen et al. (2018))
0.758
0.114
7.646
0.647
Kupershmidt (Kupershmidt ot al. (2022))
0.764
0.771
0.585
0.135
8.761
0.606
0,386
sub
FCCGAN (Wang et al. (2022))
0.713
0.773
0.596
0.118
11.432
0.589
0.402
6.348
Mind-video (Chen et al. (2024))
0,792
0.853
0.587
0.171
8.662
0.760
0.408
6.119
Ours
0.812
0.841
0.602
0.321
9.124
0.774
0.425
5.580
Nishimoto (Nishimeto ot al. (2011))
0.787
0.112
8.592
0.713
Wen (Wen et al. (2018))
0.783
145
8.415
0.626
Kupenshmidt (Kupershmidt et al. (2022))
0.776
0.766
0.591
0. 157
11.914
0.601
0.382
sub U2
FCYGAN (Wang et al. (2022))
0.727
0.779
0.596
0.107
10.940
0.589
0.404
6.277
Mind-video (Chen al. (2024))
0.789
0.842
0.595
0.172
8.929
0.773
0.409
6.062
Ours
0.811
0.827
0.615
0.292
9.250
0.791
0.429
5.329
Nishimato (Nishimato et al. (2011))
0.712
0.128
8.546
0.746
Wen (Wen et (2018))
Kupershmidi (Kupershmidt al. (2022)
0.767
0.766
0.597
0.128
11.237
0.641
0.377
sub 03
FCVGAN (Wang et al. (2022))
0.722
0.778
0.584
0.098
10.758
0.572
0.392
6.408
Mind-vidco (Chen al. (2024))
0.811
0.848
0.597
0.187
9.013
0.771
0.410
6.193
Ours
0.792
0.823
0.607
0.349
9.287
0.794
0.421
5.356
The individual quantitative comparison results on the CC2017 dataset for the three subjects are displayed in Table 14.
E.6.3
MORE RECONSTRUCTION RESULTS ON MULTIPLE SUBJECTS
Lese
Subl
Sub2
Sub3
Figure 15: The reconstruction results three subjects from the CC2017 dataset.
30

---

Published as a conference paper at ICLR 2025
Due to the anatomical and functional connectivity differences among subjects, when presented with the same stimulus video, different brain signals are elicited. Consequently, we train the Mind- Animator three subjects from the CC2017 dataset separately, with the reconstruction results shown in Figure 15. can be observed that, despite training and applying the model directly to the three subjects without additional modifications, the reconstruction outcomes largely consistent, which substantiates the effectiveness of our model.
However, as is challenging obtain large volume of brain signals from the same subject, the direct extension of model trained on one subject to others, or the direct training using data from multiple subjects, represents future research direction that requires further investigation.
Ours
GT
Ours
GT
Dure
Figure 16: More reconstruction results on the CC2017 dataset.
In Figure 16, we present additional reconstruction results from the CC2017 dataset, demonstrating that our model does not overfit despite the limited data volume. It is capable of decoding rich array of video clips, such as two people conversing, an airplane in flight, and dog turning head, among others. These video clips encompass a wide range of natural scenarios found in everyday life, including human activities, animal behaviors, and natural landscapes.
Additionally, to provide comprehensive and objective assessment of our model, we also include some reconstruction failure cases in Figure 17. The primary reasons for these failures are twofold. Firstly, the inherent data acquisition paradigm leads to abrupt transitions in content at the junction of video clips, which are evenly segmented from complete videos watched by the subjects during data collection. As illustrated in the first row of Figure 17, the model often struggles to recognize such abrupt changes, resulting in the reconstruction of only the scene prior to the transition. Secondly, emror in feature decoding occur, shown in the second row of Figure 17. Although the model
31

---

Published as a conference paper at ICLR 2025
GT
Failed resulss
Falled results
Figure 17: Reconstruction failure cases.
successfully decodes an ocean scene, structural decoding errors led to the appearance of numerous fish in the reconstructed frames.
E.7 FURTHER RESULTS THE HCP DATASET
To validate the broad effectiveness of our proposed model across different datasets, also conduct experiments on the HCP dataset. The selection of three subjects (100610, 1028 16, and 104416) aligns with that of Wang al. (2022), and the parameter settings for Mind-Animator during both training and testing are identical to those used on the CC2017 dataset. We replicate Nishimoto et al. (2011) 's model on HCP, utilizing video clips from the training sets of both CC2017 and HCP video priors. Wang only reports SSIM and PSNR metrics on the HCP dataset. Visual comparisons are presented in Figure 18, while quantitative comparisons are detailed in Table 15. The results indicate that our model still performs well across multiple subjects in the HCP dataset.
sub1
GT
sub2
sub3
Figure 18: The reconstruction results on three subjects from the HCP dataset.
E.8 FURTHER RESULTS ON THE ALGONAUTS2021 DATASET
We conduct experiments on the A1gonauts2021 competition dataset with 10 subjects, employing the same parameter settings for Mind-Animator during both training and testing as those used on the CC2017 dataset. Since no video reconstruction results has previously been published on this dataset, we replicate Nishimoto et al. 's model as a baseline and utilize video clips from the training sets of both CC2017 and HCP video priors. Visual comparisons and quantitative assessments across various metrics are depicted in Figure 19 and detailed in Table 16, respectively. Although Table 16 indicates that model outperforms the earlier baseline in nearly all metrics across the 10 subjects, the reconstructed results presented Figure 19 are not entirely satisfactory, with some video frames semantically misaligned with the stimulus video. We attribute this to the scarcity of training data,
32

---

Published as a conference paper at ICLR 2025
Table 15:
Quantitative comparison of reconstruction results across three subjects from the HCP dataset. For the 2-way-I and 2-way-V metrics, 100 repetitions were conducted, while other metrics were evaluated using 100 bootstrap trials. All metrics are averaged over the entire test set. The best results are highlighted in bold, and the second-best are underlined. Colors indicate statistical significance (Wilcoxon test for paired samples) compared our model. p (purple); p
(pinky p 0.05 (yellow); > 0.05 (green).
Semantic-levee
Pixel-Level
ST-level
Models
2-way-I
2-way-V
VIFI-score
SSIM
PSNR
Hue-pee
CLIP-pect
EPE]
Nishimoto (Nishimoto (201 1))
0.658
0.307
11.711
0.649
Wen (Wen et al. (2018))
0.728
0,052
10.805
0.751
sub 01
F-CGAAN (Wang et al. (2022))
0.154
13.200
Mind-video (Chen al. (2024))
0.798
0.752
0.605
0.123
9.302
0.774
0.486
12.746
Ours
0.819
0.783
0.613
0.325
10.757
0.820
0.476
7.825
Nishimoto (Nishimoto (201 I))
0.661
0.338
11.249
0.643
Wen (Wen et (2018))
0.688
0.055
10.475
0.720
sub 02
F-VGAA (Wang al. (2022))
0,178
13.700
Mind-video (Chen al. (2024))
0.761
0.777
0.611
0.115
9.414
0.804
0.483
7.358
Ours
0.756
0.759
11.894
0.834
0.485
6.624
Nishimoto (Nishimoto
(201 I))
0.657
0.319
10.988
0.645
Wen (Wen et al. (2018))
0.691
0.067
9.312
0.710
sub 03
1-CYAAN (Wing al. (2022))
0,147
12,200
Mind-video (Chen al. (2024))
0.779
0.778
0.612
0.118
9.109
0.803
0.529
7.767
Ours
0.781
0.793
0.634
0.336
11.018
0.834
0.573
6.792
GT
sub2 m 11
GT
sub5
GT
sub8
GT
sub9
Figure 19: The reconstruction results on four subjects from the Algonauts2021 dataset.
which renders the video reconstruction task on this dataset challenging, given that the data volume subject is approximately one-fifth of that in the CC2017 dataset.
Based on the experimental results across multiple datasets, we can draw the following conclusions: (1) The volume of training data from single subject significantly influences the performance of current vidco reconstruction models, with greater sample size and data diversity leading to better reconstruction performance. (2) There is urgent need to develop new model using incremental
33

---

Published as a conference paper at ICLR 2025
Table 16: Quantitative comparison of reconstruction results across ten subjects from the Algo- nauts2021 dataset. For the 2- 2-way-I and 2-way-V metrics, 100 repetitions were conducted, while other metrics were evaluated using 100 bootstrap trials. All metrics are averaged over the entire test set. The best results are highlighted in bold, and the second-best are underlined. Colors indicate statistical significance (Wilcoxon test for paired samples) compared to our model. < 0.0001 (purple); < 0.01
(pink): p (yellow): p > 0.05 (green).
Semantic-leved f
Pixel-level f
ST-level
ID
Models
2-way-I
2-way-V VIFI-score
SSIM
PSNR
Hue-pee
CLIP-poof
EPEL
Nishimoto (Nishimoto al. (2011))
0.688
0.672
Wen (Wen al. (2018))
0.653
0.147
9.802
0.653
sub 01
Mind-video (Chen et al. (2024)
0.702
0.761
0.568
0.135
8.642
0.794
0.277
8.368
Ours
0.722
0.790
0.599
0.401
10.088
0.824
0.439
4.420
Nishimoto (Nishimoto al. (2011))
0.682
0.443
9.553
0.676
Wen (Wen al. (2018))
0.626
0.231
8.456
0.677
sub 02
Mind-video (Chen et al. (2024))
0.698
0,769
0.573
0,132
9.004
0.773
0.265
7.458
Ours
0.734
0.765
10.932
0.796
0.425
3.806
Nishimoto (Nishimoto al. (2011))
0.679
0.441
9.576
0.682
Wen (Wen al. (2018))
0.647
0.172
8.973
0.611
sub 03
Mind-video (Chen et al. (2024))
0.701
0.729
0.564
0.117
8.796
0. 0.806
0.271
7.659
Ours
0.679
0.794
11.089
0.863
3.406
Nishimoto (Nishimoto al. (2011))
0.702
0.446
9.537
0.665
Wen (Wen al. (2018))
0.592
0.087
8.473
0.534
sub 04
Mind-video (Chen et ul. (2024))
0.665
0.785
0.556
0,126
8.439
0.811
0.254
8.011
Ours
0.673
0.810
11.410
0.848
0.381
3.089
Nishimoto (Nishimoto al. (201 1))
0.676
0.442
9.498
0.650
Wen (Wen et al. (2018))
0.651
0.136
7,599
0.589
sub 05
Mind- video (Chen et al. (2024)
0.664
0.757
0.529
0.140
8.597
0.792
0.263
8.124
Ours
0.689
0.810
0.592
0.458
10.814
0.807
0.406
3.237
Nishimoto (Nishimoto al. (201 1))
0.694
0.444
9.526
0.665
Wen (Wen et al. (2018))
0.642
0.131
9.675
0.690
sub
Mind- video (Chen et al., (2024))
0.690
0.751
0.549
0.137
9.011
0.795
0.266
7.431
Ours
0.709
0.783
0.597
0.489
11.337
0.834
0.446
3.399
Nishimoto (Nishimoto al. (201 1))
0.674
0.446
9.630
0.672
Wen (Wen et al. (2018))
0.628
0.215
8,578
0.648
sub 07
Mind- video (Chen et all, (2024))
0.687
0.721
0.574
0.109
8.409
0.783
0.209
7.652
0.681
0.802
0.578
10.889
0.857
0.329
3.845
Nishimoto (Nishimoto al. (201 1))
0.696
0.444
9.664
0.662
Wen (Wen et al. (2018))
0.596
0.205
9.431
0.610
sub
Mind-video (Chen et al. (2024))
0.658
0.764
0.590
0.114
8.251
0.817
0.204
6.597
Ours
0.709
0.802
0.592
0.467
10.893
0.820
0,376
3.757
Nishimoto (Nishimoto al. (2011)) 1))
0.673
0.445
9.573
0.661
Wen (Wen et al. (2018))
0.574
0.135
8.675
0.614
Mind-video (Chen et al. (2024))
0.679
0.780
0.609
0.117
8.673
0.784
0.267
8.102
Ours
0.731
0.788
0.594
0.502
11.310
0.820
0.400
3.551
Nishimoto (Nishimoto al. (2011))
0.685
0.441
9,598
0.662
Wen (Wen et al. (2018)
0.638
0.265
8.565
0.644
Mind-video (Chen et al. (2024))
0.663
0.770
0.563
0.108
8.912
0.809
0.185
7.524
Ours
0.684
0.777
0.590
0.465
11.128
0.858
0.408
3.533
34

---

Published as a conference paper at ICLR 2025
learning cross-subject learning methods that can be trained using data collected from different subjects, which we consider as a direction for future research.
F FURTHER RESULTS ON INTERPRETABILITY ANALYSIS
F.1 WHY UTILIZE CONTRASTIVE LEARNING?
is noted that there has been an increasing body of work utilizing contrastive learning for neural decoding (Chen et al. (2024); Scotti et al. (2024); Benchetrit et al. (2024)). To explore whether decoders trained with contrastive learning loss demonstrate advantages over those trained with mean squared error (MSE) loss in accuracy and generalization, we conduct the retrieval task and employ t-distributed stochastic neighbor embedding (t-SNE) for visualization.
The retrieval task involves identifying which specific visual stimulus has evoked a given fMRI response from a predefined set. This task differs from the classification task, which only recognizes the category of the visual stimulus that evoked the fMRI response. In contrast, the retrieval task demands a finer level of detail, requiring not just the correct classification but also the precise identification of the particular visual stimulus. During the training process of the semantic decoder, we align the fMRI representation to the pre-trained CLIP representational space via tri-model contrastive learning loss. This design enables the semantic decoder to be not only applicable video reconstruction task but also extends its utility to retrieval task.
To the best of our knowledge, no prior work has been reported on conducting retrieval task using the CC2017 dataset. Therefore, we train a simple linear regression model and a three-layer MI. as our baselines. We employ top- 10 accuracy' and top-100 accuracy as evaluation metrics, To validate the model's generalization capability, comprising 1200 samples and termed the 'small test set', , but also expand the stimulus set to enhance its scope. Specifically, we integrate 3040 video clips from the HCP dataset into our collection, creating an extended stimulus set totaling 4,240 samples, which we label as the 'large test set'.
not only conduct the retrieval task on the CC2017 test set,
Table 17: Results of retrieval task on CC2017 dataset. For the 'small test set', the chance-level accuracies for top-10 and top-100 accuracy are 0.83% and 8.3%, respectively. For the 'large test set', the chance-level accuracies for top-10 10 and top- 100 accuracy are 0.24% and 2.4%, respectively.
CC2017
Dataset
Suhjetl
Subjet2
Subjet3
Average
Model
Test set
top-10
top-100
top-10
top- 100
top-10
top-
top- 10
top-1000
Lineur MSE
Small
2.25
16.92
3.17
22.25
2.75
19.75
2.72
19.64
3-layer MLP+ + MSE
Small
1.00
9.42
1.42
11.75
0.92
9.50
1.11
10.22
3-layer MLP+ Contrast (Ours)
Small
3.08
22.58
4.75
26.90
4.50
24.67
4.11
24.72
Lineur + MSE
Large
L.08
7.83
1.92
10.75
1.92
9.92
1.64
9.50
3-layer MLP + MSE
Large
0.42
3.58
0.75
6.83
0:25
1.08
0.47
3.83
3-layer MLP+ + Contrast (Ours)
Large
2.17
12.50
2.25
17.00
2.75
16.42
2.39
15.31
The experimental results of the retrieval task on the CC2017 dataset across three subjects are depicted in Figure 20 and Table 17, As demonstrated Table 17, our model outperforms the baseline models under both 'small test set' and 'large test set" conditions. Notably, when the stimulus set is expanded to nearly four times its original size, the performance of our model does not experience sharp decline. This suggests that aligning the model's 's latent space the CLIP representational space through contrastive learning loss is beneficial for enhancing the model's generalization capability. As illustrated in Figure 21, we utilize t-SNE to visualize the predicted CLIP representations of the models. The visualization results indicate that the linear regression model exhibits the weakest generalization when performing neural decoding tasks. Increasing the number of layers in the linear model can improve its generalization ability to some extent. However, model trained with a contrastive learning loss demonstrate superior generalization performance.
'Top-k accuracy is the percentage of queries where the correct item is among the top results returned by retrieval model.
35

---

Published as a conference paper at ICLR 2025
Retrieved video
top-1
top-2
top-3
top-4
top-5
top-6
top-7
top-8
top-9
top-10
Figure 20: The retrieval performance of our model (trained with contrastive learning loss) the CC2017 dataset. This figure showcases the top ten video stimuli retrieved based on fMRL. Owing to space limitations, single frame is randomly selected for display from each of the video stimuli.
F.2 WHICH BRAIN REGIONS ARE RESPONSIBLE FOR DECODING DIFFERENT FEATURES, RESPECTIVELY?
To supplement the bar chart in Figure 9, normalized the importance values of each ROI in decoding the three features (semantic, structure, motion) and visualized them in Figure 22. In this figure, darker colors represent higher contributions of a given ROI during decoding. A horizontal comparison for each feature reveals the following:
(1) Motion Decoding: V1, V2, MT, and TPOJ regions contribute more significantly motion decoding. Notably, MT and TPOJ are regions in the dorsal pathway responsible for processing motion information. Meanwhile,
and V2 transmit motion-related attributes such as speed and direction directly to MT when processing motion. This finding aligns well with prior results in neuroscience (Zeki & Shipp (1988); Nassi & Callaway (2009)).
(2) Structure Decoding: Lower-level regions like V1, V2, and V3 show greater contributions to structure decoding, while higher-level regions such as MT contribute less.
(3) Semantic Decoding: Mid-to-high-level regions, including V4, MT, and MST, play more significant role in semantic decoding, while lower-level regions such as VI and V2 contribute less. Interestingly. MT, typically dorsal pathway region for motion processing, shows the highest contribution to semantic decoding. This phenomenon may be explained by the interplay between the dorsal and ventral pathways in processing dynamic visual input (Ingle et (1982)). Specifically, the dorsal-dorsal pathway is concerned with the control of action, whereas the ventral-dorsal pathway is involved in action understanding and recognition (Rizzolatti & Matelli (2003)).
The visualization of voxel-wise and ROI-wise importance maps on Subject 2 and 3 is depicted in the aforementioned Figures.
36

---

Published as a conference paper at ICLR 2025
1998 VEAIICHHOT Of The OL2012 Owasee for SUBJEC
1-2NE wew.iitttiinccc of the CC2037 cotanst Liber,,
1-DE shanfostion CC2S17 cabent Uiterr
Creand
(a) 1-SNE visualization of the output from linear model
1-NE vouu wtth MoE 1956. 0.0339
dataact S
Ouir ART AILE Vos OVER
Ours wtth MG1 lces,
I-SE visualization culput from our model (with MSE loss)
itsusitraiion CC2637 susject cante
CC2017 detasse subrect Dara contraciin los.
virunt bation CC2017 ditase sbject Outs contrastiv los, 2-345
Pretited INn
combooli brach
Trodlcod pitticbtzoi trach
-22
(c) t-SE visualization of the output from model (with contrastive loss)
Figure 21: t-SNE visualization presents comparative analysis of the representations predicted by three decoders on 1200 samples from the CC2017 test set: a simple linear regression model trained with MSE loss, semantic decoder trained with MSE loss, and our semantic decoder trained with contrastive learning loss. The red dots represent the real CLIP representations, while the blue dots denote the representations predicted by the decoders. The absolute Pearson correlation coefficient (i.e. r) between the real and predicted representations is displayed above each subfigure.
Semantic
ROI contribution feattuee decoding
0.2
0.3
0.34
0.37
0.39
0.35
0.4
0.39
0.32
Stucture
0.37
0.34
0.35
0.32
0.34
0.24
0.33
0.32
0.30
Molion
0.25
0.43
0.36
0.32
0.28
0.29
0.31
0.35
0.28
0.36
V1
V2
V/3
V3A
V38
V4
MT
MST
TPOJ
Figure 22: Supplementary ROI-wise importance map on subject 1.
37

---

Published as a conference paper at ICLR 2025
100
MT
LPPA
Deral
Decaal
Auderiar
Antertor
Antertor
Asteriie
Astictiar
Veatral
Viairal
weights decoding ng
Nomallzzd weights decoding structure
Normalizo zod weights spotial tempora attention
(a)
(c)
Figure 23: Voxel-wise importance maps projected onto the visual cortex of subject The lighter the color, the greater the weight of the voxel in the interpretation of feature.
1,9
1750
6-90
107
000
305
0.59
8.15
0.20
0.25
000
6m
0.55
020
8d2
ate
0de
0.12
016
wdggt pmpprtton
Aegght proporton
waight propention
(a) S Semantic
(b) Structure
(c) Moton
Figure 24: ROI-wise importance maps in the visual cortex of subject 2.
Str
L.P
MET.
Mi
Doraat
Darsal
Anterior Auterter
Antertor Asterioe
Autritr
Ventral
Yeatral
Normalzzed weights decoding semantie feature
Normalzzod weighta for decoding structure feature
Norma lzed weights for spatisl-te mproral attention
Semantic
Structure
Motan
Figure 25; Voxel-wise importance maps projected onto the visual cortex of subject 3. The lighter the color, the greater the weight of the voxel the interpretation of feature.
<2911
1.W
1.4
43903
19
182
L100
595
2,6
M0
930
0.15
0.0
008
6.0
032
wright
gi1
[31 Semanticc
(b) Structure
(cl Motion
Figure 26: ROI-wise importance maps in the visual cortex of subject 2.
38

---

Published as a conference paper at ICLR 2025
G SUPPLEMENTARY KNOWLEDGE ON THE MECHANISMS OF DYNAMIC VISUAL INFORMATION PROCESSING IN THE VISUAL CORTEX.
Thick
Pale
Thin V2
MT
V3
Dorsal pathway
d-d pathway
SPL
V1
x-d pathway
IPL
LGN
2/3
4A
MST SAIT
48
LGN
retina
4Cp
Ventral pathway
(a) Parallel visual pathways in humans. The image is cited from Yamasaki & Tobimatsu (2011).
(b) Multiple input streams to MT. The image is cited from Nassi & Callaway (2009).
Figure 27: LGN, lateral geniculate nucleus; V1, 2, 3, and are the primary, secondary, tertiary, quaternary and senary visual cortices, respectively; V5/MT, quinary visual cortex/middle temporal area; MST, medial superior temporal area; IPL, inferior parietal lobule, SPL, superior parietal lobule; and IT, inferior temporal cortex; d-d pathway, dorso-dorsal pathway; v-d pathway, ventro-dorsal pathway.
This study focuses on reconstructing dynamic visual information from brain responses. To validate the interpretability of our model, we employed cortical projection techniques for visualization. To facilitate readers' understanding of the mechanisms underlying how the human brain's visual cortex processes motion information, we provide additional explanations in this section.
As shown in Figure 27a, light signals electrical signals are transmitted through bipolar cells and modulated by horizontal cells and amacrine cells, reaching the final station of the retina-the the retinal ganglion cells (RGCs). The RGCs organize and compress the stimulus information before forwarding it to the lateral geniculate nucleus (LGN). Upon receiving signals from the LGN, the visual areas of the cerebral cortex initially process the information in the VI region. Subsequently, the signals are divided into two parallel pathways: the dorsal pathway and the ventral pathway. The dorsal pathway primarily handles motion perception and spatial vision, while the ventral pathway is mainly responsible for object recognition (Gilbert Li (2013)). Although the dorsal and ventral streams clearly make up two relatively separate circuits, the anatomical segregation between the two streams is by no means absolute (Nassi & Callaway (2009); Andersen et al. (1990); Blatt et al. (1990); Maunsell & van Essen (1983)). Recently, the dorsal stream was shown to be divided into two functional streams in primates to mediate different behavioural goals: the dorsal-dorsal and ventral-dorsa streams (Rizzolatti & Matelli (2003)). The dorsal-dorsal pathway concerned with the control of action 'online' (while the action is ongoing) and the ventral-dorsal pathway concerned with space perception and 'action understanding (the recognition of actions made by others).
converted into electrical signals by photoreceptors. The
The ventral pathway exhibits hierarchical information extraction process (Markov et al. (2014)). From VI, V2, V4 to the inferior temporal (IT) cortex, neurons progressively encode information, transitioning from basic visual features (such as orientation and color) to intermediate shape characteristics, and finally to high-level visual semantics (such as faces, limbs, and scenes). In contrast, the dorsal pathway does not typically follow a hierarchical information extraction process but rather adopts a parallel processing approach (Callaway (2005)). As shown in Figure 27b, there multiple input streams from the LGN to the MT area. The major ascending input to MT traverses the magnocellular layers of the LGN (yellow) and proceeds through layers 4C'a and 4B of the VI (Shipp& & Zeki (1989); Nassi & Callaway (2009)). Experimental evidence suggests that the direct pathway from V1 to MT primarily conveys information about motion speed and direction, while the indirect pathway is responsible for transmitting disparity information (Ponce et al. (2008)).
39


Reviews:
[{"review_id": "1N6yJYR8E9", "reviewer": "Reviewer_1xQ5", "summary": "In this work, the authors present a novel method for video reconstruction from fMRI recordings. There are two core components to this novelty, (1) the method is interpretable and learns separate semantic, structural, and motion features from the fMRI - later used to generate video frames, and (2) the motion component of the generated video is solely based on the motion predicted from the fMRI because the video generator is an inflated image diffusion model. The videos are generated in two stages, fMRI-to-feature which has trainable components, and feature-to-video which is completely frozen. At training time of stage 1, a Semantic decoder using frozen CLIP encoders, a Structure decoder using a frozen VQ-VAE, and a custom transformer-based Consistency Motion Generator with a masked causal frame prediction task are trained to learn the semantic, structural, and motion features respectively. The performance of the reconstruction model is evaluated on 3 public fMRI datasets with 8 different (semantic, pixel, and spatiotemporal) evaluation metrics, compared against previous work, analyzed with ablation studies, and assessed under two different interpretability analyses. The model is found to exceed state-of-the-art both in quantitative and qualitative metrics, yield sensible component contributions in the ablation, and offer interpretability insights that include neurobiological plausibility.", "questions": "The reviewer would like the authors to address the points outlined in the “Weaknesses” section, in each case either by making the suggested change or another change that to the authors’ opinion fixes the issue better, or lastly by giving a sufficient (and convincing) explanation of why no change is needed. This way the reviewer’s opinion of the paper would be improved, to a rating of 6 or above.\n\n**Update after rebuttal:** The score has been updated from an initial 5 to an 8, since all of this reviewer's points were addressed in a very careful, timely, and complete manner. More specifically, the crucial points outlined above in \"Soundness\" were fixed and the additional ablation experiments strengthen the paper and alleviate this reviewers concerns, which on its own brings the paper rating above acceptance threshold. On top of this, the paper's presentation was substantially improved, making it a consistent and clear read, and overall a good paper (8).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper has good quality, with sound methodology and extensive experiments. It is an original, systematic, and rational approach that addresses important issues in previous work; namely the entanglement of different types of features that are decoded from fMRI (semantic, structural, motion), and the entanglement of motion information learned by external training of the video generator with the motion information derived from fMRI. The improvements brought forward by this approach are clearly indicated in the quantitative and qualitative results and consist significant contributions in the research aiming to recover, as well as understand, dynamic visual information from brain recordings.", "weaknesses": "**Presentation** could be more clear at many points throughout the paper. There are technical parts that require more explanation or are currently creating some confusion, which are outlined as follows in order from more major to more minor issues.\n* In section 3.2, it is not mentioned at all how the ground truth text condition $\\bf{c}_i$ is obtained - are these from the training dataset of the (frozen) inflated image generator or are these the same as the video captions $\\bf{t}$? It would help to have an intuitive explanation of this in figure 3 as well ($\\bf{c}$ is not shown at all in the training pipeline).\n* In section 5.1, it is not clear why the videos are external in the “External Videos (EV)” case; This is described as “we further fine-tuned the image diffusion model using videos from the CC2017 dataset (with the training set)” which is still videos from within the video-fMRI dataset, already seen during training in earlier stages of the pipeline.\n* In section 3.2, authors are not explicit enough in the description of the trainable modules Semantic & Structure decoder; From the notation it can be assumed that the Semantic decoder is just one trainable vector $\\bf{f}$ initialized by the fMRI vector, and the Structure decoder is an MLP $D_{Structure}$ (of unknown number of layers) but it needs to be outlined explicitly.\n* In the same section, it is also not explicit how the frames are aggregated to create $\\bf{v}$ - is it the average of the CLIP visual embedding across all frames or another aggregation?\n* In section 4.2, the description of End-Point-Error(EPE) is missing - the reader has no clue what it is, if it is something existing in the literature (missing citation) or introduced by the authors (missing formula). There are also missing citations for some of the other metrics (Hue-pcc, CLIP-pcc).\n* In section 3.3, a mention to the VQ-VAE decoder depicted in the figure is missing - the reader is left to wonder about this. Additionally, this section would benefit from more information on the inflation process (e.g. at least an in-line equation).\n* In section 5.2, there is no mention of which dataset the ablation study is on.\n* In section 3.2, figure 4 is missing $\\bf{E_{pos}}$ and would benefit from a more informative caption. Also, the abbreviation LDM used in the text is probably not familiar to all readers (either remove or expand it).\n* In the rightmost box of the training pipeline in figure 3 (CMG), the bracket of $\\bf{L_{consistency}}$ is placed in a misleading way - a suggestion would be to move the mask matrix to the bottom and have the input and output frames on the right, with $\\bf{L_{consistency}}$ connecting the original (masked) future frames with the predicted future frames.\n* In figure 2c the arrow for the noise would be more accurate if it pointed to the Motion instead of the Structure box.\n* Prior work is sometimes too generically described e.g. “Subsequently, Han et al. (2019), Wen et al. (2018) and Wang et al. (2022) map brain responses to the feature spaces of deep neural network (DNN) to reconstruct video stimuli.” - what DNN?\n* In section 6.2, where authors describe “visual cortices”, probably a more adept term is to say “visual cortical areas” or “areas of the visual cortex”.\n\n**Soundness** related issues exist in parts of the methodology, or sometimes its interpretation.\n* The reader is not yet completely convinced that motion information is from the fMRI and not from the training videos. It would help to see an ablation with the CMG trained without the fMRI entirely (output from temporal module is then the Q, K, and V of spatial module), and see if the next frames can be predicted at inference from the structural latent fMRI embedding of the first frame. If the CMG module is still good, then the motion information is not from the fMRI but from the videos in the training set.\n* The reader finds several issues with the analysis in  6.1. First the y-axes should end at 1.0 as the maximum value of $\\sum _i \\delta _i$ is 100. Additionally the y-axes are different across the 3 panels which is also misleading. Second, it seems that although p-values are lower with the CMG, they are still very high and much higher than 0.05, meaning that the order of the generated frames does not matter significantly for these metrics. This is not commented on at all by the authors. Here, the reader’s suggestion for a better baseline to compare against (instead of the standard threshold of 0.05) is the p-value of the shuffle test with the ground-truth video, as it is not certain that even this would be below 0.05. Third, it is not clear why the authors are examining the structural metrics for the shuffle test instead of solely the spatio-temporal metrics, and why for the latter only CLIP-pcc is shown and not EPE. In the view of the reader, only CLIP-pcc and EPE are relevant and should be shown. Finally, the results are vastly different across the 3 subjects which is also not (and should be) commented on in the text. \n* In section 5.2, table 5, it is observed that the structure metric Hue-pcc is increasing significantly when the structure module is removed. This seems like an important inconsistency in the results, yet the authors do not comment on it. It is expected to provide some sort of explanation, perhaps based on how this specific metric works and on how the w/o Structure videos look, since the other pixel-level metrics seem to decrease.\n* In the analysis in 6.2, it is noticeable that the weight proportion of V1 in the motion information is almost double that of the next highest-weighted areas (e.g. TPOJ, MT, V2, V3), which seems very significant and is not (and should be) commented on - this effect is hidden when the whole of LVC weights are added up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MA5o1kICWS", "reviewer": "Reviewer_GXPV", "summary": "Paper presents an approach the reconstruct video clips for fMRI brain recordings.\nThe reconstruction is broken to 3 streams: structure, semantic and motion.\nFirst the fMRI signal is transformed to align with image embeddings, the fMRI embedding is used with pretrained image diffusion models to generate reconstructions.\nThe results are compared against multiple previous works on a variety of metrics.", "questions": "- Figure 7 color scheme is not consistent across plots(relevant to all similar plots) \n- I think it makes sense to put retrieval results in a more centric place in the paper. \n- The work \" Kupershmidt22\" provides a retrieval metric for itself and \"Wen18\", you should consider adding this results.\n- It would be helpful to add standard error/ statistical tests to the comparisons between the models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Competitive results on multi evaluation metrics, the provided reconstruction look good visually.", "weaknesses": "- Full reconstruction of the video clips not provided, this would make the work more transparent, and would allow future works to easily compare on new metrics.\n- No comparison to other methods is provided for retrieval metric (Which I think is one of the most objective/relevant metrics)\n- Authors don't show results for sequences with actual motion, give that one of the focuses of the work is motion, it would make sense to show the ability to reconstruct motion. (for example the clip with the soldier)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CiYkB4Mhbs", "reviewer": "Reviewer_5KvP", "summary": "This paper addresses the challenge of video reconstruction from fMRI data, identifying three key components for an effective decoder: semantics, visual structure, and motion patterns. Building on prior works that successfully capture semantic content using CLIP and Stable Diffusion, the authors aim to improve fidelity to the visual structure and motion dynamics present in the original videos. They propose a decoupled, multi-step reconstruction approach, utilizing distinct decoders and specialized reconstruction criteria. Their method combines tri-modal CLIP (fMRI, image, text), single-frame Stable Diffusion (T2I), and a uniquely learned internal motion prior that avoids biases from external video datasets. Applied to three publicly available movie-fMRI datasets, the approach demonstrates moderate gains in some metrics across different configurations, with notable improvements in preserving visual structure.", "questions": "1) Consider adding robust statistical analyses and uncertainty estimates for the primary results. This would clarify the significance of the observed gains, if any.\n\n2) For the claim regarding improved recovery of motion patterns, stronger evidence is needed. As presented, the motion generator appears modest/limited in comparison to the tools used for recovering structure and semantics. Since improvement in motion fidelity is a key goal, it would be impactful to demonstrate and emphasize substantial gains in this area. If stronger evidence cannot be shown, it may be best to reconsider this claim.\n\n3) The authors note that MT, an area associated with motion processing, shows significant activation for semantics, typically associated with the ventral stream. If this outcome supports the motion recovery, it could benefit from additional context. Some clarification on MT’s role in semantics would be helpful for the reader’s interpretation of these findings.\n\n4) For Figure 8, consider enhancing its visual accessibility to make it easier to interpret. Improvements in layout or clarity could aid the reader in understanding the figure’s contribution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors’ effort to achieve simultaneous reconstruction of visual structure, semantics, and motion patterns represents a significant and timely contribution. Many prior works prioritize visually appealing reconstructions or focus on semantic fidelity, often overlooking finer alignment with the ground truth video structure. Advancing fidelity in this regard is important; without it, decoders risk becoming general natural video generators with limited adherence to fMRI cues.\n\nThe primary improvement over previous methods appears in the preservation of visual structure, evidenced both qualitatively and quantitatively, though this improvement is moderate. The authors’ thorough analysis across three datasets, coupled with an extensive benchmarking on various metrics assessing semantics and visual structure, adds robustness to the work. Additionally, the website offers a helpful visual supplement to this complex work, making it more accessible and digestible.", "weaknesses": "The results, both qualitatively and quantitatively, lack compelling evidence of substantial improvement over prior work, particularly when compared to Chen et al. (2024). In fact, it appears that with proper statistical analysis (e.g., Table 2-4), there may be no significant gains. Methodologically, the advancement over Chen et al. (2024) seems minimal, with certain versions of the authors' approach even reintroducing external videos—the very bias this paper aims to avoid. Furthermore, the claim of improved motion pattern reconstruction is insufficiently supported, as the authors compare only against shuffled frames. A meaningful comparison should have been made with prior works, similar to the comparisons on semantic and visual structure fidelity.\n\nAdditional comments:\n- [36-38] Statement is unclear; consider rephrasing.\n- [54-59] The text appears to confuse BOLD integration duration with fMRI sampling rate. The BOLD signal integrates neural activity over a period greater than 10 seconds (~300 video frames), which is a major limitation in recovering any motion patterns potentially encoded in neural activity. The fMRI sampling rate is limited for other technical reasons but even if was sampling at a higher rate this would likely not resolve this fundamental issue.\n- [100-101] The claim about enabling video reconstruction may be overstated, as prior methods have also achieved this to an extent.\n- [102-104] Similar to above, there is a mix-up regarding temporal aspects. Clarification of this contribution would help.\n- [104-106] The visualizations mentioned are not novel to this work; they have been used in prior studies and are reintroduced here rather than proposed anew.\n- [107-110]  It remains unclear what metric is used to evaluate successful recovery of motion patterns. Detailing this metric is necessary for interpreting the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NLhwkJ3VKg", "reviewer": "Reviewer_fFuu", "summary": "This work addresses the problem of reconstructing high-quality video from fMRI data. The authors suggest that the key to achieving high-quality video reconstruction lies in decoupling and modeling semantic, structural, and motion information, also carefully handling the frequency discrepancy between fMRI data and videos. To this end, the authors develop a tri-modal contrastive learning scheme along with a next-frame prediction task. Lastly, to ensure that the generated videos are derived purely from the fMRI data, the input is fed into an untuned inflated Stable Diffusion model. Empirical evaluations show promising results and strong interpretability performance.", "questions": "1. How did the author chose $\\lambda_1$ and $\\lambda_2$ in equation 4? Can the author provide the ablation study/ discussion on how these choices affect the downstream performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[Quality, Significance]:\n\nThe structure of this work is very easy to understand and follow, with sufficient context and supported citations. The model notations and figure presentations are excellent. \n\nThe authors provides comprehensive empirical metric evaluations and compared their results with many other state-of-the-art approaches, and provide comprehensive ablation study and interoperability results.\n\n[Novelty]: One particular point that I find this work novel is that the authors do not fine-tune the stable diffusion, ensuring us that the videos that we see are not coming from the overfitting.", "weaknesses": "I do not find this work particularly having major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)