NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction
Authors: Zixuan Gong
Guangyin Bao
Qi Zhang
Zhongwei Wan
Duoqian Miao
Shoujin Wang
Lei Zhu
Changwei Wang
Rongtao Xu
Liang Hu
Ke Liu
Yu Zhang
Venue: NeurIPS 2024 oral (2024)
Keywords: fMRI visual decoding; fMRI-to-video Reconstruction
Primary area: neuroscience_and_cognitive_science
TL;DR: This paper proposed NeuroClips, a new state-of-the-art fMRI-to-video reconstruction framework, achieving smooth high-fidelity video reconstruction of up to 6s at 8FPS.

Abstract: Reconstruction of static visual stimuli from non-invasion brain activity fMRI achieves great success, owning to advanced deep learning models such as CLIP and Stable Diffusion. However, the research on fMRI-to-video reconstruction remains limited since decoding the spatiotemporal perception of continuous visual experiences is formidably challenging. We contend that the key to addressing these challenges lies in accurately decoding both high-level semantics and low-level perception flows, as perceived by the brain in response to video stimuli. To the end, we propose NeuroClips, an innovative framework to decode high-fidelity and smooth video from fMRI. NeuroClips utilizes a semantics reconstructor to reconstruct video keyframes, guiding semantic accuracy and consistency, and employs a perception reconstructor to capture low-level perceptual details, ensuring video smoothness. During inference, it adopts a pre-trained T2V diffusion model injected with both keyframes and low-level perception flows for video reconstruction. Evaluated on a publicly available fMRI-video dataset, NeuroClips achieves smooth high-fidelity video reconstruction of up to 6s at 8FPS, gaining significant improvements over state-of-the-art models in various metrics, e.g., a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics. Our project is available at https://github.com/gongzix/NeuroClips.

NeuroClips: Towards High-fidelity and Smooth fMRI-to- Video Reconstruction
Zixuan Gong', Guangyin Bao', Qi Zhang Zhongwei Wan', Duoqian Mi Shoujin Wang', Lei Zhu', Changwei Wang', Rongtao Xu', Liang Hu', Ke Liu', Yu Zhang'
'Tongii University
OOio State University
University of Technology Sydney
& *Chinese Academy of Sciences
BBejjing Anding Hospital
{gongzx, baoog baogy, zhangqi_cs, dqmiao,
o i2yg){tnnji..eil cn
Abstract
Reconstruction of static visual stimuli from non-invasion brain activity fMRI achieves great success, owning to advanced deep learning models such CLIP and Stable Diffusion. However, the research on iMRI-to-video reconstruction remains limited since decoding the spatiotemporal perception of continuous visual experiences is formidably challenging. We contend that the key to addressing these challenges lies accurately decoding both high-level semantics and low-level per- ception flows, perceived by the brain in response to video stimuli. To the end, we propose NeuroClips, an innovative framework to decode high-fidelity and smooth video from fMRT. NeuroClips utilizes semantics reconstructor to reconstruct video keyframes, guiding semantic accuracy and consistency, and employs a perception reconstructor capture low-level perceptual details, ensuring video smoothness. During inference, it adopts pre-trained T2V diffusion model injected with both keyframes and low-level perception flows for video reconstruction. Evaluated on publicly available fMRI-video dataset, NeuroClips achieves smooth high-fidelity video reconstruction of up to 6s at 8FPS, gaining significant improvements over state-of- --the-arr models various metrics, c.g., 128% improvement in SSIM and an 81 % improvement in spatiotemporal metries. Our project is available https///giibbbboooongggg/z///ini
Introduction
Decoding visual stimuli from neural activity is crucial and prospective to unraveling the intricate mechanisms of the human brain. In the context of non-invasive approaches, visual reconstruction from functional magnetic resonance imaging (fMRI), such as fMRI-to-image reconstruction, shows 3 largely benefiting from advanced deep learning models such CLIP (5 6 and Stable Diffusion 7 This convergence of brain science and deep learning presents a promising data-driven learning paradigm to explore a comprehensive understanding of the advanced perceptual and semantic functions of the cerebral cortex. Unfortunately. IMRI-to-vided reconstruction still presents significant hurdles that discourage researchers, since decoding the spatiotemporal perception of continuous flow of scenes, motions, and objects is formidably challenging.
high fidelity N 2
At first glance, fMRI measures blood oxygenation level-dependent (BOLD) signals by snapshotting few seconds of brain activity, leading to differential temporal resolutions between fMRI (low) and
Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

videos (high). The previously advisable solution to address such differential granularity is to perform self-interpolatiin on fMRI and downsample video frames to pre-align fMRI and videos. Going further, decoding accurate high-level semanfics and low-level perception flows has more profound impact on the ability to reconstruct high-fidelity videos from brain activity. Early studies before 2022 struggled with achieving satisfactory reconstruction performance, as they failed to acquire precise semantics from powerful (pre-trained) diffusion models. The latest research MinD-Vided [8] guides the diffusson model conditioned on visual IMRI features, making initial attempt to address the semantic issue. However, it lacks a design of low-level visual detailing, so it significantly diverges from the brain's 's visual system, exhibiting limitations in perceiving continuous low-level visual details.
The brain's reflection of video stimuli crucial factor that influences and enlightens the visual decoding of fMRI-to-vide reconstruction. Notably, the human brain perceives videos discreetly [ 10 due to the persistence of vision [1] 12
15) and delayed memory [16]. is impractical to perceive every video frame, and instead, only keyframes elicit significant responses in the brain's 's visual system. Reconstructing keyframes from fMRI avoids the issue of differential temporal resolutions between IMRI and videos. Also, the precise semantics and perceptual details in keyframes ensure the high fidelity and smoothness of reconstructed videos, both within and across successive IMRI inputs. Accordingly, we argue that utilizing keyframe images as anchors for transitional video reconstruction aligns with the brain's 's cognitive mechanisms and holds greater promise.
13
14
However, relying solely on manipulating poral conditions for successive image reconstruction, generate keyframes, easily yields suboptimal outcomes. Research dating back to as early as 1960 [17] has shown that the fleeting sequence of images perceived by the retina is hardly discernible during the process of perception. Instead, what emerges is a phenomenal scene or its intriguing features, which can be regarded as non-detailed, low-level images. Initially, the retina captures these low-level perceptions, and subsequently, the higher central nervous system in the brain focuses and pursues the details, generating high-level images in the cerebral cortex [8.. This process is reflected in the fMRI signal 19 20 21 22 23 24
MIRI-to-image models, e.g.. incorporated with spatiotem-
The video-to-IMRI process naturally incorporates combination of both low-level and high-level images. Therefore, the reverse fMRI-to-viie reconstruction intuitively benefits from taking into account both the high-level semantic features and the low-level perceptual features. Specifically, it is necessary to decode the low-level perception flows, such as motions and dynamic scenes, from brain activity to complement keyframes, which enhances the reconstruction of high-fieelity frames and produces smooth videos.
In light of the above discussion, we propose novel IMRI-to-video reconstruction framework NeuroClips that introduces two trainable components of Perception Reconstructor and Semantics Reconsiructor for reconstructing low-level perception flows and keyframes, respectively. 1) Percep- tion Reconstructor introduces Inception Extension and Temporal Upsampling modules to adaptively align fMRI with video frames, decoding low-level perception flows, i.e., a blurry video. This blurry video ensures the smoothness and consistency of subsequent video reconstruction. 2) Semantics Reconstructor adopts diffusion prior and multiple training strategies to concentrate quantities of high-level semantics from various modalities into fMRI embeddings. These fMRI embeddings are mapped the CLIP image space and decoded to reconstruct high-quality keyframes. 3) During infer- ence, Clupo adopts a pre-trained T2V diffusion model injected with keyframes and low-level perception flows for video reconstruction with high fidelity, smoothness, and consistency.
Extensive experiments have validated the superior performance of NeuroClips, which substantially ahead of SOTA baselines in pixel-level metrics and video consistency. NeuroClips achieves 0.219 improvement (128%) in SSIM and a 0.330 improvement (81 1)) in spatiotemporal metrics and also performs better overall on most video semantic-leve metrics. Mcanwhile, using multi- MfRi fusion, NeuroClips pioneers the exploration of longer video reconstruction up to 6s at 8FPS.
2 Related Work
2.1 Visual Reconstruction
After its initial exploration (25), static image reconstruction from fMRI has witnessed remarkable success in recent years. Due to the lack of information in fMRI adapted to deep learning models, a path has been gradually explored that aligns fMRI to specific modal representations such as the and CLIP's [5) rich representation of space is
common image and text modality [26] 27 28 29

---

pretty favored. Then, the aligned representations can then be fed into the diffusion model to complete the image generation. Along this path, a large body of literature demonstrates that reconstructing images at both pixel level and semantic level achieves great results 2 2 30 However, the field of IMRI-to-viden reconstruction remains largely unexplored. Early studies reconstruct low-level visual content from fMRI, using the embedded fMRI as conditions to guide GANs or AEs in generating multiple static images. The reconstructed videos contained little to no clear semantics recognizable by humans. Due to the excellent performance of diffusion models, MinD-Video [ reconstructed 3FPS videos from fMRI. Despite this success, the smoothness and semantic accuracy of these videos remain unsatisfactory, leaving substantial space for improvement.
attempted to
2.2 Video Diffusion Model
Diffusion models for image generation have gained significant attention in research communities recently [34 35 36]. DALLE-2 (37) improved text-image generation by leveraging the CLIP [5] joint representation space. Stable Diffusion 7 enhanced generation efficiency by moving the diffusion process to the latent space of VQVAE 38) To achieve customized generation with trained diffusion models, many works focused on adding extra condition control networks, such as ControlNet [3] and T21-Adapter [40]. In the realm of video generation, it is typical to extend existing diffusion models with temporal modeling 41 42
Animatediff 44 trained plug-and- and-play motion module that can be seamlessly integrated into any customized image diffusion model to form video generator. 45] fine-tunes pre-trained diffusion models using high-quality video datasets from multiple views to achieve powerful generation.
Stable Video Diffusion
3 Method
The overall framework of NeuroClips is illustrated in Figure NeuroClips consists of three essential components: 1) Perception Reconstructor (PR) generates the blurry but continuous rough video from the perceptual level while ensuring consistency between its consecutive frames. 2) Semanties Reconstructor (SR) reconstructs the high-quality keyframe image from the semantic level. 3) Inference Process is the fMRI-to-video reconstruction process, which employs T2V diffusion model and combines the reconstructions from PR and SR to reconstruct the final exquisite video with high fidelity, smoothness, and consistency, Furthermore, NeuroClips also pioneers the exploration of Multi-fMRI Fusion for longer video reconstruction.
Semontics Reconstructor (SR)
BLI-2
VIT bigE 14
plase Auing iN due spp
Teat Centrnutive Los
keylinane
Laune
keyfrume
big6-14
Imagg Contraction Los
DUP-2
Cusetacc
guidance
ContralNet
Trudictdd
unCUP
Keyfrann:
Diffusion Pricr
(MHI Kelsshhing
Recesstmation Exbeedings
Perception Reconstructor (PR)
guidance
Flint Brews)
VAE Decoder
VAC Decoder
Inception Extension
MIPP
Temporal Upsampling
Blarry
Erabsdiiig
VAE Enceder
MAE LasK Cistrartvv 2ans
Figure 1: The overall framework of NeuroClips. The red lines represent the infernence process.
3.1 Perception Reconstructor
Perception reconstruction is essential not only for video reconstruction but also for semantics re- construction. Additionally, smoothness and consistency are crucial metries of video quality, and Perception Reconstructor (PR) plays a key role in ensuring these attributes.

---

We split a video into several clips at two-second intervals (i.e., fMRI time resolution). For each clip c. we downsample it and retain part frames at fixed intervals, resulting a of frames X = = (2, .......... XN,1 1. X; is the i-th retained frame image, with total of N₃ retained frames. Ve the corresponding fMRT signal of clip c. Here we introduce Inception Extension module to extend one fMRI to N fMRIs, denoted as V. Y. = [2. ,Nyy)
.N,,l-
Sequentially applying a simple MLP and Temporal Upsampling module obtain Y's embedding set Ey = [ , ... VVx, which can be fed into the Stable Diffusion 7 VAE decoder to produce series of blurry images. We regard this sequence of blurry images blurry video. We expect the blurry video to lack semantic content, but to exhibit state-of-the-ar art perceptual metrics, such position, shape, scene, etc. Thus, consider using frame set X align Y.
Training Loss. Mapping X the latent space of Stable Diffusion's 's VAE obtain the perception em- bedding set Ex = lex; C.22' Cxxx,1 We adopt mean absolute error (MAE) loss and contrastive loss train the PR, the overall loss LPR of PR can be described as:
1
N
exp(simlexx,
LPR
ex.
exil
log IE exp(sim(ex,,
NG
2
j=l
(I)
N
sxxs
log
2N
exp(sim(ey,, exx)
j=1
where 7 is temperature Syper-parameter. The function sim(,) is used to compute the similarity.
Temporal Upsampling. Due to the low signal-to-noise ratio of fMRI, the direct alignment of fMRI to VAE's pixel space is highly susceptible to overfitting noise, and the learning task is too complex to guarantee the generation of decent blurry images. A common method is aligning to a coarser-grained fine-grained pixel space. The temporal relationship of the
pixel space and then upsampling frames also needs to be considered in the video task to maintain consistency. Therefore, to achieve consistency between the retained frames, we integrated temporal attention into the upsampling operation. The fMRI embedding Ey has five dimensions, i.e. Ey E b xexxxw, where denotes the batch size, h is the dimension of embedding space. The upsampling operation merely models spatial relationship. receiving reshaped embedding E E [ [BbxxN))xxxxXX
imput. To model temporal relationship of N2 fMRI, we first reshape Ey as E E B(bxhxw)xNNxx Then, we use learnable mapping to compute the query value Q = W and the key value K = k EjSTP, Finally, the output of temporal attention layer is E = Softmax( (" K
E
Ve
We utilize a learnable mixing coefficient vi to conduct residual connection:
Ey = r) E
Etemp + - r1) E'y.
(2)
Based on the above design, PR generates the blurry rough video with initial smoothness and great consistency, laying the foundation for subsequent video reconstruction.
3.2 Semantics Reconstructor
argue that key--frmmes' play crucial role in how the human brain recalls and connects relevant memories with unfolding events, and other research [48]49 49 also demonstrates that video key-frames can be used as representative features of the entire video clip. Building on these conclusions, the core objective of Semantics Reconstructor (SR) is to reconstruct high-qualin keyframe image that can used to address the issue of frame rate mismatch between visual stimuli and fMRI signals, thereby enhancing the fidelity of the final exquisite video. The facilitate our objective, detailed below:
Recent cognitive neuroscience studies 46
existing IMRI-to-imaee reconstruction studies [
IMRI Low-dimensional Processing. For each clip c, its corresponding fMRI signal is Ve. We use ridge regression to map Ve to lower-dimensional 3% for easier follow-up:
y" = x -
where X design matrix, X regularization parameter, and / identity matrix. Although the human brain processes information in highly complex and non-linear way, empirical evidence [] 26

---

underscores the effectiveness and sufficiency of linear mapping for achieving desirable reconstruction, due to nonlinear models will easily overfit to fMRI noise and then lead poor performance [50].
Alignment of Keyframe Image with fMRI. Randomly choose one frame in the clip es its keyframe X,, and use OpenCLIP ViT-bigG/14 [51] to obtain 6x,, the embedding of keyframe image X, in CLIP image space. ey. is the IMRI embedding of V via another MLP. Consequently, we perform contrastive learning between ex and ey. to align the keyframe image X, and the IMRI Ve resulting in enhancing the semantics of Ey.. It is worth noting that the MLP gets a bidirectional contrastive loss. Previous research [T] has demonstrated that introducing MixCo [52] data augmentation, an extension of mixup utilizing the InfoNCE loss, can effectively help model convergence, especially for scarce fMRI samples. Therefore, the bidirectional loss called BiMixCo LBMiiXCo- which combines MixCo and contrastive loss, needs to be used for training.
Generation of Reconstruction-Embeerte Emhedding. Since the embeddings in the CLIP VIT image space are more approximate to real images compared to fMRI embeddings, transforming fMRI embedding ey. into CLIP ViT's image embedding will significantly benefit the reconstruction quality of the keyframe. Therefore, we have to generate the reconstrutioo-eebeeddin ext for the keyframe image Xe. essentially, which is the image embedding that will be fed to the subsequent generative model for reconstruction. Inspired by DALLE-2 (37), diffusion prior is an effective approach to transforming embedding. So, we map the fMRI embedding ey. to the OpenCLIP ViT-bigG/14 image space to generate X. Here, we use the same prior loss Lprio in DALLE-2 37 for training.
Reconstruction Enhancement from Text Modality. Original fMRI-to-image reconstruction only relies on visual modality embedding. For instance, reconstructing images conditional on the image embeddings generated by diffusion prior. However, text is another critical modality. Incorporating text with higher semantic density can help improve the semannic content of reconstruction embedding, resulting in making semanties reconstruction more straightforward and effective. We adopt BLIP- 2 [53] to introduce the text modality, i.e., the caption T. of the keyframe images X, Then, we embed Te obtain the text embedding eT. Inspired by contrastive learning, we perform contrastive learning between ex and eT, to enhance reconstruction eeconstructoon-mbedding eX via additional text modality. The contrastive loss serves as the training loss Letfm of this process, similar to Equation J
omitted here.
Training Loss. As discussed above, the overall training loss LSR in SR is composite. Therefore, We set mixing coefficients ō and μ to balance multiple losses:
LSR
LRIMiXCo + &Criior HL
(4)
3.3 Inference Process
The inference of NeuroClips the process of fMRI-to-video reconstruction. We jointly utilize the blurry rough video, the high-quality keyframe image, and the additional text modality, which are a, B8 and y guidance, to reconstruct the final exquisite video with high fidelity, smoothness, and consistency. And we employ a text-to-video diffusion model to help reconstruct video.
Text-to-video Difftusion Model. Pre-training texi-to-video (T2V) diffusion models possess signifi- cant amount of prior knowledge from the graphics, image, and video domains. However, like other diffusion models, they face huge challenges in achieving controllable generation. Therefore, directly using the text corresponding to fMRI to reconstruct videos will result in unsatisfactory outcomes, as the semanties of embeddings only originate from the text modality. We also need to enhance the embeddings with semantics from the video and image modalities to produce "composite semantics" embeddings, which aid in achieving controllable generation for the T2V diffusion model.
Guidance. We consider the blurry rough-video V Murry output from PR as a Guidance. Treating V Biurry as an intermediate noisy video between target video Vo and noise video VT, the originally required T steps for the complete forward process can now be reduced to 0T steps. By applying the latent space translation and reparameterization trick, noise 2T can be formalized as:
T
0T
2T = CōT C EMurry + /1 oT/ e, āT = II t=1
Ct
II t=1
O2z
(5)
where or represents the noise schedule parameter at time step & and N(O0 1) Gaussian noise. The reverse process involves iteratively denoising the noise video from T steps back to 0 steps.
5

---

Adopting pretrained T2V diffusion model to predict the mean and variance of the denoising distribution each step:
2-11 (0(21-112)) = (aala)))-((aaa1 (0 Eo
(6)
where 1,,,, 1. After translating to pixel space, the reconstructed video Vu is obtained.
Guidance. Guidance only directs the video generation of the T2V diffusion model at the percep- tion level, leading to significant randomness in the semantics of the reconstructed videos. To resolve this issue, need to incorporate keyframe images with more supplementary semanties to control the generation process, thereby enhancing the fidelity of the reconstructed videos. Compared to directly reconstructing keyframe images from fMRI embeddings, combining perception embeddings will be more beneficial for maintaining the consistency of structural and semantic information. Therefore, select the first frame Vi of blurry V Vuurry Input Vi 's embedding and IMRI embedding to SDXL unCLIP [2] (See Appendix D for more discussion about SDXL unCLIP) in SR to reconstruct the keyframe image Xkey as Guidance. We employ ControlNet [54] to add / Guidance the T2V diffusion model, in which the keyframes are used as the first-frame to guide video generation.
y Guidance. The text is the necessary input for the T2V diffusion model. In order to maintain the consistency of visual semantics, we adopt BLIP-2 [53 to generate the caption for the keyframe image Kkey. which is used as 7 Guidance (prompt) for video reconstruction.
The inference process inputs a, 3, y Guidance into the T2V diffusion model, and the IMRI-to-video reconstruction can be completed, resulting in the exquisite video with high fidelity and smoothness,
3.4 Multi-MMRI Fusion
While it is important to emphasize that single-frame fMRI generates higher frame rate video, the more realistic question is how to recover longer video (longer than fMRI temporal resolution). Previous methods treat single-frame fMRI as sample, and temporal attention is computed at the single-frame fMRI level, thus failing to generate coherent videos longer than 2s. With the help of NeuroClips' SR, we explored the generation of longer videos for the first time. Current video generative models are built on diffusion-based image generation models and attention-based transformer architectures, both of which incur significant computational overhead. As the number of frames increases, the content scales linearly, highlighting the limitations in generating long and complex videos efficiently. Therefore, we chose a more straightforward fusion strategy that does require additional GPU training. In the inference process, we consider the semantic similarity of two reconstrueted keyframes from two neighboring fMRI samples (here we directly determine whether they belong the same class of objects, e.g., both are jellyfish). Specifically, we obtain the CLIP representations of reconstructed neighhoring keyframes and train shallow MI.P based on the representations to distinguish whether two frames share the same class. If semantically similar, we replace the keyframe of the latter fMRI with the tail-frame of the former fMRI's reconstructed video, which will be taken the first-frame of the latter fMRI to generate the video. shown in Figure 2 with this strategy, we achieved continuous video reconstruction of up 6s for the first time.
C1
Yeals meae Facm
Figure 2: Visualization of Multi-IMRI fusion. With the semantic relevance measure, we can generate video clips up to 6s long without any additional training.

---

Experimental Setup
4.1 Dataset and Pre- Pre-processing
Dataset. In this study, we performed IMRI-to-video reconstruction experiments using the open- source fMRI-video dataset (cc2017 datase() [3. For each subject, the training and testing video clips were presented and 10 times, respectively, and the testing set was averaged across trials. The dataset consists of training set containing 18 8-minute video clips and a test set containing 5 8-minute video clips. The MRI (T1 and 12-weighted) and fMRI data (with 2s temporal resolution) were collected using& 3-T MRI system. Thus there are 8640 training samples and 1200 testing samples of IMRI-video pairs.
Pre-processing, The IMRI data on the cc2017 were preprocessed using the minimal preprocessing pipeline [55] The fMRI volumes underwent artifact removal, motion correction (6 DOF), registration to standard space (MNI space), and were further transformed onto the cortical surfaces, which were coregistered onto a cortical surface template [56]. We calculated the voxel-wise correlation between the fMRI voxel signals of each training movie repetition for each subject. The correlation coefficient for each voxel underwent Fisher 7-transformation, and the average Z scores across training movie segments were tested using the one-sample t-test. The significant voxels (Bonferroni correction, P 0.05) were considered to be stimulus-activated 13447, 14828, and 91 14 activated voxels were observed in the visual cortex for the three subjects. Following the previous works B 57 588, we used the BOLD signals with delay of 4 seconds as the movie stimulus responses to account for the hemodynamic delay.
activated voxels and used for subsequent analysis A total of
4.2 Implementation Details
In this paper, videos from the cc2017 dataset were downsampled from 30FPS to 3FPS to make a fair comparison with the previous methods, and the blurred video interpolated 8FPS to generate the final 8FPS video during inference. We split semantic reconstruction into two parts: image contrastive learning and the fine-tuning of the diffusion prior adapting to the new image distribution space. In PR, all downsampled frames were utilized, and the inception extension was implemented by a shallow MLP. Theoretically, our approach can be used in any text-to-video diffusion model, and we choose the open- available Diff [44] as our inference model. 1 is set to 0.3 in Guidance, and the inference is performed with 25 DDIM 35 steps (See Appendix A for more implementation details). All experiments were conducted using single A100 GPU.
4.3 Evaluation Metrics
We conduct the quantitative assessments through frame-based and video-based metrics. Frame-based metrics evaluate each frame individually, providing a snapshot evaluation, whereas video-based metries evaluate the quality of the video, emphasizing the consistency and smoothness of the video frame sequence. Both are used for a comprehensive analysis from semantic or pixel perspective.
Frame-based Metrics We evaluate frames the pixel level and semantic level. We use the structural similarity index measure (SSIM) and Peak Signal-toNNoise Ratio (PSNR) as the pixel-level metric and the N-way top-K accuracy classification test (total 1,000 image classes from ImageNet (59) as the semantics-level metric. To conduct the classification test, we essentially compare the ground truth (GT) classification results with the predicted frame (PF) results using an ImageNet classifier 33) g We consider the trial successful if the GT class is among the top-K probabilities in the PF classification results (We used top-1), 1). selected from N randomly chosen classes that include the GT class. The reported success rate is based on the results of 100 repeated tests.
Video-based Metries We evaluate videos at the semantic level and spatiotemporra(ST)--eerr For semantic-lever metrics, similar classification test (total 400 video classes from the Kinetics-400 dataset (60) is used above, with video classifier based on VideoMAE [6]]. For spatiotemporal metrics that measure video consistency, we compute CLIP image embeddings on each frame of the predicted videos and report the average cosine similarity between all pairs of adjacent video frames, which is the common metric CLIPP-pc in video editing.
7

---

cland
Hirn
Video reconstruction the cc2017 dataset. On the left the results of the comparison with previous studies, and the right are additional comparisons with previous SOTA methods. Best viewed with zoom-in. As shown in the leftmost figure group, Mind- Sand---io''ss 's reconstruction fails to go for detail consistency on the character's s face, but our NeuroClips achieves an extremely high consistency.
Figure 3:
Table 1: Quantitative comparison of NeuroClips reconstruction performance against other methods. Bold font signifies the best performance, while underlined text indicates the second-best performance. MinD-Vided and NeuroClips are both results averaged across all three subjects, and the other methods are results from subject I. Results of baselines are quoted from 62]
Video-hased
trame-hased
Method
Semantic-level
ST-level
Semantic-level
Pixel-level
2-way1
50-wayt
CIIIP-peet
2-wayf
50-wayt
SSIM
PSNR
Nishimoto
0.727=0.04
0.1 16400.0
8.012+2.31
Wen 31
0.166=0.02
0.758=0.03
0.070=0.01
0.1 14
7.646::3.48
Wang 32
0.003
0.402
0.713
0.1 18
11.432:#242
Kupershmidt
33
0.77 +:000
0.386
0.764:20.03
0.179
0.135
8.761
HinD-Video
0.839 1:399.0.03
0.197
0.408
0.79620.03
0.174
0.171
8.662
NeuroClips
0.834:10.00
0.220
0.7
0.806
0.203
0.390
9,21 ::146
subject
0.830 03
0.208
0.736
0.79920.05
0.187
0.392:10.08
9.226::1:42
subject 0.837 03
0.230=0.01
0.742.10.1
0.81
0.210=0.01
0.39210.08
9.336:11.22
subject
0.835=9.03
0.221 1 =0.01
0.735=20.20
0.807=0.03
0.213=0.01
0.387=0.00
9.072=1.44
5 Results
In this section, compare NeuroClips with previous video reconstruction methods on the cc2017 dataset. We provide a visual comparison in FigureBaan report quantitative metrics Table [ We purposely focus on the comparison with the previous SOTA method on the right side of Figure 3 due to the lack of obvious semantic information the premature method. All methods report results for all test sets except Wen 31,, whose results are available for only generates videos at 8fps and even higher frame rates. For a fair comparison with previous studies, downsampled the SFPS videos to 3FPS. Unless otherwise noted, the experimental results and visualizations shown below are all at 3FPS.
segment. Our method
As can be seen in Figure 3 earlier methods were unable to produce videos with complete semantics but guaranteed some of the low-level visual features. Compared MinD- Video, our NeuroClips generates single- frame images with higher quality, more precise semantics (e.g., people, turtles, and airplanes), and smoother movements. At the same time, due to the limited data in the training set, some objects in the test set videos did not appear in the training set, such the moon, and perfect reproduction the semantics is difficult. However, thanks to our perception reconstructor (aka a Guidance), NeuroClips basically reproduces the shape and motion trajectory of the moon, although semantically it more similar to the aperture, demonstrating the pixel-level reconstruction ability
https://purr, http://purrruuuuuueue purdue. ettu/publiscatiins/2889

---

of the video. In terms of quantitative metrics, NeuroClips significantly outperformed 5 of the 7 metrics, with a 128% improvement in SSIM performance, indicating that the PR complements the lack of pixel-level control. At the semantic level, our metrics overall outperform previous methods, demonstrating the better semantic alignment paradigm of NeuroClips. For the ST-llvel metric, which evaluates video smoothness, NeuroClips substantially outperforms MinD-Videe because we introduce blurry rough-video (aka a Guidance) for the frozen video generation model, incorporating initial smoothness for video reconstruction. In contrast, MinD-Vides Viddo lacks perception control, resulting in substantial decrease in smoothness, as can also seen in Figure 3 where the human deformations and scene switches are too large within fMRI frame. In addition, benefiting from our keyframes for first-frame guidance (aka 3 Guidance) and blurry videos, we can connect semantically similar videos to generate longer videos, which may be the reason for the slightly lower video 2- way metrics, as neighboring reconstructed videos are more similar after multi-fMRI fusion.
Ablations
In this section, we discuss the impact of three important intermediate processes on video reconstruction, including keyframes, blurry videos, and keyframe captioning. The quantitative results in Table [ and the visualization re- sults are shown in Figure where all the results of the ab- lation experiments are from subject Since the keyframe captioning must be obtained through the keyframe, by default eliminate keyframe and keyframe captioning at the same time. From the quantitative results, it can be seen that NeuroClips exhibits better pixel-level metries without keyframes while showing better semantic- metrics without blurry clips. This indicates a trade-off between semantic and perception reconstruction, which similar to the results of a large body of literature on fMRI-to-image reconstruction []] The perception recon- struction improves the pixel-level performance and the semantic reconstruction improves the semantic metrics. In addition to this, the best ST-level results for the full model demonstrate the contribution of each module to video consistency.
jellyfish flouring in the ocein
GT
jellyfishh floating the
BLIP2 captioning
large body water
GIT captioning
sutoodo vodeo
w/o Key
Figure 4: Visualization of ablation study.
Table 2: Ablations the modules of NeuroClips, and all results are from subject 1.
Video-based
Frame-
Method
Semantic-leved
ST-level
Semantic-level
Pixel-level
2- 2-wayt
50-wayf
CLIP-pect
2-way?
50-wayt
SSIM
PSNR
w/o keyframe
0.751
0.164-10.22
0.695
0.702+0.04
0.128
0.413
9.334
w/o blurry clip
0.008
0.213
0.718=0.11
0.805 03
0.193
0.256#0.11
8.989=1.37
GIT captioning
0.828:4.00
0.195
0.728
0.785 +0.03
0.174++000
0.399
9.297+1.43
NeuroClips
0.830=0.03
0.208
0.736
0.799+0.03
0.187++0.0
0.392+0.08
9.226=1.42
For the image captioning method, previous studies 2 have used the GIT 63 model to generate keyframe captions directly from fMRI embeddings in image space, and generated it from BLIP- 2 [53) Here we compare the text generation results of these two approaches as shown in the Figure 4 We found that GIT's text generation is slightly homogeneous, filled with large number of similar descriptions, such as 'a large body of water', 'a man is standing". For the diffusion model, the influence of text significant, GIT's captions degrade the quality of the semantic reconstruction of the video, e.g., generating flowers on water from jellyfish keyframes. This shows that keyframe- based captioning is more flexible compared to representation based captioning. Finally, we remove the keyframes and keyframe captions and use only blurred videos to guide the reconstruction, with the text input replaced with the generic description smooth video'. With this approach, we find that the model generates the video completely blindly, with poor semantic control, demonstrating the strong semantic support that keyframes bring to the NeuroClips generation capability. More ablation analysis can be found in Appendix C.4

---

7 Interpretation Results
To explore the neural interpretability of the model, we visualized voxel-level weights on brain flat map, where we can see comprehensive structural attention throughout the whole region, as shown in 5 It can be seen that the visual cortex occupies an important position regardless of the task. In addition, for semantic-leve reconstruction, the weight distribution of voxels is more spread out over higher visual cortex, indicating the semantic level of the video. For perceptual reconstruction, weight distribution of voxels is more concentrated on the lower visual cortex, corresponding to the low-level perception of the human brain. See Appendix C.3 for more subjects' interpretation results.
Semantic
Percepitual
Reconsdructio
Resesstructio
Figure 5: Visualization voxel weights the first ridge regression layer subject with each voxel's weight averaged and normalized to between 0 and I and we set the colorbar to 0.25-0.75 for a clear comparison.
8 Conclusion
In this paper, we present NeuroClips, a novel framework for fMRI-to-video reconstruction. we implement pixel-level and semantic-level visual learning of fMRI through two paths: perception reconstruction and semantic reconstruction. With the learned components, we can configure them into the latest video diffusion models generate higher quality, higher frame rate, and longer videos without additional training. NeuroClips recovers videos with more accurate semantic-level precision and degree of pixel-level matching, establishing new tfte-off-the-at in this domain. addition to this, we visualized the neuroscience interpretability of NeuroClips with reliable biological principles.
9 Limitations
Although NeuroClips has achieved high-fidelity, smooth, and consistent multi-fMRI to video recon- struction, there are still slight flaws. Specifically, our framework is slightly bulky and it relies on extending keyframes to the reconstructed video. A model that can reconstruct videos from the CLIP latent space will avoid this intermediate process. Unfortunately, there is no such available model now. In addition, our method does not reconstruct cross-scene fMRI well, i.c., fMRI recorded during video clip switching. Even if such fMRI scans in tiny minority, this will be direction for future research. Moreover, additional subjects and fMRI recordings should be considered in order to reflect real-world visual experiences sufficiently. However, The alleviation of these limitations will require joint advances in multiple areas and significant further effort will required. This because improvements in these areas need be supported by common developments including machine learning, computer vision, brain science, and biomedicine.
Ethics Statements
The dataset paper [3] states that informed written consent was obtained from every study participant according to the research protocol approved by the Institutional Review Board at Purdue University.
Acknowledgements
This research is supported by the National Key Research and Development Program of China (No. 2022YFB1104000 the National Natural Science Foundation of China (No. 61976158. No. 62376198), Shanghai Baiyulan Pujiang Project (No. 08002360429).
10

---

References
[1] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Demp- ster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.
[2] Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with hour data. arXiv preprint vrXiv:2403.11200 2024.
[3] Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(11):1641-1646, 2011.
[4] Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. arXiv preprint arXiv:2404. /2630, 2024.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8763 8763. PMLR, 2021.
[6] Yu Zhang, Qi Zhang, Zixuan Gong, Yiwei Shi, Yepeng Liu, Duoqian Miao, Yang Liu, Ke Liu, Kun Yi, Wei Fan, et al. Mlip: Efficient multi-perspectivve language-image pretraining with 2024.
exhaustive data utilization. arXiv preprint
[7] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[8] Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou, Cinematic mindscapes: High-quality video reconstruction from brain activity. Advances Neural Information Processing Systems, 36, 2024.
[9] Simon Scholler, Sebastian Bosse, Matthias Sebastian Treder, Benjamin Blankertz, Gabriel Curio, Klaus-Robert Muller, and Thomas Wiegand. Toward direct measure of video quality perception using eeg. IEEE transactions on Image Processing, 21(5):261922622 2012.
[10] EVE RABINOFF. Human Perception, pages 43-70. Northwestern University Press, 2018.
[11] Ervin S Ferry. Persistence of vision. American Journal of Science, 3(261):192-20) 1892.
[12] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Ultra-high temporal resolution visual reconstruction from fovea-like spike camera via spiking neuron model. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(1): 1233-1299, 2022.
[13) Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangehen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process. arXiv preprint arXiv:2401. 145555550 2024.
[14] Aditya Patel, Ashish Kumar Khandual, Kavya Kumar, and Shreya Kumar. Persistence of vision display-a review. review. IOSR Journal of Electrical and Electronics Engineering (IOSR-JEEE) 10(4):36-40, 2015.
[15] Andreas Brøgger and Karen Newman. Persistence of vision: the interplay of vision. Vision, Memory and Media, page 13, 2010.
[16] Leslie G Ungerleider, Susan M Courtney, and James V Haxby. A neural system for human visual working memory. Proceedings of the National Academy of Sciences, 95(3):883-890, 1998.
11

---

[17] James J. Gibson. Pictures, perspective, and perception. Daedalus, 89(1):216-2)) 2211 1960.
[18] Jonathan Vacher, Aida Davila, Adam Kohn, and Ruben Coen- Texture interpolation for probing visual perception. Advances in neural information processing systems, 33:22146-22157, 2020.
[19] Changde Du, Changying Du, Lijie Huang, and Huiguang He. Reconstructing perceived images from human brain activities with bayesian deep multiview learning. IEEE transactions on neural networks and learning systems, (0(8)22310-2233 2018.
[20] Mo Chen, Junwei Han, Xintao Hu, Xi Jiang, Lei Guo, and Tianming Liu. Survey of encoding and decoding of visual stimulus via fmri: an image analysis perspective. Brain imaging and behavior, 8:7-23, 2014.
[21] Kendrick N Kay, Jonathan Winawer, Aviv Mezer, and Brian Wandell, Compressive spatial summation in human visual cortex. Journal of neurophysiology. 110(2):881444 -94,, 2013.
[22] Panagiotis Sapountzis, Denis Schluppeck, Richard Bowtell, and Jonathan W Peirce. A com- parison of fmri adaptation and multivariate pattern classification analysis in visual cortex. Neuroimage, 49(2):163 1640, 2010.
[23] Kun Wang, Tianz Jiang, Chunshui Yu, Lixia Tian, Jun Li, Yong Liu, Yuan Zhou, Lijuan Xu, Ming Song, and Kuncheng Li. Spontancous activity associated with primary visual cortex: a resting-state finr study. Cerebral cortex, 18(3):6777704 2008.
[24] Kalanit Grill-Sppetor and Rafael Malach. The human visual cortex. Annu. Rev, Neurosci., 27(1):649-677, 2004.
[25] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1 811):10037, 2017.
[26] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453-14463, 2023.
[27] Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07422. 2023.
[28] Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Yu Zhang, KE LIU, Liang Hu, and Duoqian Miao. Lite-mind: Towards efficient and robust brain representation learning. ACM Multimedia 2024, 2024.
[29] Guangyin Bao, Zixuan Gong, Qi Zhang, Jialei Zhou, Wei Fan, Kun Yi, Usman Naseem, Liang Hu, and Duoqian Miao. Wills aligner: A robust multi-subject brain representation learner. arXiv preprint arXiv:2404. 13282, 2024.
[30] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22710-22720, 2023.
[31] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12) 28(12):413-4111 2018
[32] Chong Wang, Hongmei Yan, Wei Huang, Jiyi Li, Yuting Wang, Yun-Shuang Fan, Wei Sheng, Tao Liu, Rong Li, and Huafu Chen. Reconstructing rapid natural vision with fmriiconditional video generative adversarial network. Cerebral Cortex, 32(20):4502-44 -515500221511,11 2022.
[33] Ganit Kupershmidt, Roman Beliy, Guy Gaziv, and Michal Irani. A penny for your (visual) thoughts: Self-supervised reconstruction of natural movies from brain activity. arXiv preprint arXiv:2206.0 03544, 2022.
[34] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
12

---

[35] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[36] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06//5, 1(2):3 2022.
[38] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[39] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[40] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[41] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[42] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209, 2479.,42., 2022.
[43] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15860 2022.
[44] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0477: 2023.
[45] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[46] Janne Kauttonen, Yevhen Hlushchuk, liro P Jääskeläinen, and Pia Tikka, Brain mechanisms un- derlying cue-based memorizing during free viewing of movie memento. Neurolmage, 172:313- 325, 2018
[47] Junwei Han, Kaiming Li, Ling Shao, Xintao Hu, Sheng He, Lei Guo, Jungong Han, and Tianming Liu. Video abstraction based on fmri-driven visual attention model. Information sciences, 281:781-796, 2014.
[48] Xintao Hu, Kaiming Li, Junwei Han, Xiansheng Hua, Lei Guo, and Tianming Liu. Bridging the semantic gap via functional brain imaging. IEEE Transactions on Multimedia, 14(2)3314-325, 2011.
[49] Pulkit Narwal, Neelam Duhan, and Komal Kumar Bhatia. A comprehensive survey and mathematical insights towards video summarization. Journal of Visual Communication and Image Representation, 89:103670, 2022.
[50] Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Through their eyes: multi-subject brain decoding with simple alignment techniques. Imaging Neuroscience, 2:1-21, 2024.
13

---

[51] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 211222777222944 2022.
[52] Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint 3rXiv:2010.06300 2020.
[53] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[54] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836-388 33367387,,00 October 2023.
[55] Matthew F Glasser, Stamatios Sotiropoulos, Anthony Wilson, Timothy Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80:105-124, 2013.
[56] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171-113 2016
[57] Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fimri activity in visual cortex. Neurolmage, 198: 125-136, 2019.
[58] Chong Wang, Hongmei Yan, Wei Huang, Li, Yuting Wang, Yun-Shuang Fan, Wei Sheng, Tao Liu, Rong Li, and Huafu Chen. Reconstructing rapid natural vision with fmri-conditional video generative adversarial network. Cerebral Cortex, 32(20):4502 4511, 2022.
[59] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages
Icce, 2009.
[60] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 776500650, 2017.
[61] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are lata-efficient learners for supervise video pre-training. Advances in neural information 10093, 2022.
processing systems,
[62] Yizhuo Lu, Changde Du Chong Wang, Xuanliu Zhu, Liuyun Jiang, and Huiguang He. Animate your thoughts: Decoupled reconstruction of dynamic natural vision from slow brain activity. arXiv preprint arXiv: 20550:2405.03280, 2024.
[63] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint
2022.
[64] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051001 2017.
[65] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 1 1006, pages 369-386. SPIE, 2019.
[66] Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.1693: 2023.
14

---

A More Implementation Details
Training Details. For Semantic Reconstructor, We first train the IMRI-to-keyframe alignment for 30 epochs with a batch size of 240 and then tune the Diffusion Prior for 150 epochs with a batch size of 64. For Perceptual Reconstructor, we train it for 150 epochs and the batch size is set to 40. We use the AdamW [64] for optimization, with a learning rate set to 3e-4, to which the OneCircle learning
rate schedule 65
was set. Mixing coefficients ô and μ set to 30 and 1.
Inference Process. In the phase of inference, we use Animate Diff (44) with Stable Diffussin v1.5 based motion module. The scene model is selected as Realistic Vision V60, and keyframe control is implemented by RGB condition with SparseCtrl [66]. For the video diffusion model, the text prompt is set to the keyframe captioning + ,,8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3' and the negative prompt fixed to semiirealistic,, cgi, 3d, render, sketch, cartoon, drawing. anime, text, close up, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, matilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck'.
We use the Connectome Workbench from Human Connectome Project (HCP), and flatmap templates were selected as 'Q1-Q6 R440.L.ffat..22. fs S.._.R.sfur.ii and Q1- Q6 (440.R.flat.322. fs surfuff.g Additionally, the cortical parcellation was manually delineated by neuroimaging specialists and neurologists, and aligned with the public templates in FreeSurfer software with verification. We normalized the voxel weights, scaling them to between and 1. Finally, to show a better comparison, the Colorbar was chosen to be 0.25-0.75.
Visualization Software.
B More Details about Method
B.1 More Details about Perception Reconstructor
We elaborate on the architecture of our Temporal Upsampling in Sec 3.1 here. As illustrated in Figure the Temporal Upsampling module consists of Spatial Layer, Temporal Attention, Learnable Residual Connection, and Upsampling.
The input embedding Ey of Temporal Upsampling has five di- mensions, i.c. Ey E R Ny xexhxw (To express concisely, we have drawn only its N2, y, h, and in Figure 6) Before feeding Ey into the spatial layer, reshape it to E First, we conduct modeling in spatial level, using 3D convolu- tion and spatial attention, formalized as Ej. = Spatial (Ejjet) Note that the spatial layer maintains consistent input and output dimensions so that the dimensions of E'y are exactly the same as those of Ey spat Then, applying the learnable residual connection: Ey 1/1 Exprct + (1 ) Subsequently, we reshape Ey B E x xe and input Ejer to the temporal layer E (Emp which is achieved using Temporal Attention as mentioned in main text. Similarly, we apply the learn- able residual connection again: Ey +- Eyemp + (1 1/2) E Finally, we conduct upsampling the result Ey, formalized as Ey t Upsampling(RR) (Ey) ( E RbxNy xk' xw"
Spatial Layer
17 Egy + (1 77) Eg
Temporal Attention
17 Eyy nj) Eg
Upsampling
We repeat employing the above module in different (c, h. w) until Ey is upsampled the target dimensions.
Figure 6: The detailed archi- tecture of Temporal Upsampling module.
B.2 More Details in Semantic Reconstruction
Here elaborate the three loss functions in Equation 4
15

---

BiMixCo Loss. BiMixCo aligns the keyframe Ae and its corresponding fMRI signal Ve using bidi- rectional contrastive loss and MixCo data augmentation. The MixCo needs to mix two independent fMRI signals. For each Ve, we random sample another fMRI Vmm.. which is the keyframe of the clip index by ma. Then, we mix Ve and Ymm using a linear combination:
y mix()) Mm.) A. V + (1 X,)0
where D denotes mixed fMRI signal and X is hyper-parameter sampled from Beta distribution. Then, adapt the ridge regression map 2 lower-dimensional ) and obtain the embedding ey via the MI.,, i.e. ey2² &()²)) Based on this, the BiMixCo loss can be formed as:
1
N₂
expsimmley,<xx)//
LIMMiXCo
A log
2
k= exp (simleeey exx)
i=1
1
N
(ppssim(e;;,ex... )/T)
(1 X1) log
2N
Ex exp(sim(exr,
(8)
1
Ny
exp(simley,.exx)11
X loog
2N
E exp(sim(eyy, 6x2
j=1
N
Σ (1 (1-1) A5) log cxp(sim(ey,, ex,)
Σ
2N
EN exp(sim(eyz ex, ex,)/T) 3/T)
[1|==]]
where ex. denotes the OpenCLIP embedding for keyframe Xe
Prior Loss We use the Diffusion Prior to transform 1MRI embedding ey, into the reconstructed OpenCLIP embedding of keyframe ex. Similar to DALLE-2, Diffusion Prior predicts the target embeddings with mean-squared error (MSE) as the supervised objective:
Priio Bex.ee...-- -x,
Reftm Loss. In addition to image representation-leve alignment, the assistance of text can aid in generating semantically more compatible images. Considering the inconsistency in the number and dimension of image and text tokens, the previous approaches mostly align the 257 tokens of the image with the CLS token of the text after globally averaging the pooling projection. In this paper, we ignore the CLS token and only map fMRI to 256 tokens of images, so the projection layer of the alignment needs to be adjusted. We develop the Reftm to achieve the alignment between the reconstraction-embeeddng exe, and its corresponding text embedding eT. (the text Te is generated by BLIP2 captioning according to the GT keyframe). To align the dimension of reconstruction- embedding Ex, and that of text embedding eT., add an adjusted projector P(-). The adjusted projector directly maps 256 tokens of ex to the dimension text embedding. We fine-tuned the projector for 20 epochs on the MSCOCÓ dataset, dataset, which consists of 73k images and 5 texts in each image. And the training phase of NeuroClips, the projector was frozen. Table 3 displays the effect of fine-tuning.
Table 3: The effect of fine-tuning on the MSCOCO 2017.
Method
Image2Text
Texr2lmage
Before fine-tuning
81.3% 95.2 %
78.7%
After fine-tuning
95.2% %
Finally, for our Semantic Reconstruction, we use CLIP contrastive loss to align reconstruction- embedding Ex and its corresponding text embedding eT.:
1
exp(sim(PP(E)).cr)))) log Ex exp(sim (ppp(aim(((())))).))))))
+ log exp(sim(PP(c)).c7)))
Lieftm
2N,
E exp(sim(P(e))) em)
(10)
16

---

C More Experimental Results
C.1 Visualizing Reconstructed Keyframe and Text
We display the reconstructed keyframes and their corresponding text in Figure 7 It can be observed that the reconstructed keyframes and ground truth exhibit extremely high semantic similarity. These results demonstrate the ability of our Semantic Reconstructor to reconstruct semantic-accuracy keyframes and corresponding text descriptions. This reconstructed keyframe contains not only the correct object categories but also detailed information such as object position, color, scene structure, etc., which is crucial for reconstructing high-fidelity videos.
Ground Truth Keyframe
Reconstructed Keyframe
Corresponding Text
Ground Truth Keyframe
Reconstructed Keyframe
Coresponding Text
colorful underwater ter scene with many fish
cat bathtub with its mouth open
city with large building in middle of it
boat traveling the water
a pink flower shown photo
man in suit and sitting down
black bear standing its hind legs
large airplane flying in the sky
Figure 7: The reconstructed keyframe and its corresponding text. Best viewed with zoom in.
C.2 Visualizing More Successful Reconstructions
We visualized more successfully reconstructed videos in Figure 8 As it can be seen, our NeuroClips is capable of successfully reconstructing videos with correct semantics, covering various categories such portraits, objects, animals, and natural scenes, demonstrating the effectiveness of our Semantic Reconstructor, By conducting detailed comparisons with the ground truth visual stimulus, we find that the structural information and motions in the videos could also be reconstructed, such the size of the main objects, their positions within the scene and the direction in which person walks. This demonstrates the critical addition of blurred video to motion and structural information.
We further show videos that are successfully reconstructed by all three subjects in Figure which demonstrated generalization of the model. What's more, we find that these videos often consist of simple backgrounds featuring portraits, animals, or objects, as well as some natural scenes. By comparing the reconstruction results of the three subjects, we discover an interesting phenomenon: the perceived size of objects in simple scenes varies between individuals. For example, the sizes the jellyfish and airplane reconstructed from different subjects' fMRI show significant differences. These results also suggest, to some extent, the differences in the human visual system among individuals.
C.3 Visualizing Incorrect Reconstructions
Incorrect results often provide more insights. Therefore, we present some of the incorrect reconstruc- tion results generated by NeuroClips in Figure [0, Most incorrect reconstruction clips arise from semantic erros,, leading to confusion between semantically similar items, such as fish and turtle, man and woman, and cat and dog. This type of error may arise from two main reasons. On the one hand, the semantic accuracy of the keyframes reconstructed may be insufficient. We attempt to
17

---

fMRI embedding to retrieve from a pool of keyframe image representations. When the size of the retrieval pool is set 300, the retrieval accuracy on the test is approximately 0.22, indicating that there is still significant improvement room for visual semantics encapsulated by the fMRI embedding. the other hand, the test set may encounter out-of aut-o-ddistribution issues relative the training set, where semantic categories present in the test set are not seen in the training set, making semantic reconstruction challenging to generalize on the test set.
GT
Ours
GT
Ours
Ours
GT
Ours
GT
GT
Onn
GT
Ours
Figure 8: Visualization of more successful reconstruction results. We displayed 6 frames from each 16-frame vidco generated by fMRL. Best viewed with zoom in.
18

---

GT
Subjot
Subjo2
Sibj03
Subjo1
Subj02
Subj01
Subj02
Subj03
Figure 9: Visualization of successful reconstruction results by three subjects. We displayed frames from cach 16-frame video generated by fMRI. Best viewed with zoom in.
GT
Ours
GT
Ous
Figure 10: Visualization of some incorrect reconstruction results. displayed 6 frames from each 16-frame video generated hy fMRL. Best viewed with zoom in.
19

---

C.4 More Ablation Results
The first frame of the blurry video provides structural information for the reconstruction of the keyframes and motion information for the generation of the video. Here, we perform the ablation visualization of the blurry video to observe its role perception reconstruction. Although the building can still be generated with the blurry video removed as shown in Figure [1] the shape difference from the original im- age is too large, indicating the structural information provided by the blur video. In addition, with the ad- dition of the blurred video guidance, the camera view progressively upward, similar the original image, but with the removal of the blurred video, the cam- view is encircled, suggesting that the perception reconstruction provides motion guidance.
GT
NeuroClips
blurry video
w/o blurry video
Figure 11: Visualization of ablation study.
C.5 More Interpretation Results
Semantic Reconstructor
Perception Reconstructor
subject
subject 2
subject
Figure Additional visualization of voxel-wise weights. Best viewed with zoom in.
In Section 7 we visualized voxel-Ievel weights on brain flat map for subject Here we provide detailed visualization results for all three subjects from the cc2017 dataset, as shown in Figure 12 As can be seen from the Figure, the three subjects learned similar weights in the Perception Reconstructor, which echoes the subjects' comparable SSIM metrics in Table [and demonstrates the existence commonality low-level vision in humans. However, the weights learned by subject in the Semantic Reconstructor are different from the other subjects, which also leads to its image retrieval accuracy being at a lower value, as shown in Table 4 This may indicate that there were differences in the understanding of the video between subjects.
20

---

C.6 Results of Retrieval Metrics
We further evaluate the performance of our Semantic Reconstructor using Top-1 keyframe retrieval accuracy and Top-1 fMRI retrieval accuracy. For keyframe retrieval, each test fMRI Vi is first converted an fMRI embedding ey,, and we compute the cosine similarity its respective CLIP keyframe embedding ex, and 299 other randomly selected CLIP keyframe embedding in the test set. For each test sample, success is determined if the cosine similarity greatest between the fMRI embedding and its respective CLIP keyframe embedding (aka top-1 retrieval performance, random chance is 1/300). The test set contains 1, 1,200 fMRII-eyframe pairs, and we randomly divide into parts (thus each part contains 300 test pairs) for retrieval evaluation. We report the average retrieval accuracy among 4 parts. The same procedure used for fMRI retrieval, except fMRI and keyframe are flipped. T displays the retrieval performance.
Table 4: The top-1 retrieval accuracy for each subject.
Subject
Keyframe Retrieval (t)
fMRI Retrieval (1)
subject 1
23.1%
20.7%
subject 2
26.3%
19.8%
subject 3
17.3%
15.9%
Unlike the latest techniques for IMRI image reconstruction (where retrieval accuracy can reach more than 90%), NeuroClips' retrieval accuracy on the cc2017 dataset averages around 22%, which is at the lower end of the scale. Leaving aside the difference between fMRI in image stimuli and video stimuli, we found that the cc2017 test set had a large number of categories of objects that did not appear in the training set. This likely to be the main reason for the low retrieval accuracy and is also the reason why most of the previous studies have focused on low-level visual reconstruction. In the future, large-scale fMRI-video dataset more compatible with deep learning is worth looking forward to.
D Is NeuroClips Credible?
A big difference in NeuroClips' video generation inference is that freeze the weights of the video diffusion model, while MinD-Video fine-tunes it. This raises the question of whether our generation relies exclusively the knowledge base that the video diffusion model has been trained on, and is not consistent with the distribution of video training data related to fMRI.
Firstly, since our keyframes are control conditions as first frames, the content of our video generation closely related to the keyframes, both semantically and structurally. So NeuroClips, the video diffusion model will not rely entirely its own semantic knowledge to generate videos, but on the keyframes. Secondly, for keyframe generation, keyframes are generated by SDXL unCLIP. The normal version of SDXL unCLIP generates images from larger dimensional text representation space, which was fnnettuned for 6 epochs by Scotti et al. (2 on MSCOCO with image representations. The fine-tuned version of SDXL unCLIP provides a 256 1664 large dimensional CLIP representation space, which has a strong correspondence with the pixel space of the image, shown in Figure 13 The SDXL unCLIP
is this fine-tuned version. The videos captured and part of the COCO dataset, but SDXL unCLIP can still restore the original images as shown in Figure [3] demonstrating the equivalence of its representation space and pixel space. Since NeuroClips trained Semantic Reconstructor align IMRI to this representation space, this also shows that our keyframe reconstruction is closely related fMRI training.
from the cc2017 dataset used this paper are from Videoblocks http:://www..ideollcctt.cc
YouTube (https://www.. http://wwwwwubb..ooomoo These videos
bag5-14
unCLPP
ViI bigg-14
UPCLUP
SDKL
Original from
Original
Figure 13: Visualisation of the generalization ability of SDXL unCLIP.
21

---

E
Broader Impacts
Our research is dedicated exploring the possibilities of neural decoding using deep learning techniques, which have a positive impact on the field of neuroscience. With the expansion of model scales and the improvement of corresponding hardware devices, this research will also make positive contributions to the field of brain-computer interfaces. However, this research also highlights the importance of personal privacy and security. Governments and research institutions should take appropriate measures to protect the privacy data collected and prevent any potential misuse. In our research, we use publicly available and de-identified datasets, so the study strictly adheres to relevant ethical requirements.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state the claims made, including the contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We create a separate "Limitations" section in our Appendix to discuss the limitations of our works.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when the image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: The novelty of our work is focused on the application level and does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed experimental details in the paper and appendix to ensure the reproducibility of our work. In addition, we will make the code and weight files public if the paper accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We use publicly available datasets and provide anonymous links to project.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess httae//guiieee/oeeebbbmiisse
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips ccpps.cc//pp wttpppulllllllpie https//oooooouaassssooooooooo11e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the key experimental setup in the paper and provide the complete experimental setup in our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the average and standard deviation of multiple runs to make the experimental results statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttls://eeuuuuee wtw.rpsu.p...// htcppp.ccppppppe cccpuullic/thhicccuueeeel......
Answer: [Yes]
Justification: Our research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts of our work in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited. And the license and terms of use are explicitly included in our code repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
Our anonymized URL includes these new codes, new results, and related
documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "pGUO6cv3yW", "reviewer": "Reviewer_85K8", "summary": "This paper proposes NeuroClips, a framework that decodes high-fidelity and smooth video from fMRI. NeuroClips uses a semantics reconstructor for video keyframes to ensure semantic accuracy and consistency, and a perception reconstructor for capturing low-level perceptual details, ensuring video smoothness.", "questions": "- Why is there no model or loss function for movement(motion)?\n- Are you willing to make the code publicly available?", "limitations": "As mentioned above, motion reconstruction has not been resolved.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "NeuroClips is the framework to decouple high-level semantics and low-level perception flows for fMRI-to-video reconstruction, achieving high-fidelity and smooth video outputs.\nThe framework addresses the temporal resolution gap between fMRI and video data, ensuring smooth and consistent video outputs through innovative modules like Inception Extension and Temporal Upsampling.", "weaknesses": "The video captures the semantic meaning well but fails to accurately follow the ground truth movement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for taking the time to review our work. Your effort has ensured that our submission received adequate attention and review. We address the questions and clarify the issues accordingly as described below.\n\n> **Weakness 1: NeuroClips Fails to Accurately Follow the Ground Truth Movement**\n\n+ Thanks for pointing out your concern. The fMRI-to-video decoding task presents significant challenges related to both high-level semantic comprehension and low-level motion perception. On the one hand, fMRI data is characterized by its high dimensionality, with hundreds of thousands of voxel signals captured across the entire brain. Even after rigorous voxel selection processes, approximately 10,000 voxels remain. On the other hand, fMRI data is heavily influenced by human behavior in response to visual stimuli, resulting in an extremely low signal-to-noise ratio.\n\n+ The aforementioned hurdles complicate the decoding of precise and comprehensive ground truth movement from fMRI data in fMRI-to-video reconstruction. Nevertheless, through this endeavor, we present elaborated designed components and loss functions tailored for motion perception. These refinements enable NeuroClips to efficiently capture motion details and show impressive video reconstruction results. It is a valuable and motivating contribution to the fMRI decoding community. Further elaboration on these advancements is presented in the following responses.\n\n> **Question 1: No Model or Loss Function for Movement**\n\n***NeuroClips is equipped with a specific model design and a tailored loss function, which guarantees its ability in capturing motion information.***\n\n* **Regarding Model Design.** The Perception Reconstructor (PR) module we designed can perceive movement (motion) information. In this work, we focus on capturing ***generic motion*** within videos by modeling the motion information from ***two perspectives***: perceiving the ***structured information within frames*** (pertaining to objects' shape, position, and orientation) and grasping the ***dynamic information across frame sequences*** (related to the movements of an object or the dynamics of the scene). To accomplish this, we introduce a spatio-temporal attention module within **Temporal Upsampling**, which is explicitly designed to decode spatiotemporal (structural-dynamic) details, i.e., motion cues, from fMRI data. We utilize the cues to guide the Video Diffusion model toward a more nuanced perception of motion dynamics, ultimately guaranteeing the motion-awareness of the video generation of NeuroClips. This is our meticulous design from the model perspective for capturing movement (motion) information.\n\n* **Regarding Loss Function.** The Mean Absolute Error (**`MAE`**) part within the loss function (**Eq. 1**) quantifies the disparity between the generated video and the groundtruth video frame-by-frame, facilitating the perception of generic motion within videos. Note that the perception embeddings of video frames, denoted as $\\mathbf{E}_\\mathcal{X}$, are aligned to the latent space of the Stable Diffusion Variational Autoencoder, which can be the equivalent of the pixel space at the frame level. Numerous recent video generation models have shown that the effective perception of generic motion can be achieved through two fundamental yet straightforward designs: **temporal attention mechanisms** and **frame-level loss functions** [1, 2]. Consequently, the design inspiration is embraced in this study.\n\n* **Regarding Experimental Evidences.** In **Figure 3**, the turtle's swimming direction, the airplane's orientation and flight direction, and the motorcycle rider's posture, along with their corresponding motion details, are **accurately reproduced**. If NeuroClips be exclusively crafted at a semantic level, it becomes evident that semantics alone are inadequate for capturing these motions. In addition, the exceptional accuracy of NeuroClips, as evidenced by the **pixel-level** and **ST-level** metrics in **Table 1**, further corroborates these visualization outcomes.\n\n* **Regarding Fine-grained Motion Decoding.** Currently, decoding fine-grained movements from fMRI data poses a **significant challenge** due to its **intricate and noisy nature**. Unlike text, fMRI data lacks straightforward cues such as motion-descriptive words, making the motion perception more complex. NeuroClips stands apart from previous text-to-video models that are capable of generating motions closely aligning with textual motion semantics. Note that NeuroClips has achieved significant progress in fMRI-to-video decoding tasks. Certainly, NeuroClips is by no means the final solution for fMRI-to-video decoding tasks; undoubtedly, superior solutions will emerge in the future. Looking ahead, there are **promising prospects** for enhancing the accuracy of decoding **fine-grained** movements from fMRI data in the future. This will be the focal point of our upcoming efforts.\n\n\n> **Question 2: Code Release**\n\nWe promise to release the code at the earliest opportunity, following the acceptance of the paper. We appreciate and support open source because it helps more researchers contribute to the field and advances its development. This is an effective way to further enhance the significance and value of our research.\n\n**Reference**\n\n[1] Stable video diffusion: Scaling latent video diffusion models to large datasets, *Stability AI*\n\n[2] Align your latents: High-resolution video synthesis with latent diffusion models, *CVPR 2023*\n\n&nbsp;\n***\nYou have offered many constructive and valuable suggestions, making our submission more solid and complete. Once again, we sincerely express our best gratitude for your effort and time!\n\nBest wishes,\n\nAll authors of Submission 351", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CpukNy9v9K", "reviewer": "Reviewer_mcBd", "summary": "The proposed framework NeuroClips introduces a strong pipeline for fMRI-to-Video reconstruction in the field of Brain Visual Decoding. The Perception Reconstructor(PR) maintains the motion of the video and the Semantic Reconstructor(SR) ensures the semantic information of the video. Multi-fMRI Fusion raises upper video length limit, and overall model achieves impressive results.", "questions": "1. As discussed in mind-video [1], the nature of hemodynamic response has been considered and specific modules are designed, which seems not to be included in NeuroClips. Are there other considerations or is there no need to take into account for the BOLD signals?\n2. Since a number of cross-subject models already exist in the image reconstruction field [2], does NeuroClips need to train a separate model for each subject?\n3. As you mention in the limitation section of the appendix, the cc2017 dataset test set contains too many no-show categories, and I'm curious if the unsatisfactory results of previous methods are more due to the low quality of the dataset, less than the method itself?\n4. Why text contrastive loss is placed after diffusion prior and not before like in [3]?\n\n[1] Chen, Zijiao, Jiaxin Qing, and Juan Helen Zhou. \"Cinematic mindscapes: High-quality video reconstruction from brain activity.\" Advances in Neural Information Processing Systems 36 (2024)\n\n[2] Scotti, Paul S., et al. \"MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data.\" arXiv preprint arXiv:2403.11207 (2024).\n\n[3] Sun, Jingyuan, et al. \"NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties.\" arXiv preprint arXiv:2402.01590 (2024).", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clearly written and easy to read, with clear diagrams and charts.\n2. Experiments and discussions are conducted extensively, including rich video reconstruction and neural interpretation visualization content to validate the model's performance, which strengthens the results and the paper in general.\n3. The proposed rough-video reconstruction in Perception Reconstructor(PR) and the strategy of Multi-fMRI Fusion are generally innovative, and have greatly contributed to break through the original low frame rate and fixed length of 2s video limitations in previous methods. In addition, the designs of NeuroClips for textual semantics and pre-trained diffusion models for video generation are also unique. Considering NeuroClips' powerful results and methods, I think it can be a strong baseline for the emerging fMRI-to-video reconstruction field.", "weaknesses": "1. I browsed the anonymous site, and the generated results are impressive. However, the lighting of some of the reconstructed videos varies considerably compared to the ground truth, which can also be seen on the right side of Figure 2, and the authors need to rationalize this.\n2. Existing state-of-the-art video generation models can generate high-frame rate videos such as 24 fps for up to 1 min or even longer, however NeuroClips at this stage has not yet reached this level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for taking the time to review our work. We value each of your suggestions and provide the following responses:\n\n> **Weakness 1: Light Variations**\n\nWe highlight that **light variations** are also a feature that distinguishes video from images. Our initial hypothesis was that this phenomenon was caused by the presence of partial light variations in the blurred video of the Perception Reconstructor (PR). However, subsequent experiments involving the **removal** of the blurred video revealed that the effect persisted. This may be caused by the pre-training of AnimateDiff, where a corresponding discussion will be further explored in the final version of the paper. It is still important to emphasise that although the phenomenon exists, it is only observed in a **limited number** of videos.\n\n\n> **Weakness 2: Longer and High-frame Videos**\n\n+ **From the Dataset.** After counting the video clips in our dataset, we discovered that the longest video spanned merely around 10 seconds. Hence, there is no need to reconstruct videos that extend up to a minute in duration.\n\n+ **From an availability standpoint.** Currently, the most cutting-edge video generation technology has now advanced to create longer videos, like **`Sora`**. However, the quality of the generated content still raises concerns due to the current limitations of the technology. Additionally, it's important to highlight that the majority of these technologies are **not open source.**\n\n+ **From a Research Perspective.** fMRI-to-video reconstruction is an emerging field. At this critical stage, we believe that the applicability and scalability of innovative methods are of greater consequence. Empirically, once longer blurred videos or video generation tools are available, **NeuroClips** can generate **24-frame and longer videos**. However, this will **remarkably increase the GPU usage**. From a research standpoint, we believe that the current reconstructed video is impressive and sufficient.\n\n> **Question 1: Hemodynamic Delay**\n\nIndeed, a considerable number of **exploratory experiments** have been conducted to study **hemodynamic delay**, a topic also considered in Mind-Video. In the current version of NeuroClips, with a fixed 4-second delay, it was observed that the semantics of some of the generated keyframes exhibited latency, particularly in instances where the video clips of the scene were of a **greater duration**. To address this, a **sliding window** comprising two or three fMRI frames was devised for the purpose of actively learning the aforementioned delay. It was discovered that the application of a sliding window resulted in a **notable reduction** in the final evaluation metrics, with a **more pronounced negative impact**, particularly in the case of shorter video clips. It may be the case that longer videos have a more enduring effect on human brain fMRI signals. In light of the experimental outcomes, we ultimately opted to discard this approach and instead maintain a fixed delay.\n\n> **Question 2: Cross-Subject**\n\nYes, it is necessary to train a **distinct** model for each subject when using NeuroClips. As cross-subject approaches in this area still fail to achieve satisfactory results, we finally choose to utilise and explore a **single-subject** model in this paper. However, in light of the recent advancements in **fMRI-to-image reconstruction**, a series of cross-subject models have emerged [1, 2, 3]. We believe that extending NeuroClips to encompass cross-subject content will be a **promising avenue** for future research.\n\n\n**Reference**\n\n[1] Mindbridge: A cross-subject brain decoding framework, *CVPR 2024*.\n\n[2] Psychometry: An omnifit model for image reconstruction from human brain activity, *CVPR 2024*.\n\n[3] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data, *ICML 2024*\n\n> **Question 3: Low Quality of Dataset**\n\nSince both NeuroClips and the previous baselines are founded upon the **same dataset**, the ensuing comparison is deemed to be **equitable**. The assertion that the CC2017 dataset is of poor quality is intended to **highlight** the difficulty in achieving impressive fMRI-to-video reconstruction results in **comparison to fMRI-to-image reconstruction**. The capacity to produce superior outcomes on datasets of inferior quality serves to illustrate the **resilience** of the NeuroClips' method. It would be of interest to ascertain whether NeuroClips could achieve even more impressive results with a higher-quality dataset.\n\n\n> **Question 4: Enhancement from Text Modality**\n\nConsidering that the diffusion prior loss is rooted in MSE at the representation level, this prior is inherently **unstable** and the semantic information is susceptible to **bias**. It is important to recognise that text modality possesses its own **distinctive characteristics** and **robust semantic support**, which serve to complement the image representation space. To enrich the semantic depth of the representation, we strategically place text assistance subsequent to the prior.\n\n&nbsp;\n***\nYou have offered many constructive and valuable suggestions, making our submission more solid and complete. Once again, we sincerely express our best gratitude for your effort and time!\n\nBest wishes,\n\nAll authors of Submission 351", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ECK6k8YsaE", "reviewer": "Reviewer_oPVf", "summary": "The paper introduces NeuroClips - a framework for reconstructing high-fidelity videos from fMRI data. It combines pixel-level and semantic-level visual learning through perception and semantic reconstruction pathways. The Perception Reconstructor (PR) ensures smoothness and consistency by creating a rough, continuous video, while the Semantics Reconstructor (SR) generates high-quality keyframes. These components are integrated into a video diffusion model, resulting in high-quality, detailed videos. Also, no additional post-diffusion training is required. NeuroClips sets a new standard in fMRI-to-video reconstruction, demonstrating significant advancements in semantic precision and pixel-level matching by improving SSIM by 128% and spatiotemporal metrics by 81%.", "questions": "- The paper mentions accounting for the hemodynamic delay (BOLD signal delay of approximately 4 seconds), there is limited discussion on how this delay specifically impacts the reconstruction quality and temporal alignment with video frames. \n\n- The use of ridge regression to map fMRI signals to lower-dimensional embeddings assumes a relatively linear relationship between neural activity and visual stimuli. However, the brain's processing of visual information is highly nonlinear and complex. At least, it should be mentioned in the Limitations too.\n\n- Fig 12: The weights of subject 3 are really similar for PR and SR tasks, can you elaborate on it?\n - Fig 2: Example images are too small to perceive it even with a zoom, I recommend to make them bigger (as in Fig 7).\n- I recommend adding the major limitations in the Conclusion section of the main text.\n\n- Authors should include in the Supplement section which software was used to build brain maps with weights and if any data pre-processing/normalization was applied.\n\n- Lines 75-76: “NeuroClips achieves a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics and also performs better overall on most video semantic-level metrics.” -  report not only the percent of improvement, but also the values of the metrics.\n\n- In the frame validation the main image quality metrics are PSNR and SSIM. Authors should mention that those metrics have flaws (Zhang et al, The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, 2018). Will it be possible to consider other quality evaluation metrics? Like, for example, Visual information fidelity metrics (Sheikh et al., Image information and visual quality, 2006). It is also interesting why PSNR and SSIM were used for frames-based evaluation but not for video-based evaluation with modification to catch spatio-temporal data (ST-PSNR and ST-SSIM)? Another question is why authors choose to evaluate performance only with “CLIP image embeddings on each frame” and omit other metrics like Fréchet Video Distance (FVD), MSE or already mentioned ST-PSNR and ST-SSIM?\n\n- Fig 12: The visualization of voxel weights on a brain flat map to interpret neural activity is a significant strength. However, now it is difficult to comprehend the differences in weights from the image with naked eyes. The addition of a third row with the difference between the weights of participants can increase the clarity of what authors are trying to say.\n\n- The study focuses primarily on the visual cortex. While this is appropriate for the simplest visual stimuli (i.e., dots, simple geometrics shapes, etc.), it may limit the generalization of the approach to other types of brain activity or cognitive functions. Discussing potential extensions to other brain regions and types of neural data could provide a more comprehensive neurobiological perspective.\n\n-  Lines 316-318: “From the quantitative results, it can be seen that there is a trade-off between semantic and perception reconstruction” - there is no clear explanation or reasoning why there is a trade-off between SR and PR\n\n- Line 38: “However, the visual decoding of MinD-Video significantly diverges from the brain’s visual system, exhibiting limitations in perceiving continuous low-level visual details.” - the more precise listing of what hinders MinD-Video to perceiving continuous low-level visual details would be more constructive, rather than comparison with the brain's visual system. \n\n- Lines 41-42: “Notably, the human brain perceives videos discretely [8, 9] due to the persistence of vision [10, 11] and delayed memory [12].”\n\t1. discretely -> discreetly\n\t2.  Was it intentional to use the paper [10] of 1892 year for the “persistence of vision” concept? Also, the fact that the brain has a “persistence of vision” was criticized. Actually, the second citation you’ve used [11] is the critique of this concept (J Anderson et al., The myth of persistence of vision revisited, 1993). The citations of the novel research papers are required to justify this statement.\n\t\nThe paper will be significantly improved by more valid justification of claimed neurobiological concepts. \n\n\nMinor:\n- Line 644: “However, these limitations will not be tackled overnight” - this phrase is informal and somewhat colloquial, which may not be entirely appropriate for a scientific paper\n- Line 56: “This process is reflected in the fMRI signal” - can you provide the link to the paper supporting this statement?\n- Line 129: “...loss to train the PR, the overall loss L_{PR} of PR can be donated as…” - “donated” is a confusing word to use, please use another word.\n\n-  Lines: 222-223: “Since the technical field of long video generation is still immature, we chose a more straightforward fusion strategy that” - can not see why immatureness of the field can be used as a reason to use a straightforward strategy. Please, make the reason for usage more understandable. \n\n- Figure 4 in page 8: I recommend not to use comic sans font in images. \n\n- Lines 596-597: “It proves that there may be a large difference in the understanding of the video” - please refrain from using the word “prove” here, since it implies comprehensive research or, at least, references to other works that proves it. \n\n- Could you report the number of classes you have at least in training and testing dataset (for the “N-way top-K accuracy classification test as the semantics-level metric”)\n\n- Line 55-56: “generating high-level images in the cerebral cortex” - it is also knows that brain interpolates the seen scenes (Vacher, Jonathan, et al. Texture interpolation for probing visual perception, 2020), which could be used as justification to use keyframe approach.", "limitations": "Important limitation - the dataset of only 3 participants - was not mentioned. This may result in varying interpretations with a careful review by a neurobiologist. Additional evaluation is needed with other datasets and more participants.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Framework Design introduces a dual-component approach with a Semantics Reconstructor and a Perception Reconstructor to handle both high-level semantics and low-level perceptual details, respectively. Which is a novelty.\n\n- Authors provide a comprehensive overview of existing methods for fMRI-to-video and fMRI-to-image tasks.\n\n- The paper provides a thorough explanation of the NeuroClips framework, including the components, training procedures, and inference process. The implementation details are well-written so that readers can follow the technical aspects of the work. It also ensures reproducibility.\n\n- The validation procedure is well-structured and clearly explained. The division of validation metrics in two (frames and video flow) allows one to better understand the performance of the NeuroClips framework.\n\n- The newly introduced NeuroClips’ SR which allows the generation of longer videos. The novelty of the Neuroclip pipeline, the high validation metrics scores and creative approach to dealing with fMRI data makes the proposed framework significant.", "weaknesses": "- The multi-fMRI fusion strategy is briefly described, but the implementation details and the rationale behind specific design choices are not fully elaborated. I suppose the scheme of the multi-fMRO strategy in the supplement would increase the clarity of this paragraph. \n\n-  As it can be seen from the examples provided with the code repository, the proposed framework does not account for a change of the scene in the video (which was briefly mentioned by the authors in the Limitations). The pipeline with chosen keyframe might hinder the NeuroClips  to catch this rapid change in the video. No ablation is done in this direction, which could explain how neural NeuroClips decodes fMRI signals. \n\n- The paper primarily evaluates the method on a specific dataset (with only 3 patients), which, obviously, may not fully capture the diversity of real-world video content and fMRI recordings.This should be at least mentioned in the Discussion (or Conclusion) of the work.\n\n - The neurobiological justification of keyframe usage seems ambiguous. The improvement of text clearance and up-to-date references would increase the significance of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your careful review. In particular, you have provided us with more than twenty constructive suggestions and insightful questions, which is extremely precious at a time when the quality of reviews in the community is deteriorating. We value each of your suggestions and provide the following responses:\n\n> **Weakness 1: Details of Multi-fMRI Fusion**\n\nWe acknowledge and accept your suggestion. Considering that Multi-fMRI Fusion is merely an extra merit of NeuroClips and the page limitations, we were unable to present more details regarding multi-fMRI fusion.\nThanks to the design of keyframes in the Semantic Reconstructor (SR), Multi-fMRI Fusion can be effectively achieved. In this study, we obtain the CLIP representations of reconstructed neighboring keyframes and train a shallow MLP based on the representations to distinguish whether two frames share the same class. The exact process is also shown as the GIF in the code repository. This training process operates at the image level. Despite our efforts to assess whether neighboring fMRI frames belong to the same scene class at the fMRI frame level using an MLP with fMRI frames as inputs, aiming for fMRI anomaly detection, our results were suboptimal, as illustrated in the following table. As can be seen, the fMRI level analysis frequently led to excessive false fusions. Conversely, established techniques for categorizing images perform reliably, particularly within the proficient CLIP space we employ.\nWe promise to include more technical details in the final version. Thank you for your suggestion.\n\n| Fusion-level            | Subject 1 | Subject 2 | Subject 3 |\n| ----------------------- | --------- | --------- | --------- |\n| fMRIs                   | 59.7%     | 58.8%     | 57.2%     |\n| Reconstructed keyframes | **86.3%** | **87.1%** | **85.6%** |\n\n> **Weakness 2: Rapid Scene Change**\n\n**We will present four perspectives on the inability of NeuroClips to perceive rapid changes in the scene**\n\n* **Regarding Dataset.** The rapid scene change depicted in the code repository is a **distinct** alteration influenced by **human intervention**, diverging from authentic real-world visuals. In the CC2017 dataset, video sequences were randomly divided into discrete segments, then amalgamated and presented to subjects. Such spliced videos are seldom encountered in everyday visual encounters, as scenes typically unfold continuously. This is what we have mentioned in Limitation about *cross-scene fMRI*.\n\n* **From the fMRI Data.** We must acknowledge that **continuous** scene transitions occur in actual visual perception, such as when a person turns their head. However, it remains unknown whether and how rapid scene changes occurring within an fMRI frame (e.g., 2 seconds) are reflected in the fMRI signal due to its intricate nature. Consequently, decoding scene changes from fMRI data poses a significant challenge.\n\n* **From Keyframe Design.** When scene changes occur within one fMRI frame (e.g., 2s), the design of the keyframes may adequately capture it, although the blurred video would be continuous. However, the keyframe-based method to guide the generation of continuous video can serve as a foundation for the subsequent design of decoding two consecutive scenes separated rapid scene change.\n\n* **From Cross-scene Diffusion Models.**  Presently, most of the current video generation diffusion models are not capable of very silky smooth scene switching, like SORA, to the best of our knowledge. We believe that, from a technical point of view, the issue can be better alleviated if consecutive semantics can be decoded from a single fMRI frame, or if the temporal resolution of fMRI can be refined.\n\nThanks again for your suggestions. We will provide the results of some of our exploratory experiments and a detailed discussion in the final version.\n\n> **Weakness 3: Generalization Capability**\n\n**Diverse real-world video content**. The videos in the CC2017 dataset used are, somehow, sufficiently reflective of real-world visual experiences. As the original dataset paper states 'All video clips were chosen from Videoblocks and YouTube to be diverse yet. For example, individual video clips showed people in action, moving animals, nature scenes, outdoor or indoor scenes, etc' [1]. \n\n**Diversity of fMRI recordings.** To ensure consistency with the baselines [2] and to make a fair comparison, we experimented on the dataset, which unfortunately has only 3 subjects. We appreciate you pointing out that this should at least be mentioned in Limitations, and we will add it in the final version. Publicly available datasets for fMRI-to-video reconstruction are valuable and not easy to find. Until now, we discovered that the Algonauts 2021 [3] dataset can also be used for video reconstruction, but it is a great pity that this dataset is currently in an unpublished stage. To show the generalization capability of our approach, we finally chose to perform fMRI-to-image reconstruction on the Natural Scenes Dataset (NSD) [4] to assess the keyframe effect of our Semantic Reconstructor (SR). The visual results of the reconstructed image and groundtruth images are shown in the **`PDF`** appendix. Notably, our method exhibited satisfactory reconstruction outcomes even when applied to fMRI data with a distinct distribution, signifying the generalization capabilities of NeuroClips.\n\n**Reference**\n\n[1] Neural encoding and decoding with deep learning for dynamic natural vision, *Cerebral cortex 2018*\n\n[2] Cinematic mindscapes: High-quality video reconstruction from brain activity, *NeurIPS 2023*\n\n[3] The algonauts project 2021 challenge: How the human brain makes sense of a world in motion, *2021*\n\n[4] A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence, *Nature neuroscience 2022*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers, AC, and SAC for their valuable time and selfless dedication. We are very pleased to see that the reviewers recognize the quality of our presentation (mcBd, oPVf), consider our experiments solid and extensive (mcBd, oPVf), and approve the novelty or soundness of NeuroClips (85K8, mcBd, oPVf). In particular, we are greatly encouraged by reviewer oPVf's recognition of the significance of our work. Meanwhile, we deeply value the reviewers' precious suggestions and questions, which we have addressed one by one in the rebuttal.

Meta-review: This paper proposes NeuroClips, a framework that decodes high-fidelity and smooth video from fMRI. NeuroClips uses a semantics reconstructor for video keyframes to ensure semantic accuracy and consistency, and a perception reconstructor for capturing low-level perceptual details, ensuring video smoothness.
Additionally, the reviewers are convincingly shown that NeuroClips is equipped with a specific model design and a tailored loss function, which guarantees its ability in capturing motion information.
Questions on rate and resolution have been convincingly addressed.
Cross-subject modelling will be addressed in future work, which is fine.
Overall, the authors gave  detailed and convincing answers to all technical questions made by reviewers.
The contribution is both impressive and interesting, and thus certainly deserves a wide visibility within NeurIPS conference.
Decision: Accept (oral)