NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction

Authors: zixuan  gong
guangyin  bao
qi  zhang
zhongwei  wan
duoqian  miao
shoujin  wang
lei  zhu
changwei  wang
rongtao  xu
liang  hu
ke  liu
yu  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 14:00:54

Abstract: Reconstruction of static visual stimuli from non-invasion brain activity fMRI achieves great success, owning to advanced deep learning models such as CLIP and Stable Diffusion. However, the research on fMRI-to-video reconstruction remains limited since decoding the spatiotemporal perception of continuous visual experiences is formidably challenging. We contend that the key to addressing these challenges lies in accurately decoding both high-level semantics and low-level perception flows, as perceived by the brain in response to video stimuli. To the end, we propose NeuroClips, an innovative framework to decode high-fidelity and smooth video from fMRI. NeuroClips utilizes a semantics reconstructor to reconstruct video keyframes, guiding semantic accuracy and consistency, and employs a perception reconstructor to capture low-level perceptual details, ensuring video smoothness. During inference, it adopts a pre-trained T2V diffusion model injected with both keyframes and low-level perception flows for video reconstruction. Evaluated on a publicly available fMRI-video dataset, NeuroClips achieves smooth high-fidelity video reconstruction of up to 6s at 8FPS, gaining significant improvements over state-of-the-art models in various metrics, e.g., a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics. Our project is available at https://github.com/gongzix/NeuroClips.

Main paper: 
 introduction: 
 Decoding visual stimuli from neural activity is crucial and prospective to unraveling the intricate mechanisms of the human brain. In the context of non-invasive approaches, visual reconstruction from functional magnetic resonance imaging (fMRI), such as fMRI-to-image reconstruction, shows high fidelity [1, 2, 3, 4] , largely benefiting from advanced deep learning models such as CLIP [5, 6] and Stable Diffusion [7] . This convergence of brain science and deep learning presents a promising data-driven learning paradigm to explore a comprehensive understanding of the advanced perceptual and semantic functions of the cerebral cortex. Unfortunately, fMRI-to-video reconstruction still presents significant hurdles that discourage researchers, since decoding the spatiotemporal perception of a continuous flow of scenes, motions, and objects is formidably challenging.At first glance, fMRI measures blood oxygenation level-dependent (BOLD) signals by snapshotting a few seconds of brain activity, leading to differential temporal resolutions between fMRI (low) and videos (high). The previously advisable solution to address such differential granularity is to perform self-interpolation on fMRI and downsample video frames to pre-align fMRI and videos. Going further, decoding accurate high-level semantics and low-level perception flows has a more profound impact on the ability to reconstruct high-fidelity videos from brain activity. Early studies before 2022 struggled with achieving satisfactory reconstruction performance, as they failed to acquire precise semantics from powerful (pre-trained) diffusion models. The latest research MinD-Video [8] guides the diffusion model conditioned on visual fMRI features, making an initial attempt to address the semantic issue. However, it lacks a design of low-level visual detailing, so it significantly diverges from the brain's visual system, exhibiting limitations in perceiving continuous low-level visual details. The brain's reflection of video stimuli is a crucial factor that influences and enlightens the visual decoding of fMRI-to-video reconstruction. Notably, the human brain perceives videos discreetly [9, 10] due to the persistence of vision [11, 12, 13, 14, 15] and delayed memory [16] . It is impractical to perceive every video frame, and instead, only keyframes elicit significant responses in the brain's visual system. Reconstructing keyframes from fMRI avoids the issue of differential temporal resolutions between fMRI and videos. Also, the precise semantics and perceptual details in keyframes ensure the high fidelity and smoothness of reconstructed videos, both within and across successive fMRI inputs. Accordingly, we argue that utilizing keyframe images as anchors for transitional video reconstruction aligns with the brain's cognitive mechanisms and holds greater promise.However, relying solely on manipulating fMRI-to-image models, e.g., incorporated with spatiotemporal conditions for successive image reconstruction, to generate keyframes, easily yields suboptimal outcomes. Research dating back to as early as 1960 [17] has shown that the fleeting sequence of images perceived by the retina is hardly discernible during the process of perception. Instead, what emerges is a phenomenal scene or its intriguing features, which can be regarded as non-detailed, low-level images. Initially, the retina captures these low-level perceptions, and subsequently, the higher central nervous system in the brain focuses on and pursues the details, generating high-level images in the cerebral cortex [18] . This process is reflected in the fMRI signal [19, 20, 21, 22, 23, 24] . The video-to-fMRI process naturally incorporates a combination of both low-level and high-level images. Therefore, the reverse fMRI-to-video reconstruction intuitively benefits from taking into account both the high-level semantic features and the low-level perceptual features. Specifically, it is necessary to decode the low-level perception flows, such as motions and dynamic scenes, from brain activity to complement keyframes, which enhances the reconstruction of high-fidelity frames and produces smooth videos.In light of the above discussion, we propose a novel fMRI-to-video reconstruction framework NeuroClips that introduces two trainable components of Perception Reconstructor and Semantics Reconstructor for reconstructing low-level perception flows and keyframes, respectively. 1) Perception Reconstructor introduces Inception Extension and Temporal Upsampling modules to adaptively align fMRI with video frames, decoding low-level perception flows, i.e., a blurry video. This blurry video ensures the smoothness and consistency of subsequent video reconstruction. 2) Semantics Reconstructor adopts a diffusion prior and multiple training strategies to concentrate quantities of high-level semantics from various modalities into fMRI embeddings. These fMRI embeddings are mapped to the CLIP image space and decoded to reconstruct high-quality keyframes. 3) During inference, NeuroClips adopts a pre-trained T2V diffusion model injected with keyframes and low-level perception flows for video reconstruction with high fidelity, smoothness, and consistency.Extensive experiments have validated the superior performance of NeuroClips, which is substantially ahead of SOTA baselines in pixel-level metrics and video consistency. NeuroClips achieves a 0.219 improvement (128%) in SSIM and a 0.330 improvement (81%) in spatiotemporal metrics and also performs better overall on most video semantic-level metrics. Meanwhile, using multi-fMRI fusion, NeuroClips pioneers the exploration of longer video reconstruction up to 6s at 8FPS.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract and introduction clearly state the claims made, including the contributions made in the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
visual reconstruction: 
 After its initial exploration [25] , static image reconstruction from fMRI has witnessed remarkable success in recent years. Due to the lack of information in fMRI adapted to deep learning models, a path has been gradually explored that aligns fMRI to specific modal representations such as the common image and text modality [26, 27, 28, 29] , and CLIP's [5] rich representation of space is pretty favored. Then, the aligned representations can then be fed into the diffusion model to complete the image generation. Along this path, a large body of literature demonstrates that reconstructing images at both pixel level and semantic level achieves great results [1, 2, 4, 30] . However, the field of fMRI-to-video reconstruction remains largely unexplored. Early studies [31, 32, 33] attempted to reconstruct low-level visual content from fMRI, using the embedded fMRI as conditions to guide GANs or AEs in generating multiple static images. The reconstructed videos contained little to no clear semantics recognizable by humans. Due to the excellent performance of diffusion models, MinD-Video [8] reconstructed 3FPS videos from fMRI. Despite this success, the smoothness and semantic accuracy of these videos remain unsatisfactory, leaving substantial space for improvement.Diffusion models for image generation have gained significant attention in research communities recently [34, 35, 36] . DALLE•2 [37] improved text-image generation by leveraging the CLIP [5] joint representation space. Stable Diffusion [7] enhanced generation efficiency by moving the diffusion process to the latent space of VQVAE [38] . To achieve customized generation with trained diffusion models, many works focused on adding extra condition control networks, such as ControlNet [39] and T2I-Adapter [40] . In the realm of video generation, it is typical to extend existing diffusion models with temporal modeling [41, 42, 43] . Animatediff [44] trained a plug-and-play motion module that can be seamlessly integrated into any customized image diffusion model to form a video generator. Stable Video Diffusion [45] fine-tunes pre-trained diffusion models using high-quality video datasets from multiple views to achieve powerful generation.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We create a separate "Limitations" section in our Appendix to discuss the limitations of our works.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when the image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The overall framework of NeuroClips is illustrated in Figure 1 . NeuroClips consists of three essential components: 1) Perception Reconstructor (PR) generates the blurry but continuous rough video from the perceptual level while ensuring consistency between its consecutive frames. 2) Semantics Reconstructor (SR) reconstructs the high-quality keyframe image from the semantic level. 3) Inference Process is the fMRI-to-video reconstruction process, which employs a T2V diffusion model and combines the reconstructions from PR and SR to reconstruct the final exquisite video with high fidelity, smoothness, and consistency. Furthermore, NeuroClips also pioneers the exploration of Multi-fMRI Fusion for longer video reconstruction. Perception reconstruction is essential not only for video reconstruction but also for semantics reconstruction. Additionally, smoothness and consistency are crucial metrics of video quality, and Perception Reconstructor (PR) plays a key role in ensuring these attributes.We split a video into several clips at two-second intervals (i.e., fMRI time resolution). For each clip c, we downsample it and retain part frames at fixed intervals, resulting in a set of framesX = [X 1 , X 2 , • • • , X N f ]. X iis the i-th retained frame image, with a total of N f retained frames. Y c is the corresponding fMRI signal of clip c. Here we introduce Inception Extension module to extend one fMRI to N f fMRIs, denoted asY c → Y, Y = [Y 1 , Y 2 , • • • , Y N f ].Sequentially applying a simple MLP and Temporal Upsampling module to obtain Y's embedding setE Y = [e Y1 , e Y2 , • • • , e Y N f ],which can be fed into the Stable Diffusion [7] VAE decoder to produce a series of blurry images. We regard this sequence of blurry images as blurry video. We expect the blurry video to lack semantic content, but to exhibit state-of-the-art perceptual metrics, such as position, shape, scene, etc. Thus, we consider using frame set X to align Y.Training Loss. Mapping X to the latent space of Stable Diffusion's VAE to obtain the perception embedding setE X = [e X1 , e X2 , • • • , e X N f ].We adopt mean absolute error (MAE) loss and contrastive loss to train the PR, the overall loss L P R of PR can be described as:EQUATIONwhere τ is a temperature hyper-parameter. The function sim(, ) is used to compute the similarity.Temporal Upsampling. Due to the low signal-to-noise ratio of fMRI, the direct alignment of fMRI to VAE's pixel space is highly susceptible to overfitting noise, and the learning task is too complex to guarantee the generation of decent blurry images. A common method is aligning to a coarser-grained pixel space and then upsampling to a fine-grained pixel space. The temporal relationship of the frames also needs to be considered in the video task to maintain consistency. Therefore, to achieve consistency between the retained frames, we integrated temporal attention into the upsampling operation. The fMRI embedding E Y has five dimensions, i.e. E Y ∈ R b×N f ×c×h×w , where b denotes the batch size, c × h × w is the dimension of embedding space. The upsampling operation merely models spatial relationship, receiving reshaped embedding E spat Y ∈ R (b×N f )×c×h×w as input. To model temporal relationship of N f fMRI, we first reshape E Y as E temp Y ∈ R (b×h×w)×N f ×c . Then, we use learnable mapping to compute the query value Q = W Q E temp Y and the key valueK = W K E temp Y . Finally, the output of temporal attention layer is E ′ Y = Softmax( Q ⊤ K √ c ) • E temp Y .We utilize a learnable mixing coefficient η to conduct residual connection:EQUATIONBased on the above design, PR generates the blurry rough video with initial smoothness and great consistency, laying the foundation for subsequent video reconstruction.Recent cognitive neuroscience studies [46, 47] argue that 'key-frames' play a crucial role in how the human brain recalls and connects relevant memories with unfolding events, and other research [48, 49] also demonstrates that video key-frames can be used as representative features of the entire video clip. Building on these conclusions, the core objective of Semantics Reconstructor (SR) is to reconstruct a high-quality keyframe image that can be used to address the issue of frame rate mismatch between visual stimuli and fMRI signals, thereby enhancing the fidelity of the final exquisite video. The existing fMRI-to-image reconstruction studies [1, 2, 4] facilitate our objective, detailed below:fMRI Low-dimensional Processing. For each clip c, its corresponding fMRI signal is Y c . We use ridge regression to map Y c to a lower-dimensional Y ′ c for easier follow-up:EQUATIONwhere X is design matrix, λ is regularization parameter, and I is identity matrix. Although the human brain processes information in a highly complex and non-linear way, empirical evidence [1, 26, 2] underscores the effectiveness and sufficiency of linear mapping for achieving desirable reconstruction, due to nonlinear models will easily overfit to fMRI noise and then lead to poor performance [50] .Alignment of Keyframe Image with fMRI. Randomly choose one frame in the clip c as its keyframe X c , and use OpenCLIP ViT-bigG/14 [51] to obtain e Xc , the embedding of keyframe image X c in CLIP image space. e Yc is the fMRI embedding of Y ′ c via another MLP. Consequently, we perform contrastive learning between e Xc and e Yc to align the keyframe image X c and the fMRI Y c , resulting in enhancing the semantics of e Yc . It is worth noting that the MLP gets a bidirectional contrastive loss. Previous research [1] has demonstrated that introducing MixCo [52] data augmentation, an extension of mixup utilizing the InfoNCE loss, can effectively help model convergence, especially for scarce fMRI samples. Therefore, the bidirectional loss called BiMixCo L BiMixCo , which combines MixCo and contrastive loss, needs to be used for training.Generation of Reconstruction-Embedding. Since the embeddings in the CLIP ViT image space are more approximate to real images compared to fMRI embeddings, transforming fMRI embedding e Yc into CLIP ViT's image embedding will significantly benefit the reconstruction quality of the keyframe. Therefore, we have to generate the reconstruction-embedding e re Xc for the keyframe image X c , essentially, which is the image embedding that will be fed to the subsequent generative model for reconstruction. Inspired by DALLE•2 [37] , diffusion prior is an effective approach to transforming embedding. So, we map the fMRI embedding e Yc to the OpenCLIP ViT-bigG/14 image space to generate e re Xc . Here, we use the same prior loss L Prior in DALLE•2 [37] for training. Reconstruction Enhancement from Text Modality. Original fMRI-to-image reconstruction only relies on visual modality embedding. For instance, reconstructing images conditional on the image embeddings generated by diffusion prior. However, text is another critical modality. Incorporating text with higher semantic density can help improve the semantic content of reconstruction embedding, resulting in making semantics reconstruction more straightforward and effective. We adopt BLIP-2 [53] to introduce the text modality, i.e., the caption T c of the keyframe images X c . Then, we embed T c to obtain the text embedding e Tc . Inspired by contrastive learning, we perform contrastive learning between e reXc and e Tc to enhance reconstruction-embedding e re Xc via additional text modality. The contrastive loss serves as the training loss L Reftm of this process, similar to Equation 1, omitted here.Training Loss. As discussed above, the overall training loss L SR in SR is composite. Therefore, We set mixing coefficients δ and µ to balance multiple losses:L SR = L BiMixCo + δL Prior + µL Reftm .(4)The inference of NeuroClips is the process of fMRI-to-video reconstruction. We jointly utilize the blurry rough video, the high-quality keyframe image, and the additional text modality, which are α, β, and γ guidance, to reconstruct the final exquisite video with high fidelity, smoothness, and consistency. And we employ a text-to-video diffusion model to help reconstruct video.Text-to-video Diffusion Model. Pre-training text-to-video (T2V) diffusion models possess a significant amount of prior knowledge from the graphics, image, and video domains. However, like other diffusion models, they face huge challenges in achieving controllable generation. Therefore, directly using the text corresponding to fMRI to reconstruct videos will result in unsatisfactory outcomes, as the semantics of embeddings only originate from the text modality. We also need to enhance the embeddings with semantics from the video and image modalities to produce "composite semantics" embeddings, which aid in achieving controllable generation for the T2V diffusion model.α Guidance. We consider the blurry rough-video V blurry output from PR as α Guidance. Treating V blurry as an intermediate noisy video between target video V 0 and noise video V T , the originally required T steps for the complete forward process can now be reduced to ϑT steps. By applying the latent space translation and reparameterization trick, noise z T can be formalized as:EQUATIONwhere α t represents the noise schedule parameter at time step t and ϵ ∼ N (0, 1) is Gaussian noise. The reverse process involves iteratively denoising the noise video from T steps back to 0 steps.Adopting a pretrained T2V diffusion model p θ to predict the mean and variance of the denoising distribution at each step:EQUATIONwhere t = T, T -1, ..., 1. After translating z 0 to pixel space, the reconstructed video V 0 is obtained.β Guidance. α Guidance only directs the video generation of the T2V diffusion model at the perception level, leading to significant randomness in the semantics of the reconstructed videos. To resolve this issue, we need to incorporate keyframe images with more supplementary semantics to control the generation process, thereby enhancing the fidelity of the reconstructed videos. Compared to directly reconstructing keyframe images from fMRI embeddings, combining perception embeddings will be more beneficial for maintaining the consistency of structural and semantic information. Therefore, we select the first frame V 1 of blurry V blurry . Input V 1 's embedding and fMRI embedding to SDXL unCLIP [2] (See Appendix D for more discussion about SDXL unCLIP) in SR to reconstruct the keyframe image X key as β Guidance. We employ ControlNet [54] to add β Guidance to the T2V diffusion model, in which the keyframes are used as the first-frame to guide video generation.γ Guidance. The text is the necessary input for the T2V diffusion model. In order to maintain the consistency of visual semantics, we adopt BLIP-2 [53] to generate the caption for the keyframe image X key , which is used as γ Guidance (prompt) for video reconstruction.The inference process inputs α, β, γ Guidance into the T2V diffusion model, and the fMRI-to-video reconstruction can be completed, resulting in the exquisite video with high fidelity and smoothness.While it is important to emphasize that single-frame fMRI generates higher frame rate video, the more realistic question is how to recover longer video (longer than fMRI temporal resolution).Previous methods treat single-frame fMRI as a sample, and temporal attention is computed at the single-frame fMRI level, thus failing to generate coherent videos longer than 2s. With the help of NeuroClips' SR, we explored the generation of longer videos for the first time. Current video generative models are built on diffusion-based image generation models and attention-based transformer architectures, both of which incur significant computational overhead. As the number of frames increases, the content scales linearly, highlighting the limitations in generating long and complex videos efficiently. Therefore, we chose a more straightforward fusion strategy that does not require additional GPU training. In the inference process, we consider the semantic similarity of two reconstructed keyframes from two neighboring fMRI samples (here we directly determine whether they belong to the same class of objects, e.g., both are jellyfish). Specifically, we obtain the CLIP representations of reconstructed neighboring keyframes and train a shallow MLP based on the representations to distinguish whether two frames share the same class. If semantically similar, we replace the keyframe of the latter fMRI with the tail-frame of the former fMRI's reconstructed video, which will be taken as the first-frame of the latter fMRI to generate the video. As shown in Figure 2 , with this strategy, we achieved continuous video reconstruction of up to 6s for the first time. Pre-processing. The fMRI data on the cc2017 were preprocessed using the minimal preprocessing pipeline [55] . The fMRI volumes underwent artifact removal, motion correction (6 DOF), registration to standard space (MNI space), and were further transformed onto the cortical surfaces, which were coregistered onto a cortical surface template [56] . We calculated the voxel-wise correlation between the fMRI voxel signals of each training movie repetition for each subject. The correlation coefficient for each voxel underwent Fisher z-transformation, and the average z scores across 18 training movie segments were tested using the one-sample t-test. The significant voxels (Bonferroni correction, P < 0.05) were considered to be stimulus-activated voxels and used for subsequent analysis A total of 13447, 14828, and 9114 activated voxels were observed in the visual cortex for the three subjects.Following the previous works [3, 57, 58] , we used the BOLD signals with a delay of 4 seconds as the movie stimulus responses to account for the hemodynamic delay.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
implementation details: 
 In this paper, videos from the cc2017 dataset were downsampled from 30FPS to 3FPS to make a fair comparison with the previous methods, and the blurred video was interpolated to 8FPS to generate the final 8FPS video during inference. We split semantic reconstruction into two parts: image contrastive learning and the fine-tuning of the diffusion prior to adapting to the new image distribution space.In PR, all downsampled frames were utilized, and the inception extension was implemented by a shallow MLP. Theoretically, our approach can be used in any text-to-video diffusion model, and we choose the open-source available AnimateDiff [44] as our inference model. ϑ is set to 0.3 in α Guidance, and the inference is performed with 25 DDIM [35] steps (See Appendix A for more implementation details). All experiments were conducted using a single A100 GPU.We conduct the quantitative assessments through frame-based and video-based metrics. Frame-based metrics evaluate each frame individually, providing a snapshot evaluation, whereas video-based metrics evaluate the quality of the video, emphasizing the consistency and smoothness of the video frame sequence. Both are used for a comprehensive analysis from a semantic or pixel perspective.We evaluate frames at the pixel level and semantic level. We use the structural similarity index measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR) as the pixel-level metric and the N-way top-K accuracy classification test (total 1,000 image classes from ImageNet [59] ) as the semantics-level metric. To conduct the classification test, we essentially compare the ground truth (GT) classification results with the predicted frame (PF) results using an ImageNet classifier [30, 8] .We consider the trial successful if the GT class is among the top-K probabilities in the PF classification results (We used top-1), selected from N randomly chosen classes that include the GT class. The reported success rate is based on the results of 100 repeated tests.We evaluate videos at the semantic level and spatiotemporal(ST)-level. For semantic-level metrics, a similar classification test (total 400 video classes from the Kinetics-400 dataset [60] ) is used above, with a video classifier based on VideoMAE [61] . For spatiotemporal-level metrics that measure video consistency, we compute CLIP image embeddings on each frame of the predicted videos and report the average cosine similarity between all pairs of adjacent video frames, which is the common metric CLIP-pcc in video editing.  
results: 
 In this section, we compare NeuroClips with previous video reconstruction methods on the cc2017 dataset. We provide a visual comparison in Figure 3 and report quantitative metrics in Table 1 . We purposely focus on the comparison with the previous SOTA method on the right side of Figure 3 due to the lack of obvious semantic information in the premature method. All methods report results for all test sets except Wen [31] , whose results are available for only one segment. Our method generates videos at 8fps and even higher frame rates. For a fair comparison with previous studies, we downsampled the 8FPS videos to 3FPS. Unless otherwise noted, the experimental results and visualizations shown below are all at 3FPS.As can be seen in Figure 3 , earlier methods were unable to produce videos with complete semantics but guaranteed some of the low-level visual features. Compared to MinD-Video, our NeuroClips generates single-frame images with higher quality, more precise semantics (e.g., people, turtles, and airplanes), and smoother movements. At the same time, due to the limited data in the training set, some objects in the test set videos did not appear in the training set, such as the moon, and perfect reproduction of the semantics is difficult. However, thanks to our perception reconstructor (aka α Guidance), NeuroClips basically reproduces the shape and motion trajectory of the moon, although semantically it is more similar to the aperture, demonstrating the pixel-level reconstruction ability of the video. In terms of quantitative metrics, NeuroClips significantly outperformed 5 of the 7 metrics, with a 128% improvement in SSIM performance, indicating that the PR complements the lack of pixel-level control. At the semantic level, our metrics overall outperform previous methods, demonstrating the better semantic alignment paradigm of NeuroClips. For the ST-level metric, which evaluates video smoothness, NeuroClips substantially outperforms MinD-Video because we introduce blurry rough-video (aka α Guidance) for the frozen video generation model, incorporating initial smoothness for video reconstruction. In contrast, MinD-Video lacks perception control, resulting in a substantial decrease in smoothness, as can also be seen in Figure 3 , where the human deformations and scene switches are too large within an fMRI frame. In addition, benefiting from our keyframes for first-frame guidance (aka β Guidance) and blurry videos, we can connect semantically similar videos to generate longer videos, which may be the reason for the slightly lower video 2-way metrics, as neighboring reconstructed videos are more similar after multi-fMRI fusion.6 Ablations In this section, we discuss the impact of three important intermediate processes on video reconstruction, including keyframes, blurry videos, and keyframe captioning. The quantitative results are in Table 2 and the visualization results are shown in Figure 4 , where all the results of the ablation experiments are from subject 1. Since the keyframe captioning must be obtained through the keyframe, by default we eliminate keyframe and keyframe captioning at the same time. From the quantitative results, it can be seen that NeuroClips exhibits better pixel-level metrics without keyframes while showing better semantic-level metrics without blurry clips. This indicates a trade-off between semantic and perception reconstruction, which is similar to the results of a large body of literature on fMRI-to-image reconstruction [1] . The perception reconstruction improves the pixel-level performance and the semantic reconstruction improves the semantic metrics. In addition to this, the best ST-level results for the full model demonstrate the contribution of each module to video consistency. For the image captioning method, previous studies [2] have used the GIT [63] model to generate keyframe captions directly from fMRI embeddings in image space, and we generated it from BLIP-2 [53] . Here we compare the text generation results of these two approaches as shown in the Figure 4 . We found that GIT's text generation is slightly homogeneous, filled with a large number of similar descriptions, such as 'a large body of water', 'a man is standing'. For the diffusion model, the influence of text is significant, so GIT's captions degrade the quality of the semantic reconstruction of the video, e.g., generating flowers on water from jellyfish keyframes. This shows that keyframe-based captioning is more flexible compared to representation-based captioning. Finally, we remove the keyframes and keyframe captions and use only blurred videos to guide the reconstruction, with the text input replaced with the generic description 'a smooth video'. With this approach, we find that the model generates the video completely blindly, with poor semantic control, demonstrating the strong semantic support that keyframes bring to the NeuroClips generation capability. More ablation analysis can be found in Appendix C.4.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We use publicly available datasets and provide anonymous links to our project.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide the key experimental setup in the paper and provide the complete experimental setup in our Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
interpretation results: 
 To explore the neural interpretability of the model, we visualized voxel-level weights on a brain flat map, where we can see comprehensive structural attention throughout the whole region, as shown in Figure 5 . It can be seen that the visual cortex occupies an important position regardless of the task. In addition, for semantic-level reconstruction, the weight distribution of voxels is more spread out over the higher visual cortex, indicating the semantic level of the video. For perceptual reconstruction, the weight distribution of voxels is more concentrated on the lower visual cortex, corresponding to the low-level perception of the human brain. See Appendix C.5 for more subjects' interpretation results. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report the average and standard deviation of multiple runs to make the experimental results statistically significant.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
conclusion: 
 In this paper, we present NeuroClips, a novel framework for fMRI-to-video reconstruction. we implement pixel-level and semantic-level visual learning of fMRI through two paths: perception reconstruction and semantic reconstruction. With the learned components, we can configure them into the latest video diffusion models to generate higher quality, higher frame rate, and longer videos without additional training. NeuroClips recovers videos with more accurate semantic-level precision and degree of pixel-level matching, establishing a new state-of-the-art in this domain. In addition to this, we visualized the neuroscience interpretability of NeuroClips with reliable biological principles. 
limitations: 
 Although NeuroClips has achieved high-fidelity, smooth, and consistent multi-fMRI to video reconstruction, there are still slight flaws. Specifically, our framework is slightly bulky and it relies on extending keyframes to the reconstructed video. A model that can reconstruct videos from the CLIP latent space will avoid this intermediate process. Unfortunately, there is no such available model now. In addition, our method does not reconstruct cross-scene fMRI well, i.e., fMRI recorded during video clip switching. Even if such fMRI scans are in a tiny minority, this will be a direction for future research. Moreover, additional subjects and fMRI recordings should be considered in order to reflect real-world visual experiences sufficiently. However, The alleviation of these limitations will require joint advances in multiple areas and significant further effort will be required. This is because improvements in these areas need to be supported by common developments including machine learning, computer vision, brain science, and biomedicine.Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential societal impacts of our work in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited. And the license and terms of use are explicitly included in our code repository.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our anonymized URL includes these new codes, new results, and related documentation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Training Details. For Semantic Reconstructor, We first train the fMRI-to-keyframe alignment for 30 epochs with a batch size of 240 and then tune the Diffusion Prior for 150 epochs with a batch size of 64. For Perceptual Reconstructor, we train it for 150 epochs and the batch size is set to 40. We use the AdamW [64] for optimization, with a learning rate set to 3e-4, to which the OneCircle learning rate schedule [65] was set. Mixing coefficients δ and µ are set to 30 and 1.Inference Process. In the phase of inference, we use AnimateDiff v3 [44] Visualization Software. We use the Connectome Workbench from Human Connectome Project (HCP), and flatmap templates were selected as 'Q1-Q6_R440.L.flat.32k_fs_LR.surf.gii' and Q1-Q6_R440.R.flat.32k_fs_LR.surf.gii'. Additionally, the cortical parcellation was manually delineated by neuroimaging specialists and neurologists, and aligned with the public templates in FreeSurfer software with verification. We normalized the voxel weights, scaling them to between 0 and 1. Finally, to show a better comparison, the Colorbar was chosen to be 0.25-0.75.We elaborate on the architecture of our Temporal Upsampling in Sec 3.1 here. As illustrated in Figure 6 , the Temporal Upsampling module consists of Spatial Layer, Temporal Attention, Learnable Residual Connection, and Upsampling.The input embedding E Y of Temporal Upsampling has five dimensions, i.e. E Y ∈ R b×N f ×c×h×w (To express concisely, we have drawn only its N f , h, and w in Figure 6 .) Before feeding E Y into the spatial layer, we reshape it to E spat Y ∈ R (b×N f )×c×h×w . First, we conduct modeling in spatial level, using 3D convolution and spatial attention, formalized as E ′ Y = Spatial(E spat Y ). Note that the spatial layer maintains consistent input and output dimensions so that the dimensions of E ′ Y are exactly the same as those of E spat Y . Then, applying the learnable residual connection:E Y ← η 1 • E spat Y + (1 -η 1 )E ′ Y . Subsequently, we reshape E Y as E temp Y ∈ R (b×h×w)×N f ×c and input E temp Y to the temporal layer E ′′ Y = Temporal(E temp Y), which is achieved using Temporal Attention as mentioned in main text. Similarly, we apply the learnable residual connection again:E Y ← η 2 • E temp Y + (1 -η 2 )E ′′ Y .Finally, we conduct upsampling to the result E Y , formalized asE Y ← Upsampling(E Y ) ∈ R b×N f ×c ′ ×h ′ ×w ′. We repeat employing the above module in different (c, h, w) until E Y is upsampled to the target dimensions. Here we elaborate on the three loss functions in Equation 4.BiMixCo Loss. BiMixCo aligns the keyframe X c and its corresponding fMRI signal Y c using bidirectional contrastive loss and MixCo data augmentation. The MixCo needs to mix two independent fMRI signals. For each Y c , we random sample another fMRI Y mc , which is the keyframe of the clip index by m c . Then, we mix Y c and Y mc using a linear combination:EQUATIONwhere Y * c denotes mixed fMRI signal and λ c is a hyper-parameter sampled from Beta distribution. Then, we adapt the ridge regression to map Y * c to a lower-dimensional Y * ′ c and obtain the embedding e Y * c via the MLP, i.e. e Y * c = E(Y * ′ c ). Based on this, the BiMixCo loss can be formed as:EQUATIONwhere e Xc denotes the OpenCLIP embedding for keyframe X c .Prior Loss. We use the Diffusion Prior to transform fMRI embedding e Yc into the reconstructed OpenCLIP embedding of keyframe e re Xc . Similar to DALLE•2, Diffusion Prior predicts the target embeddings with mean-squared error (MSE) as the supervised objective:EQUATIONReftm Loss. In addition to image representation-level alignment, the assistance of text can aid in generating semantically more compatible images. Considering the inconsistency in the number and dimension of image and text tokens, the previous approaches mostly align the 257 tokens of the image with the CLS token of the text after globally averaging the pooling projection. In this paper, we ignore the CLS token and only map fMRI to 256 tokens of images, so the projection layer of the alignment needs to be adjusted. We develop the Reftm to achieve the alignment between the reconstruction-embedding e re Xc and its corresponding text embedding e Tc (the text T c is generated by BLIP2 captioning according to the GT keyframe). To align the dimension of reconstructionembedding e re Xc and that of text embedding e Tc , we add an adjusted projector P(•). The adjusted projector directly maps 256 tokens of e re Xc to the dimension of text embedding. We fine-tuned the projector for 20 epochs on the MSCOCO dataset, which consists of 73k images and 5 texts in each image. And in the training phase of NeuroClips, the projector was frozen. Table 3 displays the effect of fine-tuning. We display the reconstructed keyframes and their corresponding text in Figure 7 . It can be observed that the reconstructed keyframes and ground truth exhibit an extremely high semantic similarity. These results demonstrate the ability of our Semantic Reconstructor to reconstruct semantic-accuracy keyframes and corresponding text descriptions. This reconstructed keyframe contains not only the correct object categories but also detailed information such as object position, color, scene structure, etc., which is crucial for reconstructing high-fidelity videos. We visualized more successfully reconstructed videos in Figure 8 . As it can be seen, our NeuroClips is capable of successfully reconstructing videos with correct semantics, covering various categories such as portraits, objects, animals, and natural scenes, demonstrating the effectiveness of our Semantic Reconstructor. By conducting detailed comparisons with the ground truth visual stimulus, we find that the structural information and motions in the videos could also be reconstructed, such as the size of the main objects, their positions within the scene and the direction in which a person walks. This demonstrates the critical addition of blurred video to motion and structural information.We further show videos that are successfully reconstructed by all three subjects in Figure 9 , which demonstrated generalization of the model. What's more, we find that these videos often consist of simple backgrounds featuring portraits, animals, or objects, as well as some natural scenes. By comparing the reconstruction results of the three subjects, we discover an interesting phenomenon: the perceived size of objects in simple scenes varies between individuals. For example, the sizes of the jellyfish and airplane reconstructed from different subjects' fMRI show significant differences. These results also suggest, to some extent, the differences in the human visual system among individuals.Incorrect results often provide more insights. Therefore, we present some of the incorrect reconstruction results generated by NeuroClips in Figure 10 . Most incorrect reconstruction clips arise from semantic errors, leading to confusion between semantically similar items, such as fish and turtle, man and woman, and cat and dog. This type of error may arise from two main reasons. On the one hand, the semantic accuracy of the keyframes we reconstructed may be insufficient. We attempt to use fMRI embedding to retrieve from a pool of keyframe image representations. When the size of the retrieval pool is set to 300, the retrieval accuracy on the test set is approximately 0.22, indicating that there is still significant improvement room for visual semantics encapsulated by the fMRI embedding.On the other hand, the test set may encounter out-of-distribution issues relative to the training set, where semantic categories present in the test set are not seen in the training set, making semantic reconstruction challenging to generalize on the test set. The first frame of the blurry video provides structural information for the reconstruction of the keyframes and motion information for the generation of the video. Here, we perform the ablation visualization of the blurry video to observe its role in perception reconstruction. Although the building can still be generated with the blurry video removed as shown in Figure 11 , the shape difference from the original image is too large, indicating the structural information provided by the blur video. In addition, with the addition of the blurred video guidance, the camera view is progressively upward, similar to the original image, but with the removal of the blurred video, the camera view is encircled, suggesting that the perception reconstruction provides motion guidance. In Section 7, we visualized voxel-level weights on a brain flat map for subject 1. Here we provide detailed visualization results for all three subjects from the cc2017 dataset, as shown in Figure 12 . As can be seen from the Figure, the three subjects learned similar weights in the Perception Reconstructor, which echoes the subjects' comparable SSIM metrics in Table 1 and demonstrates the existence of commonality in low-level vision in humans. However, the weights learned by subject 3 in the Semantic Reconstructor are different from the other subjects, which also leads to its image retrieval accuracy being at a lower value, as shown in Table 4 . This may indicate that there were differences in the understanding of the video between subjects.We further evaluate the performance of our Semantic Reconstructor using Top-1 keyframe retrieval accuracy and Top-1 fMRI retrieval accuracy. For keyframe retrieval, each test fMRI Y c is first converted to an fMRI embedding e Yc , and we compute the cosine similarity to its respective CLIP keyframe embedding e Xc and 299 other randomly selected CLIP keyframe embedding in the test set. For each test sample, success is determined if the cosine similarity is greatest between the fMRI embedding and its respective CLIP keyframe embedding (aka top-1 retrieval performance, random chance is 1/300). The test set contains 1,200 fMRI-keyframe pairs, and we randomly divide it into 4 parts (thus each part contains 300 test pairs) for retrieval evaluation. We report the average retrieval accuracy among 4 parts. The same procedure is used for fMRI retrieval, except fMRI and keyframe are flipped. Table 4 displays the retrieval performance. Unlike the latest techniques for fMRI image reconstruction (where retrieval accuracy can reach more than 90%), NeuroClips' retrieval accuracy on the cc2017 dataset averages around 22%, which is at the lower end of the scale. Leaving aside the difference between fMRI in image stimuli and video stimuli, we found that the cc2017 test set had a large number of categories of objects that did not appear in the training set. This is likely to be the main reason for the low retrieval accuracy and is also the reason why most of the previous studies have focused on low-level visual reconstruction. In the future, a large-scale fMRI-video dataset more compatible with deep learning is worth looking forward to.A big difference in NeuroClips' video generation inference is that we freeze the weights of the video diffusion model, while MinD-Video fine-tunes it. This raises the question of whether our generation relies exclusively on the knowledge base that the video diffusion model has been trained on, and is not consistent with the distribution of video training data related to fMRI.Firstly, since our keyframes are control conditions as first frames, the content of our video generation is closely related to the keyframes, both semantically and structurally. So in NeuroClips, the video diffusion model will not rely entirely on its own semantic knowledge to generate videos, but on the keyframes. Secondly, for keyframe generation, keyframes are generated by SDXL unCLIP. The normal version of SDXL unCLIP generates images from a larger dimensional text representation space, which was fine-tuned for 6 epochs by Scotti et al. [2] on MSCOCO with image representations. The fine-tuned version of SDXL unCLIP provides a 256×1664 large dimensional CLIP representation space, which has a strong correspondence with the pixel space of the image, as shown in Figure 13 . The SDXL unCLIP we use is this fine-tuned version. The videos captured from the cc2017 dataset used in this paper are from Videoblocks (https://www.videoblocks.com) and YouTube (https://www.youtube.com). These videos are not part of the COCO dataset, but SDXL unCLIP can still restore the original images as shown in Figure 13 , demonstrating the equivalence of its representation space and pixel space. Since NeuroClips trained Semantic Reconstructor to align fMRI to this representation space, this also shows that our keyframe reconstruction is closely related to fMRI training. Our research is dedicated to exploring the possibilities of neural decoding using deep learning techniques, which have a positive impact on the field of neuroscience. With the expansion of model scales and the improvement of corresponding hardware devices, this research will also make positive contributions to the field of brain-computer interfaces. However, this research also highlights the importance of personal privacy and security. Governments and research institutions should take appropriate measures to protect the privacy data collected and prevent any potential misuse. In our research, we use publicly available and de-identified datasets, so the study strictly adheres to relevant ethical requirements.NeurIPS Paper ChecklistJustification: Our research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://purr.purdue.edu/publications/2809/1 


Decision: This paper proposes NeuroClips, a framework that decodes high-fidelity and smooth video from fMRI. NeuroClips uses a semantics reconstructor for video keyframes to ensure semantic accuracy and consistency, and a perception reconstructor for capturing low-level perceptual details, ensuring video smoothness.
Additionally, the reviewers are convincingly shown that NeuroClips is equipped with a specific model design and a tailored loss function, which guarantees its ability in capturing motion information.
Questions on rate and resolution have been convincingly addressed.
Cross-subject modelling will be addressed in future work, which is fine.
Overall, the authors gave  detailed and convincing answers to all technical questions made by reviewers.
The contribution is both impressive and interesting, and thus certainly deserves a wide visibility within NeurIPS conference.

Reviews: review: paper_summary: paper_summary: This paper proposes NeuroClips, a framework that decodes high-fidelity and smooth video from fMRI. NeuroClips uses a semantics reconstructor for video keyframes to ensure semantic accuracy and consistency, and a perception reconstructor for capturing low-level perceptual details, ensuring video smoothness.
strength_weakness: strength_weakness: The video captures the semantic meaning well but fails to accurately follow the ground truth movement.
questions: questions: - Why is there no model or loss function for movement(motion)?
- Are you willing to make the code publicly available?
limitations: limitations: As mentioned above, motion reconstruction has not been resolved.
paper_summary: paper_summary: The proposed framework NeuroClips introduces a strong pipeline for fMRI-to-Video reconstruction in the field of Brain Visual Decoding. The Perception Reconstructor(PR) maintains the motion of the video and the Semantic Reconstructor(SR) ensures the semantic information of the video. Multi-fMRI Fusion raises upper video length limit, and overall model achieves impressive results.
strength_weakness: strength_weakness: 1. I browsed the anonymous site, and the generated results are impressive. However, the lighting of some of the reconstructed videos varies considerably compared to the ground truth, which can also be seen on the right side of Figure 2, and the authors need to rationalize this.
2. Existing state-of-the-art video generation models can generate high-frame rate videos such as 24 fps for up to 1 min or even longer, however NeuroClips at this stage has not yet reached this level.
questions: questions: 1. As discussed in mind-video [1], the nature of hemodynamic response has been considered and specific modules are designed, which seems not to be included in NeuroClips. Are there other considerations or is there no need to take into account for the BOLD signals?
2. Since a number of cross-subject models already exist in the image reconstruction field [2], does NeuroClips need to train a separate model for each subject?
3. As you mention in the limitation section of the appendix, the cc2017 dataset test set contains too many no-show categories, and I'm curious if the unsatisfactory results of previous methods are more due to the low quality of the dataset, less than the method itself?
4. Why text contrastive loss is placed after diffusion prior and not before like in [3]?

[1] Chen, Zijiao, Jiaxin Qing, and Juan Helen Zhou. "Cinematic mindscapes: High-quality video reconstruction from brain activity." Advances in Neural Information Processing Systems 36 (2024)

[2] Scotti, Paul S., et al. "MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data." arXiv preprint arXiv:2403.11207 (2024).

[3] Sun, Jingyuan, et al. "NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties." arXiv preprint arXiv:2402.01590 (2024).
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces NeuroClips - a framework for reconstructing high-fidelity videos from fMRI data. It combines pixel-level and semantic-level visual learning through perception and semantic reconstruction pathways. The Perception Reconstructor (PR) ensures smoothness and consistency by creating a rough, continuous video, while the Semantics Reconstructor (SR) generates high-quality keyframes. These components are integrated into a video diffusion model, resulting in high-quality, detailed videos. Also, no additional post-diffusion training is required. NeuroClips sets a new standard in fMRI-to-video reconstruction, demonstrating significant advancements in semantic precision and pixel-level matching by improving SSIM by 128% and spatiotemporal metrics by 81%.
strength_weakness: strength_weakness: - The multi-fMRI fusion strategy is briefly described, but the implementation details and the rationale behind specific design choices are not fully elaborated. I suppose the scheme of the multi-fMRO strategy in the supplement would increase the clarity of this paragraph. 

-  As it can be seen from the examples provided with the code repository, the proposed framework does not account for a change of the scene in the video (which was briefly mentioned by the authors in the Limitations). The pipeline with chosen keyframe might hinder the NeuroClips  to catch this rapid change in the video. No ablation is done in this direction, which could explain how neural NeuroClips decodes fMRI signals. 

- The paper primarily evaluates the method on a specific dataset (with only 3 patients), which, obviously, may not fully capture the diversity of real-world video content and fMRI recordings.This should be at least mentioned in the Discussion (or Conclusion) of the work.

 - The neurobiological justification of keyframe usage seems ambiguous. The improvement of text clearance and up-to-date references would increase the significance of the work.
questions: questions: - The paper mentions accounting for the hemodynamic delay (BOLD signal delay of approximately 4 seconds), there is limited discussion on how this delay specifically impacts the reconstruction quality and temporal alignment with video frames. 

- The use of ridge regression to map fMRI signals to lower-dimensional embeddings assumes a relatively linear relationship between neural activity and visual stimuli. However, the brain's processing of visual information is highly nonlinear and complex. At least, it should be mentioned in the Limitations too.

- Fig 12: The weights of subject 3 are really similar for PR and SR tasks, can you elaborate on it?
 - Fig 2: Example images are too small to perceive it even with a zoom, I recommend to make them bigger (as in Fig 7).
- I recommend adding the major limitations in the Conclusion section of the main text.

- Authors should include in the Supplement section which software was used to build brain maps with weights and if any data pre-processing/normalization was applied.

- Lines 75-76: “NeuroClips achieves a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics and also performs better overall on most video semantic-level metrics.” -  report not only the percent of improvement, but also the values of the metrics.

- In the frame validation the main image quality metrics are PSNR and SSIM. Authors should mention that those metrics have flaws (Zhang et al, The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, 2018). Will it be possible to consider other quality evaluation metrics? Like, for example, Visual information fidelity metrics (Sheikh et al., Image information and visual quality, 2006). It is also interesting why PSNR and SSIM were used for frames-based evaluation but not for video-based evaluation with modification to catch spatio-temporal data (ST-PSNR and ST-SSIM)? Another question is why authors choose to evaluate performance only with “CLIP image embeddings on each frame” and omit other metrics like Fréchet Video Distance (FVD), MSE or already mentioned ST-PSNR and ST-SSIM?

- Fig 12: The visualization of voxel weights on a brain flat map to interpret neural activity is a significant strength. However, now it is difficult to comprehend the differences in weights from the image with naked eyes. The addition of a third row with the difference between the weights of participants can increase the clarity of what authors are trying to say.

- The study focuses primarily on the visual cortex. While this is appropriate for the simplest visual stimuli (i.e., dots, simple geometrics shapes, etc.), it may limit the generalization of the approach to other types of brain activity or cognitive functions. Discussing potential extensions to other brain regions and types of neural data could provide a more comprehensive neurobiological perspective.

-  Lines 316-318: “From the quantitative results, it can be seen that there is a trade-off between semantic and perception reconstruction” - there is no clear explanation or reasoning why there is a trade-off between SR and PR

- Line 38: “However, the visual decoding of MinD-Video significantly diverges from the brain’s visual system, exhibiting limitations in perceiving continuous low-level visual details.” - the more precise listing of what hinders MinD-Video to perceiving continuous low-level visual details would be more constructive, rather than comparison with the brain's visual system. 

- Lines 41-42: “Notably, the human brain perceives videos discretely [8, 9] due to the persistence of vision [10, 11] and delayed memory [12].”
	1. discretely -> discreetly
	2.  Was it intentional to use the paper [10] of 1892 year for the “persistence of vision” concept? Also, the fact that the brain has a “persistence of vision” was criticized. Actually, the second citation you’ve used [11] is the critique of this concept (J Anderson et al., The myth of persistence of vision revisited, 1993). The citations of the novel research papers are required to justify this statement.
	
The paper will be significantly improved by more valid justification of claimed neurobiological concepts. 


Minor:
- Line 644: “However, these limitations will not be tackled overnight” - this phrase is informal and somewhat colloquial, which may not be entirely appropriate for a scientific paper
- Line 56: “This process is reflected in the fMRI signal” - can you provide the link to the paper supporting this statement?
- Line 129: “...loss to train the PR, the overall loss L_{PR} of PR can be donated as…” - “donated” is a confusing word to use, please use another word.

-  Lines: 222-223: “Since the technical field of long video generation is still immature, we chose a more straightforward fusion strategy that” - can not see why immatureness of the field can be used as a reason to use a straightforward strategy. Please, make the reason for usage more understandable. 

- Figure 4 in page 8: I recommend not to use comic sans font in images. 

- Lines 596-597: “It proves that there may be a large difference in the understanding of the video” - please refrain from using the word “prove” here, since it implies comprehensive research or, at least, references to other works that proves it. 

- Could you report the number of classes you have at least in training and testing dataset (for the “N-way top-K accuracy classification test as the semantics-level metric”)

- Line 55-56: “generating high-level images in the cerebral cortex” - it is also knows that brain interpolates the seen scenes (Vacher, Jonathan, et al. Texture interpolation for probing visual perception, 2020), which could be used as justification to use keyframe approach.
limitations: limitations: Important limitation - the dataset of only 3 participants - was not mentioned. This may result in varying interpretations with a careful review by a neurobiologist. Additional evaluation is needed with other datasets and more participants.

Comments: title: Response to Ethics Reviewer 9jC7
Heartfelt Thanks
Post-rebuttal thoughts
Response to Minors of Reviewer oPVf
Response to Questions of Reviewer oPVf
Response to Questions of Reviewer oPVf
Response to Weaknesses of Reviewer oPVf
comment: We greatly appreciate for your constructive feedback. We would like to express our sincere gratitude for your increased rating and further support towards our work! We hope that this paper achieves satisfactory results, not in vain of your efforts and suggestions.

Once again, thank you, Reviewer mcBd.

Best wishes,

All authors of Submission 351
We greatly appreciate for your constructive feedback. We would like to express our sincere gratitude for your increased rating and further support towards our work! We hope that this paper achieves satisfactory results, not in vain of your efforts and suggestions.

Once again, thank you, Reviewer 85K8.

Best wishes,

All authors of Submission 351
Thanks for your well-constructive and persuasive review. I will raise the score.
I would like to thank the authors for their responses. It is good to see that my concerns have been sufficiently addressed. I think the submission is solid, and I will raise my rating.
Thank you very much for the reminder. We have confirmed that there are **no Ethical Issues**. The original dataset paper[1] states that 'Three healthy volunteers (female, age: 22–25; normal vision) participated in the study, with informed written consent obtained from every subject according to the research protocol approved by the **Institutional Review Board** at Purdue University'. We'll add it to the paper as a separate paragraph discussing the ethical issues. Thanks again!

**Reference**

[1] Neural encoding and decoding with deep learning for dynamic natural vision, Cerebral cortex 2018
We greatly appreciate your constructive feedback and meticulous review. And it is good to know that our response has addressed some of your concerns. Although some others have not yet reached a mutual agreement, we are committed to resolving them promptly to ensure the high quality of the work. Lastly, we would like to express our sincere gratitude for your increased rating and further support towards our work! We hope this paper achieves satisfactory results, not in vain of your efforts and suggestions.

Best wishes,

All authors of Submission 351
Thank you for a rebuttal as thoroughly prepared as the submission itself. Despite the limitation of a low-n study and some debatable conclusions for the neurobiology (which had motivated my original score) AND given the authors keep their 5 rebuttal promises for the camera ready, I am now willing to give an extra point and be a proponent of this work during the discussion with ACs. Thank you for interesting read and good luck!
> **Minor 1**

We would modify this sentence to "***The alleviation of these limitations will require joint advances in multiple areas and significant further effort will be required.***" Thank you.

> **Minor 2**

fMRI measures brain activity by detecting changes in blood flow, and can thus reflect and quantify brain activity evoked by visual stimuli which has been widely applied in studies. The following are relevant references:

**Reference**

[1] Reconstructing perceived images from human brain activities with Bayesian deep multiview learning, *TNNLS 2019*

[2] Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective, *Brain imaging and behavior 2023*

[3] Compressive spatial summation in human visual cortex, *Journal of Neurophysiology 2013*

[4] fMRI evidence for areas that process surface gloss in the human visual cortex, *Vision research 2015*

[5] A comparison of fMRI adaptation and multivariate pattern classification analysis in visual cortex, *Neuroimage 2010*

[6] Spontaneous activity associated with primary visual cortex: a resting-state FMRI study, *Cerebral cortex 2008*

[7] The human visual cortex, *Annual review of neuroscience 2004*

> **Minor 3** 

Thanks for your advice!  We will replace ‘donated’ with **‘described’** in the final version.

> **Minor 4** 

In the context of the current **diffusion model** and **attention-based transformer model** as the dominant models for image generation, the computational overhead required for image generation models is sufficiently large. The content of video grows linearly with the number of frames, so the technical field of long video generation is still immature. 

We value your opinion and also believe that a clearer explanation is needed here. Therefore, we will revise the above statement to make it more understandable.

> **Minor 5** 

Thank you for your professional suggestion. We decide to change the font in the Figure 4 to 'Times New Roman' in the final version.

> **Minor 6**

Thanks for your kind reminder. Upon careful consideration, we acknowledge that using **'prove'** is indeed inappropriate since the paragraph is an exposition of the results of the experiment. We agree with your suggestion and will change the sentence to ***"This may indicate that there were differences in the understanding of the video between subjects."***

> **Minor 7**

The classifiers we use are **frozen pre-trained classifiers**, so the total number of categories is fixed independent of the CC2017 dataset. The image classifier is an ImageNet classifier, pre-trained on ImageNet-1K[1] hence **1000 image classes**. The video classifier based on VideoMAE [2] is trained on Kinetics-400 [3], an annotated video dataset with **400 classes**, including motions, human interactions, etc.

**Reference**

[1] Imagenet: A large-scale hierarchical image database, *CVPR 2009*

[2] Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, *NeurIPS 2022*

[3] The kinetics human action video dataset, *ArXiv 2017*

> **Minor 8** 

Thank you for your valuable addition. We will carefully review this literature and include it as a justification for using the keyframe approach in the final version.

&nbsp;
***

We would like to extend our heartfelt gratitude to you! Our sincerest thanks! Your insights and suggestions from a neuroscience perspective have provided us with numerous inspiring ideas, greatly benefiting not only our current work but also our future research endeavors. Additionally, your detailed feedback has made our submission more solid and complete. We hope our response adequately addresses your questions, and we eagerly look forward to further communication and discussions.

Once again, we sincerely express our deepest gratitude and highest respect for your effort and time!

Best wishes,

All authors of Submission 351
> **Question 8: More Metrics**

The two new metrics ST-SSIM and ST-PSNR you mentioned are very interesting and enlightening, and we have carried out comparative experiments with MinD-Video in terms of the two metrics. We also evaluated NeuroClips with Visual Information Fidelity (VIF) metrics, and the results are shown in the table below (All results are averages of 3 subjects). Notably, NeuroClips' performance at the ST-level far exceeds that of MinD-Video, and even more than pixel-level, which further indicates that NeuroClips has a smoother video reconstruction capability.

| Method|ST-SSIM|SSIM|ST-PSNR|PSNR|VIF|
| - | - | - | - | - | - |
| MinD-Video |0.489|0.171|11.595|8.662|0.113|
| NeuroClips | **0.785** | **0.390** | **17.200** | **9.211** | **0.170** |


As commonly used evaluation metrics in the field of **video generation**, *Fréchet Video Distance (FVD)* is more often used to assess the performance of video diffusion generation models. However, since we **freeze** the pre-trained parameters of the advanced generation model *Animatediff* [1], the effect of our video generation model must be better than the previous video models. Therefore, the evaluations on these video metrics may not be a **fair** one. Instead, we consider the CLIP rerepsentations for evaluation. Note that the consistency in CLIP representation space is more revealing of the degree of **semantic consistency**, reflecting the superiority of Semantic Reconstructor (SR) in NeuroClips. 
In the table above, we conducted an evaluation using the Visual Information Fidelity metric (VIF), instead of the FVD metric. When assessing FVD, all of Pytorch's open-source methods necessitate a video frame rate exceeding 10fps due to their need for some level of downsampling. Despite NeuroClips making significant advancements in frame rate, it falls to meet this requirement. 

[1] Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, *ICLR 2024*

> **Question 9: Difference Voxel Weight Visualization**

We will add a third row with the difference between the weights of subjects as described in *Question 3*, thanks again!

> **Question 10: Voxel Selcetion**

In fact, we did **not** select voxels **specifically for the visual cortex**. We calculated the voxel-wise correlation between the fMRI voxel signals of each training movie repetition for each subject. **The significant voxels**(Bonferroni correction, **P < 0.05**) were considered to be **stimulus-activated voxels** and used for subsequent analysis, as described in the pre-processing paragraphs of **Section 4.1**. We agree with you that other brain regions may also contribute to video decoding as well, so we expanded the range of significance (**MinD-Video using P < 0.01**), and the voxels used in NeuroClips are more than MinD-Video. The following table shows the number of voxels selected by the two methods.

|Method|Subject 1|Subject 2| Subject 3|
|-|-|-|-|
| MinD-Video|6016| 6224|3744|
| NeuroClips|13447|14828|9114|

We believe that our voxel-selective paradigm is more easily migrated to other fMRI decoding tasks and provides a more comprehensive neurobiological perspective.

> **Question 11: Trade-off Between PR & SR**

Due to the parallel design of our Semantic Reconstructor (SR) and Perception Reconstructor (PR), SR focuses more on the decoding of **video semantics**, while PR is more geared towards the reconstruction of **pixel-level information**. SR can significantly improve semantic-related metrics, and PR can improve pixel-level related metrics. So in the end, when the two are combined together, there is a trade-off between semantic and perception reconstruction. During training, the Video Diffusion model will achieve a compromise effect between semantic and perception reconstruction. We will provide more discussions and deeper insighs on the compromise effect.

> **Question 12: Low-level Visual Details**

Thanks for your suggestion. We willl modify it to '***MinD-Video lacks design of low-level visual detailing, so it significantly diverges from the brain’s visual system, exhibiting limitations in perceiving continuous low-level visual details.***'

> **Question 13: Persistence of Vision**

Thanks for your suggestion. Regarding the persistence of vision, we will careful check the references and discussions and supplement more supporting references outlined below.

**Reference**

[1] Ultra-High Temporal Resolution Visual Reconstruction From a Fovea-Like Spike Camera via Spiking Neuron Model, *TPAMI 2023*

[2] CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process, *Arxiv 2024*

[3] Persistence Of Vision Display-A Review, *IOSR-JEEE e-ISSN 2015* 

[3] POV: Persistence of Vision: International Journal of Ethics in Engineering & Management Education

[4] Persistence of vision: the interplay of vision, *Vision, Memory and Media 2010*
> **Question 1: Hemodynamic Delay**

Indeed, a considerable number of **exploratory experiments** have been conducted on the topic of **hemodynamic delay**, which has also been considered in Mind-Video. In the current version of NeuroClips, which employs a fixed 4-second delay, it was observed that the semantics of some of the generated keyframes exhibited latency, particularly in instances where the video clips of the scene were of a **greater duration**. Accordingly, a **sliding window** comprising two or three fMRI frames was devised to actively learn the aforementioned delay. However, it was discovered that employing a sliding window resulted in a **notable reduction** in the final evaluation metrics, with a **more pronounced negative impact**, particularly in the case of shorter video clips. It may be the case that longer videos have a more enduring effect on human brain fMRI signals. In light of the experimental outcomes, we ultimately opted to discard this methodology and instead fix the delay. 

> **Question 2: Ridge Regression**

As you mentioned, the human brain processes information in a highly complex and non-linear way. However, empirical evidence [1, 2, 3] underscores the **effectiveness and sufficiency of linear mapping** for achieving desirable reconstruction outcomes. Notably, **complex nonlinear models will easily overfit to fMRI noise**, leading to poor performance in the test set [4]. We will add more discussion in the Method Section.

**Reference**

[1] Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors, *NeurIPS 2023*

[2] High-resolution image reconstruction with latent diffusion models from human brain activity, *CVPR 2023*

[3] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data, *ICML 2024*

[4] Through their eyes: multi-subject Brain Decoding with simple alignment techniques, *Imaging Neuroscience 2024*

> **Question 3: Voxel Weight Visualization**

In Figure 12, each column represents the PR and SR visualization weights for the same subject. The voxel distribution is the same in each column since the **voxels initially selected by each subject are fixed**. For each subgraph, the weights of the voxels were visualized after normalization, with **brighter regions** representing **higher weights** and **darker regions** representing **lower weights**. For subject 3, the distribution of voxel weights in PR and SR is quite different. For example, in the right region of V4 of the right brain, PR is brighter and SR is darker. We appreciate your suggestion to add another line to visualize and highlight the **difference** between PR and SR. In this way, we can better show the difference in voxel weights between the two modules, and we will add it to the final Supplement.

> **Question 4: Larger Example Images**

We will modify it to a vector figure and adjust the number of images to make it clearer.

> **Question 5: Major Limitations**

Since NeurIPS allows an additional page to be added to the final version, we will move Limitation from the appendix to the main body. Thanks for your suggestion.

> **Question 6: Visualization Software**

We use **Connectome Workbench** from Human Connectome Project (*HCP*),  and flatmap templates were selected as 'Q1-Q6_R440.L.flat.32k_fs_LR.surf.gii' and Q1-Q6_R440.R.flat.32k_fs_LR.surf.gii' . Additionally, the cortical parcellation was manually delineated by **neuroimaging specialists** and **neurologists**, and aligned with the public templates in **FreeSurfer software** with verification. We **normalised** the voxel weights, scaling them to between 0 and 1. Finally, to show a better comparison, the **Colorbar** was chosen to be 0.25-0.75. We will add this note to the final Supplement section.

> **Question 7: Metric Improvement**

We'll add extral analysis to indicate the improvement values on the metrics.
> **Weakness 4: Neurobiological Justification**

Thank you for sharing your expert advice from neuroscience. 

In numerous studies within **cognitive neuroscience** and **computational neuroscience**, researchers have delved into the mechanisms related to the **brain's visual information processing** and memory functions. Building on these studies, we innovatively propose using keyframes to guide our research, addressing the issue of frame rate mismatch between visual stimuli and fMRI signals, and enhancing the model's accuracy in fMRI-to-Video Reconstruction. 

Specifically, [1] demonstrate that **'key-frames'** play a crucial role in how the human brain recalls and connects relevant memories with unfolding events. In [2], a novel video abstraction paradigm which use the brain response reflected by fMRI to guide the extraction of visually informative segments from videos was proposed to quantitatively reveal the attentional engagement of human brain in the comprehension of video. In [3], the key frames in a video clip were used to extract these features, with the combined features from all **keyframes** representing the entire **video clip**. [4] provided a framework and explanation for video summarization.

We will revise and update our text clearance and references in the final version.

**Reference**

[1] Brain mechanisms underlying cue-based memorizing during free viewing of movie Memento, *NeuroImage 2018*

[2] Video abstraction based on fMRI-driven visual attention model, *Information sciences 2014*

[3] Bridging the semantic gap via functional brain imaging, *IEEE Transactions on Multimedia 2011*

[4] A comprehensive survey and mathematical insights towards video summarization, *Journal of Visual Communication and Image Representation 2022*