A Cognitive Process-Inspired Architecture for Subject-Agnostic Brain Visual Decoding
Authors: Jingyu Lu
Haonan Wang
Qixiang ZHANG
Xiaomeng Li
Venue: ICLR 2026 Poster (2026)
Keywords: fMRI
brain decoding
video reconstruction
cross-subject generalization
visual cortex
contrastive learning
zero-shot decoding
Primary area: applications to neuroscience & cognitive science
TL;DR: A cognitive process-inspired fMRI-to-video framework that hierarchically aligns brain features with CLIP representations and enables subject-agnostic applicability.

Abstract: Subject-agnostic brain decoding, which aims to reconstruct continuous visual experiences from fMRI without subject-specific training, holds great potential for clinical applications. However, this direction remains underexplored due to challenges in cross-subject generalization and the complex nature of brain signals.
In this work, we propose Visual Cortex Flow Architecture (VCFlow), a novel hierarchical decoding framework  that explicitly models the ventral-dorsal architecture of the human visual system to learn multi-dimensional representations. By disentangling and leveraging features from early visual cortex, ventral, and dorsal streams, VCFlow captures diverse and complementary cognitive information essential for visual reconstruction.
Furthermore, we introduce a feature-level contrastive learning strategy to enhance the extraction of subject-invariant semantic representations, thereby enhancing subject-agnostic applicability to previously unseen subjects. 
Unlike conventional pipelines that need more than 12 hours of per-subject data and heavy computation, VCFlow sacrifices only 7\% accuracy on average yet generates each reconstructed video in 10 seconds without any retraining, offering a fast and clinically scalable solution.

Published as a conference paper at ICLR 2026
A COGNITIVE PROCESS-INSPIRED ARCHITECTURE FOR SUBJECT-AGNOSTE BRAIN VISUAL DECODING
Jingyu Lu', Haonan Wang'. Qixiang Zhang', Xiaomeng Li
'The Hong Kong University of Science and Technology, Hong Kong SAR, China 2 "Shnnzhen Loop Area Institute, Shenzhen, China
ABSTRACT
Subject-agnostic brain decoding, which aims to reconstruct continuous visual ex- periences from fMRI without subject-specifie training, holds great potential for clinical applications. However, this direction remains underexplored due to chal- lenges in cross-subject generalization and the complex nature of brain signals. In this work, we propose Visual Cortex Flow Architecture (VCFLOW), a novel hierarchical decoding framework that explicitly models the ventral-dorsal archi- tecture of the human visual system to learn multi-dimensional representations. By disentangling and leveraging features from early visual cortex, ventral, and dor- sal streams, VCFLOW captures diverse and complementary cognitive informa- tion essential for visual reconstruction. Furthermore, we introduce a feature-level contrastive learning strategy to enhance the extraction of subject-invarinn invariant seman- tie representations, thereby enhancing suhject-agnostic applicability to previously unseen subjects. Unlike conventional pipelines that need more than 12 hours of per-subject data and heavy computation, VCFLOW sacrifices only 7% accuracy average yet generates each reconstrueted video in 10 seconds without any re- training, offering a fast and clinically scalable solution. The code is available at https://gitii com xmed aa/VCFFLO
INTRODUCTION
The world, as perceived by the human brain, unfolds as a continuous flow of visual experiences-not static images, but dynamic videos rich in motion, context, and meaning. Capturing this fluid nature of perception, IMRI-to-vidco decoding plays critical role in revealing how the brain processes complex visual information over time, encompassing fine visual details, abstract semantic under- standing, and temporal coherence. Previous research (Chen et al., 2023) Gong et al.) 2024 Wang al. 2025) has primarily focused on training and evaluating models on the same specific subjects, aiming to achieve the highest reconstruction quality. However, these methods suffer from funda- mental limitation: they often overlook subject-agnostic applicability. a factor that is even more crit- ical in clinical settings. Specifically, when applied to new patients, these models often require more than 12 hours of subject-specific training. making them impractical for downstream tasks such as large-scale screening clinical rehabilitation. For instance, in detecting conditions like schizophre- nia, hallucinations, or cognitive impairments, subject-specifie models become impractical due to their reliance extensive retraining, which is both time-consuming and expensive. As result, developing subject-agnostic model capable of directly evaluating previously unseen subjects holds far greater clinical and practical value. However, this area remains largely underexplored.
To achieve subject-agnostic capability, a straightforward approach is to modify the preprocessing pipeline and encoding scheme of existing subject-specifie models (Gong al. 2024 Wang al.) 2025) to map into shared representational space, thereby enabling a subject-agnostic setting. How- ever, when applied to unseen subjects, these subject-specific methods performed poorly, primarily due to their inability to extract universal semantic information across subjects, as demonstrated by the results of N in Table To address this, the GLFA (Li al., 2024) method proposed data-level functional alignment strategy that projects data from different subjects into universal space. However, its reliance on pretraining with fMRI data from all subjects substantially reduces its
"Corresponding author; cexmli @ust.hk

---

Published as a conference paper at ICLR 2026
Previous: Bad Applicability
Ours: Good Applicability
Yraining
Subject-agnostic
Unseed
Unseen
new patient training free 10s test
patient: 12h required train spacifct- model
L
large-scale screening
rehabilitation
Figure 1: Former methods are typically subject-dependent, meaning that when encountering new patient, approximately 12 hours of training are required to build a subject-specific model. Such requirements severely constrain the practical applicability and clinical utility of these approaches. By contrast, our method ensures applicability at the subject-agnostic level, allowing inference new patient without any additional training and requiring only about 10 seconds of testing, which provides substantial advantages for downstream tasks.
practical applicability and runs counter to the subject-agnostic paradigm. In this case, it is necessary to redesign truly subject-agnostic model that enables robust decoding across new subjects the level of cognitive features.
To this end, we introduce Visual Cortex Flow Ar- chitecture (VCFL.OW), a novel subject-agnostic ar- chitecture inspired by the dual-stream mechanism of the human visual cortex (Kandel et al., 22000]
Vi groceeday
Neuroscience Motivation
Huff et al. 2018 DiCarlo et al.) 2012; Dumoulin
& Wanddell 2008 Yamins & DiCarlo (016). As
Vestral Stremm
Ventral
illustrated in Fig. human visual cognition pri- marily proceeds along two pathways: the ventral stream, extending from early vision to encode high- level semantics such as abstract concepts and object recognition, and the dorsal stream, extending from early vision capture dynamic features including movement direction, velocity. and spatial transfor- mations. Following this, we split the fMRI brain fea- tures into three components: (I) early visual areas, which are aligned with CLIP low-level features to capture perceptual and structural properties; (2) ven- tral stream areas, which are aligned with CLIP high- level features learn abstract semantics; and (3) dorsal stream areas, which are aligned with CLIP video embeddings, isolating and explicitly modeling motion-related components. This design enhances the precision and controllability of dynamic feature capture, which is essential for video reconstruction tasks. To further enhance subject-agnostic applica- bility, we propose a disentanglement module to sep- arate subject-specife and subject-agnostic semantic components. By employing contrastive learning techniques, effectively extract robust semantic representations that are directly applicable to pre- viously unseen subjects. (Fig. D.
vinaa excedigg
V2 Tretum
Early Vis
Visual cortex
NTT
IPS Vianegerial
Natza
Docsal
The visual cortex can be broadly divided into three types of areas: early vi- sual, ventral, and dorsal. Early visual ar- eas are primarily responsible for detecting low-level features including edges, orienta- tion, and color. Ventral areas are associated with the processing of higher-level and ab- stract visual information. In contrast, dorsal areas are specialized for encoding dynamic features and spatial representations.
Figure 2:
Our contributions can be summarized as follows: (1) We are the first to formulate IMRI-to-video decoding in a subject-agnostic setting, enabling the model to apply to previously unseen subjects without any retraining. (2) We propose VCFLOw, novel subject-agnostic framework inspired by the ventral-dos dorsal dual-stream architecture of the visual cortex, which hierarchically extracts and aligns features in accordance with CLIP's cognitive hierarchy, reinforced by disentanglement
2

---

Published as a conference paper at ICLR 2026
Hierarchical Cogmitive Alignment
Keyframe
Hierarchical
Subject
Visual Stimali
Explicit Decoder
Early
Low-level
loperss
CA
Align
Early Vis
CLIP Vision
Encoder
EMRE Pull Brain
Brain Model
Eversl
Froanal
pare Grader
High-Jevel
CA
Align
Ventral
Spatr
Exene
Form
Motion
CA
Aligm
CLIP Vided Embeddings
Vision Dorsal
Prior
Diffusion
[|0 о|l
Inference
Figure 3: The overall framework VCFLOW consists of three core components: (1) Hicrarchical Cognitive Alignment Module (HCAM), (2) Subject-Agnostic Redistribution Adapter (SARA), and (3) Hierarchical Explicit Decoder (HED). VCFLOW learns three types of semantic representations through HCAM, which are then fused with subject-agnostic common features extracted by SARA. These enriched representations are subsequently decoded by HED to explicitly reconstruct informa- tion across multiple semantic levels.
strategy and feature-level contrastive learning. (3) Conventional subject-specific models require more than 12 hours of subject-specific data and substantial computational resources. In contrast, our subject-agnostic approach incurs only a 7 % average drop across all evaluation metrics compared to fully subject-specific approach while achieving 10-second inference per video, making it fast, practical, and scalable for clinical use.
2 RELATED WORKS
2.1 EMRI-TO-VIDEO RECONSTRUCTION
In recent years, the field of IMRI signal reconstruction has witnessed rapid advancement, largely driven by the proliferation of IMRI technology and the concurrent development of deep learn- ing architectures. Compared to the reconstruction of other modalities from fMRI data, the task of fMRI-to-Vee MiI-to-video reconstruction presents significant challenges due its comprehensive require- ments for both semantic and temporal information. Despite these complexities, the IMRI-to-vider reconstruction task holds considerable potential for understanding the cognitive processes underly- ing the perception of dynamic stimuli. Several notable studies have emerged recently to address this challenge (Chen et 2023; Gong et al. 2024; Wang al.) 2025 Lu al.). For instance, MinD-Vidco (Chen et al. 2023) leverages diffusion models to generate vidcos with competitive semantic quality, yet falls short in accurately capturing fine-grained visual details and maintain- ing consistent dynamic information across frames. NeuroClipp (Gong et al. (224)) further mitigates this limitation by jointly leveraging keyframe reconstruction and temporally blurred video guidance. However, its robustness remains constrained, primarily due to an implicit alignment strategy with CLIP vision features. More recently, the NEURONS (Wang et al. 2025) approach attempts to in- tegrate multidimensional information via the design of explicit training tasks. Nevertheless, critical information essential to these explicit tasks may not be sufficiently preserved within the diffusion prior formulation.
3

---

Published as a conference paper at ICLR 2026
2.2
SUOSS-S LEARNING
In light of individual differences in brain structure and cognition, generalization across subjects typ- ically yields significantly inferior performance compared to within-subject generalization. Nonethe- less, cross-subject generalization remains essential, given its significance for real-time applications in clinical medicine and neuroscience (Wang et al. 2018: Sorger et al., 2012). In the iMRI-to- image domain, many studies have transitioned from individual-centte
to holistic spatial methodolo- gies (Scotti et al. 2024 Huo et al., 2024, Gong et al. 2025 Kong et al.] 2024 Zhou et al.) 2024);
however, these approaches often still require subject- speecifc fine-tuning for previously unseen sub- jects. Similar trends have been observed within the video reconstruction domain (Li et al.] 2024] Fosco et al., 2024). For instance, the GLFA et al. (2024) method addresses this issue by function- ally aligning data to a unified representational space. Nevertheless, such alignment strategies tend to lack semantic hierarchy and robustness, indicating room for further improvement cross-subject generalization.
METHOD
The overall framework of VCFLOW is illustrated in Fig. 3 Inspired by the hierarchical cognitive processes of the human brain, VCFLOw integrates cross-subject semantic alignment multiple cognitive levels. Specifically, VCFLOW comprises the following three modules: 1) Hierarchical Cognitive Alignment Module (HCAM), which extracts cognitive features across hierarchical lev- els and aligns them within a unified semantic $3.1e 2) Subject-Agnostie Redistribution Adapter (SARA), designed to map indivicual-specific semantic representations into common, subject-invariant semantic space for robust cross-subject generalization( (33.2; and 3) Hierarchical Explicit Decoder (HED), which decodes complementary features from different semantic dimen- sions and fuses them synergistically for accurate reconstruction( 3.3
3.1 HIERARCHICAL COGNITIVE ALIGNMENT MODULE
3.1.1 FUNCTIONAL ROI-BASED VOXEL PARTITIONING
Inspired by the dual-stream hypothesis of human visual processing, our cognitive workflow begins with etrly-stage perception, which primarily associated with low-level information such as edges, color, orientation, and spatial structure. Subsequently, along the ventral stream, we select ROIs that capture higher-level semantics and associative memory. In addition, we select ROIs along the dorsal stream enrich our representation with motion perception and spatial information (Kan-) del et al.) 2000; Huff et al. 2018). Further discussion of the ROI selections is provided in the Appendix A Let X E XXX denote the full fMRI voxel sequence, which is extracted from Xinput E IR x Muon Wine, where B is the batch size, S is the number subjects, and V is the voxel length. Then,
XROIs = Xlis TROis] E R Bx8xxvam Vaow
(1)
3.1.2 MULTI-LEVEL FEATURE EXTRACTION
Due to the continuity and hierarchical nature of human cognition, directly extracting information from subset of voxels for specific dimension may disrupt the integrity of semantic represen- tations. In contrast, a learning-based approach that integrates level-specific semantics with global representations provides more coherent and interpretable modeling of brain activity.
Given the input fMRI signal Ximpues we extract four types of features: a full-brain representation Eprain: early visual features Eearly, ventral stream features Eventaa, and dorsal stream features Egorat- The global representation Enrain E T BxDDX is obtained by applying a VIT backbone to the entire voxel input Xinput- Meanwhile, the early visual subset Xeaty. ventral stream subset Xventrals and dorsal stream subset X are independently projected into the same latent feature space, result- ing in Egatys Eventrals and Edersal, respectively. For the global representation Evcan. after extracting universal features through SARA (Sec.
3.1 follow prior work and employ expressive dif- 13 Sx Laap x Cap,, optimized using the same loss Cprior as
fusion prior to transform it into F Fnmin
R
adopted in DALL-E 2 (Ramesh al.
2022
This prior effectively facilitates the transformation
4

---

Published as a conference paper at ICLR 2026
of features from the fMRI domain to the OpenCLIP embedding space, enabling subsequent recon- struction. Consequently, we utilize a learnable Cross-Attention module to guide the integration of features across different cognitive levels, yielding the representations F Fearly F ventral. and F dorsal-
3.1.3 HIERARCHICAL ALIGNMENT
To align multi- lintisniimenii features within the OpenCLIP embedding space, we identify the most representative ground-truth features for each dimension. High-level features can be naturally aligned using CLIP vision embeddings F(L) clipp given their rich semantic content. Aligning low-level infor-
mation is relatively challenging; hence, we align these features with embeddings F(!)
clip from early CLIP VIT layers, guided by neuroscientific insights into the hierarchical correspondence between deep neural networks and the human visual system (Yang et al., 2024). In the alignment process, we adopt the BiMixCo loss (Kim et al., 2020), which leverages MixCo-based data augmentation construct bidirectional contrastive objective that facilitates model convergence, detailed in the Appendix A
3.2 SUBJECT-AGNOSTCC REDISTRIBUTION ADAPTER
3.2.1 REDISTRIBUTION LAYER
Inspired by previous VTT register-based feature extraction strategies (Darcet al. 2023), ), we propose unified semantic feature extraction framework based redistribution block, which serves token-level information classifier to separate and structure semantic information. To accommodate more general scenario, consider multi-subject feature sequences as the input our framework. Let the input feature be defined as E IR XxXXxxX where B is the batch size, S is the number of subjects, L denotes the number of temporal frames or patch tokens, and C' is the original feature dimension. To enhance the model's 's representational capacity, the feature is first expanded along the token dimension:
Eexp Expand(E) E TR. Bx Luass
(2)
The expanded representation Eexp is then passed through redistribution layer to generate two dis- tinct sets of latent tokens:
[T se;; suij Redistribution where the semantic tokens T E BBxSxIXC and the subject-specific tokens T T E
RBxSxLaxxC correspond to the original and expanded token groups, respectively. This redistri- bution mechanism enables the model to explicitly disentangle generalizable semantic content from subject-dependent features, thereby facilitating more robust and interpretable cross-subject align- ment. In this task, L and C' correspond to the token length and channel dimension of features extracted by OpenCLIP, respectively.
3.2.2 TRAINING OBJECTIVES
To achieve the above objectives, we define set of training losses based on the semantic and subject tokens produced by the redistribution block.
First, we apply a BiMixCo-based alignment loss to encourage the semantic tokens to align with CLIP vision embeddings. Given semantic tokens Teem and corresponding CLIP embeddings Feip-- the semantic alignment loss is defined as:
Calign BiMixCo Fetip)
(4)
Building upon the semantic alignment, we further aim to project subject-specific semantics into shared latent space that ensures semantic consistency across individuals. To achieve this, we adopt an inter-subject alignment strategy based on bidirectional contrastive learning scheme. Specif- ically, we construct moving window across subjects and apply symmetric InfoNCE loss to enforce mutual alignment. Notably, as the number of subjects increases, this training paradigm becomes more effective and stable due to the richer inter-subject comparisons.
Cgenetie
2(S - 1) 1=2
InfoNCE
T norm
1-1.sem 1.sem 5,sem
morm
+ InfoNCE (Trorm morm 1-1..mm)
(5)
5

---

Published as a conference paper at ICLR 2026
To complement the semantic alignment, it essential to preserve subject-specific identity informa- tion embedded within the individualized semantic deviations as well. To this end, we introduce a subject classifier trained on T suhj. where each token, along the subject dimension, corresponds to subject-specifie representation. The classifier is supervised using a cross-entropy loss that encour- ages the retention of discriminative individual features, where Usubj denotes the one-hot ground-truth label for subject and 2(x) represents the classifier's ' logit score corresponding to the prediction that the input belongs subject k:
K
exp(x(A)) R Σラ= exp(z(0))
Sub
Suu log
(6)
The total loss function is weighted combination of all objectives:
LSARA Calign AsubjCsubj Agoneric
3.3 HIERARCHICAL EXPLICIT DECODER
In the reconstruction of videos, directly utilizing ex- tracted feature embeddings may fail adequately inte- grate information across multiple cognitive levels. How- ever, decoding these embeddings into existing auxiliary modalities such as textual descriptions, segmentation masks, and blurry video representations-car signifi- cantly enhance reconstruction quality. This conversion effectively achieved by formulating explicit auxiliary tasks. Within the Hierarchical Cognitive Alignment Module, we systematically generate three distinct fea- tures: F Fearly: Fvemmal, and Foorsal. each associated with tailored explicit tasks that refine feature representations within their respective cognitive dimensions, thus im- proving reconstruction performance (Fig. 4.
Dorsal
frames
Early Vis
M000 Ventral
smad bad perihed
-100 o0l
For the ventral stream feature F veatrat. which encodes ab- stract semantic content such object identity and cat- egorical meaning, we introduce two explicit tasks: im- age caption generation and object category classification, yielding the losses Lesption and Lets, respectively. For the early visual feature F carryy which represents perceptual and structural properties such edges, tex- tures, and spatial layout, we design segmentation task to effectively capture morphological details, resulting in the segmentation loss Lug. Regarding the dorsal stream feature F Farrsal, which charac- terizes spatial-temporal dynamics and motion-related cues, we align it with blurry video modalities to explicitly capture nnttial-temporal motion information. Specifically, the video frame dimension, obtaining F dorsall IR BxFxxxx Fxxxxxxxxaa xxp Cap, where F represents the frame dimension. This transformed feature is further projected into the latent space of Variational Au- toencoder (VAE) to achieve alignment, yielding the alignment loss Luotionn
The inference stage of VCFL.ww integrates multi-level seman- tie embeddings to facilitate comprehen- sive decoding.
Figure 4:
first project F deesal into
Finally, the total loss is formulated as:
LIED Acaption Cesptiin + Aets Lets + Aaay Lueg + Amatioo Emotion
Following NEURONS Wang et al. 2025), we progressively adjust loss coefficients during train- ing, shown in Appendix
4 EXPERIMENT
4.1 FMRI-IMAGE DATASETS AND PRETRAINING
We pretrain the backbone on the DIR dataset (Shen et al. 2019) and the GOD dataset (Horikawal
& Kamitani 2017) to acquire initial capacity for capturing semantic representations from neural

---

Published as a conference paper at ICLR 2026
activity. In the DIR and GOD datasets, fMRI signals were recorded from eight subjects while they viewed 1,250 natural images spanning 200 categories using a 3.0-Tesla Siemens MAGNETOM Verio scanner. The visual stimuli used in the image presentation experiments for both datasets were identical and sourced from ImageNet (Deng et al. 2009). Among these images, 1,200) images from 150 categories were allocated to the training sessions, whereas 50 images from 50 categories were reserved for the test sessions.
Due to the lack of dynamic information in image datasets, the primary goal of our pretraining stage is to train the backbone to acquire an initial understanding of semantics and to develop the ability to separate subject tokens. Therefore, we adopt the SARA training objective, as shown in Equation 7
4.2 DATASET AND PREPROCESSING
In this study, we conducted IMRI-to-video reconstruction experiments using the publicly available fMRI-video dataset, cc2017 dataset (Wen et al. 2018). For each subject, 18 training video clips and testing video clips-each minutes duration were presented. The training clips were shown twice. while the testing clips were presented 10 times, and the corresponding fMRI signals in the test set were averaged across repetitions to improve signal-to-noise ratio. MRI data (including T1- and T2-weighted anatomical scans) and fMRI data (with temporal resolution of 2 seconds) were acquired using a 3-Tesla MRI scanner. In total, the dataset contains 8640 training samples and 1200 testing samples of synchronized IMRI-video pairs per subject.
For fMRT data preprocessing, we follow the strategy proposed in fMRI-PTE (Qian et al. 2023) map fMRI signals into common representational space, upon which brain regions further segmented. Considering the hemodynamic response delay between stimulus onset and the peak of the BOLD signal, we introduce temporal shift of approximately seconds to the fMRI data. Further details are provided in the Appendix B
4.3 EVALUATION METRICS
To comprehensively evaluate the reconstruction quality, we adopt two-fold assessment strategy, considering both frame-level and video-level performance. At the frame level, we evaluate both semantic consistency and pixel-wise similarity. For semantic evaluation, we perform an N-way top- K classification task based on 1,000 ImageNet categories. Each trial compares the classification result of predicted frame against its ground truth counterpart. A trial is deemed correct if the ground truth label appears within the top-K predictions (top-1 in our case) of the predicted frame, randomly selected from N candidate labels. The final accuracy is averaged over 100 repeated trials. For pixel-level assessment, we report SSIM and PSNR scores to capture structural and intensity- level differences between the reconstructed and original frames. At the video level, we examine two key aspects: semantic accuracy and spatiotemporal (ST) continuity. Semantic accuracy is also measured through an N-way top-K action classification task (top in our case) involving 400 classes from the Kinetics-400 dataset (Kay et al. 2017). using a VideoMAE- as the classifier. To evaluate spatiotemporal coherence, we calculate the CLIP embedding for each video frame and compute the mean cosine similarity between every pair of adjacent frames. This metric, commonly referred to as CLIP-pec in the video editing litcrature (Wu et al.) 2023), reflects how smoothly semantic content transitions over time.
model (Tong et al.) 2022)
4.4
MAIN RESULTS
4.4.1
QUANTITATIVE RESULTS
We evaluated the performance of VCFLow against existing methods, conducting comparisons across three subjects well as their averaged results. As shown in Table our approach achieves substantial improvements over the subject-agnostic baselines NEURONS and GLFA*, and even surpasses GLFA, which pretrains the encoder fMRI data from all subjects. These results high- light the superiority of our framework.
To be specific, the frame-based metrics exhibit substantial improvements. At the semantic level, our method attains an accuracy of 14.0% on the 50-way classification task, representing a 46% relative gain over GL_AA (9.6%). Consistent improvements are also observed in pixel-level metrics,
7

---

Published as a conference paper at ICLR 2026
Frame-bused
Video-baxed
Method
Semantic level
Pixel level
Semantic level
leve
50-way?
2-wayl
SSIMT
PSNRT
50-wayt
2-way1
CLIP-pect
12.3%
77.1%
0.151
17.9%
GLEA Li
2014
12.6%
18.1%
0,181
17.9%
83.7%
NEURONS
9.7%
74.3%
GLFA" Li mt 2024
9.0%
74.1%
0.133
16.3%
83.3%
VCFLOW
14.2%
0.389
84.8%
0,944
IMRI-PTE-V
[T al
2024
x
10.4%
76.2%
0,130
17.4%
84.4%
NEURONS Wane et al.
2025
10.6%%
75.5%
0.385
10,000
16.8%
84.3%
0,936
23
[Laet
74.9%
84.6%
VCFLOW
13.2%
77.6 %
0.424
10.629
18.0%
84.3%
0.937
IMRI-PTE-V (T. al.
GLEFA [i et al RUA
x
11.6%
77.7 %
0.172
19.3%
84.7%
1.24
NEURONS
2025
10.0%
74.9%
0.931
GLEFN" [La ct Mi 2024
9.5%
75.3%
0.137
17.0%
84.1%
VCFLOW
IMRI-PTE-V et 2024
x
11.1%
76.6%
0,147
17.8%
84.1
GLFA et g 2014
x
11.6%
77.5%
0.173
18.2%
NEUROTS Wiigg e 2025
10.15 1%
74.9%
0,380
9.612
16.1%
83.6%
0,931
9.6%
74.8%
VCFLoW
14.0%
77.9%
0.396
10.478
18.2%
84.5%
0.940
Comp.
VCFLOW vs. GLFA" (A%)
+45.8%
+4.1%
+189.1 1%
+7.1%
+0.6%
VCFLoW vs. NEURONS' (4 %))
+38.655 6%
+4 +4.0%
+4.22%
+9.0%
+13.0%
+1.1%
+1.0%
(A%)
+20.7%
Table 1: Quantitative comparison of VCFLOW with representative methods. All results are based on subjects provided by the cc2017 dataset (Wen et al. 2018). GLFA refers to GLFA results with test subject data excluded during pretraining. NEURONS refers the NEURONS model adapted to subject-agnostic setting by modifying its data processing pipeline and encoder components. w/o Pretrain indicates whether the encoder is pretrained using the fMRI data of the test subject.
including SSIM and PSNR. Moreover, the video-based metrics achieve state-of-the-ar performance under the current setting, highlighting the model's superior ability to capture dynamic information. These results collectively underscore the effectiveness of VCFLOw both motion modeling and semantic feature extraction. The results comparing with subject-specific methods provided in Appendix C
4.4.2 QUALITATIVE RESULTS
We compare the qualitative performance of VCFLOW with GLFA, as illustrated in Fig. The results demonstrate that our method achieves superior performance in both semantic accuracy and the ability to capture motion dynamics. Benefiting from the incorporation of dedicated motion features, the reconstructed videos exhibit more coherent and faithful representations of spatial and temporal information. Additional results provided in the Appendix C
4.5 ABLATION STUDIES
In this section, we evaluate the effectiveness of four key components: IMRI-to-image pretraining, HCAM, SARA, and HED. All experiments follow a subject-agnostic setting, where models are trained on subjects and 3 and directly tested on subject tributes to performance gains across both video-based and imsge-based metrics. Adding HCAM enhances semantic understanding, while SARA boosts cross-subject transferability, reflected in CLIP-pec and PSNR. Introducing HED provides the most significant gains, particularly in high- level semantics and reconstruction quality.
shown in Table each module con-
4.6 INTERPRETATION RESULTS
By adopting the cortical projection technique introduced in YYang et al. 2024). we visualize the hier- archical embeddings the brain surface map. As illustrated in Fig. 6 the early vision embeddings exhibit a clear correspondence with early visual areas, notably V1 through V4. The ventral stream embeddings show strong activation in high-level visual regions such the FFA and PPA, while also
8

---

Published as a conference paper at ICLR 2026
VoFfow
Figure Compared with GLFA (Li et al. 2024). the qualitative comparison demonstrates that
VCFLOW achieves superior semantic fidelity and temporal coherence, effectively capturing fine- grained semantics and preserving motion information a subject-agnostic setting.
Bruin
Image
Modules
Frume-based
Video-based
Model Pretrain
HCAM
SARA
HED
50-wayf
2-wayt
SSIM
PSNR1
50-way1
2-wayt
CLIP-pcc
11.3%
73.1%
0.401
9.720
12.6%
81.3%
0.908
10.4%
75.0%
0.382
9.866
15.3%
81.8%
0.918
/
11.8%
75.9%
0.357
9.583
14.7%
82.4%
0.919
d
12.4%
77.2%
0.389
10.442
15.2%
83.1%
0.934
/
14.2%
78.6%
0,389
10.469
18.9%
84.8%
0.944
Table 2: Ablations on the key components of VCFLOW, and all results are from subject 1.
demonstrating more diffuse projections across broader cortical areas, likely due their encoding of abstract global semantics. In contrast, the dorsal stream embeddings align well with motion-related regions, particularly those associated with MST. These interpretation results are highly consistent with the neurocognitive structure underpinning our proposed framework.
Early Vis
Ventral
Dorsal
Figure 6: By projecting the three types of embeddings from different semantic dimensions onto the cortical surface, we can clearly observe their respective correspondences with different brain regions.
CONCLUSION
In this work, we propose VCFLOW, the first subject-agnostic framework for fMRI-to-video recon- struction. We design novel dual-stream hierarchical feature extraction architecture inspired by the

---

Published as a conference paper at ICLR 2026
human brain's cognitive pathways, enabling the model extract multi-level features for accurate video reconstruction. A key innovation of our approach lies in the utilization of CLIP embeddings from multiple layers to achieve fine- greeggraindd semantic alignment with fMRI signals. To further enhance generalization across individuals, we introduce a redistribution-basee cross-subject learn- ing strategy that captures subject-invariant representations. Without relying any subject-specific training data, VCFLOW achieves efficient video reconstruction with only a marginal drop in ac- curacy, offering rapid and generalizable solution suitable for clinical applications. Additionally, our interpretation analyses provide compelling evidence of the alignment between the reconstructed features and human visual cognition mechanisms.
ACKNOWLEDGEMENTS
This work was supported in part by the Joint Research Scheme (JRS) under the National Natural Sci- ence Foundation of China (NSFC) and the Research Grants Council (RGC) of Hong Kong (Project No. N.HKUST654/24); the Research Grants Council of the Hong Kong Special Administrative Re- gion, China (Project Reference No. AoE/E-601/24-N and Project No. R6005-24); and the National Natural Science Foundation of China (Grant No. 62306254).
REFERENCES
Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video recon- struction from brain activity. Advances in Neural Information Processing Systems, 36:24841- 24858, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-434 2012.
Serge O Dumoulin and Brian Wandell. Population receptive field estimates in human visual cortex. Neuroimage, 39()):647 660) 2008.
Camilo Fosco, Benjamin Lahner, Bowen Pan, Alex Andonian, Emilie Josephs, Alex Lascelles, and
Aude Oliva. Brain netflix: Scaling data reconstruct videos from brain signals. In European Conference on Computer Vision, pp. 474. Springer, 2024.
Matthew Glasser, Stamatios N Sotiropoulos. J Anthony Wilson, Timothy Coalson, Bruce Fischl, Jesper L Andersson. Junqian Xu. Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80:105-124, 2013.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):711-77, 2016.
Zixuan Gong, Guangyin Bao, Qi Zhang, Zhongwei Wan, Duoqian Miao, Shoujin Wang, Lei Zhu, Changwei Wang. Rongtao Xu, Liang Hu, et al. Neuroclips: Towards high-fidelity and smooth fmrii-o-video reconstruction. Advances in Neural Information Processing Systems, 37:51655- 51683, 2024.
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Rongtao Xu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14247-14255, 2025.
10

---

Published as a conference paper at ICLR 2026
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint 20KV::2307.04725, 2023.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.
Shuo Huang, Wei Shao, Mei-Ling Wang, and Dao-Qiang Zhang. fmri-based decoding of visual information from human brain activity: brief review. International Journal of Automation and Computing, 18(2):170-188 2021.
Trevor Huff, Navid Mahabadi, and Prasanna Tadi. Neuroanatomy, visual contex. 2018.
Jingyang Huo, Yikai Wang, Yun Wang, Xuelin Qian, Chong Li, Yanwei Fu, and Jianfeng Feng. Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi- level modulation. In European Conference Computer Vision, pp. 56-73. Springer, 2024.
Eric Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, volume 4. McGraw-hill New York, 2000.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1rXiv:/705.06900 2017.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXi::2010006300, 2020.
Xiangtao Kong, Kexin Huang, Ping Li, and Lei Zhang. Toward generalizing visual brain decoding to unseen subjects. arXiv preprint 2rXiv:22100 10. 14445, 2024.
Chong Li, Xuelin Qian, Yun Wang. Jingyang Huo, Xiangyang Xue, Yanwei Fu, and Jianfeng Feng. Enhancing cross-subject fmriito-video decoding with global-local functional aligmment. In Euro- pean Conference on Computer Vision, pp. 353-369. Springer, 2024.
Yizhuo Lu, Changde Du, Chong Wang, Xuanliu Zhu, Liuyun Jiang, Xujin Li, and Huiguang He. Animate your thoughts: Reconstruction of dynamic natural vision from human brain activity. In The Thirteenth International Conference on Learning Representations.
Karla Miller, Fidel Alfaro-Almagro, Neal K Bangerter, David L Thomas, Essa Yacoub, Jumqian Xu, Andreas Bartsch, Saad Jbabdi, Stamatios N Sotiropoulos, Jesper LR Andersson, et al. Mul- timodal population brain imaging in the uk biobank prospective epidemiological study. Nature neuroscience, (9(11):1523-155 2016
Xuelin Qian, Yun Wang, Jingyang Huo, Jianfeng Feng, and Yanwei Fu. fmri-pte: A large-scale fmri pretrained transformer encoder for multi-subject brain activity decoding. arXiv preprint wwXiv:2311.003422 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115,, 1(2):3, 2022.
Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu. Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with I hour of data. arXiv preprint arXiv:2403.1 2007, 2024.
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruc- tion from human brain activity. PLoS computational biology, 15(1)ce1006633, 2019.
Bettina Sorger, Joel Reithler, Brigitte Dahmen, and Rainer Goebel. A real-time fanri-based spelling device immediately enabling robust motor motor-independen communication. Current Biology, 22(14)) 1333-1338, 2012.
11

---

Published as a conference paper at ICLR 2026
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 35:10078-10095 2022.
Patrick Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Ra- sul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github. com/
huuginggface//fffuseee 2022.
Haonan Wang, Qixiang Zhang, Lehan Wang, Xuanqi Huang, and Xiaomeng Li. Neurons: Emulating the human visual cortex improves fidelity and interpretability in fmri-to-video reconstruction. arXiv preprint arXiv:2503. 1677, 2025.
Tianlu Wang, Dante Mantini, and Celine R Gillebert. The potential of real-time finr neurofeedback for stroke rehabilitation: A systematic review. cortex, 107:148-166 165, 2018
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12): 4136 1136-4160, 2018
Jay Zhangjie Wu, Xiuyu Li, Difei Gao, Zhen Dong, Jinbin Bai, Aishani Singh, Xiaoyu Xiang, Youzeng Li, Zuwei Huang, Yuanxi Sun, et al. Cvpr 2023 text guided video editing competition. arXiv preprint arXiv:2310116000 2023.
Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356 2016.
Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13330-23040) 2024.
Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. Clip-mused: Clip-guide multi- subject visual neural information semantic decoding. arXiv preprint 249894440,.0899 2024.
A
TECHNICAL DETAILS
A.1
MIXCO LOSS
The MixCo needs to mix independent fMRI signals. For each Vc. we random sample another fMRI Vm.. which the keyframe of the clip index by m.- Then, we mix Ve and Vm, using a linear combination:
y mix( Vm.) = A. Vo + A.)3
sampled from Beta distribution. Then, we adapt the ridge regression to map 3 to a flwer-dimensions y and obtain the embedding eyz via the MIP, i.e. eyz = E()x') Based on this, the BiMixCo loss can be formed as:
where D denotes mixed fMRI signal and >e is
(xssim(ey,,xx)//
LBIMixCo
A log
2N,,
N exp(sim(ex; ex,
1
N
exp(sim(ey; ex )/T)
Σ
(1
X1) log
2N,
N, k= exp (xxsssi(ey)
i=1
(10)
1
N
exp (sim(ey), ,xx;) ex2)/7)
X
log
2N
Σ k= exp(sim((e))) ex,,,
/=1
N
(1- 21) log exp(sim(Es) ex2 )/T)
2N,
f
EN exp(sim ex, 3/T)
J=1
{ [||m=]}
where ex, denotes the Ommheiiiemn for keyframe X.,
12

---

Published as a conference paper at ICLR 2026
A.2 PRIOR LOss
We use the Diffusion Prior to transform fMRI embedding ey, into the reconstructed OpenCLIP embeddings of video ey.- Similar to DALLE-2, Diffusion Prior predicts the target embeddings with mean-squared error (MSE) as the supervised objective:
Lpiir
Bey, ey ((..)llee)) ev.ll-
(11)
A.3 FUNCTIONAL CLASSIFICATION OF ROIS
In constructing the dual-stream framework, distinguish three main groups: early vision ROIs, dorsal ROIs, and ventral ROIs. However, the dual-stream theory does not provide a definitive as- signment for every ROI in the visual cortex, and the categorization of certain borderline regions re- mains ambiguous. To address this uncertainty, we adopted two alternative ROI partitioning schemes and compared their effectiveness. Considering both cognitive relevance and the balance in voxel distribution, we ultimately selected scheme A as our final partitioning strategy. The comparative results on subject are presented in Table
Scheme A
Early vision: VI, V2, V3, V4
Dorsal stream: V3A, V3B, V6, V6A, V7, IPS1, LO1, LO2, L03, EST, MT, MST, V3CD, V4t, PH, IPO
Ventral stream: FFC, PIT, V8, VMVI, VMV2, VMV3, VVC, PHAI, PHA2, PHA3, TE2p
Scheme B
Early vision: V1, V2, V3, V4
Dorsal stream: V3A, V3B, V6, V6A, V7, IPSI, FST, MT. MST, V3CD, V4t, IPO
Ventral stream: FFC, PIT, V8, VMVI, VMV2, VMV3, VVC. PHAI, PHA2, PHA3, TE2p.
LO1, LO2, L03, PH
Frame-based
Video-based
Scheme
Semantic-level
Pixel-level
Semantic-level
ST-levll
50-way1
2-way1
SSIM
PSNR1
50-wayt
2-wayt
CLIP-pect
Scheme A
14.2%
78.6%
0.389
10.469
18.9%
84.8%
0.944
Scheme B
12.4%
71.9%
0.353
9.366
9.8%
80.6%
0.913
Table 3: Comparison of ROI partitioning schemes on subject 1.
A.4 PROGRESSIVE LEARNING STRATEGY
We adopt a progressive learning strategy (Wang et al. 2025) to jointly optimize multiple loss func- tions. This strategy regulates the weight of each loss along smooth schedule: starting from 1, increasing to 10, and then decaying back to I within a predefined period. The scheduling is applied four loss functions, each with distinct offset period promote balanced training.
Formally, let E denote the epoch index, B the batch index, and Ng the number of batches per epoch.
The total number of batches within a period P is defined as T P. Ngg. For a period that starts at epoch S, the position of the current batch within this period is computed as
= (E S) NB + B.
(12)
The weight w at this position is then given by
C )
1+9 sin
(13)
If E. lies outside the interval [S,S + P), the weight remains constant, i.e., 1.
13

---

Published as a conference paper at ICLR 2026
A.5 SUBTASKS DESCRIPTIONS
Our task setup is inspired by NEURONS [Wang et al. 2025), and we describe each subtask in detail below. We first present the ventral-related tasks, which are mainly designed capture high-level semantic information along the visual processing stream.
Concept Recognition. To enhance conceptual understanding, introduce a concept recognition lask by adding a multi-label classifier (-) that predicts the key concepts in each frame from the IMRI-derived visual embeddings. Coneretely, we apply cross-entropy loss between the classifier prediction and the ground-trute truth (GT) concept list:
Cele = Lce (Paeeee())")
(14)
where E" denotes the mean of e" along the frame axis, and the GT concept list.
Scene Description. To further model scene-level semantics, we incorporate a scene description task that aims generate a descriptive caption for each video frame. Specifically, we finetune pre-trained text decoder Deaption (-), which takes the fMRI-derived text embeddings e as input and produces the caption. We adopt GPT-2 (Radford al.] (019) as the text decoder and train using prefix language modeling. Given a GT caption token sequence S {so, $1..... $1ss} {|s)) corresponding text embedding e', the decoder Deaption (-) is trained to reconstruct & conditioned on e' as the prefix. The training objective Leaption is defined as:
and the
[
Leaption
log D Peaption (8; ec)s,"),
(15)
[
where 8; denotes the i-th token in the GT sequence S.
Next, we consider early-vision tasks, which primarily aim to learn coarse object contours and spatial masks.
Key Object Segmentation. To better capture object-level information, we design a text-driven video decoder based the VAE video decoder (von Platen al. 2022). This decoder takes the video embeddings e" together with the text embeddings ef as inputs. For this task, the text embeddings obtained by encoding the category name of the key object with the CLIP text encoder. cross- attention module is then applied to activate specific patches in e" (used as queries Q) corresponding to el (used as keys K and values V): eeg softmax QK
The activated feature e*eg upsampled higher resolution for pixel-level prediction. We then employ simple segmentation head (-) (-) to generate the binary segmentation masks Yseg for the key object in the video. The training objective for this task is binary cross-entropy loss, denoted Lseg-
Vi
Finally, we introduce the dorsal-related task, which focuses on modeling global motion information.
Blurry Video Reconstruction. We reuse the same VAE decoder in the key object segmenta- tion task, but replace the segmentation head (-.(-) with a reconstruction head Der(-). Given the video embeddings e", the text-driven video decoder together with Dwr (-) generates blurry video ye moooioo, We then map mottoon into the latent space of the Stable Diffusion VAE to obtain the latent embeddings V/.. This subtask trained with mean absolute error (MAE) loss, defined as:
F
Emotion F
Σ
motion Vers
(16)
is worth noting that, unlike NEURONS, which jointly optimizes all tasks over single global feature representation, we explicitly associate each task with particular cognitive process and train it on the corresponding process-specifie features.
A.6 INFERENCE
At inference time, pipeline leverages pre-trained T2V diffusion model (Guo al. 2023). in a setup similar to NEURONS (Wang et al., 2025). We conditions the model jointly on a control image, bluryy video, and text description. These three inputs assembled from the outputs of
14

---

Published as a conference paper at ICLR 2026
the decoupled tasks: control image is reconstructed from each frame via unCLIP (Ramesh al. 2022). while Dete and Peaption provide the predicted concepts and generated captions, respectively. The embedding of the top-1 predicted concept, together with the caption embedding, is used to guide video-mask prediction through D.s and blurry video reconstruction through Dur- To further enhance the prominence of the key object, rescale its binary mask from 0, 1 (0.5, 1] and multiply it with both the control image and the blurry video before feeding them into the diffusion model.
Because inference for unseen subjects does not require any subject-specific finetuning, their data can be directly processed by the model. With this setup, the inference time-normalized by the total number of videos remains below 10 seconds per video. All measurements are obtained on an NVIDIA RTX 4090 GPU.
A.7 DETAILS OF BRAIN MODEL
As shown in Fig. 3 our Brain Model extracts four types of representations from the input fMRI sig- nals: full-brain representation Eggain. early visual features Esarly. ventral-stream features E Eventral- and dorsal-streat features Edorsal- Here, the Brain Model refers to the module responsible for deriv- ing these IMRI-based features, consisting of a ViT backbone together with several linear projection heads.
The linear heads operate directly on the flattened voxel signals to obtain the three cognitively grounded feature sets sets-Eeerry- Eventral, and Estoral- each with shape Bxs'xD x For the global full- brain representation Eprain: we adopt ViT-based fMRI encoder (fMRI-PTE (Qian et al. 2023), pre- trained on the UK Biobank dataset (Miller et al. 2016)), which produces features of shape RBx5xD
B DETAILS OF DATA PRE-PROCESSING
We employ both the fMRI-to tmmee datasets and the fMRI-to- dataset in our experiments (Wen al., 2018 Shen al.) 2019. Horikawa & Kamitani 2017). For fMRI data preprocessing, first aligned all fMRI volumes to the 32k_fsLR brain surface space based on anatomical struc- ture (Glasseree al. 2013). Unlike conventional fMRI decoding methods that flatten each fMRI frame into a one-dimensional signal and select subject -specicc activated voxels, our approach transforms the fMRI data into unified surface-based representation across subjects. After surface alignment, we applied voxel-wise 2- z-transformation to normalize the fMRI signals and unfolded the cortical sur- face into a two-dimensional plane, thereby preserving spatial relationships between adjacent voxels. Given that only subset of brain regions is typically activated during visual stimulation tasks (Huang) ct al. 202I), we further restricted the analysis to early and higher visual cortical Regions of Interest (ROls). These ROIs, encompassing total of 8,405 vertices, were defined according to the HCP- MMP atlas (Glasser et al. 2016) the 32k 22k surface space. Each processed fMRI volume was then converted into single single-channel 256 256 image. For datasets with multiple runs corresponding to the same video stimulus, we averaged the aligned fMRI frames across runs. Finally, for fMRO- to- Video dataset, to account for the hemodynamic response delay- the time lag between stimulus presentation and the peak of the BOLD response- temporal shift of approximately 6 seconds was applied to the fMRI data.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I COMPARISON WITH SUBJECT-SPECII METHOPS
We also compare our method with state-of-the-art approaches under different settings shown in Table 4 Notably, when compared with the subject-specific state-of-the-art method NEU- RONS WWng et al. 2025), our approach exhibits only modest decrease of 7% in average per- formance. However, it achieves the advantage of direct testing without the need for retraining.
C.2 VISUALIZATION RESULTS
We further conduct a qualitative comparison among VCFLOw, GLEA (L al. 2024), and NEU- RONS (Wang et al. 2025), illustrated in Fig. Compared to GLFA, VCFLOW consistently
15

---

Published as a conference paper at ICLR 2026
Frame-based
Video-based
Method
Setting
Semantic-level
Pixel-level
Semantic- -enaa
ST-level
50-way1
2-wayt
SSIM
50-way1
2-way1
CLiP--poe
Subject
NEURONS (Wans clal.
2025
Subjet-speeifie
20.6%
81.0%
0.373
25.4%
86.2%
0.932
GLEA eL 2024
Subject- addppiive pretraining
11.6%
77.7%
0.172
19.3%
84.7%
VCFLOW
Subject-annostic
14,2%
78.656
0.389
10.469
18.9%
84.8%
0.944
Subject
NEURONS Wong et GLFA eL
2025
Supeoct-speeifle
21.4%
81.0%
0.353
9.502
25.2%
86.0%
0.933
Subject-adaprive pretaining
76.8%
17.5%
83.8%
VCFLOW
Subject-agnesti
13.2%
77.6%
0.424
10.629
18.0%
84.3%
0.937
Subject
NEURONS (Wagg et GLFA [Li a. 2024
Subject-specife
81.7%
Subject-udupiivee udaptive pretraining
12.6%
78.1%
0.181
17.9%
83.7%
VCFLOW
Subject-agnostic
14.7%
77.6%
0.375
10.335
17.8%
84.5%
0.940
Average
NEURONS Wang el 2025
Subjoct-speeific
21.0%
81.2%
0.365
9.527
26.1%
86.3%
0.934
GLFA
2024
Subject-adaptive pretraining
11.6%
77.5%
0.173
18.29
84.1%
VCFLOW
Subject-agnestic
14.0%
77.9%
0.396
10.478
18.2%
84.5%
0.940
VCP GL.FA (A%)
+20.7%
+0.5%
+128
0.0%
40.5%
VCFLOW vs. NEURONS (4%)
-3.33%
-4.1%
+8.5%
+10009
-30.3%
-2.1%
+0.6%%
GLFA NEURONS (A%)
-4.8%
-4.6%
-52.69
-30.3%
-2.69
Table 4: Comparison of VCFLOW with GLEA (Li et al., 2024 and NEURONS (Wang et al. across subjects. GLEA adopts subject-adaptive pretraining by using fMRI data from all subjects, while NEURONS is trained and evaluated on the same subject.
2025)
delivers superior performance in both semantic fidelity and dynamic coherence. In comparison with the subject-specific model NEURONS, which exhibits high visual fidelity and strong temporal con- sistency, VCFLOW achieves similarly high-quality reconstructions while maintaining robust motion dynamics. Notably, in scenarios involving common semantic structures, VCFLOW even surpasses NEURONS by producing smoother motion trajectories and more semantically accurate content, de- spite being trained in a subject-agnostee manner.
C.3 FAILURE CASES
Since our evaluation is conducted in cross-subject subject-agnostic setting and the amount of training data is limited, number of failure cases are observed. As illustrated in Fig. X the major failure cases can be broadly categorized into two types. The first type arises when the primary objects in the stimulus vidcos belong to very categories, making it difficult for the model to effectively learn their semantics, as exemplified by the first two cases. The second type occurs when the semanties in the stimulus videos are overly complex and intertwined, which makes it challenging to recover the most salient semantic components, as shown in the latter two cases.
C.4 ADDITIONAL ABLATION RESULTS
To strengthen the persuasiveness of our SARA and HED designs, conducted ablation studies on both components, as shown in Table 5 and Table 6
SARA
Frame-hased
Video-based
HCAM
HED
Callign
Lunti
Cyaneric
50-wayt
2-wayt
SSIM
PSNR1
50-way?
2-way?
CLIP-pect
7.52%
68. 2%
0.392
9.090
9.67%
77. 3%
0.903
10.7%
75.0%
0.368
9.983
13.9%
81.7%
0.924
14.2%
78.6% %
0.389
10.469
18.9%
84.8%
0.944
Table 5: Ablations on the components of SARA, and all results are from subject 1.
C.5 VISUALIZATION COMPARISON
We conducted a comparative visualization of the preprocessed data and the semantic tokens, as shown in Fig. 9
16

---

Published as a conference paper at ICLR 2026
GT
VCFlow
NEURONS
GLFA
GT
VCFlow
NEURONS
GLFA
Figure 7: Qualitative comparison results among VCFLOW, GLFA and NEURONS.
17

---

Published as a conference paper at ICLR 2026
GT
VCFlow
GT
VCFlow
GT
VCFlow
GT
VCFlow
Figure 8: Representative failure cases under the cross-subject subject-agnostic setting, including (1) rare object categories (first two examples) and (2) overly complex and intertwined semantics (last two examples).
D USE OF LLMS
Large Language Models (LLMs) were employed solely for spelling and writing refinement in the preparation of this paper. They were not involved in research ideation, methodological design, experimental execution, or substantive content generation.
18

---

Published as a conference paper at ICLR 2026
HCAM
SARA
HED
Frame-based
Video-based
Logian
Lik
Lug
Lmatio
PL
50-wayt
2-wayt
SSIM1
PSNR
50-wayt
2-way?
CLIP-pecT
10.0%
72.6%
0.356
9.370
12.8%
81.4%
0.907
8.2%:
71.1%
0.360
10.197
14.6%
82.3%
0.970
13.2%
78.9%
0.408
10.737
16.4%
84.1%
0.942
11.3%
76.0%
0.383
10.123
15.2%
82.4%
0.926
14.2%
18.9%
0.944
Table 6: Ablations the components of HED, and all results from subject
preprocessed by subject)
subject)
Subject
10
Subject
subject (unseen)
subject
subject
Dm
Bim
1-SME (colored subject)
semantic (colored by subject)
Subject
subject
60
-40
subject (unseon)
-80
subject 2
-00
-25
25
50
25
-60
40
-20
0
20
Figure 9: Visualization of the preprocessed data and the semantic tokens.
19


Reviews:
[{"review_id": "bRoyrNHq8i", "reviewer": "Reviewer_9GCm", "summary": "I have carefully read this manuscript. The research area of this paper is fMRI-to-video reconstruction, with a focus on the challenge of cross-subject transfer (generalization to new/unseen subjects). This is an important problem, as the authors describe in the introduction: in practical applications, it is impossible to collect large amounts of data from a single subject for training decoding models. Although the authors emphasize the cross-subject issue, only about one-third of the proposed methods (Section 3.2) are actually related to this problem, with the majority focusing on designing fMRI model architectures that better align with human visual mechanisms and on improving representation learning methods.\n\nFurthermore, I believe the paper lacks certain methodological and evaluation details regarding cross-subject transfer; the authors do not explain how the model adapts when new subjects are introduced. Putting these issues aside, I think the contribution of the methods proposed in this paper to cross-subject visual decoding is still limited.", "questions": "1. In Equation 2, does 𝐹 refer to the fMRI voxels or the result obtained through the diffusion prior? According to the equation, it seems to be the former, but according to Figure 3, it appears to be the latter.\n\n2. Equation 5 is puzzling—why is the InfoNCE loss computed using fMRI representations from two different subjects within the batch? How is the InfoNCE loss computed with only two samples?\n\n3. What do 𝑧 and 𝑦 represent in Equation 6?\n\n4. The authors state in line 101 that no training using new-subject data is needed. Does this mean the new subject’s test data is fed directly into the model? If so, where does the subject-specific extended token T_{subj} come from?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ This paper provides a good overview of the work on fMRI-to-video reconstruction.\n\n+ This paper focuses on an important issue.", "weaknesses": "1. The abstract and the first two paragraphs of the introduction strongly emphasize the issue of transfer/generalization to unseen subjects. However, in the third paragraph of the introduction (lines 73–94), the methods described by the authors have no relation to this issue. This creates a jarring shift in the writing. Similarly, after reading the Introduction, I think the authors should focus on how to achieve transfer to unseen subjects. Yet, in the methods section, only one design is actually related to this issue. Therefore, I believe the introduction of the manuscript requires substantial revision and should not solely emphasize the transfer to unseen subjects.\n\n2. I am fairly familiar with the related work. Essentially, the token extension method proposed by the authors in Section 3.2 assigns subject-specific tokens to each participant. This approach has actually appeared previously in fMRI-to-Image studies [1]. However, it should be acknowledged that in the fMRI-to-Video task, the authors may indeed be the first to apply this method.\n\n3. I believe that some of the methodological and experimental descriptions in this manuscript are not sufficiently clear, and I could not understand how the model adapts when a new subject is introduced; for details, please refer to the Questions section.\n\n4. The citation format in the main text of this manuscript needs to be corrected.\n\n5. According to Figure 5, the video reconstruction results are not very satisfactory.\n\n[1] Zhou et al. CLIP-MUSED: CLIP-guided multi-subject visual neural information semantic decoding. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5Xk576J8n", "reviewer": "Reviewer_HxmM", "summary": "This paper introduces Visual Cortex Flow Architecture (VCFLOW), which is a subject-agnostic framework for reconstructing videos from fMRI.  One key idea of the pipeline is to extract hierarchical features from brain regions which are the early visual areas, ventral and dorsal stream areas and  align them with different types of visual representations to capture complementary information such as low-level, semantic and motion cues. Besides, the authors design a subject-agnostic model by proposing a module called SARA that projects subject-specific information into a shared space.", "questions": "- Could the authors explain more clearly evaluation metrics ? N-way top k is cited for frame-level metrics but it seems to also be used for semantic accuracy in video metrics \n- Could you add an ablation on the different SARA/HED losses displayed in line 267 and line 298, on the effect of adjusting the loss coefficients in line 299\n- Could you give more details on what is the image pretraining phase ? To which representations do they align brain features ? ( as i guess there is no motion representations for instance)\n- Would be nice to compare to other baselines such as the recent work ANIMATE YOUR THOUGHTS: RECONSTRUCTION OF  DYNAMIC NATURAL VISION FROM HUMAN BRAIN ACTIVITY (ICLR25)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written and mostly easy to follow (i.e up to the complexity of the presented method)\n- The motivation for subject-agnostic model is clearly explained and detailed.\n- Most fMRI-to-video decoding methods are subject-specific which is a significant bottleneck for clinical applications while this method addresses this fundamental challenge, making it a first important step towards better practicality.", "weaknesses": "- The method seems quite complex with many components while the ablation seems quite incomplete and doesn’t ensure all the elements are crucial to the final performance.\n- Missing information on how they generate the final videos in less than 10 seconds (claim in abstract). Could they be more specific into which generative model they use and exactly which brain-predicted embeddings they give to it (as only stable diffusion is mentioned in figure 3 but without further details, maybe explained in fig 4 but didn’t really understand... )?\n- Difficult to draw parallels between the letters/concepts introduced in method section 3 and drawing in figure 3. Could you represent on fig 3 at which stage E_early, E_ventral, E_dorsal and F_brain, F_ventral, T_sem, T_subj are present in the figure for instance ?\n- The authors claim ‘A key innovation of our approach lies in the utilization of CLIP embeddings from multiple layers to achieve fine-grained semantic alignment with fMRI signals’ but it has already been done in previous brain-to-image papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fVxsukMTfZ", "reviewer": "Reviewer_y215", "summary": "This paper proposes the Visual Cortex Flow Architecture (VCFLOW), a hierarchical brain decoding framework capable of learning multi-dimensional representations from different brain regions (early visual cortex, ventral, and dorsal streams). It also introduces a feature-level contrastive learning strategy to help the model generalize to unseen subjects. The article achieves this by sacrificing a certain degree of accuracy, enabling the model to perform well on unseen test subjects.", "questions": "Please see Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the article is very good. I completely agree that the brain decoding model should further focus on decoding for multiple individuals, especially for unseen subjects. Because we cannot expect users to undergo prolonged collection processes in practical use. Generalization among the subjects is a problem that must be addressed.\n\n2. The article is well-written and clearly expressed. The experiment was also conducted thoroughly.\n\n3. The model not only designs multiple different modules to extract features at different levels for alignment, but also adopts multiple tasks to conduct a more comprehensive alignment.", "weaknesses": "1. I agree that generalization to unseen subjects is very important. However, I do not fully agree that the reliability of the method can be demonstrated in the video reconstruction task. As is well known, current brain video reconstruction largely depends on the pre-trained Diffusion model to achieve its effects. Then, we calculate the indicators based on the generated videos. I believe the results are heavily influenced by the generation model rather than actual brain decoding. (In other words, brain decoding might only produce very high-dimensional and not detailed semantic information, which the diffusion model then uses to generate the video. In fact, the results shown in Figure 5 also reflect this point. Current brain decoding is likely to have provided only very vague information. We evaluate everything based on the generated video, and there are too many factors in between that can influence the outcome.)\n\n2. Based on the article's indicators, the improvement is minimal. In the 50-way scenario, it only increased by 2% to 4% compared to other pre-training methods, resulting in just 2 to 4 more images correctly identified out of 100 samples. Additionally, the pixel-level metrics are very low, so they offer limited comparative value. Even a completely black image can reach a PSNR of around 10. \n\n3. The study had too few subjects. With only three participants involved, it is very difficult to rule out coincidental factors and establish a clear pattern. The works Clip-Mused [1] and TGBD [2] focus on retrieving image/video frames based on brain signals for multiple subjects. Particularly, TGBD is also designed for unseen subjects, utilizing the HCP[3] dataset (with over 170 participants). The main conclusion is that as the number of participants increased, the generalization ability on unseen participants significantly improved. I think TGBD could provide inspiration for this work and should be included by related works, as one of the few studies that focused on the unseen subjects. Maybe the author's method is limited by the number of participants. When the number of participants is sufficient, a much greater performance gap might be achieved.\n\n\n[1] CLIP-MUSED: CLIP-guided multi-subject visual neural information semantic decoding, ICLR2024\n\n[2] Toward Generalizing Visual Brain Decoding to Unseen Subjects, ICLR2025\n\n[3] The WU-Minn Human Connectome Project: An Overview. Neuroimage, 2013.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykPGhYBxQa", "reviewer": "Reviewer_ebCw", "summary": "This paper proposes VCFLOW, a cognitive process–inspired architecture for cross-subject fmri-to-video decoding. By disentangling early visual, ventral, and dorsal pathways and introducing a SARA module, the method can effectively generalizes to unseen subjects without any finetuning.", "questions": "1. How exactly are the video processed and aligned with fMRI signals? More details should be clarified.\n2. In Figure 3, what exactly is the “Brain Model” shown in the framework? Additionally, what are the input dimension of the early, ventral, and dorsal features, and what is the dimension of the shared projection space? It would be helpful if these details could be included in the appendix for clarity and reproducibility.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper conducts comprehensive ablation studies, and the overall framework design is well visualized.\n2. Compared with NEURONS, the proposed SARA module and fine-grained hierarchical alignment mechanism achieve good performance improvements in the cross-subject setting.", "weaknesses": "1. The dataset used only contains three subjects, which weakens the persuasiveness of the cross-subject evaluation. Future work could validate on datasets with larger subject pools, such as CineBrain[1].\n2. The study only reports results in the cross-subject setting. Given that the overall performance on the fMRI-to-video task is still relatively limited, including single-subject results would better show whether the proposed architecture truly enhances decoding quality rather than merely improving transfer robustness.\n3. The claimed 10s inference time appears only in the introduction and lack specific  configuration. The authors should clearly state the experimental conditions and provide comparative results with other methods.\n4. Section 3.3 introduces multiple auxiliary tasks, but the paper lacks detailed explanations of how these are constructed and how multi-feature fusion is handled during inference.\n5. In terms of neuroscientific interpretation, VCFLOW relies heavily on strong anatomical priors (early/ventral/dorsal separation). Thus, the model validates existing neuroscientific hypotheses rather than discovering such structure in a data-driven way, which limits novelty compared to NEURONS.\n\n[1] CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)