MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Authors: Yuqin Dai
Zhouheng Yao
Chunfeng Song
Qihao Zheng
Weijian Mai
Kunyu Peng
Shuai Lu
Wanli Ouyang
Jian Yang
Jiamin Wu
Venue: ICML 2025 poster (2025)
Keywords: Brain Decoding
Functional Alignment
Cross-subject Decoding
Neuroscience
Neuroimaging
Visual Perception
Primary area: applications->neuroscience_cognitive_science
TL;DR: We propose MindAligner, an explicit functional alignment framework for cross-subject brain decoding with limited fMRI data.

Abstract: Brain decoding aims to reconstruct visual perception of human subject from fMRI signals, which is crucial for understanding brain's perception mechanisms.  Existing methods are confined to the single-subject paradigm due to substantial brain variability, which leads to weak generalization across individuals and incurs high training costs, exacerbated by limited availability of fMRI data. To address these challenges, we propose MindAligner, an explicit functional alignment framework for cross-subject brain decoding from limited fMRI data. The proposed MindAligner enjoys several merits. First, we learn a Brain Transfer Matrix (BTM) that projects the brain signals of an arbitrary new subject to one of the known subjects, enabling seamless use of pre-trained decoding models. Second, to facilitate reliable BTM learning, a Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli with a multi-level brain alignment loss, uncovering fine-grained functional correspondences with high interpretability. Experiments indicate that MindAligner not only outperforms existing methods in visual decoding under data-limited conditions, but also provides valuable neuroscience insights in cross-subject functional analysis. The code will be made publicly available.

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Yuqin 2
Zhouheng Yao"!
Chunfeng Song'
Qihao Zheng'
Weijian Mai
Kunyu Peng
Shuai Lu² Wanli Ouyang
Jian Yang? Jiamin Wuf1.3
https://githes cow htt.t//i....o//aaaagggg/////aaaa htwpaaaiiiaaaaiaieeeee....e
Abstract
demic significance (Naselaris et al., 2011). To investigate the brain's perceptual mechanisms, functional magnetic res- onance imaging (fMRI) (Naselaris et al., 201 1) has been widely used due to its noninvasive acquisition and precise localization of the functional regions. The advances fMRI facilitate the research brain visual decoding, which aims to recover visual stimuli seen by humans from their brain activity, contributing to the progress of cognitive science research and Brain-Computer Interfaces (BCI) (Qian et al., 2020; Horikawa & Kamitani, 2017)
Brain decoding aims to reconstruct visual percep- tion of human subject from fMRI signals, which is crucial for understanding brain's perception mechanisms. Existing methods are confined the single-subject paradigm due to substantial brain variability, which leads to weak general- ization across individuals and incurs high train- ing costs, exacerbated by limited availability of fMRI data. To address these challenges, we pro- pose MindAligner, an explicit functional align- ment framework for cross-subject brain decod- ing from limited fMRI data. The proposed Min- dAligner enjoys several merits. First, we learn Brain Transfer Matrix (BTM) that projects the brain signals of an arbitrary new subject to one of the known subjects, enabling seamless use of pre- trained decoding models. Second, to facilitate reli- able BTM learning, a Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli with multi-level brain alignment loss, uncovering fine- ineegraiied functional correspondences with high interpretability. Experiments indicate that Min- dAligner not only outperforms existing methods in visual decoding under data-limited conditions, but also provides valuable neuroscience insights in cross- functional analysis.
Despite the success fMRI-based visual decoding, the ma- jority methods (Seeliger et al., 2018; Lu al., 2023; Ozcelik & VanRullen, 2023) are confined the less practical single- subject paradigm, where customized decoding model is trained for each person subject. Due to substantial brain differences among subjects, the decoding model trained on one subject cannot be effectively transferred to others, lim- iting its practicality in BCI and clinical applications. In fact, variations in individual cognitive patterns and brain structures, result in significant fMRI differences (Naselaris et al., 2011). Moreover, the high acquisition cost of fMRI limits the data availability for new subjects. Thus, adapting brain decoding models to new subjects with limited data is crucial.
To address the cross- cross-subject issue, several methods (Scotti et al., 2024b; Li et al., 2025; Wang et al., 2024) adopt brain alignment techniques in implicit manner. They align IMRI signals from different subjects to latent space that is assumed to capture common cognition patterns across subjects by learning subject-specift parameters. However, this implicit alignment approach has two limitations. (1) In- sufficient brain alignment: learning a shared space that effectively aligns all subjects remains challenging due to noisy and limited IMRI data. As individuals have vast brain differences even when viewing identical stimuli, enforcing all subjects to be simultaneously aligned to a single latent space is prone to suboptimal alignment and compromised representation. Even with extensive multi-subject fMRI training, the shared latent space shows limited generalizabil- ity to unseen subjects. For instance, Unibrain (Lei et al., 2023) shows 50% performance drop when transferring the
1. Introduction
The brain serves as the center of human cognition and un- raveling its underlying mechanisms holds profound aca-
"Equal contribution. 'Shanghai Artificial Intelligence Laboratory "Najjing University of Science and Technology 'The Chinese University of Hong Kong "Tsinghua Univer- sity 'Karlsruher Institut für Technologie. Correspondence to: Jian Yang <cs rcsiyangenjuste edu. en>, Jiamin Wu 3jlaminwu@cuhte edu. hk>.
Proceedings of the 42nd International Conference on Ma- chine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Implicit Functional Aligument Shared Latent Space Compromise
Baselins
Ground Truth
Decoding Model
Embedding
Explicit Functional Alignment
MindAligner
Decoding
Model
Our
Brain Functional Analysis
Novel Subject
Known Subject
Fumetional Atlas
Figure 1. Different approaches to functional alignment in brain decoding: Prior works (Scotti et al., 2024b; Li al., 2025) adopt implicit alignment approach that aligns all subjects into single latent space, which may lead to suboptimal alignment. Differently, MindAligner employs an explicit alignment strategy, mapping novel subject signals to seen ones by establishing fine-grained functional correspondences. MindAligner not only enables high-quality visual reconstruction from fMRI signals but also facilitates brain functional analysis across subjects.
decoding model to new subjects. (2) Lack of functional interpretability: existing latent alignment methods (Scotti et al., 2024b; Wang et al., 2024; Li et al., 2025) fail to ex- plicitly account for cognitive pattern relationships between subjects. Their alignment is incapable of localizing the brain regions for functional differences and commonalities. This lack of interpretability not only limits subject knowl- edge transfer in new subject adaptation, but also hinders analysis of neural functional mechanisms underlying human perception process. Given these limitations, an important question arises: can we create brain alignment frame- work enabling effective new-subject adapiation and brain functional analysis under data scarcity?
one of the known subjects in the voxel-level. To overcome the lack of strictly paired fMRI signals, we propose Brain Functional Alignment module (BFA) to perform soft cross- subject alignment between fMRI signals from different but similar visual stimuli, facilitating the mapping of function- ally equivalent cortical areas. Specifically, BFA first de- composes the brain transfer matrix into two low-rank linear layers, enhancing parameter efficiency to facilitate effective adaptation with limited data. In the latent space of the BTM, cross-stimulus neural mapper is designed to transform the fMRI under different visual stimuli, with stimulus dif- ferences as mapping condition. To achieve sufficient and fine ane--graied aligmment, we design a multi-level brain align- ment losses that incorporates a signal-level reconstruction loss and latent alignment loss guided by visual seman- tic similarities. In this way, the resulting brain transfer matrix not only facilitates fine-grained alignment without shared stimuli constraint, but also encodes cross-subject brain relations for enhanced functional interpretability and neuroscience analysis.
To answer this question, our motivation is to establish an explicit brain functional alignment framework that maps the novel subject's 's signal to seen subject's s signal. Given an arbitrary new subject, our explicit alignment can estab- lish fine- ine-inaieed functional correspondences between the new subject and seen subjects in the original brain space, as shown in Fig. 1. The aligned fMRI signal can not only seamlessly be integrated into the pre-trained decoding model of seen subjects but also reveals brain region-level cross- subject variability. However, achieving such brain align- ment is challenging, it requires paired IMRI from sub- jects performing the same task (i.e., viewing identical visual stimuli (Bazeille et al., 2021)), a condition not met by the existing dataset (Allen et al., 2022).
In summary, our contributions are as follows:
We propose Mind/Aligner, the first explicit brain align- ment framework that enables cross-subject visual de- coding and brain functional analysis in the lata-Iimited setting.
We propose a brain transfer matrix to establish fine- grained functional correspondences between arbitrary subjects. This matrix optimized through a brain func- tional alignment module, which employs multi-level alignment loss to enable soft cross-subject mapping.
Based on the above observations, we propose Mind Aligner, an explicit functional alignment framework for cross-subject visual decoding with limited fMRI data. The core of our method is to train a cross-subject Brain Transfer Matrix (BTM) that projects the brain signals of a new subject to
Experiments demonstrate that MindAligner outper-
2

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
forms state-of-the-art methods in visual decoding with only 6% of the whole model's parameters learned.
signal to that of another, thereby preserving functionality and facilitating knowledge transfer across subjects. How- ever, such ideal large-scale shared stimuli are often absent. To better adapt to these scenarios, current methods focus on either anatomical alignment (Bao et al., 2025; Jiang et al,, 2024; Shen et al., 2024) or functional alignment in latent space (Scotti et al., 2024a:b; Wang et al., 2024; Gong et al., 2025). MindEye2 (Scotti al., 2024b) employs ridge re- gression to align different subjects into a shared latent space, followed by shared decoding module. MindBridge (Wang et al., 2024) generates pseudo stimuli to create shared stim- uli for brain alignment. However, these alignment methods either rely on shared stimuli, restricting their applicability to datasets without such conditions, or utilize latent space alignment, which impedes their ability to uncover inter- subject neural variability. We introduce an explicit brain functional alignment model to conquer the restriction of the shared stimuli and enhance the interpretation of inter-subject neural variability.
We conduct cross-subject brain functional visualization and discover that the early visual cortex shows similar activities across subjects, while the higher visual cor- tex related to memory and spatial navigation exhibits significant inter-subject variability.
2. Related Work
2.1. fMRI-Based Brain Decoding
Brain decoding seeks to reconstruct the visual stimuli per- ceived by subjects based on their brain activity, offering a deeper understanding of the brain's mechanisms for pro- cessing external information (Naselaris et al., 201 I). Farlier work (Horikawa & Kamitani, 2017) reveals a correlation between Deep Neural Networks (DNNs) image representa- tions and neural activity in the visual cortex using sparse lin- ear regression. With the advent of generative models (Good- fellow et al., 2020; Ho al., 2020) and extensive fMRI datasets (Allen et al., 2022), visual decoding has shifted towards mapping brain signals to the latent spaces of large models, facilitating the reconstruction of diverse visual stim- uli (Gu et al., 2022; Ozcelik et al., 2022; Gao et al., 2024a; Zhou et al., 2024; Mai et al., 2024; Gao et al., 2024b). This approach has proven effective in utilizing latent diffusion models for image reconstruction (Lin et al., 2022: Takagi & Nishimoto, 2023; Mai & Zhang, 2023; Scotti et al., 2024a; Chen et al., 2023), addressing inter-subject differences by either training separate models for individual subjects or employing partially unified models with subject-specific pa- rameters. However, influenced by neural variability, cross- subject brain signals in the latent space are prone to seman- tic confficts, which can lead to convergence suboptimal points. MindAliigner addresses this by leveraging an plicit functional alignment framework across brains, this approach more effectively utilizes shared functionalities among subjects, thereby mitigating semantic conflicts.
3. Preliminary
We begin with the illustration of the problem definition, and preliminary on cross-subject brain decoding baseline that reconstructs visual stimuli in the data-limited setting.
Problem Definition. The acquisition of fMRI data is both time-intensive and costly, leading to brain decoding scenar- ios frequently constrained by limited data. Therefore, this study focuses on investigating cross-subject brain decod- ing in a data-limited setting. We follow MindEye2 (Scotti et al., 2024b) to build this setting the Natural Scenes Dataset (NSD) (Allen et al., 2022). Specifically, the de- coding model is first pre-trained for one or several subjects SK using their full 40 scanning sessions of fMRI signals. Subsequently, the pre-trained decoding model is transferred to a new subject SN using only a single session of scanned IMRI (approximately 1 hour of data, representing just 2.5% of the full dataset). Finally, the adapted model is tested on 1000 images shared across all subjects for subject SN.
Cross-subject Brain Decoding Baseline. Here we intro- duce our cross-subject decoding baseline method (Scotti et al., 2024b). To reduce inter-subject differences, the base- line model first employs linear layers to map brain signals from different subjects into shared latent space. Then the fMRI embeddings are aligned with the latent space of CLIP model (Radford et al., 2021) through a diffusion prior (Ramesh et al., 2022), thereby leveraging generative models' capabilities for visual reconstruction. The output embeddings are then fed through a low-level submodule and a retrieval submodule. Two corresponding losses are utilized: a low-level reconstruction loss Liow-level between the blurry images generated by the low-level submodule and the ground truth, and a bidirectional MixCo loss LBiMisCo
2.2. Cross-Subject Functional Alignment
As brains differ both in size and processing mechanisms (Allen et al., 2022: Finn et al., 2017), the resulting variabil- ity in fMRI signals has spurred research into brain align- ment methods. The ideal condition for functional align- ment methods often depends on shared stimuli. requiring paired data from multiple subjects exposed to identical vi- sual inputs, with alignment achieved through reconstruc- tion loss optimization (Dadi et al., 2020; Rastegarnia et al., 2023). Relying on this ideal shared stimuli condition, new paradigm (Bazeille et al., 2019; Thual et al., 2022; 2023; Ferrante et al., 2024) has emerged, enabling explicit align- ment under shared stimuli by transforming one subject's
3

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Training
Bruin Functional Alignment
Trainable
Simmlar Images
Frozen
Stimaka Samilariy
Boan Functiond
Sinilaity
Subjan
Brain Functional Alignment
Functional Embedder
Decoding
Model
Cros--ttimllas Neural Mapper
fv
Novel Subject 5x
Inferenee
Fanctional Mapping
Lyes + Les.
Matrix
Decoding Model
E
Eiesage
Siimaaas Sauilasity
Ererge
Subject
Fx
1x
Figure 2. Overview of MindAligner. To achieve explicit brain functional alignment, given a pre-trained brain decoding model, we design Brain Functional Alignment Module (BFA) learns Brain Transfer Matrix (BTM) M fMRI mapping between known novel subjects. BTM is decomposed into two low-rank matrices A and B to create latent space for further alignment. The Cross-Stimulay Neural Mapper proposed to create fMRI pairs under shared stimuli. addition to the alignment losses Cree CKL between generated and real IMRI, latent alignment loss Liatent guides functional alignment based on stimulus similarities. In the inference stage, only BTM is utilized for functional mapping, enabling cross-subject brain decoding.
to perform contrastive optimization between the retrieval module's output and the CLIP image embeddings. The final loss for the decoding model's training formulated Q22 BiMizoo where Sprior denotes the diffusion prior loss that measures dis- crepancies between the CLIP image embedding and the outputs produced by the diffusion prior. More details can be found in (Scotti et al., 2024b).
during pre-training through a linear transformation:
F M
(1)
dee priir + Clow-lesee
where Fx k denotes the IMRI signal projected to the brain space of the seen subject Sk. To improve parameter effi- ciency in the data-limited setting, we decompose M into two low-rank matrices A and B,
M
(2)
4. MindAligner
where A E Raxh and B Raxk n and are the voxel dimensions of the novel and known subjects' fMRL, and is the hidden dimension. The matrix decomposition creates shared latent space between two subjects for subsequent alignment. M encodes transfer weights that capture region- level inter-subject brain correlations and can be utilized for functional alignment during inference. Moreover, these correlations provide valuable insights for analyzing inter- subject variability.
4.1. Overview
Building on the pre-trained brain decoding model, Min- dAligner utilizes Brain Transfer Matrix (BTM) trans- form signals from novel subjects into the signal space of known subject with limited data, thereby enabling cross- subject brain decoding. To achieve reliable and fine-grained brain alignment, we design the Brain Functional Align- ment Module (BFA). Notably, the alignment module is utilized only during the training phase assist BTM learn- ing: during the inference phase, only the lightweight BTM retained. Next, provide detailed illustration of each module of MindAligner.
4.3. Brain Functional Alignment Module
learn a reliable brain transfer matrix, the Brain Func- tional Alignment Module (BFA) conducts soft cross-subject alignment both the brain space and the shared latent space of the BTM. As strietly-paires fMRI data under iden- tical stimuli is provided, we employ cross-stimulus neu- ral mapper to facilitate stimulus transformation, rendering fMRI-pairs under visually similar stimuli. Based on these fMRI-pairs, a multi-level brain alignment loss is employed to achieve final alignment.
4.2. Brain Transfer Matrix
Given the fMRI signal FN of arbitrary novel subject Sy as input, the Brain Transfer Matrix (BTM) M aims to transform it into the fMRI signal of a subject Sx seen

---

MindAligner:
Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Table 1. Visual decoding performance comparison. 1h means
hour of data. Bold indicates the best performance.
Method
Low-Level
High-Level
Retrieval
PixCorrt
SSI
Alex(2)*
A
Incept
CLIP
Emr
SwAV.
Imager
Brain
(1 h) MindBridge
0.109
0.324
79.0%
86.1%
78.7%
82.7%
0.866
0.528
(1 h) MindEye2
0.195
0.419
84.2%
90.6%
81.2%
79.2%
0.810
0.468
79.0%
57.4%
(1 h) Ours
0.206
0.414
85.6%
91.6%
83.0%
81.2%
0.802
0.463
79.0%
75.3%
(subj1) MindBridge
0.124
0.333
80.55%
86.51%
78.72%
83.56 %
0.868
0.526
(subji) MindFye2
0.235
0.428
88.02%
93.33%
83.56%
80.75%
0.798
0.459
93.96 %
77.63%
(subj1) Ours
0.226
0.415
88.19%
93,26%
83.48%
81.76%
0.800
0.459
90.90%
86.88%
(subj2) MindBridge
0.107
0,330
78.21% %
85.85%
77.96%
81.04%
0.874
0,532
(subj2) MindEye2
0.200
0,433
85.00%
92.13%
81.86%
79.39%
0.807
0.467
90.53% %
67. 67.18%
(subj2) Ours
0.218
0.426
88.08 %
93.33%
84.13%
82.47 %
0.791
0,452
90.04%
85.61 %
(subj5) MindBridge
0.105
0.326
80.18%
87.70%
81.95%
85.38 %
0.840
0.507
(subj5) MindEye2
0.175
0.405
83.11%
91.00%
84.33%
82.53%
0.781
0.444
66.94%
46.96%
(subj5) Ours
0.197
0.409
84.69% %
91.61% %
84.63%
82.76%
0.784
0.454
70.62% %
65.95%
(subj7) MindBridge
0.101
0.308
76.99%
84.51
76.31%
80.98 %
0.882
0.546
(subj7) MindEye2
0.170
0.408
80.70%
85,90%
74.90%
74.29%
0.854
0.504
64.44%
37.77%
(suhj7) Ours
0.183
0.407
81.45%
88.31
79.92%
77.82%
0.834
0.487
64.18%%
62.58%
Due to the lack of stimuli- pair where two subjects view the same image, we turn to select cross-subject fMRI pairs with similar stimuli IN and Ik viewed by two subjects. The cross-stimulus neural mapper aims transform the fMRI embedding næ under stimuli In into those corresponding to stimuli I of the known subject SK- However, due to the absence of brain prior knowledge, directly generating fMRI signals is still challenging. Therefore, we leverage the differences between IN and k as conditions for generating fMRI. Based on the fMRI embedding N projected by the low-rank matrix A. i.e., NZ A x F N. we use the visual stimuli difference condition perform
Cross-stimulus Neural Mapper:
Multi-level Brain Alignment Loss. The brain alignment loss integrates both signal-Level reconstruction loss between the generated and real fMRI signals and an embedding-level alignment loss to achieve more refined alignment across different visual stimuli. To ensure the quality of synthesized fMRI F an fMRI reconstruction loss is designed to force the consistency between F k and the real fMRI F k of the known subject:
Cree 1| FxlE
(7)
To further enhance the alignment performance, we use the Kullback-Leible (KL) Divergence loss to enforce the con- sistency between distributions of the generated and real fMRI signals:
Editt viewed by the two subjects linear modulation to generate the stimuli embedding = corresponding to the stimuli x of the known subject:
=KL=KL(FK K Fx).
(8)
Edit = Eimage ( xN) - Eimage (I));
(3) To enable fine-grained functional mapping under different stimuli, we leverage intrinsic correlations between visual semantics to guide the alignment of the corresponding brain activities in the latent space. Specifically, we design a la- tent alignment loss Liatent by enforcing the consistency between IMRI embedding pairs and stimuli pairs:
Jdiff = Eait x Mair.
(4)
zK= = MM(ZN. zair);
(5)
where Emaage is the image encoder of pretrained CLIP (Rad- ford et al., 2021). The cross-stimulus neural mapper (c) is a linear modulation that splits the condition Zair to scale and shift parameters using Man E _ax2h, a is the clip embedding's dimension. These parameters can be used to modulate the input n,. thereby facilitating the cross- subject stimulus transformation in the latent space. The transformed embedding æ is then projected the known subject's space by the low-rank matrix B, rendering syn- thesized fMRI embedding Fx k to be aligned with the known subject's 's real fMRI embedding F k:
Ciatent [(N((E(ZN),, E - R(EN, Ex 1I), (9)
where EN and Ex denote the CLIP embeddings of IN and K R(-) calculates the dissimilarity matrix between em- bedding pairs. & denotes functional embedder for fMRI embeddings for better dissimilarity calculation. Hence, the final brain alignment loss Latign is formulated as:
Calign Ldec aak totaatiitnn (10)
where O'yen; Orkt, Gia are the loss coefficients, and Lace de- notes the decoding loss in the baseline method. The com- bination of these losses can improve the semantic accuracy of visual reconstruction and also reduce the reliance on same-stimulus data.
(6) The effectiveness of linear structures for cross-sujject align- ment is justified by both theoretical and empirical evidence (Appendix F).
FK = K x B.
5

---

MindAligner:
Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
4.4. Inference
to evaluate low-level properties and use Inception (Szegedy et al., 2016); CLIP (Radford et al., 2021), EffNet-B (Tan & Le, 2019) and SwAV (Caron et al., 2020) to evaluate high- level properties. These metrics assess the fidelity of the reconstructed images by comparing them with the ground truth. Metric details can be found in Appendix A.
During inference. only the trained BTM is used for func- tional alignment (Eq. 1). The generated Fx is then directly fed into the pre-trained model for brain decoding. Min- dAligner is lightweight functional alignment module that enables efficient cross-subject visual decoding.
To evaluate functional alignment, we use two metries: fMRI Spatial Correlation (ISC) (Conroy et al., 2009) and Transfer Quantity (TQ). fSC measures the Pearson correlation be- tween corresponding brain regions of two subjects (i + j), assessing global alignment consistency. TQ captures voxel- level differences by analyzing the weights of the BTM M, which maps voxels between subjects. For a source voxel indexed by i, TQ is defined as TQ; = Eogjep |MMil where p the number of voxels in the target brain. High TQ values indicate regions with greater activation differences and more intricate functional alignment requirements.
5. Experiments
In this section, present the implementation details, fol- lowed by IMRI-to-image reconstruction results and brain functional alignment analysis. The Appendix includes ad- ditional metrics, qualitative and quantitative results, model efficiency, and further visualizations.
5.1. Implementation Details
The BTM is composed of two bias-free linear layers with hidden size h = 4096. The input dimension n and output dimension of BTM are determined by the specific subject transfer pairs. For subjects 1, 2. 5, and 7. the dimensions are 15,724, 14,278, 13,039, and 12,682, respectively. The cross- stimulus neural mapper is implemented using the Feature- wise Linear Modulation model (Perez et al., 2018), where the input dimension of Mann is a = 768, matching the CLIP emhedding dimension, and the output is 2h = 8192. The functional embedder is a linear layer with input and output sizes of = 4096. The loss coefficients are set aia aKL 0.001, 01 = 0.033, and 02 0.016. The learning rates for the brain transfer matrix, cross-stimulus neural mapper, and functional embedder are all set to 1e-5. We use a batch size of 16 and optimize using Adam. Training on a single NVIDIA A A100 GPU achieves convergence in approximately 12 hours.
5.4. IMRI-based Visual Decoding
We evaluate the visual decoding performance of Min- dAligner in both qualitative and quantitative manners. The compared state-of-the-arr methods include our baseline MindEye2 (Scotti et al., 2024b) and MindBridge (Wang et al., 2024).
Qualitative Comparison. We train our model using data from a single session (1 hour of data) and visualize the re- sults in Fig. 3. MindAligner delivers decoding results that are more consistent with the original visual stimuli seman- ties compared to the baseline, highlighting its effectiveness. The performance improvement can be attributed to the effec- tive learning of brain transfer matrix that accurately aligns the novel subject with the known subject, thus well transfer- ring the pre-trained decoding model the new subject with limited data.
to Gree
5.2. Dataset
Quantitative Comparison. The first row (Ih) of Tab. I represents the average of the results from the 2nd to the 5th rows, providing a fair comparison with other methods to demonstrate our superiority. "(subjl) Ours" refers to the average result obtained by aligning the novel subject (subj 1) every subject in the known subject list (subj 2, 5, 7). Detailed results are provided in Appendix Tab. 7. As sum- marized in Tab. 1, MindAligner surpasses ofte-offthhe-at methods across almost all metrics. Notably, it achieves a 17.9% improvement in brain retrieval performance. The observed improvement can be attributed to the inherent chal- lenges in implicit alignment strategies employed in previous methods. Aligning multiple subjects with substantial indi- vidual differences remains complex task that may result in information loss during the alignment process. Our method addresses this challenge by adopting an explicit alignment strategy that aligns one subject at a time, avoid conflicts arising from multi- subject alignment. Our model focuses on
We use the Natural Scenes Dataset (NSD) (Allen et al., 2022), the largest publicly available set, widely used for brain visual decoding. It includes neural responses from subjects viewing complex images from the MSCOCO-2017 dataset(Lin et al., 2014). In line with MindEye2's data- limited setting, our approach uses only a single session of neural recordings, corresponding to one hour of data.
5.3. Metrics
To evaluate fMRIIto-image reconstruction performance, we assess both low- and high-level properties of the recon- structed images. Low-level properties capture fundamental visual elements like pixel similarity and edges, while high- level properties reflect semantic information. Following pre- vious works (Scotti et al., 2024a;b), we adopt the PixCorr, SSIM, AlexNet(2),, and A5exNet(5) (Krizhevsky al., 2012)
6

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Stimilus
MindEyee2
Ours
3GU
Figure 3. Visualization of MindAligner's decoding results from training one hour of data.
Stimulus
5-42
5-
subi2
subji
subj$
subii
subj2
Subjs
Figure 5. Visualization of transfer quantity brain heatmaps.
0.42
MindEye2
MindAlignar
Figure Visualization results of aligning subject with dif- ferent known subjects.
0.38
0.36
0.32
fine-grained cross-subject brain mapping, thereby achieving better decoding performance with high fidelity.
0.30
0.26
Ablation Study. To evaluate the effectiveness of each model design in MindAligner, perform ablation study ing Subject 2 as the novel subject and Subject 1 as the known subject. The results exclude the refinement step of MindEye2 for generated images. As shown in Tab. 2, training MindAligner with only the visual decoding loss Caee yields suboptimal cross-subject reconstruction perfor- mance, underscoring the difficulty of directly generalizing pre-trained models to subjects without effective align- ment. Adding signal reconstruction loss Cree significantly enhances performance as it leads to more accurate brain activity reconstructions. The incorporation of C Kr further strengthens alignment by enforcing consistency between the distributions of the generated and real signals. Lastly,
Figure 6. Comparison of fSC results between MindAligner and the baseline.
Ltatent exploits the correlation visual stimuli and fMRI embeddings to guide the brain alignment the latent space, thereby improving model's ability to capture visual seman- ties in brain activity and enhancing low-level reconstruction performance. These losses together work in synergy to re- fine alignment and improve cross-subject decoding fidelity.
Impact of Aligning to Different Subjects. We visualize the results fixing new subject and aligning with different known subjects in Fig. 4. The reconstruction performance

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Table Ablation study results. The combination CreeetCreetCeee CKL+ Ctatent is our final model setting.
Method
PixCorr
SSIM
Alex(2)1
Alex(5)t
Incept
CLIPt
E.T.
SwAV
0.072
0.318
63.50%
71.44%
66.07%
62.59%
0,905
0.550
+ +Loecthrrec
0,186
0.340
86.83%
93.51%
84,55%
82.42%
0.811
0,465
+ Ldcct C
0.191
0,407
87.98%
92.99%
86.61 %
82.16%
0.780
0.453
+ +L Creet + Latent
0.195
0.408
88.25 %
93.51 %
86.24%
82.72%
0.782
0.454
are stable when aligned with different known subjects, the generated images nearly identical. This demonstrates the robustness of our brain functional alignment. The choice of the known subject has minimal impact on visual decod- ing performance. This is because MindAligner leverages the intrinsic correlation between visual semantics to guide the alignment of corresponding brain activities, thereby fa- cilitating robust alignment performance. We provide more detailed cross-subject results in Appendix C.
suby
subyz
subj5 suby1
baur
Fill datzeet
Table 3. Efficiency comparison results. "Tr. Param." refers to the model's trainable parameters when adding a new subject.
Method
Tr. Param.
Total Param.
Inference
Figure 7. Visualization of transfer quantity in brain heatmaps from MindAligner training using hour and full datasets.
MindEye2
2.21G
2.21G
5.000 s
MindAligner
139.23M
2.21G
5,056
commonality across subjects. contrast, higher visual re- gions handle more complex cognitive processes, including categorical perception and semantic understanding. lead- ing to higher variability across individuals. 2) The ventral pathway exhibits the greatest inter-subject variability. The ventral pathway- anatomically positioned on the brain's ss ventral surface (lower section in Fig. 5) and encompassing functional regions like PPA and FFA demonstrates the highest variability among visual pathways. This variability arises from its important role in high-level visual processing, such as object recognition, face perception, and semantic interpretation. The ventral stream integrates sensory input with prior knowledge, experiences, and cognitive biases. This results in greater individual differences, as factors like familiarity, attention, and personal experiences shape how visual information is interpreted and understood.
Computational Efficiency. We compare the computational efficiency between our model and baseline MindEye2 w.r.t. parameter count and inference time per image. As shown in Tab. 3, MindAligner achieves superior decoding perfor- mance while significantly reduces the fine-tuning require- ment, with just 6% of MindEye2's s learnable parameters, demonstrating its efficiency. Moreover, the addition of our alignment module only slightly increases the inference time.
5.5. Brain Functional Alignment Analysis
To deepen the understanding of the brain functional align- ment process, we provide detailed visualizations of brain regions along with corresponding functional analysis, of- fering insights into subsett variability and underlying neuroscience mechanisms.
To assess the align- ment effect of MindAligner, we measure the IMRI Spatial Correlation (fSC) for different subject pairs, comparing our functional alignment strategy with the baseline in reducing inter-subject differences, as shown in Fig. 6. We use 1 + 2 to denote aligning Subject I to Subject 2. The results demon- strate that our method significantly outperforms the existing baseline in ISC in all transfer configurations, demonstrating the superiority of our explicit alignment manner against the implicit alignment. By establishing fine-grained voxel cor- respondences between subjects, MindAligner significantly enhances alignment performance even without paired fMRI signals under shared stimuli, leading to a better visual de- coding performance.
Cross-subject Correlation Analysis.
Region-level Functional Mapping. We apply the Trans- fer Quantity (TQ) metric on MindAligner's 's brain transfer matrix to assess cross-subject associations and visualize the results through brain heatmaps. As shown in Fig. 5, the vi- sualization results highlight two key neuroscience findings. 1) The visual system exhibits hierarchical pattern of inter-subject variability. The early visual region (labeled as "EsrlyVis" in Fig. 5) presents lower inter-subject variability while higher visual regions (including OPA, FFA, PPA, and EBA) show larger variability. This graded variability aligns with established neuroscientific principles. The early visual region processing fundamental features like lines/textures show more conserved neural mechanisms, sharing larger
8

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Brain Alignment with More Data. Furthermore, to eval- uate MindAligner's robustness in limited data scenarios, we compare its performance using only 1 hour of fMRI data (2.5% of the full dataset) to using the full scanning sessions. As shown in Fig. 7, even with limited data, the TQ distribution closely resembles that of the full dataset, effectively identifying regions with significant inter-subject variability. This highlights the robustness of our explicit brain alignment strategy under data scarcity.
optimal transport for functional brain template estimation.
In Information Processing in Medical Imaging, pp. 237- 248. Springer, 2019.
Bazeille, T., DuPre, E., Richard, H.,, Poline, J., and Thirion, В. An empirical evaluation of functional alignment using inter-subject decoding. Neurolmage, 245: 118683, 2021. doi: 10.
URL ceceence/art.icce/pp/////ssse
bttps::/www.sccienceeeeee
com/
6. Conclusion
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 9924, 2020.
We present MindAligner, a functional alignment framework for cross-subject brain visual decoding. Unlike existing methods, it addresses insufficient alignment and lack of interpretability by learning a brain transfer matrix for voxel- level correspondences and proposing brain functional alignment module for cross-subject mapping. Experiments validate the effectiveness of our method.
Chen, Z., Qing, J., Xiang, T., Yue, W., and Zhou, See- ing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710-22720 2023.
Impact Statement
Conroy, B., Singer, B., Haxby, J., and Ramadge, P. J. fmri- based inter-subject cortical alignment using functional connectivity. Advances in Neural Information Processing Systems, 22, 2009.
MindAligner enables high-quality visual perception recon- struction from single fMRI session, potentially advancing the clinical diagnosis and brain computer interface applica- tions. This approach holds significant potential for enabling alignment across diverse data formats and uncovering com- monalities in brain organization across species, such as between humans and monkeys. Moreover, it could play a pivotal role in advancing the creation of unified brain at- lases. The datasets used are publicly available, ensuring transparency and participant privacy.
Dadi, K., Varoquaux, G., Machlouzarides-SSaliic A., Gor- golewski, K. J., Wassermann, D., Thirion, B., and Men- sch, A. Fine-grain atlases of functional modes for fmri analysis. Neurolmage, 221:117122 126, 2020.
Ferrante, M., Boccato, T., Ozcelik, F., VanRullen, R., and Toschi, N. Through their eyes: multi-subject brain decod- ing with simple alignment techniques. Imaging Neuro- science, 2:1-21, 2024.
Acknowledgements
This work was done during the internship at Shanghai Ar- tificial Intelligence Laboratory. This work is supported by Shanghai Artificial Intelligence Laboratory and the NSFC under Grant Nos. U24A20330 and 62361 166670.
Finn, E. S., Scheinost, D., Finn, D. M., Shen, X., Pa- pademetris, X., and Constable, R. T. Can brain state be manipulated to emphasize individual differences in func- tional connectivity? Neurolmage, 160:140-15:: 151, 2017.
References
Gao, J., Fu, Y., Wang, Y., Qian, X., Feng, J., and Fu, Y. Mind-3d: Reconstruct high-quality 3d objects human brain. In European Conference on Computer Vision, pp. 312-329. Springer, 2024a.
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F,, Charest, I., et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116-12) 126, 2022.
Gao, J.. Fu, Y., Wang, Y., Qian, X., Feng, J., and Fu, Y. fmri- 3d: A comprehensive dataset for enhancing fmri-based 3d reconstruction. arXiv preprint arXiv:2409. 1315, 2024b.
Bao, G., Zhang, Q., Gong, Z., Zhou, J., Fan, W. Yi, K., Naseem, U., Hu, L.., and Miao, D. Wills aligner: Multi- subject collaborative brain visual decoding. In Proceed- ings of the AAA/ Conference on Artificial Intelligence, pp. 11194-14202, 2025.
Gong, Z., Zhang, Q., Bao, G., Zhu, L.,, Xu, R., Liu, K., Hu, L., and Miao, D. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. Proceedings of the AAAI Conference on Artificial In- telligence, pp. 14247-14255 2025.
Bazeille, T., Richard, H., Janati, H., and Thirion, B. Local

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(11))139- 134, 2020.
Proceedings of the ACM International Conference on Multimedia, pp. 5899-5908, 2023.
Mai, W. and Zhang. Z. Unibrain: Unify image reconstruc- tion and captioning all in one diffusion model from human brain activity. arXiv preprint TrXiv:2308.074288. 2023.
Gu, Z., Jamison, K., Kuceyeski, A., and Sabuncu, M. Decoding natural image stimuli from fmri data with surface-based convolutional network. arXiv preprint arXiv:2212.022009 2022.
Mai, W., Zhang, J., Fang, P., and Zhang, Brain- conditional multimodal synthesis: A survey and taxon- omy. IEEE Transactions Artificial Intelligence, 2024.
Haxby, V., Guntupalli, J. S., Connolly, A. C., Halchenko, Y. O., Conroy, B. R., Gobbini, M. I., Hanke, M., and Ramadge, P. J. A common, high- nigh-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):404-41,, 2011.
Naselaris, T., Kay, K. N., Nishimoto, S., and Gallant, J. L. Encoding and decoding in fmri. Neurolmage. 56(2):400- 410, 2011.
Ozcelik, F. and VanRullen, R. Natural scene reconstruc- tion from fmri signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Ozcelik, F., Choksi, B., Mozafari, M., Reddy, L., and Van- Rullen, R. Reconstruction of perceived images from fmri patterns and semantic brain exploration using instance- conditioned gans. In International Joint Conference on Neural Networks, pp. 1-8. IEEE, 2022.
Horikawa, T. and Kamitani, Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1):15037, 2017.
Jiang, S., Meng, Z., Liu, D., Li, H., Su, F., and Zhao, Z. Mindshot: Brain decoding framework using only one image. arXiv preprint arXiv:2405. 15278 2024.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with general con- ditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
Qian, C., Sun, X., Wang, Y., Zheng, X., Wang, Y., and Pan, G. Binless kernel machine: Modeling spike train transformation for cognitive neural prostheses. Neural Computation, 12(10):1863-1990 2020.
Lei, J., Dai. L., Jiang, H., Wu, C., Zhang, X.. Zhang, Y., Yao, J., Xie, W., Zhang, Y., Li, Y., et al. Unibrain: Uni- versal brain mri diagnosis with hicrarchical knowledge- enhanced pre-training arXiv preprint 2XXv:2309900622 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Proceedings of the Interna- tional Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Li, C., Qian, X., Wang, Y., Huo, J., Xue, X., Fu, Y., and Feng. J. Enhancing cross-subject fmri-to-video decod- ing with global-local functional alignment. In European Conference on Computer Vision, pp. 353-369. Springer, 2025.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C.,, and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06122 1(2):3, 2022.
Lin, S., Sprague, T., and Singh, A. K. Mind reader: Re- constructing complex images from brain activities. Ad- vances in Neural Information Processing Systems, 35: 29624-29636, 2022.
Rastegarnia, S., St-Laurent, M., DuPre, E., Pinsard, B., and Bellec, P. Brain decoding of the human connectome project tasks in a dense individual fmri dataset. Neurolm- age, 283: 223:15 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., Verlinde, N., Yundler, E., Weisberg, D., Norman, K., et al. Reconstructing the mind's d's eye: fmri- to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024a.
Lu, Y., Du, C., Zhou, Q. Wang, D., and He, H. Mind- diffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. In
10

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Scotti, P. S., Tripathy, M., Torrico, C., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K. A., and Abraham, T. M. Mindeye2: Shared- subject models enable fmri-to-image with hour of data. In Proceedings of the International Conference on Ma- chine Learning, volume 235 PMLR, pp. 44038-44059, 2024b.
Zhou, Y., Liu, L.,, and Gou, C. Learning from observer gaze: Zero-shot attention prediction oriented by human-object interaction recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28390-28400, 2024.
Seeliger, K., Güglüü, U., Ambrogioni, L.,, Güçlütürk, Y., and Van Gerven, M. Generative adversarial networks for reconstructing natural images from brain activity. Neu- rolmage, 181:775-785, 2018.
Shen, G., Zhao, D., He, X., Feng, L., Dong, Y.. Wang, J., Zhang, Q., and Zeng. Y. Neuro-vision to language: Enhancing brain seordding-bsedd visual reconstruction and language interaction. Advances in Neural Information Processing Systems, 37:98083-98110, 2024.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2818- 2826, 2016.
Takagi, Y. and Nishimoto, S. High-resolution image construction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 14453-14463, 2023.
Tan, M. and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning. pp-
6105-61 14. PMLR, 2019.
Thual, A., Tran, Q. H., Zemskova, T., Courty, N., Flamary, R., Dehaene, S., and Thirion, B. Aligning individual brains with fused unbalanced gromov wasserstein. Ad- vances in Neural Information Processing Systems, 35: 21792 21804, 2022.
Thual, A., Benchetrit, Y., Geilert, F., Rapin, J., Makarov, I., Banville, H., and King, J.-R. Aligning brain functions boosts the decoding of visual semanties in novel subjects. arXiv preprint ar 23 2.66677, 2023.
Wang, S., Liu, S., Tan, 7., and Wang. X. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 11333-11342, 2024.
Wang. Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 134)):600-612, 2004.

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
A. Explanation of Metrics
Following prior work (Scotti et al., 2024a;b), we evaluate the image reconstruction results based on eight metrics, which are categorized into low-level and high-level groups. Low-level metrics, including Pixelwise Correlation (PixCorr). Structural Similarity Index (SSIM) (Wang et al., 2004), AlexNet(2) (Alex(2)), and AlexNet(5) (Alex(5)) (Krizhevsky al., 2012), focus textural and structural details. High-level metrics- Inception (Incep) (Szegedy et al., 2016), CLIP (Radford et al., 2021), EfficcientNet-B (Eff) (Tan & Le, 2019), and SwAV-ResNet00 (SwAV) (Caron et al., 2020) assess semantic fidelity, Alcx(2), Alex(5), Incep, and CLIP metrics are derived by calculating Pearson correlation between the embeddings of the ground truth and reconstructed images, following the two-way identification framework of Ozcelik and VanRullen (Ozcelik & VanRullen, 2023). Eff and SwAV scores are based on the average distance between feature embeddings.
In addition to the aforementioned metrics, we also evaluate the model using retrieval-based metrics to quantify the fine- grained image information in the fMRI embeddings, following the methodology in MindEyee2 (Scotti et al., 2024b). Specifically, for image retrieval, each test fMRI scan is first transformed into its corresponding fMRI representation. We then compute the cosine similarity between this representation and the CLIP-derived image representations of 300 randomly selected images from the test set. Retrieval success is defined as the maximization of cosine similarity between the fMRI embedding and its ground truth CLIP embedding (top-1 I retrieval, with random chance at 1/300). To mitigate variability from random batch sampling, the evaluation is repeated 30 times per test sample. The same procedure is applicd for brain retrieval, with fMRI and image representations swapped.
B. Details on Model Parameters
Table 4. Parameter counts of different modules.
Module
Parameter
Used during inference
BTM
122,888,192
CNM
6,299,648
FE
16,781,311
MindFyee2
2,227,290,748
MindEye2.ridge.regree regression
64,405,504
SindEye2.backhomn
1,903,020,028
MindEye2.difftssion..rirr
259,865,216
Table 5. Trainable parameter share of different modules.
Module
Parameter %
Used during inference
MindEye2
100%
MindAlignnr (Ours)
6.2%
MindAligner.BIT BTM
5.2%
MindAligner. CNM
0.3%
X
MindAligner.FFF 0.7%
X
We list the parameter counts of different modules in the pipeline. MindAligner comprises the Brain Transfer Matrix (BTM), Functional Embedder (FE), and Cross-Stimulas Stimulus Neural Mapper (CNM). Tab. 4 shows the number of parameters for each module, while Tab. 5 displays the trainalble parameter share for each module, helping us understand their relative contributions to the overall model. The results show that our model has a relatively small parameter count, accounting for only 6% of the parameter size of the visual decoding model. Moreover, the introduction of the FE and CNM modules during the training phase does not significantly increase the model's parameters, contributing to only 1 %% of the total parameter count. The BTM only accounts for 5% of the parameter size of the visual decoding model.
12

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Stimulus
2
1
1-7
2-1
2-5
2-7
5-1
5
5-7
7-1
7-2
7
Figure 8. More visualization of brain decoding results under different novel and known subjects.
Table 6. Performance of MindAligner with different hidden sizes.
Hidden size
Low-Level
High-Level
Retrieval
PixCorrt
SSIM
Alex(2)t
Alex(5)1
Incep
CLIP
Emf]
SwAV]
Imaget
Brain
64
0.144
0.384
76.33%
85.34%
74.67%
73.99%
0.876
0.512
79.13%
64.00%
256
0.166
0.395
80.83%
87.81 %
76.89%
76.47%
0.858
0.498
86.67%
77.83%
512
0.185
0.405
83.85%
90.95%
80.74%
78.55%
0.839
0.481
89.06%
82.97%
1024
0.204
0.415
87.01%
93.30%
83.51%
80.40%
0.811
0.463
90.30%
86.19%
2048
0.215
0.422
88.30%
93.30%
83.94%
82.75%
0.798
0.458
90.16%
85.96%
4096
0.218
0.425
88.36%
93.55%
84.17%
82.57%
0.794
0.455
90.09%
86. 86%19%
C. More Results of Aligning to Different Subjects
We visualized more results of fixing new subject and aligning it with different known subjects in Fig. 8. MindAligner demonstrates robustness, as the generated images remain nearly identical when the novel subject fixed. This is because MindAligner combines fMRI reconstruction between generated and real data under similar stimuli to ensure result fidelity, while also utilizing intrinsic correlations in visual semantics to guide alignment of corresponding brain activities, enabling robust optimization.
D. Ablation Study on Hidden Size
To investigate the potential for further reducing the model size, we adjusted the hidden size and evaluated the model's s performance at different values. The experiments showed that when the hidden size is set to 1024, the model delivers comparable performance, while its size is reduced to one-quarter of the original. Compared to the = 4096 configuration, which only accounts for 6% of the framework, the d 1024 setting accounts for just 2%, further highlighting the efficiency
13

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
Table 7. Detailed performance of our model compared with the baseline. Bold means our results outperform the baseline.
Method
Low-Level
High-Level
Retrieval
PixCorrt
SSIM
Alex(2)t
Alex(5)t
Incept
CLIP
Emf;
SwAV
Imaget
Brain
(subj1) MindEye2
0.235
0,428
88.02%
93.33%
83.56%
80.75%
0.798
0.459
93.96%
77.63%
(subjl) Ours
0.226
0,415
88.19 %
93.26%
83 83.48%
81.76%
0,800
0,459
90.90%
96.88%
subjl subjl-+suj2
0.222
0,413
88.09 %
93.28%
84.01% %
81882%
0.796
0.457
91.56%
87.49%
subjl uubji
0.227
0.416
88.29 %
93.36%
83.54%
80.94%
0.803
0,461
89.76%
85.78%
subj subjl=>subj7
0.229
0.416
88.18 %
93.13%
82.90%
82.52%
0,800
0.458
91.37%
87.36%
(subj2) MindEye2
0.200
0.433
85.00%
92.13%
81.86%
79.39%
0.807
0.467
90.53%
67. 67.18%
(subj2) Ours
0.218
0.426
88.08 %
93.33%
84.13%
82.47%
0.791
0.452
90.04%
85.61
subj2-suubj1
0.218
0.425
88.36 %
93.55 %
84.17 %
82.57% %
0.794
0.455
90.09%
86.19 %
subj2=ssubj5
0.218
0.426
87.88 %
93.13%
83.39 %
82.05%
0.793
0,454
90.34%
85.67 %
subj2-+subj7
0.217
0.427
88.00 %
93.32% %
84.83%
82.78% %
0.785
0.449
89.70%
84.98%
(subj5) MindFye2
0.175
0.405
83. %
91.00%
84.33%
82.53%
0.781
0.444
66.94%
46.96%
(subj5) Ours
0.197
0.409
84.69 %
91.61 %
84.63 %
82.76%
0.784
0.454
70.62%
65.95%
subj5-+suii
0.196
0.405
84.23 %
91.28% %
84.66
82.93%
0.787
0.459
69.67
65.14%
subj5->subj
0.196
0.409
84.71 %
91.88% %
84.56 %
82.88 %
0.783
0.455
70.78 %
66.38 %
subj5-rsubj7
0.198
0,412
85.12 %
91.67 %
84,66 %
82.47%
0.781
0.450
71.41 %
66.32 %
(subj7) MindEye2
0.170
0.408
80.70%
85, 95,90%
74.90%
74.29%
0.854
0.504
64.44%
37.77%
(subj7) Ours
0.183
0.407
81.45 %
88.31 %
79.92%
77.82%
0.834
0.487
64. 18%
62.58 %
subj7->subj
0.180
0.404
80.86 %
87.47 %
78.94%
77.05%
0.840
0.492
65.62% %
63.69%
subj7
0.185
0.406
82.11 %
89.01 %
80.47%
77.53%
0.835
0.486
63.26%
61.06%
subj7 >subj5
0.183
0.411
81.38 %
88.46 %
80.36%
78.87
0.828
0.482
63.67%
62.98%
(1 h) MindEye2
0.195
0.419
84.2%
90.6%
81.2%
79.2%
0,810
0.468
79.0%
57.4%
(1 h) Ours
0.206
0.414
85.6 %
91.6% %
83.0%
81.2%
0.802
0.463
78.9%
75.3%
performance advantages of our model.
E. More Detialed MindAligner Reconstruction Performance
We provide more detailed MindAligner reconstruction evaluation results, as shown in Tab. 7. MindAligner surpasses the baseline in almost all metries, even when applying the same novel subject to different known subjects. Notably, our method achieves a 17.9% improvement in brain retrieval performance. This improvement stems from addressing the limitations of implicit alignment strategies used in prior methods. Aligning multiple subjects with significant individual differences is inherently challenging and often leads to functional information loss during the alignment process. To overcome this, our approach employs an explicit alignment strategy, aligning one subject at a time, which effectively mitigates the conflicts arising from multi-subject alignment. By focusing on region-level cross-subject brain mapping, our model not only achieves superior visual performance but also captures more comprehensive brain region features for functional representation.
F. The Linear Hypothesis Underlying MindAligner
Table 8. Comparison different methods across various metrics.
Method
PixCorrt
SSIM
Alex(2)1
Alex(5)1
Incept
CLIPt
EmT
SwAV]
FE (Transformer)
0.182
0.350
87.68%
93.55%
85.14%
81.65%
0.807
0.463
NM (Transformer)
0.169
0.339
85.40%
92.46%
84.23%
80.85%
0.819
0.482
Ours (Linear)
0.195
0.408
88.25% %
93.51%
86.24%
82.72%
0.782
0.454
The linear hypothesis in cross-subject brain difference modeling is a well-established principle in neuroscience. Haxby et al. (2011 1) demonstrated that inter-subject differences in visual representations can be climinated via linear transformations, while Naselaris et al. (201) 1) showed that linear models account for over 90% of variance in primary sensory cortex decoding. Drawing on these established linear hypotheses, MindAligner employs linear structures to model brain variations across subjects and stimuli.
14

---

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited [MRI Data
To further substantiate the validity of hypothesis, perform ablation experiments the subj2-ssubi] transfer setting by replacing the original Functional Embedder (FH) and Cross-stimulus Neural Mapper (NM) components with nonlinear architectures, specifically employing Transformer layers. This allows us to directly assess the contribution of the linear design in each module and evaluate the impact of introducing nonlinearity on model performance and generalization.
The superior performance of the linear layer further validates the linear hypothesis, demonstrating its effectiveness in modeling individual brain differences in limited data settings.
15


Reviews:
[{"review_id": "sxHnr6v9D5", "reviewer": "Reviewer_5B7Z", "summary": "This paper propose MindAligner, a explicit brain signal functional alignment framework for cross-subject brain decoding. It utilize a LoRA-based **brain transfer matrix (BTM)** to convert signals from novel subjects into signals fo a known subjects. A **brain functional alignment (BFA)** module based on linear hypothesis is designed to acomplish this. Experimental results shows superior performance compared to existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable time and insightful feedback. Your recognition is highly meaningful to us. Below is our response addressing your concerns：\n\n**Q1: Multi-level -> Bi-level.**\n\nThank you for your suggestion. We will revise it as suggested.\n\n**Q2: About the linear hypothesis in BFA.**\n\n**Literature-based Justification**: The linear hypothesis in cross-subject brain difference modeling is a well-established principle in neuroscience. Haxby et al. (2011) demonstrated that inter-subject differences in visual representations can be eliminated via linear transformations, while Naselaris et al. (2011) showed that linear models account for over 90% of variance in primary sensory cortex decoding. Drawing on these established linear hypotheses, MindAligner employs linear structures to model brain variations across subjects and stimuli.\n\n**Experimental Justification**: To further validate this, we conduct **ablation experiments on subj2->1** by replacing the Functional Embedder (FE) and Cross-stimulus Neural Mapper (NM) structures with nonlinear architectures (Transformer layer).\n\n| Method           | PixCorr↑  | SSIM↑     | Alex(2)↑   | Alex(5)↑   | Incep↑     | CLIP↑      | Eff↓      | SwAV↓     |\n| ---------------- | --------- | --------- | ---------- | ---------- | ---------- | ---------- | --------- | --------- |\n| FE (Transformer) | 0.182     | 0.350     | 87.68%     | **93.55**% | 85.14%     | 81.65%     | 0.807     | 0.463     |\n| NM (Transformer)  | 0.169     | 0.339     | 85.40%     | 92.46%     | 84.23%     | 80.85%     | 0.819     | 0.482     |\n| Ours (Linear)    | **0.195** | **0.408** | **88.25%** | 93.51%     | **86.24%** | **82.72%** | **0.782** | **0.454** |\n\nThe superior performance of the linear layer further validates the linear hypothesis, demonstrating its effectiveness in modeling individual brain differences in limited data settings.\n\n**Q3: Distance metric.**\n\nCosine similarity was employed as the distance metric.\n\n**Q4: About the improvement of Latent Alignment loss (LALoss).**\n\nTo thoroughly investigate the impact of LALoss, we conduct additional ablation experiments on subj 2→1 by only adding LALoss to the baseline visual decoding loss（DecLoss）:\n\n| Method           | PixCorr↑  | SSIM↑     | Alex(2)↑   | Alex(5)↑   | Incep↑     | CLIP↑      | Eff↓      | SwAV↓     |\n| ---------------- | --------- | --------- | ---------- | ---------- | ---------- | ---------- | --------- | --------- |\n| DecLoss          | 0.072     | 0.318     | 63.50%     | 71.44%     | 63.07%     | 62.59%     | 0.935     | 0.550     |\n| DecLoss + LALoss | **0.187** | **0.348** | **87.92%** | **92.19%** | **84.47%** | **82.78%** | **0.792** | **0.454** |\n\nThe results reveal that incorporating LALoss into the visual decoding loss DecLoss substantially enhances decoding performance. \n\n**Q5: Why is MindBridge only compared on a single entry? Additionally, why is there no visual comparison with MindBridge?**\n\nWe provide detailed MindBridge results for each subject in Tab. 1, as well as a comparison of the reconstruction results between our method and several other approaches, including MindBridge, in Fig. 1 in the anonymous link (https://mindaligner.github.io/MindAligner/).\n\n**Reference**\n\n1. Haxby et al. A common, high-dimensional model of representation in human visual cortex. PNAS 2011.\n2. Naselaris et al. Encoding and decoding in fMRI. NeuroImage 2011.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Previous cross-subject brain decoding works primarily rely on implicit alignment, whereas this work proposes explicit alignment, introducing a new perspective in the field.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "## Strengths\n\n1. The motivation is strong and well-founded, as the identified limitations of current cross-subject methods are valuable to address.\n2. The introduction of explicit soft alignment is novel and promising, particularly for handling non-common views between known and novel subjects.\n3. The interpretable functional alignment analysis is good, providing valuable insights into cross-subject variability and the underlying neural mechanisms.\n\n\n## Weaknesses\n\n1. Many technical components (e.g., linear modulation, functional embedder) rely on the linear hypothesis, yet this hypothesis is neither explicitly presented nor justified, making the paper less convincing in terms of theoretical foundation.\n3. The clarity of descriptions needs improvement, as some key aspects are missing, which hinders the reader's ability to grasp the full scope of the work. For example:\n* What distance metric is used to measure dissimilarity?\n* Why is MindBridge only compared on a single entry? Additionally, why is there no visual comparison with MindBridge?\n4. The improvement introduced by the proposed latent alignment loss is minor, based on the ablation study in Table 2.", "other_comments_or_suggestions": "The term \"multi-level\" typically implies more than two levels. If only two levels are considered, a term like \"bi-level\" might be more appropriate.", "questions_for_authors": "Please refer to the weaknesses section.\n\nIf these concerns are clearly addressed, I will consider raising my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfzF20IcmL", "reviewer": "Reviewer_iSeR", "summary": "The manuscript proposed an explicit brain functional alignment for cross-subject decoding. The method trains a cross-subject brain transfer matrix to map signals from novel subjects to the known subject. Experimental results demonstrate improved performance and provide insightful interpretations.\n\n## update after rebuttal\n\nThe authors have addressed most of my initial concerns. However, I still believe the proposed method does not demonstrate sufficient technical contributions. The core contribution appears to be learning a linear mapping to calibrate voxels between a new subject and existing subjects, which I find relatively naive. For a machine learning venue such as ICML, I expect application-specific models to incorporate nontrivial methodological or algorithmic advancements, motivated by domain-specific insights. In my view, this manuscript does not demonstrate a clear advantage in that regard. Therefore, I have decided to retain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time. We will revise the typos as suggested. **We promise to open-source our code upon acceptance.**\n\n**Q1: The settings of the Tab. 1 is unclear.**\n\nThank you for your question. The setting in Tab. 1 is as follows: \"Ours (subj 1)\" refers to the average result obtained by aligning the novel subject (subj 1) to every subject in the known subject list (subj 2, 5, 7). \nDetailed results are provided in Appendix Tab. 7, where \"1 → 2\" denotes the experiment with subj 1 as the novel subject and subj 2 as the known subject. We will clarify the setting in the revised version.\n\n**Q2: More results of MindBridge**\n\nWe provide detailed MindBridge results for each subject in Tab. 1 in the anonymous link (https://mindaligner.github.io/MindAligner/). Our method outperforms MindBridge in each subject, demonstrating the effectiveness of MindAligner.\n\n**Q3: Comparison with UniBrain**\n\n1. **Method and quantitative performance comparison**: we reimplement UniBrain using their official code and following the same setting in MindAligner in the table below. Complete results on subjects can refer to Tab. 1 in the anonymous link (https://mindaligner.github.io/MindAligner/)\n\n| Method         | PixCorr↑  | SSIM↑     | Alex(2)↑  | Alex(5)↑  | Incep↑    | CLIP↑     | Eff↓      | SwAV↓     |\n| -------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| Unibrain (avg) | 0.078     | 0.222     | 74.2%     | 82.6%     | 75.9%     | 80.1%     | 0.865     | 0.542     |\n| Ours (avg)     | **0.206** | **0.414** | **85.6%** | **91.6%** | **83.0%** | **81.2%** | **0.802** | **0.463** |\n\nThe results indicate that MindAligner demonstrates superior performance compared to Unibrain, especially in low-level metrics.\nUniBrain attempts multi-subject decoding with shared parameters by aligning semantics in the latent space. However, this implicit alignment approach may suffer from severe semantic conflicts, leading to suboptimal results.  \nIn contrast, MindAligner leverages explicit alignment, preserving semantic features while avoiding semantic conflicts, thereby achieving superior performance.  \n2. **Qualitative Comparison**:\nThe comparison of reconstruction results are available in Fig.1 on the anonymous link(https://mindaligner.github.io/MindAligner/).\nThe results indicate that compared to UniBrain, MindAligner achieves superior semantic preservation in the generated images, which align closely with the visual stimuli.\n\n**Q4: Some existing solutions that align two distributions should be considered first (e.g. optimal transport (**OT**)) and should be included as baselines.**\n\nThank you for your suggestion. We follow your suggestion to include results of using OT to fit the KL distribution of fMRI data, the results are as follows:\n\n| Method     | PixCorr↑  | SSIM↑     | Alex(2)↑  | Alex(5)↑  | Incep↑    | CLIP↑     | Eff↓      | SwAV↓     |\n| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| OT         | 0.107     | 0.201     | 77.3%     | 81.6%     | 70.5%     | 71.8%     | 0.890     | 0.534     |\n| Ours (avg) | **0.206** | **0.414** | **85.6%** | **91.6%** | **83.0%** | **81.2%** | **0.802** | **0.463** |\n\nOur model demonstrates superiority over OT method. Without shared stimulus-fMRI paris across subjects, OT methods struggle to produce effective brain alignment. MindAligner addresses this limitation through cross-stimulus mapping and multi-level alignment loss.\n\n**Q5: It would be more insightful to visualize and analyze the transferability of specific voxels.**\n\nWe visualize the transferability of the early visual cortex V2 region as a representative example. Specifically, for each voxel in the novel subject, we identify the corresponding voxel in the known subject with the highest correspondence weight in the BTM and visualize the results in Fig. 2: https://mindaligner.github.io/MindAligner/.\nDue to large cross-subject variances, the brain region exhibits location variations across novel and known subjects, and from Fig. 2, we can observe how specific region’s activity patterns map across subjects.\nInterestingly, the mapped V2 regions for novel subjects align accurately with their actual anatomical locations, e.g., visualization in \"novel subj1\" v.s. \"novel subj2 → known subj1\".\n\n**Q6: Notation issues.** We will revise them.", "claims_and_evidence": "The settings of the main table are unclear to me. Therefore, it is hard to say if the claim holds (see questions below in my review) without further clarifications from the authors.", "methods_and_evaluation_criteria": "The method makes sense and makes nontrivial contributions in terms of the explicit modeling of transferability between subjects. However, some existing solutions that align two distributions should be considered first (e.g. optimal transport) and should be included as baselines. The evaluation makes sense. However, some other baselines are still missing (e.g., [1]).\n\n[1] Wang, Zicheng, et al. \"UniBrain: A Unified Model for Cross-Subject Brain Decoding.\" arXiv preprint arXiv:2412.19487 (2024).", "theoretical_claims": "I did not find any issues regarding theoretical claims.", "experimental_designs_or_analyses": "The experiments are reasonable. The ablation study is sufficient regarding the proposed components. The analysis is interesting. However, visualizing and analyzing the transferability of specific voxels would be more interesting. For example, something like \"FFA1 shifts to the right when transferring from subject 1 to subject 2\" would be more significant.", "relation_to_broader_scientific_literature": "This could be related to cross-subjects fMRI foundation models, as well as multimodal (besides images, e.g., text) brain decoding.", "essential_references_not_discussed": "[1] works on cross-subject decoding, but is not discussed.\n\n[1] Wang, Zicheng, et al. \"UniBrain: A Unified Model for Cross-Subject Brain Decoding.\" arXiv preprint arXiv:2412.19487 (2024).", "other_strengths_and_weaknesses": "Strength:\nThe work enables cross-subject fMRI alignment with interpretations.\n\nWeakness:\nI did not find any major weaknesses other than what have been discussed elsewhere.", "other_comments_or_suggestions": "1. In section 4, I would recommend annotating the dimension of each notation (e.g. $\\mathcal{F}_N$)\n2. It is unusual to me to use a mathcal symbol as a notation for a matrix. Consider changing them to plain upper letters or referring somewhere else that uses similar notations.", "questions_for_authors": "1. The setting of the main table (table 1) is very unclear. Did you train your model on each subject individually? Or did you set each subject as the novel subject? If it is the latter, which subject did you set as the known subject? You have that information in ablation studies, but I did not find it in table 1.\n2. In table 1, why does MindBridge only appear in subject 1, but not subject 2, 5, 7?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5BxfPkNiaT", "reviewer": "Reviewer_7q81", "summary": "This work considers inter-subject alignment in the context of image reconstruction from fMRI. A light way Brain Transfer Matrix is used to align a novel subject to a known subject, which differs from the previous work that aligns subjects in a shared latent space.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We will address your concerns point by point below:\n\n**Q1: About the training parameters of MindEye2.**\n\nAccording to the open-source code of MindEye2, it performs full-parameter fine-tuning when adding a new subject. The results in Table 3 are obtained by adhering precisely to this methodology. We also evaluate the performance of MindEye2 by fine-tuning only its linear layer when adding a new subject (Finetune_ridge (avg)):\n\n| Method               | PixCorr↑  | SSIM↑     | Alex(2)↑  | Alex(5)↑  | Incep↑    | CLIP↑     | Eff↓      | SwAV↓     | Image↑    | Brain↑    |\n| -------------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| Finetune_ridge (avg) | 0.172     | 0.360     | 80.6%     | 87.8%     | 79.8%     | 78.9%     | 0.830     | 0.486     | 79.0%     | 55.2%     |\n| Ours (avg) | **0.206** | **0.414** | **85.6%** | **91.6%** | **83.0%** | **81.2%** | **0.802** | **0.463** | **79.0%** | **75.3%** |\n\nThe results demonstrate that fine-tuning only the ridge layer is still less effective compared to MindAligner.\n\n**Q2: How does this compare to LoRA fine-tuning? What are the benefits of subject-specific alignment?**\n\nCompared to LoRA, our method enjoys the following advantages:\n1. **Enhanced decoding performance**: we conduct ablation experiments to use LoRA finetuning for subj 2→1 alignment. The results show that LoRA underperforms compared to MindAligner, particularly in PixCorr and SSIM, indicating that MindAligner better preserves the low-level information of the visual stimulus.\n\n| Method               | PixCorr↑  | SSIM↑ | Alex(2)↑   | Alex(5)↑   | Incep↑     | CLIP↑      | Eff↓      | SwAV↓     |\n| -------------------- | --------- | ----- | ---------- | ---------- | ---------- | ---------- | --------- | --------- |\n| LoRA (on subj2)      | 0.175     | 0.363 | 85.75%     | 92.72%     | 85.26%     | 80.31%     | 0.812     | 0.459     |\n| Ours subj2->1        | **0.195** | 0.408 | 88.25%     | 93.51%     | 86.24%     | 82.72%     | 0.782     | 0.454     |\n| Ours subj2->1 + LoRA | 0.193     | 0.408 | **88.28%** | **93.81%** | **86.51%** | **83.11%** | **0.776** | **0.450** |\n\n2. **Neuroscience Interpretability**: MindAligner can reveal fine-grained functional correspondences that enhance neuroscientific interpretability, a capability that simply using LoRA fails to achieve.\n\n**Q3: Some results in Tab.1 are close, and a statistical test could be a better choice to support the claims.**\n\nThank you for the suggestion. We have conducted Wilcoxon signed-rank tests comparing our method with MindEye2. The results are summarized in the table below, with significant improvements (p < 0.05) highlighted in bold.\n\n| Setting           | Avg       | Subj1 | Subj2     | Subj5     | Subj7     |\n| ----------------- | --------- | ----- | --------- | --------- | --------- |\n| Ours vs. MindEye2 | **0.008** | 0.703 | **0.005** | **0.032** | **0.005** |\n\nHere, \"Subj1\" represents the average result of our method when Subj1 is used as the novel subject mapped to Subj2, 5 and 7, compared to MindEye2's results on Subj1. \"Avg\" refers to the average performance difference between our method and MindEye2 across all subjects. The results confirm statistically significant improvements in our approach in most scenarios. Updated analyses will be included in the revised version.", "claims_and_evidence": "Most claims are well-stated and supported by evidence and references. I have one concern though. \n\nIn Table 3, the Tr. Param. of MindEye2 is 2.21G (basically the total param). From my understanding, adding a new subject only requires training a linear layer in MindEye2. Am I missing something here?", "methods_and_evaluation_criteria": "The evaluation criteria make sense. But I do have a question (concern) about the method. \n\nHow does this method compare with finetuning a decoding model (say mind-eye) with LoRa using the novel subject? It would still be efficient in terms of training and avoid the trouble of finding a suitable subject to align.  What does aligning to a specific subject bring us compared with that?", "theoretical_claims": "No theoretical claims in this work.", "experimental_designs_or_analyses": "The experimental designs and analyses are reasonable, except that some results in Table 1 are very close, and a statistical test could be a better choice to support the claims.", "relation_to_broader_scientific_literature": "Cross-subject adaptation has been an important and interesting problem in computational neuroscience tasks. Cross-subject capability has been a limiting factor in many attempts to make algorithms practical in this field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper is well-written and structured. The major concerns are stated in the previous parts.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Questions are stated in the previous parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v6DIgbolm4", "reviewer": "Reviewer_Hn6R", "summary": "This paper proposes MindAligner, a framework using functional alignment to facilitate cross-subject brain decoding. Their framework consists of (1) “Brain Transfer Matrix” (BTM) that works by transforming limited novel subject brain activity into the brain activity of a known, previously seen subject via linear mappings, such that the researcher can use a pretrained model learned on someone else’s brain rather than training a new model from scratch on the new subject’s brain, and (2) a “Brain Functional Alignment” module facilitates BTM learning by estimating different alignment losses using similar images seen by both participants.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable feedback. We will incorporate the suggested modifications in the revised version.\n\n**Q1: Clarification on \"explicit\" and \"implicit\" functional alignment**\n\nWe define alignment in voxel space as explicit and methods in latent space that cannot be restored to voxel space as implicit, not based on single- or multi-subject distinction. We distinguish these two to highlight their implications for cross-subject functional interpretability.\n\n**Q2: Comparison with Ferrante et al. (2024) using Simple Alignment Techniques (SAT).**\n\n**Novelty**: Both MindAligner and SAT use linear hypothesis of cross-subject variances (see Reviewer 5B7Z Q2), but their motivations differ, and linear modeling is **NOT** MindAligner's primary novelty. SAT needs identical stimuli across subjects-a strict requirement for practical use. Conversely, MindAligner tackles alignment without identical stimuli, a critical issue also emphasized by Wang et al., 2024. We address this issue to perform soft cross-subject brain alignment under different stimuli. Thus, MindAligner is the first to achieve explicit functional alignment **WITHOUT** requiring identical stimuli.\n\n**Performance comparison**:\nWe adapt SAT to our setting for comparison, and the results are in Tab.2: https://mindaligner.github.io/MindAligner/. SAT significantly underperforms MindAligner, showing that simple linear alignment **ALONE** cannot address the lack of shared stimuli like MindAligner does.\n  \n**Q3: NSD had few images seen by all subjects, which the authors chose to use as the test set.**\n\nWe use the standard train/test splits from the original NSD dataset and previous NSD reconstruction papers, including MindEye2, to ensure a fair comparison.\n\n**Q4: About the setting of Table 1.**\n\nPlease see the Reviewer iSeR Q1.\n\n**Q5: Comparison between MindAligner and normalized flatmaps in region-level functional mapping.**\n\nCompared to normalized flatmaps, MindAligner offers several advantages:  \n1. **Pattern analysis without paired stimuli data**: By training on a small amount of data, MindAligner could capture fine-grained neural patterns without paired stimuli, rather than regional average activations.\n2. **Region-level analysis**: MindAligner could evaluate the contribution of brain regions to specific analysis, whereas simple statistical comparisons of flatmaps without training can only identify correlations. Compared to flatmaps, MindAligner achieves effective brain region mapping and enables exploration of region-level correspondence between arbitrary subjects, as shown in Reviewer iSeR.Q4.\n\n**Q6: About the Cross-subject Correlation Analysis (CCA).**\n\nCCA reveals two key insights:\n1. CCA results show that MindAligner's explicit alignment manner achieves higher brain correlation scores than implicit alignment. Benefit from effective subject alignment, MindAligner achieves superior decoding performance.\n2. We further analyze the relationship between alignment and decoding, finding that subjects with higher average correlation scores (e.g., subj1 and subj2) with other subjects consistently achieve better visual decoding performance. Please refer to Tab.3: https://mindaligner.github.io/MindAligner/.\n\nTogether, these findings confirm that improving cross-subject brain alignment is an effective and direct way to enhance novel subject decoding, offering a pathway for future research on cross-subject decoding.\n\n**Q7: About the pre-trained brain decoding model.**\n\nTo ensure fairness, we follow MindEye2 to use its multi-subject pre-trained weights and learn BTM for novel subjects under limited data. We will clarify it in the revised version. It is worth noting that the alignment layer (i.e., Linear Regression (LR)) in multi-subject MindEye2 only aligns known subjects to shared space and can not be directly applied to new subjects without fine-tuning. The results of MindEye2 in Tab.1 is its multi-subject pre-trained model (same as MindAligner) with its LR fine-tuned on novel subjects.\nIn contrast, MindAligner does not involve this process, thus does **NOT** enjoy the benefit of implicit alignment. \n\n**Q8: How the \"baseline\" results of MindEye2 in Fig. 6 is computed?**\n\nFor MindEye2, we input paired stimuli in test set into each subject’s ridge regression model and compute the fSC between embeddings. For MindAligner, we transform the novel subject’s voxels via BTM, reconstruct the known subject’s voxels, and compute the fSC between the reconstructed and paired stimuli fMRI after the linear regression in multi-subject MindEye2.\n\n\n**Q9: About the term \"atlas\".**\n\nThank you for your suggestion. In our paper, we used \"atlas\" to describe functional regions during brain alignment process. To avoid misunderstanding, we will revise it to \"region\".\n\n**Q10: About the brain ROI?** \n\nWe strictly followed the same ROI selection protocol as other works for fair comparison.\n\n**Q11: Misbolded results.** \n\nWe will revise them.", "claims_and_evidence": "Claim 1: MindAligner outperforms existing methods in fMRI-to-image reconstruction in low-sample settings.\n\nEvidence: Table 1 compares MindAligner results to past work (MindEye2 and MindBridge) using 1 hour of data from a new subject. They show MindAligner performs best across most metrics. The authors don’t seem to reveal which “known” subject was used for the results depicted in this table. They also do not explicitly mention that the top header is the average of the subsequent four rows (averaging across the 4 subjects’ metrics), nor do they clarify that the subsequent four rows were conducted with only 1 hour of training data for the subject in parenthesis. These are all things that should be easily correctable in a revision. \n\nClaim 2: MindAligner facilitates novel insights in cross-subject functional analysis.\n\nEvidence: The paper claims their method allows for novel brain functional alignment analyses, allowing for improved interpretations that past methods lack. They present two such functional alignment results: (1) Region-level Functional Mapping and (2) Cross-subject Correlation Analysis.\n\nRegion-level Functional Mapping: This result is simply presenting which regions of the brain had the greatest inter-subject variability. You could achieve such results by simply comparing the input voxels across subjects, without any model training needed. What unique advantages does MindAligner offer to region-level functional mapping that statistical comparison across normalized flatmaps of the participants can’t already do?\n\nCross-subject Correlation Analysis: The authors compute the functional correlation between brain regions of two subjects. I don’t see what novel insights can be gleaned from this analysis? The authors simply state that these results outperform MindEye2, but that doesn’t relate to the question of novel insights. Further, I do not understand how the “baseline” of MindEye2 was computed? MindEye2 doesn’t involve translating from one subject’s voxel space to another subject’s voxel space, so what is even being correlated?  \n\nClaim 3: “MindAligner [is] the first explicit brain alignment framework that enables cross-subject visual decoding and brain functional analysis in the data-limited setting.”\n\nMindAligner is not the first to tackle cross-subject decoding by explicitly mapping a new subject’s brain to an old, known subject’s brain. Specifically, Ferrante et al. (2024) seems nearly identical to the method used by the present authors except for different losses implemented in the Brain Functional Alignment module. Further, Ferrante et al. (2024) was also specifically used within the context of wanting to get decoding models to work with smaller datasets.\n\nI’m also unsure if the term “explicit” is the best way to describe their method in contrast to previous work using shared-subject spaces. Past methods mapped new subjects to a shared subject space, but in practice these past methods could just as easily have made the “shared subject space” consist of a single subject rather than multiple subjects (in fact this is one of the ablations in the MindEye2 paper, in Appendix A.11). \n\nMy understanding of the novelty here seems to be that mapping to a single subject is more advantageous (in terms of decoding performance) than mapping to multiple subjects and that the use of BTM confers novel interpretational insights (except see my concerns regarding this in Claim 2 section above). Calling this method “explicit” whereas past works are “implicit” seems confusing—why is it that mapping from one subject to another is explicit but mapping from one subject to several subjects is implicit? Would you call the works of Bazeille et al., 2019, Thual et al., 2022; 2023, and Ferrante et al., 2024 explicit or implicit approaches? \n\nOverall I think only claim #1 is sufficiently supported out of the above three claims, and even then, the novelties of their method seem largely borrowed from Ferrante et al. (2024).", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense, although further clarifications are recommended (see other comments).", "theoretical_claims": "See my Claims And Evidence section comments.", "experimental_designs_or_analyses": "I find it misleading that Figure 1 depicts the authors approach as mapping different functional regions of a new subject’s brain to various functional “atlas” in a known subject. Neuroimagers use the term “atlas” to mean a predefined region-of-interest or parcellation scheme (which could confer additional interpretational benefits). However, the use of an “atlas” or “parcel” is not used anywhere else in the paper. The method section seems to suggest that it’s simply linear mappings from one voxel space to another, with no atlas involved, making the Figure 1 potentially misleading.", "relation_to_broader_scientific_literature": "Broader scientific literature is well-covered, with exception to my points regarding Ferrante et al. (2024) (see Claims And Evidence)", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Methods novelty is limited and claims regarding novel interpretational benefits seem not well-justified. That said, it's still useful and important to show that simple linear alignment to someone else's voxel space can outperform the more complicated shared-subject latent space approaches used in other work.", "other_comments_or_suggestions": "The authors should re-examine their tables for proper bolding of the best performing metrics. Some metrics are misbolded and other metrics that should be bolded are not.\n\n“However, achieving such brain alignment is challenging, as it requires paired fMRI from subjects performing the same task (i.e., viewing identical visual stimuli (Bazeille et al., 2021)), a condition not met by the existing dataset (Allen et al., 2022).”\nNote that Allen et al. (2022) did contain a subset of images seen by all subjects. The authors choose not to use this in favor of using it as the test set.", "questions_for_authors": "I could not find what the authors used for their “pre-trained brain decoding model”. Was it MindEye2 without shared-subject pretraining? Or something else? If it was using MindEye2 pretrained with a shared-subject latent space then that would make interpretations and comparisons with MindEye2 confusing (you'd basically be benefitting from both \"explicit\" *and* \"implicit\" advantages from other subjects).\n\nWas the same brain region of interest (nsdgeneral) used here as other works like MindEye?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers generally regarded the paper favourably and found the approach to be novel, well-motivated and to provide useful neuroscientific insights. In particular, the papers strengths include:
* The cross-subject brain decoding approach with explict functional alignment is well motivated and well-founded, and addresses limitations in current cross-subject methods
* The quantitative results show substantial advantages to this method across a broad set of metrics over the baseline in most cases
* The need for paired stimuli is relaxed with the soft alignment mechanism, which is novel and promising
* The interpretable functional alignment analysis facilitates insights into cross-subject variability and underlying neural mechanisms.

Limitations and concerns for the paper include:
* Some descriptions could be clearer, including the choice of some notation, the description of the method, conditions for experiments, the meaning of some elements of the results, and procedures for selecting samples for qualitative comparisons.
* A better discussion of the ablation results with regards to the latent alignment loss, and whether this is less critical when other auxiliary losses are included, would strengthen this aspect of the paper.
* Some indication of the significance or error bounds for quantitative results would strengthen the paper

The authors are strongly encouraged to include key rebuttal clarifications in the revised paper, including the justification and discussion of the linear assumption and the implications of explicit alignment.
Decision: Accept (poster)