$i$MIND: Insightful Multi-subject Invariant Neural Decoding
Authors: Zixiang Yin
Jiarui Li
Zhengming Ding
Venue: NeurIPS 2025 poster ()
Keywords: Brain Vision
Neural Decoding
Brain Semantic Understanding
Primary area: neuroscience_and_cognitive_science

Abstract: Decoding visual signals holds an appealing potential to unravel the complexities of cognition and perception. While recent reconstruction tasks leverage powerful generative models to produce high-fidelity images from neural recordings, they often pay limited attention to the underlying neural representations and rely heavily on pretrained priors. As a result, they provide little insight into how individual voxels encode and differentiate semantic content or how these representations vary across subjects. To mitigate this gap, we present an $i$nsightful **M**ulti-subject **I**nvariant **N**eural **D**ecoding ($i$MIND) model, which employs a novel dual-decoding framework--both biometric and semantic decoding--to offer neural interpretability in a data-driven manner and deepen our understanding of brain-based visual functionalities. Our $i$MIND model operates through three core steps: establishing a shared neural representation space across subjects using a ViT-based masked autoencoder, disentangling neural features into complementary subject-specific and object-specific components, and performing dual decoding to support both biometric and semantic classification tasks. Experimental results demonstrate that $i$MIND achieves state-of-the-art decoding performance with minimal scalability limitations. Furthermore, $i$MIND empirically generates voxel-object activation fingerprints that reveal object-specific neural patterns and enable investigation of subject-specific variations in attention to identical stimuli. These findings provide a foundation for more interpretable and generalizable subject-invariant neural decoding, advancing our understanding of the voxel semantic selectivity as well as the neural vision processing dynamics.

MMND:: Insightful Multi-subjeet Invariant Neural Decoding
Zixiang Yin, Jiarui Li, Zhengming Ding" Department of Computer Science, Tulane University {zyin, j1178, zdingl)@tulane edu https://zaciis http//t..nnioomaaooooammmeeeeeee /immnimid
Abstract
Decoding visual signals holds appealing potential to unravel the complexities of cognition and perception. While recent reconstruction tasks leverage powerful generative models to produce high-fidelity images from neural recordings, they often pay limited attention to the underlying neural representations and rely heavily pretrained priors. As result, they provide little insight into how individual voxels encode and differentiate semantic content or how these representations vary across subjects. To mitigate this gap. we present an insightful Multi-subject Invariant Neural Decoding (iMIND) model, which employs novel dual-decoding framework-bothh biometric and semantic decoding-to offer neural interpretabil- ity in data-driven manner and deepen our understanding of brain-based visual funetionalities. Our iMIND model operates through three core steps: establishing a shared neural representation space across subjects using a ViT-based masked autoencoder, disentangling neural features into complementary subject-specific and object-specific components, and performing dual decoding to support both biometric and semantic classífication tasks. Experimental results demonstrate that iMIND achieves otate-of-thheart decoding performance with minimal scalability limitations. Furthermore, iMIND empirically generates voxel-object activation fingerprints that reveal object-specific neural patterns and enable investigation of subject-specific variations attention to identical stimuli. These findings provide foundation for more interpretable and generalizable subject-inven innvaratn neural de- coding, advancing our understanding of the voxel semantic selectivity as well as the neural vision processing dynamics.
1 Introduction
Deep learning models have recently been adopted in neuroscience as powerful tools for modeling brain activity, especially in the study of vision and cognition 9 19 26). A central goal in cognitive neuroscience is to understand how the brain transforms sensory input into meaningful representations that support recognition, memory, decision-making, and attention. Unlike behavioral annotations, neural signals provide a direct readout of these internal processes, revealing perceptual 33 emotional [32), and attentional dynamics (27) that cannot be fully captured by explicit labeling. Among available neural modalities, functional Magnetic Resonance Imaging (IMRI) [29] has been especially influential for its ability to non-invasively measure distributed patterns of cortical activity, enabling the study of how complex visual concepts are represented across brain regions. As such, fMRI not only offers a promising supervisory signal for aligning neural activity with computational models, but also serves as critical tool for probing the neural basis of cognition and vision [66].
"Corresponding author. This work is partially supported by NIH 2U19 AG055373-06AI as well as the Harold L. and Heather E. Jurist Center of Excellence for Artificial Intelligence at Tulane University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Stimulus
MHA
(2 ob
Semantic
CLIP
Head
Pool
Volur
A
M
m
IMRI
B Bobj
els Suu xx
Biometric
MAE
Pool
...
Buubi
Subjects
Fruzem
Trainable
Figure 1: Overview our framework. zMIND involves biometric decoding (identifying individuals) and semantic decoding (classifying perceived objects). IMRI voxel signals are first flattened and encoded by pretrained Masked Autoencoder (MAE) to generate latent neural tokens, which are then passed through a learnable subject-object disentanglement block via orthogonal basis transformation B = (uub Bobj). The resulting biometric neural tokens are directly pooled for subject classification, while the semantic neural tokens act as keys and values latent features of the corresponding visual stimuli as queries to reflect the semantic contents captured by each subject. The fused representation is then pooled for multi-label object classification.
cross-attention module with the CLIP
Most recent research has focused on reconstructing visual stimuli from brain activity by projecting neural representations into deep visual spaces and employing generative models such GANs or diffusion models 5
38 41]. These approaches have yielded visually plausible, high- resolution images, suggesting that deep learning can serve bridge between brain activity and imagery. However, despite these remarkable achievements, we argue that reconstruction alone is fundamentally inadequate for understanding brain vision mechanism. Specifically, the following limitations exist:
Reconstruction relies heavily on pretrained generative priors, which often dominate the decod- ing process and may introduce model-specific biases that obscure genuine neural content.
Brain recording may not encode all fine-grained details necessary for accurate reconstruction- especially across ultimately shifts the focus from reconstruction to generation.
pixel-level reconstruction ill- -lo-p and often misleading. This
Thus, argue that reconstructing visual stimuli is neither reliable nor interpretable strategy for decoding neural representations. A more effective alternative is to classify the subject's perceptual experience directly from fMRI data, enabling the identification of visual concepts embedded in neural responses [44). This discriminative classification-bu based approach supports evaluation through standard metrics and allows researchers disentangle both shared and individual-speciff semantic components of brain activity-capabilice that reconstruction methods often fail to provide.
Technically, current neural decoding solutions follow main strategies: single-subject models [5) 22 24 25 35 38, which suffer from data scarcity, overfitting, and poor scalability due to the which face substantial
high cost of fMRI collection; and multi-subject models 4 36
41
44
inter-subject variability from anatomical and functional differences. This can lead to entangled neural representations, where subject-specific and object-related signals become mixed [10], degrading decoding performance and interpretability. This raises a key research question:
"How can we develop discriminative neural decoding framework that generalizes across individuals while preserving subject-specific semantic interpretations of visual stimuli?"
To answer this, we present a novel approach to decoding brain activity using deep classification models, aiming capture shared neural representations across individuals while preserving per- sonalized interpretations shaped by diverse experiences and backgrounds. Toward this goal, we introduce the insightful Multi-subject Invariant Neural Decoding (MMNDD model-a dual-decoding
"See Appendix A.l for detailed related works.

---

framework (Figure[] that supports both biometric decoding (identifying individuals) and semantic decoding (classifying perceived objects). By constructing voxel-object activation maps and examining subject-specific attentional patterns in response to complex, multi-object scenes, iMIND offers principled path to disentangling individual and shared components of visual perception, ultimately advancing our understanding of brain-vision mechanisms. To sum up, our contributions are threefold:
We introduce a multi-subject dual-decoding framework that disentangles neural signals into subject- specific and object-specific components, enabling scalable and subject-aware brain analysis with state-of-the-an semantic and biometric decoding performance.
We develop a visual-neural interaction module that identifies object-voxel activation patterns in a data-driven manner, revealing how different subjects encode object-level semantics in the brain.
We perform comprehensive analysis of attention-based variability across subjects viewing the same visual stimuli, providing insights into individualized neural responses under time-constrained conditions.
2 Proposed Method
2.1 Problem Formulation
Consider neural dataset containing brain activities of subjects S in response visual stimuli drawn from an image dataset with M samples D = {(X,, y:)] where each visual stimulus X, is three-channel image with resolution of H W, paired with non-exclusive ground-truth label y; E IR representing C' object categories. During neural recordings, all images X are viewed by a group of subjects S. For each subject s E S viewing the i-th image X,, an fMRI voxel response V1, is recorded, capturing neural activity specific to the subject-image pair. Note that the voxel signal Vi, flattened as an ID vector with various lengths (12,682 17,907) across subjects due to biometric differences in neural structure. Following (5), we apply wrap-around padding to achieve uniform voxel length L.. Details on pre-processing procedures are provided in Appendix AA322
2.2 Framework Overview
Our goal is to decouple arbitrary fMRI voxel signals into subject-invariant invariant and subject-speciffe components from both semantic and biometric perspectives by developing a visual-neural model M. Formally, this can be expressed as a mapping:
M Xx
S
image
voxcls
semantics
biometrics
where semantic decoding seeks to extract object-related neural representations that consistent across subjects, whereas biometric decoding focuses on capturing subject-specific neural patterns that are independent of the visual stimuli.
To address both tasks, we propose the insightful Multi-subject Invariant Neural Decoding model, ab- breviated as iMIND. Building the SC-MBM framework [5), our method constructs a d-dimennional shared latent neural space across subjects to reduce the noise and redundancy inherent in IMRI employ Vision Transformer (ViT)-based encoder E IR - Nxd that projects each input voxel signal V into a set of N neural feature tokens, denoted as F = (f, J where each is d-dimensional neural feature token. This yields N neural representations per fMRI input, all embedded in the shared latent space F. The encoder {() is pretrained using a masked autoencoding objective in self-supervised manner, emphasizing voxel-wise reconstruction.
signals [40]. Specifically,
Subsequently, we disentangle the learned features into object-specific and subject-specific components. The subject-specine features are used for subject classification (biometric decoding), while the object- specific features are aligned with frozen CLIP visual embeddings of the corresponding images via a cross-attention mechanism, enabling multi-label object classification (semantic decoding).
2.3 Subject-Object Disentanglement
The supe-supe pretraining reduces noise and redundancy in fMRI signals by encouraging the model to capture generalizable patterns. However, the resulting neural features J are not explicitly
3

---

tailored for downstream decoding tasks such as biometric identification or semantic classification. Because the reconstruction objective treats all latent information as equally relevant, often leads to entangled representations-miiigg subject-specific and object-specific components. This entanglement limits the interpretability of the learned features and hinders task-specific performance, especially when distinguishing between individual variability and shared visual semantics is essential
Mathematically, this can be formalized by interpreting each neural feature token f (f1, f2....,.. fa) E R2d as the coordinate representation of a corresponding neural point p in the latent neural feature space F. By default, this representation is expressed with respect to the standard basis E = fex E R. "R] of R2, denoted as [p)æ:
P := Σ facK ER.
(2)
From this perspective, the entanglement of subject-specific and object-specific information within f can be attributed to the default use of E as the basis for spanning the neural feature space F. As E is implicitly determined by the self-supervised task in the first training stage, this choice is beyond direct control, resulting in inevitable blending of both subject and object information within the feature representation f for any neural point p E F.
To enable effective downstream biometric and semantic decoding, we propose a solution from the per- We begin with assuming that the suhject-specific and object-specific information within neural point p linearly entangled in the current neural feature representation under the basis E. While this may not fully characterize the complexities of neural dynamics, it serves as simplified approximation understanding the interplay between subject-specific and object-specific information. More crucially, this assumption applies at the latent feature level, not at the original fMRI signal level. At this level, the assumption is reasonable, as it aligns with the principles of deep classification tasks, where linear MLP classifiers rely on deep neural networks transform inputs into linearly separable features for ac- curate classification. Based on this assumption, we resolve the linear entanglement by re-representing P with respect new basis B. Specifically, we seek learnable basis B (Baubj. Boby) of R space that perfectly separates the subject-specifet and object-specific features. Mathematically, this re-representation would allow z, the representation of the same neural point p with respect to the new basis B, to be distinctly split into
spective of feature disentanglement 8 14 39 43
provide meaningful step toward
Zuu and object-specific Zurj components:
[jB = ((D) (PIB... (Zsubj Zobj) R
According the mathematical property of bases, the separation of Z-uhj and Zori is guaranteed long as B forms an orthonormal basis of Rd, i.... BB Lts where L is the identity matrix of rank d. With this orthonormality condition satisfied, the transformation from the original representation of f the new representation of the same neural point p can be derived follows:
z p B B [ple B [|e B E Edd
Combined with Eq.
3 we finally arrive at our subject-object disentanglement formulation:
B Su E Rudu and
Zubi Byy E dR
Zuuhj
From a feature transformation perspective, we realize subject-object disentanglement by decomposing the original neural space F into two complementary (orthogonal) subspaces: the subject-speciffe neural subspace Fuuby and the object-specific neural subspace Fobj- This disentanglement achieved by learning two orthonormal sets, Bauby E ddxx and Boy E dxddas which span Fush and Forj respectively. In our framework, danj is treated as user-defined value, with dunrj := d - dabj to complete the basis. A formal and theoretie proof is provided in Appendix[A.2
The complementary relationship, F = Fsuty + Fobj. guarantees a clear separation of subject and object information in the transformed neural representation z = (Zsabj, Zobj) E Rd for each neural point p E F, establishing a foundation for the subsequent biometric and semantic decoding tasks.
2.4 Biometric& & Semantic Decoding
In this section, we describe our approach to decoding fMRI signals biometrically and semantically. Note that an fMRI signal V is represented by F set of N neural tokens 1 within the

---

latent neural space F. Based on Eq. (5), the subject-specific feature map Zuub) E NNdaa and object-specific feature map Zoj E R Xxdon are generated from F E R-Nxd as follows:
suj subj
and Zoy FBobj.
(6)
2.4.1 Biometric Decoding
The biometric neural decoding is driven by supervised multi-subject classification task. We apply
a Global Average Pooling (GAP) operator, Guhj RR. x - RRduas to the subject-specific neural feature map Zuu to build a subject class token:
sub
Gsuby (Zsub)).
Finally, linear multi-subject classifier Cubb :Rdd -t IR is applied for the final biometric prediction:
Yaubj Csuby (
(8)
2.4.2 Semantic Decoding
To establish a feature-level connection between the fMRI voxel signals V and the semantic content of visual stimulus X, we utilize the vision feature map Fx E R& extracted from the last layer of a frozen CLIP visual encoder (33] In contrast to most neural vision approaches that project fMRI features into the CLIP visual space [5] 3 our method takes the opposite strategy by treating the CLIP vision features as queries to extract corresponding neural object features from Zobj-
This design is motivated by the complementary roles of CLIP and fMRI signals. CLIP features encode subject-ivvariant and stimulus-driven semantics with rich spatial and conceptual structure, effectively serving
sseudo-ground-trgg -ruth reference for object existence. In contrast, fMRI captures subject-specific neural responses that reveal how different individuals attend to these semantic components. To fuse these two modalities, applied multi-head Pross-attention extractor IR Na xda,,, defined as follows:
A: Na xda x R Xxdaa x R ddai -
F A(Query Key Zotj: Value Zobz).
Here, the CLIP embeddings act as queries, while the fMRI-derived Zony serves as keys and values. This configuration ensures that the fused representation remains fundamentally neural in nature: fMRI signals determine what semantic components are prioritized, while CLIP provides the structured semantic reference frame. The resulting cross-attention has two key effects:
Semantic Prioritization CLIP embeddings query the fMRI features, and attention weights highlight which parts of the CLIP semantic space align with neural activations. This allows the model anchor predictions in semantically grounded content;
Subject-Specific Modulation The fMRI responses modulate CLIP driven semantics in subject- dependent manner, enabling the model to capture how different individuals selectively emphasize different attributes of visual stimuli that share similar semantic contents.
In this way, CLIP will not dominate or overwrite the neural signal. but rather provides a semantically structured scaffold. The fMRI features dynamically shape which aspects of that structure are emphasized, yielding a bi-directional synergy. Our semantic decoding thus remains primarily rooted in the fMRI modality, with CLIP assisting in refining object-specific neural features for final multi- label object classification. Similar to what we have done in the biometric decoding in the previous section, a global feature operator Gobj
R N2 Xday - dad transforms Z F. into an object class token: obj
ob
Garj Fx).
(10)
Following that a multi-label object classifier Cotj Rdai - RRC is applied for final semantic prediction:
Pobj Cobj (zobi).
(11)
5

---

2.5 Model Training
The training process of our model consists of two stages. In the first stage, we follow the approach of SC-MBM [5] to pre-train ViT-based masked autoencoder for fMRI data, constructing latent neural space via minimizing reconstruction error with a Mean-Square Error (MSE) loss.
In the second stage, we retain only the IMRI encoder E-) from the first stage and optimize it with all other parameters in the proposed architecture. Three loss functions guide this stage. First, for biometric decoding, we introduce a subject classification loss Lubbj, which computes the cross-entropy HcE with softmax activation against the one-hot label Yuu from the ground-trus subject index Ysubj
Suh == HCF (softmax (S (Psuti) .
(12)
For multi-label semantic decoding, employ an object loss Loby which is binary cross-entropy function HBCE with sigmoid activation a-)):
Lobj == HBCE (Pobj) Yobi).
(13)
Finally, we impose an orthonormal constraint on the learnable basis concatenation B (Bsubj, Bobi). as defined in Eq. (6). This orthonormal loss Corth ensures the perfect separation of subject-specific and object-specific features by minimizing Lonh == BBB Ialle. where ||- indicates the Frobenius matrix norm, and 1 E Rdxd is the identity matrix of rank d.
In summary, in the second training stage, the total objective L is formulated as:
L = Lubbi + Lutj + XLotth
(14)
where \ serves as a trade-off hyperparameter to balance the orthonormal constraint against the subject and object classification losses, which are considered equally important and share the same scale.
3 Experiments
3.1 Experimental Setup
Dataset. We evaluate our iMIND framework using the Natural Scenes Dataset (NSD) 12, a compre- hensive, publicly available IMRI dataset capturing brain responses from human subjects viewing natural scenes from MS-COCO (33 Each subject passively viewed a set of 10,000 images for 38, each repeated three times; 1,000 of these images were shared across all subjects, while the remaining 9,000 were unique to each individual, with no overlap between subjects. Due to incomplete sessions and data availability restrictions, not all trials are accessible for every subject, resulting in a total of 213,000 trials across all participants before pre-processing. In line with previous NSD studies we used standardized train/test splits and averaged fMRI activations over repetitions for each image within each subject. This pre-processing yielded 69,566 training samples and 7,674 test samples, allowing us to train a single multi-subject model across all subjects. Additional details on NSD data, fMRI pre-procs,,nng, and iMIND implementation can be found in Appendix A.3.3
35 36 38
3.2 Neural Decoding Performance
Semantic Decoding. For the semantic decoding task, we evaluate and compare with other models using three standard metrics for multi-label classification: mean Average Precision (mAP), the area under the receiver operating characteristic curve (AUC), and Hamming distance (Hamming). Table [ categorizes models based their ability to process multi-subject fMRI signals simultaneously on a per-subject basis, as well as the modalities used for object classification. While our iMIND single- also be adapted
model is designed process both image and fMRI modalities, modality model by simply removing the cross-attention mechanism in Eq. ( and using Zobj in Eq. (6) directly for semantic decoding. Experimental results indicate that iMIND achieves superior performance across all three metrics, establishing a new state-of-the-art for semantic decoding in both
and multi- modality settings.

---

Table 1: Semantic decoding performance on the NSD dataset.
Model Type
Methods
Modalities
mAP 1 AUC
Hamming L
MLP
44
fMRI
.258
.854
.033
Single-subject
VIT
IMRI
.238
.815
.032
MLP [4]]
IMRI
.150
.767
.039
ViT 44
IMRI
.156
.755
.038
EMB
fMRI
.220
.825
.035
Multi-subject
CLIP-MUSED
IMRI+Image+Ten
.258
.877
.030
1MRI .309
.913
.027
iMND (Ours)
IMRI+Image
.784
.984
.012
directly sourced from 44 as benchmarks due to the limited rescarch on semantic neural decoding
1.0
axte03 sute02 Sub003
autp04 sub,05 Sub/06
Activatons
0
0.0
50
100
150
200
250
Figure 2: Average activations of tokens in Zohj for the object Chair.
Biometric Decoding, To the best of our knowledge, no existing neural decoding models support subject classification using the fMRI modality. To provide a comprehensive and fair evaluation, we established baseline models following the exact preprocessing steps from our proposed method and conducted both supervised and unsupervised biometric decoding, as presented in Table 2 Top-1 accuracy (ACC) and
Table 2: Biometric decoding performance.
Method
Setting
ACC
MCC
Euclidean
.181
.068
K-Means
Cosine
.232
.126
Plain
.283
.181
MLP
1.2 norm 1.2 ReLU
.377
.290
.573
.526
Matthews Correlation Coefficient (MCC) (28) are
used as metrics. The poor performance of naive subject classification methods underscores the com- plexity of neural data across subjects, whereas the near-perfect classification achieved by iMIND highlights the effectiveness and necessity of our subject-object disentanglement approach. This demonstrates that within framework, biometric fMRI features are highly discriminative across subjects and our linearity assumption reasonable. By facilitating the extraction of task-relevant features, disentanglement method further enhances downstream semantic and biometric decoding. Details on how we build biometric decoding baselines are provided in Appendix A.4
MIND
.999
.999
3.3
Invariant Decoding
The primary motivation for the subject-object disentanglement design, introduced in Section 2.3 is to decompose the entangled neural feature into subject-specifie and object-oriented components Zuubj and Zub for a better biometric and semantic decoding. This approach expects object- bbject-wise token contributions in Zotj to remain consistent across subjects. Using the object chair as an example, we visualize 280 tokens' activations averaged across all correct predictions by subject in Figure [ It turns out that at the feature level, our method successfully achieves subject-invarians decoding, as token activations display high similarity with only negligible subject-level variations. This outcome demonstrates our model's effectiveness in extracting object-specific information from complex fMRI data, offering robust framework for multi-subject fMRI decoding.
3.4 Visual-Neural Relationship
We empirically investigate the relationship between brain activities and semantic objects in visual stimuli, leveraging both GradCAM (37 and Attention Roll-out (I].
Subject-wise 1D Activation Pattern. Taking subj01 as an example, we calculate voxel-wise activations within low-level visual regions of interest (ROIs) (V1-V4) and wider high-level visual ROI response to three objects: person, horse, and chair. This calculation takes the median activation across all true positive samples predicted by our model the test set. As shown in Figure 3 the
7

---

Higher Vision
mmm
Chale horse
1SLL
Vovels
Figure 3: ID Object- Voxel activations by brain vision ROIs for subj01.
person
car
horse
Lolans
coloms
Figure 4: 3D Object- Voxel activations of person, car, bird, and horse for subj01, subj03, and subj05.
y-axis represents median activation, while the x-axis represents voxel IDs, providing clear overview of the active voxels across both low-level and high-level visual ROIs when subj01 recognizes each object. The unique activation patterns, with specific voxels responding to each object, indicate that brain voxels function differently from image pixels. In images, object locations are believed spatially random, so evaluating pixel-wise activation does not yield consistent spatial activation patterns. In contrast, fMRI voxels exhibit specialized roles in processing visual information, suggesting that brain voxels are organized by functional responsibility with degree of spatial peeriance-especialyy in high-level visual ROIs. This conclusion aligns with the existing studies from neuroscience (20 21
Cross-subject 3D Activation Pattern. We present 3D brain activation patterns in Figure 4 for four objects-person, car, bird, horse-visualized across three subjects: subj01, subj03, and subj05. Light yellow regions denote non-visual areas that were excluded from the dataset, resulting in uniformly absent signals these regions. Based on the visualization, the following observations are made:
Consistency across subjects: the objects bird and horse show broad similarity across subjects, particularly in the region the higher visual cortex and predominantly in the left hemisphere. This consistency suggests that certain high-level features associated with animals may be processed in similar ways across individuals, reflecting stable visual processing pathways the brain.
Object sensitivity: the activation intensity for object person appears stronger and more concen- trated, indicating that the brain may allocate increased neural resources or "attention" to socially relevant stimuli (people), compared to less socially significant objects like bird. This result is supported by neuroscience research [15] 42).
Representational flexibility: while general patterns are shared across subjects, the intensity and spatial distribution of activation vary slightly for certain objects, such as car. These variations may reflect individual differences brain anatomy or prior experiences that influence object representation and visual information processing. This flexibility of the brain's 's adaptability to personal needs and experiences known as neural plasticity 7 30
3.5 Variations in Subject Attention
A key contribution of 2MIND is its use of subject-invariant CLIP visual features to explore how different subjects focus on distinct objects when receiving the same stimulus. this attention variation: the first column displays the original visual stimulus with six ground-truth
8

---

Oriihal mmaze
CEST
Figure 5: Variations in subjects' attention to different objects. The leftmost image shows the visual stimulus, labeled with six objects: person, dining table, cup,fork, hot dog, and chair. Four of them selected for visualization. Plots in the second column represent the shared attention across all subjects, and the remaining eight columns show the residual, subject-specific attention alongside predicted probabilities compare recognition confidence and priority.
annotations, second column represents the shared attention map common across subjects, and the remaining eight columns show the residual, subject-specifie attention patterns. Four object- specific attention maps are visualized on rows with predicted logits to compare recognition confidence. Considering images shown for only 3s (2). patterns of attention and object recognition offer more intriguing insights into rapid, automatic processes of visual information and neural encoding.
Temporal Constraints on Atten- tion Allocation: Within brief second viewing window, the brain must rapidly parse and prioritize elements of complex scene. No- tably, despite occupying only modest portion of the image, the object chair consistently receives high attention across all subjects. This suggests that chairs are pro- cessed in early, feed-forward manner-likely due to their high salience and distinctive visual fea- tures that enable rapid recognition under time constraints. To vali- date this, we analyzed prediction confidence for chair the train- ing set across subjects. grouped by object size (Figure (). The results confirm that chairs sufficient size are reliably identified by all participants. This immediate and confident response highlights the efficiency of the visual system in detecting familiar, contextually relevant objects with minimal cognitive effort.
subjo1
subj07
sub/34
subj06
10
Conficence
cp BxD
come
chair size
Figure Recognition of chair by object size.
Subject-specific Focus Under Time Con-
X339
straints: Under these brief viewing conditions, subject-specific differences in attention to ob- jects like cup, fork, and hot dog become espe- cially revealing. Variation in attention to cup, particularly with subj01 and subj03 achieving recognition confidence (predicted probabilities 0.5) by focusing more precisely on its loca- tion, suggests that these individuals may possess faster or more selective attentional strategies. Such patterns point to individual differences in visual processing speed, attentional control, or perceptual expertise that influence object prioritization under time constraints. These findings are consistent with training results (Figure 77. where subj01 and subj03 show the highest sensitivity to cups, measured by weighted MCC. Interestingly, although none of the subjects successfully recog-
Waighted on cup
subjo1
Sub/02
subj23
Sub/04
subio5
sub006
subjo7
subiss
Figure 7: Sensitivity to cup subjects.

---

nize fork or hot dog in Figure 5 all but subj02 allocate more attention to fork than hot dog, suggesting a subtle yet consistent attentional bias that reflects object familiarity or contextual relevance.
To the best of our knowledge, the proposed iMIND is the first model to capture subtle variations in how quickly and differently individuals allocate attention within constrained time frame, demonstrating the model's 's robustness in simulating real-world neural processes. The model's ability to account for both shared and individual-speeific attention patterns in response to brief stimulus exposure can inform the development of neural decoding approaches that better reflect human variability, especially in sensitvse applications like real-time scene analysis or autonomous driving. Complete details for all visualized figures are provided in the Appendix.
Table 3: Ablation on loss functions.
In sum, the fact that subjects allocate attention differ-
ently within just a few seconds underscores the effi ciency of neural mechanisms in prioritizing objects and
ID
Cubi
Lorhh
À
mAP (%)
Full
v
v
.1
78.36
the role of individual cognitive differences. This rapid, nuanced attention mapping highlights how our iMIND framework captures the interplay of shared and indi- vidual neural patterns, bridging cognitive neuroscience with computational modeling to decode visual attention in real-world scenarios.
/
-7.15 (L)
/
.1
-11.17 (L)
3
-11.07 (L)
4
/
v
.01
-0.94 (L)
7
v
-2.95 (L)
6
10
-0.84 (1)
3.6 Ablation Studies
Loss Functions. Our novel designs--sujjct-ohiee disentanglement and the dual-decoding framework are considered two key factors in achieving SOTA semantic decoding performance. To evaluate their effectiveness and necessity, we test combinations of the two loss functions, Lauhj and Lort. along with trade-off hyperparameter X. Table confirmm that both Lorh for subject-object disentanglement and the dual-decoding design are crucial for achieving high semantic performance. Moreover, the trade-off parameter seems to have a minimal effect on the overall results. The optimal
model utilizes all three loss functions with trade-off parameter of X = 0.1.
Model Variants. We investigate the impact of two key hyperparameters on the performance of semantic decoding: datj. the dimension of the neural object space for subject- object disentanglement in Section 2.3, and h, the number
Table 4: Ablation on heads and dobj-
ID
Hcad(s)
dabj
mAP (%)
Full
4
700
78.36
of heads in multi-head cross-attention module in Eq. (9).
700
-1.95 (4)
According to Table increasing the number of heads does necessarily lead to performance gain, as it may result in potential overfitting. In addition, we found that performance degradation remains minimal as long as there sufficient feature space allocated for the neural object information. Ultimately, the optimal object classification performance, in terms of mAP, is achieved with h = 4 and dubj = 700.
2
2
700
-1.12 (L)
700
-1.35 (L)
4
8
700
-9.18 (L)
5
4
100
(4)
200
-2.11
4
300
-1.04 (L)
400
-1.38 (L)
4
500
-0.89 (L)
4 Conclusion
10
600
-1.14 (4)
In this paper, we introduce an innovative multi-subject dual-decoding framework that decomposes latent MMRI representations into distinct subject-specific and object-specific components using robust basis transformation. This approach enables precise biometric decoding through individualized neural features, while shared object-oriented features facilitate subject-invan invariint semantic decoding by querying with CLIP-derived visual representations. Our framework not only establishes a new benchmark for semantic decoding accuracy but also reveals variations in attentional focus across subjects when viewing identical visual stimuli, Additionally, we construct object-specific activation patterns the voxel level, offering data-driven insights into the brain's visual processing mechanisms.
In future work, we aim to leverage large-scale IMRI datasets to develop more robust and informative pretrained models for extracting latent neural features. Additionally, we plan to collaborate with brain scientists to deepen our understanding of how specific voxel patterns in fMRI data relate to semantic object representations. This domain knowledge will help bridge the gap between visual features and neural signals, further enhancing the interpretability and accuracy of brain- based decoding models.
10

---

References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online, July 2020. Association for Computational Linguistics.
[2] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, al. massive fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126 2022.
[3] Antonello Baldassarre, Christopher M Lewis, Giorgia Committeri, Abraham Z Snyder, Gian Luca Romani, and Maurizio Corbetta. Individual variability in functional connectiv- ity predicts performance of a perceptual task. Proceedings of the National Academy of Sciences,
109(9):3516-35111 2012.
[4] Omar Chehab, Alexandre Défossez, Loiseau Cean-Christophe Alexandre Gramfort, and Jean- Remi King. Deep recurrent encoder: an end-to-end network to model magnetoenccephalggaapyy at scale. Neurons, Behavior, Data Analysis, and Theory, 2022.
[5] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 227 127100227,0, 2023.
[6] Davide Cortinovis, Marius V Peelen, and Stefania Bracci. Tool representations in human visual cortex. Journal of Cognitive Neuroscience, 3753)31 2025.
[7] Hans P Op de Beeck, Ineke Pillet, and Brendan Ritchie. selective areas emerge in visual cortex. Trends cognitive sciences, 23(9):784-797 2019.
Factors determining where category-
[8] Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. In 9th International Conference on Learning Representations, 2021.
[9] Changde Du Kaicheng Fu, Bincheng Wen, Yi Sun, Jie Peng, Wei Wei, Ying Gao, Shengpei Wang, Chuncheng Zhang, Jinpeng Li, et al. Human-like object concept representations emerge naturally multimodal large language models. Nature Machine Intelligence, pages 1-16, 2025.
[10] Julien Dubois and Ralph Adolphs. Building a science of individual differences from fmri. Trends in cognitive sciences, 20(6):425-444 443, 2016.
[11] Scott L Fairhall and Alfonso Caramazza. Brain regions that represent amodal conceptual knowledge. Journal of Neuroscience, 33(25):10552-1055) 2013.
[12] Bruce Fischl. Freesurfer. Neuroimage, 62(2):774-781, 2012.
[13] Bruce Fischl, Martin Sereno, Roger BH Tootell, and Anders M Dale. High-resolution intersubject averaging and coordinate system for the cortical surface. Human brain mapping, 8(4):272-284, 1999.
[14] Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf. and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36:27682-2766 2023.
[15] Michelle R Greene and Aude Oliva. The briefest of glances: The time course of natural scene understanding. Psychological science, 20(4):464-47) 2009.
[16] Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annu. Rev. Neurosci., 27(1):649-677 2004.

---

[17] Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert R. Sabuneu. Decoding natural image stimuli from fmri data with a surface- based convolutional network. In Ipek Oguz, Jack Noble, Xiaoxiao Li, Martin Styner, Christian Baumgartner, Mirabela Rusu, Tobias Heinmann, Despina Kontos, Bennett Landman, and Benoit Dawant, editors, Medical Imaging with Deep Learning, volume 227 of Proceedings of Machine Learning Research, pages 107-118. PMLR, 10-12 Jul 2024.
[18] Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily Allen, Yihan Wu, Ghislain St- Yves, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. Neurogen: activation optimized image synthesis for discovery neuroscience. Neurolmage, 247:118812, 2022.
[19] Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris I Baker. Revealing the multidi- mensional mental representations of natural objects underlying human similarity judgements. Nature human behaviour, 4(11) ((11):1173-1188)) 173-1185, 2020.
[20] Klaus Hoenig, Eun-Jin Sim, Viktor Bochev, Bärbel Hermbenggr, and Markus Kiefer. Conceptual flexibility in the human brain: dynamic recruitment of semantic maps from visual, motor, and motion-related areas. Journal of Cognitive Neuroscience, 20( (111001779-181411 2008.
[21] Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6):1210-122 1224, 2012.
[22] Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 35:29624-29633 2022.
[23] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[24] Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36, 2024.
[25] Andrew F Luo, Margaret M Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: Fine- grained natural language captions of visual cortex selectivity. arXiv preprint arXiv:2310.04420, 2023.
[26] Florian P Mahner, Lukas Muttenthaler, Umut Güçlüü and Martin N Hebart. Dimensions underlying the representational alignment of deep neural networks with humans. Nature Machine Intelligence, 76))848 859, 2025.
[27] George R Mangun, Michael H Buonocore, Massimo Girelli, and Amishi P Jha. Erp and fmri measures of visual spatial selective attention. Human brain mapping, 6(5-6):383-36) 1998.
[28] Brian W Matthews. Comparison of the predicted and observed secondary structure 14 phage lysozyme. Biochimica et Biophysica Acta (BBA) Prootein Structure, 405(2):442-45 451, 1975.
[29] Paul M Matthews and Peter Jezzard. Functional magnetic resonance imaging. Journal of Neurology, Neurosurgery & Psychiatry, 75(1):6-12, 2004.
[30] Arne May. Experience-dependent structural plasticity in the adult human brain. Trends in cognitive sciences. 15(10):475 482, 2011.
[31] John Mazziotta, Arthur Toga, Alan Evans, Peter Fox, Jack Lancaster, Karl Zilles, Roger Woods, Tomas Paus, Gregory Simpson, Bruce Pike, et al. four-dimensional probabilistic atlas of the human brain. Journal of the American Medical Informatics Association, 8(5):01 2001.
[32] K Luan Phan, Tor Wager, Stephan F Taylor, and Israel Liberzon. Functional neuroanatomy of emotion: meta-analysis of emotion activation studies pet and fmri. Neuroimage, 16(2):331- 348, 2002.
12

---

[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages
PMLR, 2021.
[34] N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Nancy Kanwisher. Computational models of category-selectivy brain regions enable throuhhuuughput tests of selectivity. Nature communications, 12(1))5540, 2021.
[35] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Demp- ster, Nathalie Verlinde, Blad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.
[36] Paul Steven Scotti, Mihir Tripathy, Cesar Torrico, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. In Forty-first International Conference on Machine Learning, 2024.
[37] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee based localization. In Proceedings of the IEEE international conference computer vision, pages 618-626, 2017.
[38] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453-14463, 2023.
[39] Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International conference on machine learning, pages 10401-10412. PMLR, 2021.
[40] Kamil Ugurbil, Junqian Xu, Edward Auerbach, Steen Moeller, An T Vu, Julio M Duarte- Carvajalino, Christophe Lenglet, Xiaoping Wu, Sebastian Schmitter, Pierre Francois Van de Moortele, et al. Pushing spatial and temporal resolution for functional and diffusion mri in the human connectome project. Neuroimage, 90:80-104, 2013.
[41] Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11333-11342 2024.
[42] Galit Yovel and Nancy Kanwisher. The neural basis of the behavioral face-inversion effect. Current biology, 15(24):2256-237) 22622, 2005.
[43] Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable contrastive learning with automatic feature importance discovery. Advances in Neural Information Processing Systems, 36, 2024.
[44] Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. Clip-mused: Clip-guider multi- subject visual neural information semantic decoding. In The Twelfth International Conference on Learning Representations, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: we clearly state our contributions that match claims made in the abstract and introduction. also did theoretic proof and conduct extensively experiments to validate our claims the result section the main paper and in Appendix
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Due to page limit, we don't include a limitation section in the main paper. Instead, we point out the limitation of the proposed method in Appendix A.6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification: We don't make any theoretic contributions in the paper, but we do rely on the assumption that the subject and object information is linear entangled in the latent fMRI space.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details necessary for reproducing the experimental results are fully de- scribed in either the main paper or appendix. The code will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: In appendix, describe the used dataset well the version of publicly available pretrained deep models we rely on in details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
sss.iioliciiiyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. wwwpccccpcpccpcu.ccccce htte://wiisss/eeee www.ooowwwissoooooooooooooooooee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to the implementation details in appeix for all the details regarding hyperparameter choice, preprocessing, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We don't do statistical significance check simply because our model outper- forms existing models by large margin that don't 't need statistical test to prove it. Also, the model is trained and tested on large dataset. It would be consuminn to do such test.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We list the computing resources needed for the proposed method, whose parameters are publicly available.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We closely follow the code of ethics by NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is an exploring work and far away from application. So, may not have any socictal impacts for now.
17

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: we don't think the rease of our data or models will have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The data and the pretrained deep models are fully described. We give credits to the original creator in the main paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode cowc datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release code upon acceptance, which include new assets such as trained model weights.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: Our research relies on human neural signals data, but it has potential risks for incurred participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our research relies on human neural signals data, but has no potential risks for incurred participants.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
20

---

A Technical Appendices and Supplementary Material
A.1 Related Work
A.1.1 Single-Subject v.s. Multi-Subject Models
The application of deep learning to neural data has initially centered on single-subject models tailored to individual participants. BrainDiVE (4 adopts a generative approach, synthesizing images pre- dicted to activate specific regions of the human visual cortex. Moving beyond visual modalities, Mind Reader [22] incorporates textual information to reconstruct complex images containing multiple objects from brain activities. Extending this multimodal approach, BrainSCUBA [25] takes advantage of contrastive langugge models and large-language models to generate voxel-wise captions, voxel-caption data. While single-subject models have
eliminating the need for achieved notable success, they face inherent limitations. These models require large amounts of subject-specifer data to train robust models, which is challenging given the high costs and effort involved in collecting fMRI data, Furthermore, they are prone to overfitting, exhibit poor general- izability across individuals, and struggle with scalability when applied to larger datasets or diverse populations.
To overcome these challenges, multi-subject models aim to unify data across participants, enabling shared representation learning. However, this approach introduces significant complexity due to inter- subject variability, which arises from static anatomical differences and dynamic functional responses. Various methods have been proposed to overcome these obstacles. employs subject embeddings and recurrent architectures to account for inter-trial and inter-subject variability, outperforming many single-subject models in predicting MEG time series. MindBridge [41 introduces a biologically inspired aggregation function and a cyclic fMRI reconstruction mechanism to achieve subject- invariant representation learning. MindEye2 B6] aligns spatial patterns of fMRI activity to shared latent space using subject-speciff ridge regression, improving out-of-subject generalization with limited training data and achieving state-of-the-art results in image retrieval and reconstruction. More recently, CLIP-MUSED 44 introduced learnable subject-specific tokens to facilitate the aggregation of multi-subject data without linear increase in model parameters. This approach integrates representational similarity analysis (RSA) to guide token representation learning based on the topological relationships of visual stimuli in the latent visual space.
These advancements demonstrate the potential of multi-subject models to surpass the limitations of single-subject approaches, providing more generalizable and scalable solutions for neural decoding tasks. However, to the best of our knowledge, existing multi multi-subject neural decoding models predominantly adopt what we term suppressive strategy for handling inter-subject variability. This approach aims to minimize subject-specific differences during learning, progressively refining features to become more task-relevant as the model deepens. these frameworks, subject-specific information is often treated as noise or an obstacle to effective decoding. In contrast, our iMIND framework proposes an instructive strategy. Rather than suppressing subject-specific differences, our model embraces this variability by explicitly disentangling subject-specific features from task-relevant ones. By doing so, iMIND not only preserves setividual-speccfee neural representations but also leverages them positively to enhance both subject-specift and shared task-related decoding. This dual-decoding approach enables iMIND to achieve superior performance while offering insights into both individual neural patterns and shared semantic representations.
A.1.2 Vision-Neural Interactions
Decoding visual information from neural signals is an inherently multi model task, involving the alignment and interaction of at least two modalities: images and neural signals (eg., IMRI). Broadly speaking, approaches for vision-neural modality alignment can be categorized into two branches based on the direction of projection between visual and neural spaces.
The first branch projects neural signals into a pre-trained latent visual space. This approach is exemplified by works such as [35], which maps flattened spatial patterns of fMRI activity across 3D cortical tissue cubes into the image embedding space of a pre-trained CLIP model. Similarly, (38] predicts latent representations of presented images from fMRI signals within the early visual cortex. Other notable works in this branch include 5 22 36 which leverage pre-trained visual generative
models, such GANs (34] and diffusion models, for reconstruction tasks. These models capitalize
21

---

on large-scale visual datasets and avoid re-training -sstucceintensivv generative architectures, which would otherwise be infeasible given the scarcity of paired neural-visual data. The second branch adopts the opposite approach by projecting latent visual image features into the neural space. This method particularly useful for generating synthetic stimuli that activate specific brain regions, enabling the study of feature preferences in different areas of the brain. Classic examples include [8] 24 25), which investigate neural activation patterns in response to synthetic stimuli derived from visual features.
Our iMIND model takes a fundamentally different approach vision-neure neural modality interaction. Rather than projecting between modalities, we use CLIP- -eePied vision features as queries to extract corresponding neural object features directly from neural representations. This design choice is motivated by several factors. First, as semantic neural decoding framework, iMIND does not rely on resource-intensive generative models. Second, direct projections between modalities often result in significant information loss and modality gaps that require careful handling. Most importantly, our approach is expected to enable the investigation subject-specific attention variations when viewing the same visual stimuli. By treating the CLIP vision features pseudo-ground -routrrru---ttthh for object presence, we leverage their subject-invarian properties as an anchor to explore how neural responses to specific objects differ across subjects. This design uniquely aligns with the goals of understanding suter-subject variability neural decoding.
A.2 Theoretic Validation for Basis Transformation
In this section, we present the basis transformation in linear algebra and establish the relationship between coordinates in different bases. The derivation ensures clarity in transitioning from one basis to another, essential for interpreting subject-specific neural space Fsuti and object-specific neural space Fabj mentioned the end of Section 2.3 in the main paper. We begin with the following formal claim:
Claim Let V a vector space of cardinality d over R, with a standard basis E = {ee, 62, e of d and an arbitrary basis B = {b), b bu} of Rdd For any vector v E V. if its coordinate with respect the standard basis E is given by [v]n Rd. then its coordinate with respect to the basis B can be derived as:
MB PE-B [V]E.
(15)
where PE-B E Rddxd is the change-of-basie matrix from E to B, defined as:
P PE+B
bi
b2
ba
(16)
Proof Since v E V and E forms a basis for the vector space V, v can be written as a linear combination of all basis vectors from E:
a
(17)
where ag E R are scalars. this case, the coordinate of v with respect to E is:
[ (a1, 02
aa) Rd.
(18)
Similarly, because B also forms basis for V. we can express v as:
b,,
(19)
where Wi are scalars. The coordinate of with respect to B is:
[v] B = (Wi, W2 ....... wa) Rd.
(20)
22

---

Since each basis vector e; within E is also an element of the vector space V, it can also be written as linear combination of all basis vectors from B:
e, Pjbj
(21)
==1
Writing the equation above in matrix form, obtain:
P11
P12
Pid
e1
e2
ed
P21
P2d
bj
b
bd
(22)
Lan
Pdz
Pad
Denoting the middle matrix as and solving it, we get:
P11
P12
Pid
p=
P21
Paa
bi
b2
bd
(23)
:
Ldd1
Pd2
Pdd_
Next, let's plug each e; into Eq.
17
using the formulation of Eq. 21
a,e.
Pjibyl = Σ
(24)
1=1
1=1
j=1
==1 111
Combining with Eq. 19
we have:
d
w,bj
a:Pyi bb,
(25)
j=1
j=1
Move everything to the left hand:
Σ
Wy
a.Pji bb 0.
(26)
According to the claim, B {bi, b only if:
ba} is a basis R Therefore. Eq.
26 holds if and
W3
a.Pj = 0 for j = 1,2.... d.
(27)
1=1
Equivalently, we have:
Wi
Pis
P11
P12
Pid
[01
W2
P2i
02
a2
P21
P2d
(28)
Pas
Pan
Paz
Paa_
Ldd
Using the coordinates expression defined in Eq. following equation:
(18
and Eq.
20
obtain the along with Eq.
23
MBB P [V]E where P
b
b
by
(29)
Finally, complete the proof of the claim.
In our iMIND model, subject-object disentanglement is achieved through basis transformation described above, where the new basis B is treated as learnable parameters optimized by loss prop- agation. We further enforce the orthonormality constraint on B, as this ensures that any subspace spanned by subset of B is orthogonal (complementary) to its counterparts in the original feature space. Specifically, our model, this constraint guarantees that the subject-specific neural space Fubyj spanned by Buuhj, and the object-specific neural space Fabj. spanned by Batj. are complementary and non-overlapping. Consequently, this orthogonal decomposition yields perfect separation of subject-specifer and object-specific neural features within the latent neural representations.
23

---

A.3
NSD dataset, Pre-processing, and Implementation
A.3.1 NSD Dataset
The Natural Scenes Dataset (NSD) 2 is a groundbreaking resource in cognitive neuroscience and artificial intelligence, designed to capture extensive, high-resolution fMRI data during natural scene perception. It includes whole-brain fMRI measurements of eight human participants at 7T field strength, with spatial resolution of 1.8 mm. Participants viewed total of 70, 566 natural scene images, with 10, 000 unique images per subject (9,000 unique to each participant and 1. 000 shared across all participants). Images were sourced from the richly annotated Microsoft COCO dataset (23), ensuring ecological relevance and diversity. The experiment was conducted over 30-40 sessions per participant, taking a rapid event-related design with continuous recognition tasks to guarantee engagement and probe both short- and long-term memory processes, During neural recording, participants were tasked with identifying objects in the image, with each visual stimulus presented for only three seconds per trial. This design makes the NSD particularly well-suited for investigating the mechanisms of rapid attention and visual recognition in human vision. Advanced hemodynamic
pre- reepproeessin completed by the authors, including denoising and response modeling, yielded high-quality single-trial beta estimates with exceptional signal-to-nn to-noese ratios. Complementing the functional data, NSD includes extensive anatomical scans, resting- data, and behavioral performance measures, enabling multi-faceted investigations of vision and memory. This dataset, with its unparalleled scale and quality, serves as valuable benchmark for developing and testing machine learning models that aim to decode brain activity and simulate neural representations of natural scenes. In addition, the NSD dataset supports multiple widely used neuroimaging atlases to facilitate data analysis and integration with existing frameworks. Functional data are provided in both native cortical surface space and standard volumetric spaces, including fsaverage [3] and MNI152 BI, enabling compatibility with tools like FreeSurfer [12] and FMRIB Software Library. Additionally, the dataset includes manually defined regions of interest (ROIs) for retinotopic mapping and sutegory-selectiv areas, such as the early visual cortex and higher-order regions in the ventral visual stream, which is the atlas that we used in our iMIND model. These comprehensive atlases allow researchers to seamlessly apply NSD data to diverse analytic pipelines and study comparisons.
A.3.2 Preprocessing
Our preprocessing pipeline begins with splitting the dataset into training and testing sets. Due to incomplete sessions and data availability constraints, not all trials are accessible for every subject, resulting in total of 213, 000 trials across all participants. Among these, neural recordings corre- sponding to 1, 1,000 images viewed by all subjects are allocated to the testing set, comprising a total of 21, 118 test trials. The remaining neural recordings, corresponding to images viewed exclusively by individual subjects, are included in the training set, resulting in 191, 882 training trials. Both training and testing trials are standardized voxel-wist using the mean and standard deviation calculated from the training set. Since each image is presented to a subject three times, we average the IMRI responses across repetitions for each image within each subject. This results in 69, 566 training samples and 7, 674 testing samples, allowing us to train and evaluate a single multi-subject model across all eight subjects. For each sample, we use the nsdgeneral atlas provided by the NSD dataset to extract visual voxel signals as ID vector. However, the number of visual voxels varies between subjects due to anatomical differences, with the voxel length L, ranging from 12, 682 to 17, 907 across subjects. To unify the input length, we apply a padding strategy inspired by Mind- Vis 5 which conducts wrap-around padding. This approach avoids issues arising from truncation or constant padding to the maximum voxel length. Additionally, since our fMRI encoder is based on Vision Transformer (ViT), which requires input lengths divisible by the user-defined patch size (64 in our model), we adjust the uniform voxel length L accordingly. The final voxel length across subjects is set to L 17, 920, ensuring compatibility with the model while maintaining consistency across participants.
A.3.3 Implementation Details
As described in Section of the main paper, our proposed architecture is trained in two stages. The first stage involves pre-training ViT-based masked autoencoder, similar to SC-MBM (.. using self-supervised fMRI reconstruction task. In this stage, we choose a patch size of 64 voxels with masking ratio of 0.75. The encoder has hidden dimension of 768 and consists of 12 layers
24

---

Table Generalizability unseen subjects
ID
Trained
Tested
mAP (%)
subj01-07
subj01-07
.7904
M7
subj01-07
subj08
.7842
subj01-08
subj01-07
.7842
M8 (Full)
subj01-08
subj08
.7909
of 6-head self-attention, while the decoder has hidden dimension of 512 and 8 layers of 8-head self-attention. In the second stage, we discard the decoder and inherit only the encoder from the first stage, which outputs pre-trained feature F R Nxd with N = 280 and d = 768. We set the object neural space dimension doty 700 and choose 4-head cross-attention module for fMRI-vision feature interactions. The CLIP visual encoder we used is clip-vit-base base-patchhhlee 16 released by OpenAL, which remains frozen at all stages the proposed framework. A trade-off parameter of 0. is set by default to enforce the orthonormal constraint of the learnable basis B. A detailed investigation is provided in Section [.6] During this stage, all parameters are optimized end-to-end for subject and object classification tasks. For either stage, train the model for 100 epochs, including 10 warm-up epochs. The learning rate initialized at 7.5 x 10 and terminated at zero adjusted dynamically by cosine scheduler. The batch size is set to 200. Optimization is performed via the Adam W optimizer with a weight decay of 0.05. All experiments are conducted on two Nvidia RTX 6000 Ada GPUs, with the first stage taking approximately 1.5 hours and the second stage around hours to complete.
A.4 Subject Classification Baselines
To the best of our knowledge, no existing models support subject classification. To provide a comprehensive evaluation, established baseline models using the exact fMRI preprocessing steps in Appendix [A.3.2 and conducted both supervised and unsupervised biometric decoding. All methods are trained and tested identical data splits and fMRI voxel sets as iMIND, ensuring fair comparison on the same held-out unseen test set.
For supervised learning, employ single linear layer trained in two ways:
Linear Regression: We minimize the mean squared error (MSE) between the input (padded fMRI voxel signals) and the target (one-hot subject IDs), using the ordinary least squares closed-form solution;
Classification: We train an identical architecture with cross-entropy loss, treating subject identifica- tion standard classification task.
For unsupervised learning, we evaluate K-Means clustering with two distance metrics:
Euclidean (L2) distance;
Cosine similarity.
Since the number of subjects is known (8), we set the number clusters well. To measure performance, we compute accuracy and MCC by optimally aligning the learned clusters with ground- truth subject IDs.
A.5 Subject Generalizability
For completeness, we conducted an additional experiment to assess subject generalizability within NSD as shown in Table our original setup (denoted as M8), iMIND was trained on data from all 8 subjects. For this experiment, we introduced variant (M7), where iMIND was trained using only first 7 subjects and tested on the held-out data of subj08. M7/M8 achieved an overall mAP of .7904/.7822 on the first 7 subjects and .7842/79909 on subj08. These results demonstrate that our proposed method exhibits reasonable and strong generalizability neural signal semantic decoding, particularly for unseen subjects.
25

---

A.6 Limitations
While our method achieves state-of-the-are art performance in both semantic and biometric decoding tasks, several limitations remain unresolved.
First, the current approach to neural feature extraction may not be optimal. Although functional, the pretrained neural reconstruction stage produces fMRI reconstructions-both numerically and visually-that underperform compared to the ground-truth voxel signals. This suggests that the masked autoencoder (MAE) backbone may not be the most effective architecture for this task, warranting further exploration.
Second, flattening voxel inputs discards crucial spatial relationships among neighboring voxels, despite neuroscientific evidence that proximal voxels exhibit functional coupling in visual processing. Future work could explore advanced architectures suchu as 3D Transfororsssrss, which are explicitly designed for volumetric fMRI data and have demonstrated efficacy in neurological discase diagnosis to better preserve spatial hierarchies and improve feature learning.
Third, we acknowledge that the brain's 's functional dynamics are fundamentally non-linear and com- plex. Our assumption that subject-specific and object-specific components are linearly entangled at the latent feature level is simplifying inductive bias introduced to enable interpretable, computa- tionally tractable disentanglement. We agree it fully captures the richness of brain representations; rather, it serves as a first-order approximation that enables clear factorization of subject identity and semantic content from fMRI signals. the linearity assumption does not fully hold, we expect the following potential implications:
If subject-object interactions are fundamentally non-linear, the disentangled object repre- sentation Zoty may still retain residual subject-specifer information, potentially introducing subject bias in semantic decoding and diminishing our model's generalizability for unseen subjects.
Conversely, enforcing strict linear disentanglement may suppress relevant non- lon-linear object features in Zobj. potentially smoothing out sharp voxel-object modulations or degrading decoding performance for fine-grained categories.
Last, our analysis of visual mechanisms relies on post hoc interpretation methods (Grad-CAM and Rollout), which provide only approximate explanations of model behavior. A more principled approach would involve explainable-by-dese architectures for fMRI feature extraction, which we leave for future work.
A.7 Broader Impacts
Our work represents pioneering step toward decoding the brain's visual processing mechanisms, with far-reaching implications for both neuroscience and artificial intelligence. By modeling how the brain transforms visual signals into neural activity and high-level semantics, we aim to uncover the fine-grained functional organization of visual regions-such as those specialized for distinguishing closely related objects (e.g., dogs vs. cats).
This understanding could enable breakthroughs in brain-computer interfaces (BCIs), where precise neural decoding could restore or augment vision for impaired individuals. Conversely, also raises ethical considerations: the same principles could theoretically be used to manipulate neural signals, artificially inducing semantic perceptions (e.g., generating "fake" visual concepts in the brain). Such capabilities would necessitate rigorous ethical frameworks to prevent misuse while maximizing socictal benefit.
Further, our computational approach bridges Al and neuroscience, offering interpretable models that could inspire more biologically plausible machine vision systems. By aligning artificial and biological vision, we may accelerate progress in both fields-from improving Al's robustness to advancing treatments for neurological disorders.
A.8 Technical Details Object- Voxel Visualization
In our experiments, we empirically investigate the relationship between brain activities and semantic objects in visual stimuli. In this section, we detail how voxel contributions to object recognition are
26

---

visualized in our framework. Given an input fMRI voxel signal V E R", we obtain its object-specific neural feature map Zaj R Nxdaa before attsntttnn module in our model. Using GradCAM 33,, we are able to build an activation map t E R.V, which quantifies the contribution of each neural token Zubj E RR in object-specific neural feature map Zobj E R dd to the correct recognition of the object of interest.
Unlike CNN-based models, which can simply resize to match the size of the input V because of their spatial invariance nature (a one-to-one correspondence between input patches and latent feature vectors), our ViT-based neural encoder in iMIND model lacks such spatial invariance properties by default. For instance. the first neural token in Zobj does not necessarily correspond the first voxel patch of V, making direct resizing infeasible.
To address this, we leverage Attention Roll-out (1 to approximate how information flows from the input voxels V to the neural tokens in feature map Zutj- Specifically, this information flow in our ViT-based encoder can be measured as follows:
A' = A.A A())
(30)
where A (A) E NNN represents the attention weights in the k-th self attention layer of the ViT encoder. Based on the mathematical property of the self-attention mechanism, the element diy of the attention weights at every transformer block defines how much attention flows from the token j in the previous layer the token i in the next layer. Therefore, each element dl within A' detined in Eq. (30 quantifies the degree of information flow from the j-th voxel patch of input IMRI signal V _ the i-th token in the feature map Zoby.
Next, we combine the GradCAM- -aseed token contributions t E RV with the cumulative information flow A' to derive the voxel-level activation measurement T:
= t=t-A' A' E ER..
(31)
Here, T measures the contribution of each voxel path immediately after embedding the original 1D voxel signal V of length I. into N patches. Since T is now positionally aligned with the input voxel patches, it can be safely upsampled from size N to L to obtain a voxel-level activation map for each fMRI sample. Unfortunately, this upsampled activation map is partially synthetic because wrap-around padding was applied during preprocessing to achieve a uniform, model-compatible voxel length L across subjects. This padding introduces artificial "fake" voxels. The advantage of the wrap-around padding strategy is that it allows us to trace the origins of these fake voxels. To restore the true voxel activation map, we trace their origins and assign their values as the maximum the original and artificial activations. This approach ensures that the restored activation map accurately represents the contributions of real voxels while mitigating the impact of synthetic padding.
retain the activations of the real voxels, while for the fake voxels,
Finally, this process is repeated across all samples containing the object of interest, enabling us to investigate voxel semantic selectivity, as illustrated Figure3of the main paper. To achieve the 3D activation like Figure in the main paper, we can just map each flattened voxel back to 3D brain space using the provided nsdgeneral atlas. This methodology allows us to map neural activations back to their voxel-level origins, providing insights into the relationship between neural representations and object recognition.
A.9 Technical Details on Figure
We first computed the pixel occupation ratio for all training images containing the object chair. This ratio was derived by dividing the number of chair pixels (using MS-COCO annotations for masking) by the total image resolution. Since the raw pixel ratios exhibited a highly skewed distribution, applied log transformation to approximate a normal-like distribution as shown in Figure
Next, we calculated the mean yı and standard deviation of the log-transformed ratios. To partition the chairs into based categories, we defined three intervals:
Small chairs: (-oe, / 0.5g)
Medium chairs: (μ 0.5or, μ + 0.5g)
Large chairs: (μ + 0.5c, 0)
27

---

5000
3000
2000
0.0
0.2
0.4
0.6
0.8
1.0
-12
-10
-2
chair ratio
chair ratio % log scale
Figure 8: Chair pixel ratio distribution in training set
Finally, for each subject, we generated a boxplot of the predicted probabilities for the chair class, stratified by these size groups.
A.10 Technical Details on Figure
To analyze differences in subjects' sensitivity to the object cup, we first computed the pixel occupation ratio for all training images containing cup and calculated the baseline MCC subject by subject as the pre-adjusted sensitivity measure. Since each subject viewed distinct images during training, accounted for distribution shifts in both the size and frequency of cup appearances across subjects. To ensure fair comparison, we adjusted the MCC by normalizing with the subject-wise average pixel ratio. The resulting weighted MCC is visualized in Figure
A.11 More Analytical and Visual Results
A.11.1
Voxel Sensitivity
To examine voxel sensitivities, analyzed the mean (x-axis) and standard deviation (y-axis) of voxel activations across five brain ROIS for objects bench and chair, as presented in Figure/A.11. The results indicate quadratic relationship in voxel sensitivity across these regions, allowing us to classify voxels into three distinct groups based on their mean activation and variability (standard deviation). Each group reflects unique role in semantic decoding within visual regions:
Bystanders This group, characterized by the lowest mean and standard deviation, consists of voxels that consistently contribute minimal information to the semantic decoding of visual stimuli. These voxels are either not responsible for distinguishing specific objects (bench and chair) or likely located in regions less involved in object discrimination, and instead providing generalized but stable responses across diverse stimuli.
Discriminators This higher mean and the highest standard deviation group includes voxels that show selective, highly variable responses, playing key role in differentiating between features and supporting object-specific sensitivity. These voxels likely drive the flexibility needed for nuanced and accurate decoding of semantic information in visual stimuli.
Supporters - The highest mean, low standard deviation voxels, characterized by strong, consistent activation, likely represent core object features and provide a stable foundation for robust and invariant decoding across subjects. They likely provide stable, foundational support for correctly classifying objects across different conditions.
These findings suggest that voxel sensitivity patterns vary across the visual hicrarchy, with each group contributing distinct vision cooomatiio--rocessiii roles in object recognition in the brain.
A.11.2 Single-subject 1D Activation Pattern
Similar to Figure in the main paper, we provide more visualization results on ID Object-Vore Voxel activation below:
28

---

righe
08
00
Figure 9: Object- Voxel activations (std v.s. mcan) by vision ROIs for subj01
29

---

ا Mn
Me
025
0.75
030
immmon Mahales
h
0.35
Nhoww Whra
Figure 10: 1D Object- Voxel activations by brain vision ROIs for subj01.
30

---

120
nighr valo
033
0.1
035
snule
1.0
0:3
639
3.30
Mum
b fu "Mmummton
mmm
MMM
0.5
m
M
Mordi
Figure 11: ID Object-Voost Voxel activations by brain vision ROIs for subj02.
31

---

10
Hight Vuie
ANA
Mn
8
Mmm
02-
84-
n
Lma
dnn A.
MMM
Ment
044
1250
2550
13227
Object-Voxel Figure 12: 1D Object- Voxel activations by brain vision ROIs for subj03.
32

---

0.3
0.7
8 0.50
0.25
sndd
Mhe when
035
8.5
8.00
823
Jum
Thart M Mnnaan
hanh
Figure 13: ID Object-Vosss Voxel activations by brain vision ROIs for subj04.
33

---

9.3
Mas Mod h pern
0.2
Mar D Man
Nam الل سua
8
0.25
0.35
022
0.2
500
lr
^
vad
0.2
0.71
0.2
Figure 14: ID Object-Vosxs Voxel activations by brain vision ROIs for subj05.
34

---

0.5
0.3
0.55
3.50
0.23
Manaha Nn
6.75
8
My
0.2
MM
Ml
Uhi
825
Hall
500
un Men
Munt Wuw
Nan
ma
Figure 15: ID Object-Voxel activations by brain vision ROIs for subj06.
35

---

High Vuie
M
Vam wW M
64-
mn mm hn me
03-
08-
02
Mr Mm
Loveld
84-
02-
0.6
wheh
02-
Wh
04-
Mn Phap
Manlhh
D0
2229
12805
Figure 16: 1D Object-
Object-Voxel Voxel activations by brain vision ROIs for subj07.
36

---

0.3
0.5
075
3.50
035
8
Mini
0.23
اياي M mm wa
0.22
sypel My
0.5
wanlli ulk numman
mm
035
835
Mm
0.75
1y
men Mu mmm.
0.75
0.50
0.3
225
Wer Mum.
Warl wornd haam
0.75
Figure 17: ID Object-Voxxt Voxel activations by brain vision ROIs for subj08.
37

---

A.11.3
Cross-subject 3D Activation Pattern
Similar to Figure 4 in the main paper, we provide more visualization results on 3D Object-Voxel Voxel activation below:
Figure 18: 3D Object- Voxel activations of person, bicycle, car, and motorcycle for subj01, subj03, subj04, and subj05.
Figure 19: 3D Object-Voox Voxel activations of airplane, train, truck, and boat for subj01, subj03, subj04, and subj05.
38

---

Figure 20: 3D Object- Voxel activations of boat, traffic light, hydrant, and stop sign for subj01,
subj03, subj04, and subj05.
Figure 21: 3D Object-Vors Voxel activations of bench, bird, cat, and dog for subj01, subj03, subj04, and subj05.
39

---

regg
Figure 22:
Variations subjects' attention to different objects. Four objects: person, cup, chair, and dining table are selected for visualization.
A.11.4 Variations in Subject Attention
Similar to Figure 4 the main paper, we provide more visualization results on variations in subjects' attention different objects in the same image. The leftmost image shows the visual stimulus. Plots in the second column represent the shared attention across all subjects, and the remaining eight columns show the residual, subject-specific attention alongside predicted probabilities to compare recognition confidence and priority.
40

---

Original maage
Cart
Cae PM
Cae 8004
Figure 23: Variations in subjects' attention different objects. Four objects: fur. knife, and dining table are selected for visualization.
41

---

Ongin
Poerr
M
Diggna Bay
PMN
1000
2000
8004
1000
10M
Enggia MAya
1.00
+200
6001
485
1,01
KAM
1000
1200
010
1110
Eniginal
MACE
Figure 24: Variations subjects' attention different objects. Three objects: person, car, and truck are selected for visualization.
42

---

B
Ciyy
8005
1:00
440
INT ERIN
Figure 25: Variations subjects' attention different objects. Three objects: person, sports ball, and tennis racket are selected for visualization.
43


Reviews:
[{"review_id": "v2oDDrw3ed", "reviewer": "Reviewer_7Y8L", "summary": "This paper proposes iMIND, a dual-decoding framework for neural signal interpretation that disentangles fMRI data into subject-specific and object-specific components through orthogonal basis transformation. The method achieves state-of-the-art performance in both biometric identification and semantic classification tasks while providing interpretable voxel-object activation patterns and revealing individual differences in visual attention.", "questions": "1. The core claim of your method is that Zsubj and Zobj are properly disentangled, but no empirical validation is provided. Can you compute and report the linear correlation coefficients between these components? What is the mutual information between them? \n2. Please clearly specify whether the voxel-object activation patterns and attention variation analyses (Sections 3.4-3.5) are based on the fMRI-only model or require the fMRI+Image multimodal version. \n3. Can you include comparisons with VAE-based self-supervised fMRI encoders and other recent neural representation learning methods? The current baselines are insufficient for a comprehensive evaluation.", "limitations": "The authors should more thoroughly address several limitations: (1) evaluation is limited to a single dataset, (2) the lack of empirical validation for the claimed disentanglement, and (3) the dependency of interpretability analyses on multimodal inputs needs clarification.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n1. The orthogonal basis transformation approach for disentangling subject-specific and object-specific neural features is theoretically sound and well-motivated.\n2. The paper demonstrates superior performance across multiple metrics (mAP, AUC, Hamming distance) and provides extensive visualizations of neural activation patterns, attention variations, and cross-subject consistency.\n3. The voxel-object activation fingerprints and subject-specific attention pattern analysis provide valuable insights into individual differences in visual processing, which is rare in neural decoding literature.\n4. Achieves state-of-the-art performance in semantic decoding (mAP: 0.310 vs 0.258 for previous best) and near-perfect biometric classification (ACC: 0.999).\n\nWeaknesses\n1.  The motivation connecting fMRI research to autonomous driving and medical diagnosis annotation problems is unconvincing and irrelevant. The paper should focus on neuroscience motivations rather than unrelated computer vision applications.\n2. The paper lacks comparison with other self-supervised neural encoding methods, particularly VAE-based approaches for fMRI representation learning. The biometric decoding baselines are too simplistic.\n3. Despite the theoretical completeness of the orthogonal decomposition, the paper fails to empirically validate that Z_subj and Z_obj are actually disentangled. Critical spillover tests are missing, such as computing linear correlation coefficients between the two components or measuring mutual information to verify independence.\n4. It's ambiguous whether insightful neural activation and attention analyses depend on the fMRI-only model or require the fMRI+Image multimodal version.\n5. Only evaluated on NSD dataset; generalization to other fMRI datasets remains unclear.", "was_revised": true, "final_justification": "Thanks for the detailed response, I would keep the positive rating.", "rebuttal": "We thank Reviewer 7Y8L for your constructive feedback. Below are our responses. (W, Q, and L stand for Weakness, Question, and Limitation, respectively.)\n\n# [W1] Motivation Rephrase \nThanks very much for your great suggestion. We admit that the original introduction placed too much emphasis on downstream applications such as autonomous driving and medical diagnosis, which may **distract** from the core neuroscience motivations of our work. In the final version, we will rephrase the introduction to better highlight key neuroscience-driven tasks—such as **how the human brain encodes, disentangles, and generalizes semantic representations across individuals**—and to emphasize the role of fMRI as a powerful non-invasive tool for probing the visual processing pipeline and subject-specific perception mechanisms. We will highlight our goal of contributing to a deeper understanding of **multi-subject neural decoding** and **inter-individual variability** in visual perception. We will retain only one or two sentences (in the conclusion) about potential long-term implications for human-AI interaction, as suggested by recent NeurIPS trends in brain-inspired AI.\n\n# [W3 Q1 L2] Empirical Validation of Disentanglement \nThis is a great point to make our work complete. To rigorously validate disentanglement, we have included the following results:\n- **Near-Zero Linear Correlation**: We calculate the correlation of each feature vector of $\\mathbf{Z}\\_\\text{obj}$ and $\\mathbf{Z}_\\text{subj}$ and report their mean and standard as $0.0027\\pm0.0012$, supporting our disentanglement claim;\n- **Orthogonality Enforcement**: The Frobenius norm $||\\mathbf{B}\\_\\text{obj}\\mathbf{B}\\_\\text{subj}^\\top||= 2.7997\\times 10^{-5}$ along with our orthonormal loss $\\mathcal{L}_\\text{orth}=7.2874\\times10^{-5}$ confirms the role of learned $\\mathbf{B}$ as an orthonormal basis for object-subject disentanglement.\n\n# [W2 Q3] Baseline Comparisons \nWe have expanded our comparisons to include state-of-the-art self-supervised fMRI encoders and strengthened our biometric decoding baselines. Below, we summarize the key results:\n- For **semantic decoding** and **neuro encoding** methods, we evaluated **three more self-supervised** fMRI encoders, all trained with identical data/hyperparameters as shown below. We found that the **Masked Autoencoder (MAE)**, which serves as the backbone of our model, outperforms the other three self-supervised methods. This is primarily due to its ability to **reduce noise and redundancy inherent in fMRI data**, as also observed in Mind-Vis. Furthermore, since prior work on semantic decoding—particularly for **both classification and multi-subject modeling**—is limited, we include MindBridge for comparison, even though it was originally designed for reconstruction tasks. To adapt MindBridge for classification, we extract its output features and train a linear classification head on top. This comparison confirms the need for **task-specific architectures like ours that are explicitly designed for semantic decoding across subjects**.\n\n| **Backbone** | **Modalities** | **mAP ↑** | **AUC ↑** | **Hamming ↓** |\n|:--------------:|----------------|-----------|-----------|:----------------:|\n|MindBridge| fMRI+Image|.1499|.8001|.0286|\n||fMRI+Image+Text|.1521|.7999|.0289|\n|AE|fMRI|.1474|.8042|.0377|\n||fMRI+Image|.7051|.9725|.0169|\n|DAE|fMRI|.1435|.8012|.0379|\n||fMRI+Image|.7135|.9748|.0159|\n|VAE|fMRI|.1290|.7230|.0374|\n||fMRI+Image|.6951|.9683|.0165|\n|MAE (Ours)|fMRI|**.3095**|**.9132**|**.0271**|\n||fMRI+Image|**.7836**|**.9840**|**.0121**|\n\n- For **biometric decoding**, we included **three** additional baseline models. Notably, we found that even a simple **two-layer MLP with ReLU activation and L2 regulation** can achieve perfect subject classification from fMRI voxel signals. This result provides strong empirical support for the core assumption of iMIND—that subject information in fMRI can be linearly separable at the latent feature level. Given that the Vision Transformer (ViT) offers **significantly greater feature extraction capacity** than a shallow MLP, this finding reinforces the validity of our disentanglement approach.\n\n|**Layer(s)**|**Regulation**|**Activation**|**ACC**|**MCC**|\n|:--------------:|:----------------:|:----------------:|:-----------:|:-----------:|\n|1|—|—|.2833|.1812|\n||L2|—|.3767|.2899|\n||L2|ReLu|.5726|.5262|\n|2|L2|ReLu|**.9999**|**.9999**|\n\n# [W5 L1] Dataset Generalization \nWe appreciate your comment on the generalization of the data set. We agree that demonstrating cross-dataset generalizability is an important step toward establishing a more comprehensive and robust framework.\n\nIn exploring additional datasets, we considered both the BOLD5000 and HCP movie-watching datasets, which involve naturalistic visual stimuli and multiple subjects. However, two key challenges currently limit their use in our study:\n- **Lack of semantic decoding benchmarks (BOLD5000)**: Unlike NSD, which provides well-curated annotations and is widely used for semantic decoding tasks, BOLD5000 lacks **standardized, reproducible benchmarks** for object-level classification. This makes it difficult to **evaluate and compare performance** under our semantic decoding setup.\n- **Computational challenges (HCP)**: The HCP movie dataset includes **158** subjects and more than **800** semantic concepts. Given the need to model multi-subject disentanglement and high-dimensional voxel features, applying our method to this dataset is **computationally demanding**. Unfortunately, due to limited time and resources during the review period, we were unable to complete a full evaluation.\n\nThat means, we recognize the importance of cross-dataset validation and plan to explore generalization on datasets like HCP and BOLD5000 in future work. We will explicitly acknowledge this limitation in the revised manuscript and elaborate on it in the Limitations and Future Work section."}, {"review_id": "OpcznRxpSu", "reviewer": "Reviewer_T9by", "summary": "This paper introduces iMIND, a framework that aims to address the core challenge of visual brain decoding (using fMRI), particularly how to handle significant neural variability across individuals.\n\nThe framework's central idea is to explicitly disentangle the \"subject-specific\" and \"object-specific\" components of neural representations through a dual-decoding task (biometric and semantic decoding). Specifically, the model comprises three key steps:\n\n1. A ViT-based masked autoencoder is used to establish a shared latent neural representation space across all subjects.\n\n2. A learnable orthonormal basis transformation method disentangles the features in this shared space into two orthogonal subspaces: a \"subject-specific\" one and an \"object-specific\" one.\n\n3. These disentangled features are then utilized for biometric identification (identifying the subject) and semantic classification (identifying the object the subject perceived).\n\nExperiments demonstrate that iMIND achieves SOTA performance on the semantic decoding task and near-perfect accuracy on the biometric identification task. More importantly, the paper provides extensive analysis showing that the model can generate neuroscientifically meaningful \"voxel-object activation maps.\" It also uncovers differences in attentional patterns among subjects viewing the same image or category, offering data-driven insights into the mechanisms of the brain's visual processing.", "questions": "1. Please see the points raised in the Weaknesses section above.\n\n2. One more question: The authors state that Table 1 lists few methods due to the limited research on semantic neural decoding. However, other methods like MindBridge [1] could likely be adapted to create a baseline by substituting the task head for classification. It would be interesting to see how other methods would perform on this task.\n\n[1] MindBridge: A Cross-Subject Brain Decoding Framework, CVPR 2024.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. Elegant and Effective Method Design: The idea of using a learnable orthonormal basis to decompose the feature space is elegant and technically sound, and the ablation study proves the effectiveness of this disentanglement. This \"instructive strategy\" is a significant conceptual advance over the prevailing \"suppressive strategies\" in the field.\n\n2. Strong Experimental Results: The performance of iMIND on the semantic decoding task is superior to that of other methods. In the novel biometric decoding task, where no prior models exist for comparison, iMIND achieves near-perfect accuracy, providing powerful evidence for the effectiveness of its feature disentanglement framework.\n\n3. In-depth and Insightful Analysis: The paper does not stop at reporting performance metrics; it dedicates significant effort to an interpretable analysis of the model, which is highly commendable. This includes:\n> * By visualizing voxel-object activation maps, the paper successfully connects the model's learned representations with findings from neuroscience, such as the functional organization of the higher visual cortex and its sensitivity to specific categories like \"person\".\n> * By examining prediction confidences and attention maps for objects like \"chair,\" \"cup,\" and \"fork,\" the paper draws insightful conclusions about visual salience and cognitive strategies under temporal constraints.\n\n\n\n# Weakness\n\n1. The \"Linear Entanglement\" Assumption: The paper's core theory rests on a key assumption: that subject-specific and object-specific information are \"linearly entangled\" in the latent space. While the authors provide a justification (that deep networks aim to learn linearly separable features), this remains a strong assumption that may oversimplify the complex, non-linear dynamics of the brain. It is recommended that the authors discuss the limitations of this assumption more prominently in the main paper and speculate on how the model's performance might be affected if the assumption does not hold.\n\n2. Large Performance Gap between fMRI+Image and fMRI-only Settings: Table 1 shows a massive performance gap between iMIND (fMRI-only) with an mAP of 0.310 and iMIND (fMRI+Image) with an mAP of 0.784. This large difference indicates that the image modality plays a decisive role. While the paper explains this is achieved via a cross-attention mechanism, it could be clearer about the precise role of the fMRI signal in this fusion. For instance, does the fMRI primarily provide a \"where to look\" (spatial attention) signal for the powerful CLIP features to then \"identify\"? Or is there a more complex synergy at play? Clarifying this would sharpen the paper's contribution.", "was_revised": true, "final_justification": "I am satisfied with the rebuttal and am happy to keep my original positive rating.", "rebuttal": "We sincerely appreciate Reviewer T9by for your valuable feedback. Below are our responses. (W and Q stand for Weakness and Question, respectively.)\n# [W1] Linear Entanglement Assumption\nWe fully acknowledge that the brain’s functional dynamics are **fundamentally non-linear and complex**. Our assumption—that subject-specific and object-specific components are linearly entangled at the latent feature level—is **a simplifying inductive bias** introduced to enable interpretable, computationally tractable disentanglement. We **do not* claim it fully captures the richness of brain representations; rather, it serves as **a first-order approximation** that enables clear factorization of subject identity and semantic content from fMRI signals.\n\n## Empirical Justification \nWhile we agree this assumption may not fully capture the brain's non-linear functional structure, we provide the following empirical evidence to support its pragmatic utility in our framework:\n- **Near-perfect biometric decoding (Table 2 in main paper)**: We achieve 99.9% accuracy in subject classification, indicating subject-specific features **can be linearly separated** from entangled fMRI features in the context of multi-subject learning;\n- **Subject-invariant object activations (Fig. 1 in main paper)**: After disentanglement, we observe that object activation patterns in the latent space exhibit **minimal cross-subject variability**, indicating that our method successfully distills **stable, shared semantic representations**—further supporting the effectiveness of linear disentanglement.\n- **Shallow MLP baselines confirm separability (Table below)**: We additionally implemented two-layer MLPs with ReLU activation for subject classification from raw fMRI. Surprisingly, even this **lightweight non-linear** model achieved near-perfect accuracy, validating the notion that subject-specific features are already largely linearly separable at the voxel level. This further justifies our design, especially considering our ViT encoder has **significantly higher representational capacity** compared with MLPs.\n\n| **Layer(s)** | **Regulation** | **Activation** | **ACC**   | **MCC**   |\n|--------------|----------------|----------------|-----------|-----------|\n| 1            | —             | —             | .2833    | .1812    |\n|             | L2             | —             | .3767    | .2899    |\n|             | L2             | ReLu           | .5726    | .5262    |\n| 2            | L2             | ReLu           | **.9999**| **.9999**|\n\n## Discussion of Limitations\nWe agree it is important to discuss potential implications if the linearity assumption **does not fully hold**:\n- If subject-object interactions are fundamentally **non-linear**, the disentangled object representation $\\mathbf{Z}_\\text{obj}$ may still **retain residual subject-specific** information, potentially introducing **subject bias** in semantic decoding and **diminishing our model's generalizability* for unseen subjects.\n- Conversely, enforcing strict linear disentanglement may **suppress relevant non-linear** object features in $\\mathbf{Z}_\\text{obj}$, potentially **smoothing out** sharp voxel-object modulations or **degrading decoding performance** for fine-grained categories.\n\n## Future Directions \nWe are actively exploring **non-linear disentanglement techniques**—e.g., manifold learning, kernel-based projection, or contrastive objectives—to model richer subject-object interactions while preserving **interpretability**. Incorporating auxiliary data such as **structural MRI** or **resting-state connectivity** could also regularize the subject basis more effectively. As suggested, we will make these limitations and future directions more explicit in the revised manuscript, particularly in the Discussion and Limitations sections.\n\n# [W2] Role of CLIP and fMRI\nWe thank you for this thoughtful suggestion. Further clarifying the role of the fMRI signal in the cross-attention fusion is essential for us to highlight our contribution.\n\nTo answer directly: in our fusion framework, the fMRI signal plays the role of **\"what to attend to\"** rather than simply \"where to look.\" The fused output remains **fundamentally neural in nature**, modulated by subject-specific brain responses rather than being dominated by CLIP features. This is reflected in the design of the cross-attention module (Eq. 9), where **CLIP embeddings serve as queries**, while **fMRI features act as keys and values**—allowing neural responses to govern which semantic components are emphasized.\nThis design achieves two complementary goals:\n- **Semantic Prioritization**: CLIP features encode **subject-invariant, stimulus-driven** semantic content and contain rich spatial and conceptual structure. During cross-attention, CLIP features **query the fMRI signal**, and the fMRI effectively **responds with what concepts or object components are prioritized**, based on subject-specific neural responses. In other words, attention weights reflect which parts of the CLIP space align with neural activations.\n- **Subject-Specific Modulation**: As shown in Fig. 4 (columns 3–10), residual attention maps reveal that fMRI signals modulate the CLIP-based predictions in a **subject-dependent** manner. This synergy allows our model to preserve semantic content from CLIP while capturing **individual-specific neural patterns**, highlighting how different subjects may focus on different semantic attributes of the same visual input.\n\nThus, rather than acting as a passive filter or spatial mask, the fMRI features dynamically shape which aspects of the CLIP semantic space are elevated—creating a **bi-directional synergy**. CLIP provides **a semantically grounded reference frame**, while fMRI reveals how the **brain selectively engages** with that semantic structure.\n\n# [Q2] Comparison against MindBridge\nWe thank you for this excellent suggestion and fully agree that including additional adapted baselines could strengthen the evaluation. In response, we adapted **MindBridge**—a reconstruction-oriented model—into a semantic decoding baseline by replacing its generative output head with a linear multi-label classification head, trained on its latent representations. While MindBridge was not originally designed for classification, this setup enables a fairer comparison of its representational utility under our semantic decoding task.\n\nDue to time and resource constraints, we followed MindBridge’s original training protocol and evaluated the model on **4 subjects (1, 2, 5, 7)**. In contrast, our model (iMIND) was trained and evaluated on **all 8 subjects**, using the same data split and evaluation metrics. The results are shown below:\n| **Methods** | **Modalities** | **mAP ↑** | **AUC ↑** | **Hamming ↓** |\n|--------------|----------------|-----------|-----------|----------------|\n| MindBridge   | fMRI+Image     | .1499| .8001| .0286     |\n|              | fMRI+Image+Text| .1521| .7999| .0289     |\n| iMind   | fMRI           | **.3095**| **.9132**| **.0271**     |\n|              | fMRI+Image     | **.7836**| **.9840**| **.0121**     |\n\nDespite the limited setup, these results demonstrate a **substantial performance gap**, especially in the **fused modality setting**. This gap reinforces a central argument of our paper: **semantic decoding—particularly in a multi-subject setting—requires models explicitly designed to handle subject variability and semantic disentanglement**. Simply reusing a reconstruction model, even with strong latent priors, is **insufficient**.\n\nWe will include this analysis and baseline comparison in the final version to strengthen the experimental evaluation and better contextualize iMIND’s advantages."}, {"review_id": "vJIwZv0FFF", "reviewer": "Reviewer_6F63", "summary": "Neural coding is a long standing question. While recent generative models provided a powerful tool for image reconstruction, it's performance largely depend on the generative model rather than the interpretability of neural activity. Furthermore, it is unclear how those models can handle subject-wise variability. While most work has been focused on minimizing inter-subject variability for generalizability , limited work aims to explicitly partition subject-specific differences and shared task-relevant features. \n\nThis work tried to disentangle subject-specific and stimulus/object-specific information by carefully designing a two stage training model and orthogonalization-constrained loss to identify corresponding subspaces in learned space of neural features. The first stage of the model is a ViT-based encoding model to identify the latent representation space of fMRI voxels in response to images. The second stage of training is dual-decoding stage with orthogonal constrains of the latent features. They achieved better performance in semantic decoding and biometric decoding tasks compared with some other models. As a standard control, model explanation and ablation analysis have been provided.\n\nKey contributions are: \n1. Modeling: a new dual-step encoding-decoding framework for subject-specific and object-specific components disentanglement. \n2. Analysis: analysis of difference and variability across subjects.", "questions": "**Opportunities for Improvement:**\n- add appropriate regularization to baseline model for a fair biometric decoding performance comparison\n- Assumption of subject and object specific information are linearly entangled may need more justification. Further explanation and discussion of linear entanglement hypothesis.\n- line 2-9: Description of gap and contribution is ambiguous. The method or idea could potentially offer more understanding but results of the paper are still limited in interpreting visual functionalities.\n- $d_{obj}$ is treated as a user-defined value (line 146) but the interpretation of hyperparameter influence is weak (A.6). Further analysis of the subspace should be investigated from perspective of representation, dimensionality and dynamics.\n- Biometric decoding is measured through subject classification. Baseline model may need more regularization such as sparsity to prevent overfitting due to high-dimensional input. (3.2) \n- line 352: No evidence to show that the basis transformation is robust.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**:\nThe strengths of this paper includes 1. a new decoding framework for subject and semantic classification; 2. visualization of object-voxel activation patterns; 3. analysis of difference and variability across subjects. \n\n- Quality: The effort to partition subject-specific vs object specific latents is of interest to the general community. The concern raised about reconstruction method for neural coding deserve more discussion as well.\n- Clarity: The paper is clearly written and well organized, including its appendix. Loss function, model training, evaluation metric are clearly explained as well. \n- Originality: This paper proposed a new model to disentangle subject-specific and stimulus/task-relevant information through data-driven multi-subject models.\n- Significance: The question of subject-wise variability has been a big hurdle in field of neural coding and brain-machine interface. Finding a way to disentangle subject-specific variance from stimulus-specific variance is of great importance. Previous work like the Fisher information, Factor analysis and latent analysis have this flavor, but need to make many assumptions. This method suggested by the paper could be a new way to solve this problem. \n\n*Weaknesses**:\n- The concern that reconstruction method heavily depend on the generative model and cannot fully reveal neural coding properties is valid. However, directly reduce the dimensionality to object level or semantic level representation in fMRI does not directly address the question. There are limitations for models that can only perform object classifications as well. It is unclear what kind of features are captured for object discrimination. \n\n- Subject and object classification loss is task-specific and may reveal only partial variance from individual-subject and stimulus. Once the latent space is disentangled, linear decoding performance could be high but the variance may not be complete. In other words, there are other variance, which could be pushed to null-space in the latent that got discarded by this method.", "was_revised": true, "final_justification": "Based on the rebuttal and discussion. I would like to keep my original rating: 4.", "rebuttal": "We thank Reviewer 6F63 for your constructive feedback. Below are our responses. (W and Q stand for Weakness and Question, respectively.)\n\n# [W1] Semantic Decoding and Feature-level Interpretability\nWe appreciate your insightful comments on the limitations of existing approaches and the challenges of interpretability in object-level neural decoding.\n\nMany recent works in neural decoding have **prioritized generating high-fidelity** visual reconstructions from fMRI data. While these outputs are visually impressive, such methods often rely heavily on pretrained generative models and focus primarily on image realism rather than understanding the **neural signals** themselves. As a result, they contribute **marginally** to advancing our knowledge of how the **brain encodes visual information**. This fundamental gap motivates our work: rather than asking **what the image looks like**, we ask **how semantic object representations are encoded and how they vary across individuals at the neural level**.\n\nIn our work, we take **a discriminative decoding approach** that simultaneously predicts object categories (semantic decoding) and subject identities (biometric decoding) from fMRI signals. We agree that object classification alone **does not** fully capture the richness of neural representations. However, we argue that our framework provides **a meaningful step** toward interpretability by enabling **structured**, **task-relevant decomposition** of neural information.\n\nOur strong performance in semantic decoding allows us to leverage **post-hoc interpretability tools** (i.e., GradCAM, Attention Rollout) to analyze both model behavior and brain function. Specifically:\n- **Voxel-object activation maps (Fig. 2–3)** and **subject-specific attention analyses (Fig. 4–6)** uncover both shared and individualized patterns of neural activity in response to visual stimuli；\n- **These analyses reveal attention dynamics and voxel selectivity** that offer a data-driven perspective on how the brain represents object-level semantics—an aspect typically overlooked in reconstruction-based methods;\n- By combining **subject-object disentanglement** with **voxel-level interpretation**, we address a largely underexplored question: **how consistent are object-specific neural encodings across subjects, and where do individual differences emerge?**\n\n# [W2] Variance Pushed to the Null Space\nWe appreciate this observation. It is indeed true that our disentanglement and decoding objectives **prioritize task-relevant variance**—specifically, for subject identification and object classification—which may lead to the **omission* of other latent dimensions not directly aligned with these tasks. This trade-off is inherent to discriminative frameworks that aim for linear separability in service of specific objectives.\n\nAs clarified in Section 2.3 and Appendix A.2, our method is designed to **isolate** linearly separable components of neural representations relevant to subject and object decoding, rather than **capturing the entire variance** present in the fMRI signal. It is therefore expected that residual information—such as low-variance components or task-irrelevant patterns—may **reside in the orthogonal null space** of the learned latent basis.\n\nWe view this as a **valuable opportunity** rather than a limitation. In fact, one of the longer-term goals of our framework is to first isolate task-relevant components in a structured, interpretable manner, and then **iteratively explore** the null space to investigate what information remains. This could include:\n- Unsupervised or contrastive objectives to identify structure in residual variance;\n- Temporal dynamics or contextual dependencies not captured by static object categories;\n- Subtle cognitive or emotional correlates that are not labeled but may still be encoded in the data;\n-Or even noise patterns that correlate with session-specific or scanner-specific artifacts.\n\nWe believe this two-stage process—(1) identifying interpretable, task-relevant features and (2) probing the remaining space—is a promising direction toward more **transparent, comprehensive, and human-aligned** neural decoding models, and we plan to pursue this in future work.\n\nRegarding your concern about **transparency at the latent feature level**: we strongly agree that interpretability of individual latent dimensions remains **a fundamental challenge in deep learning** regardless of tasks or modalities. Our method **does not claim to assign direct semantic meaning** to each latent unit. Instead, we probe the structure of the latent space through aggregate behavioral and voxel-wise analyses. This work provides a framework that moves toward more interpretable models, and we view detailed latent-space analysis (e.g., basis axis meaning, feature attribution, spectral structure) as an important direction for future investigation.\n\n# [Q1 Q5] Additional Subject Baselines \nThank you for the valuable suggestions regarding regularization for more robust and fair biometric decoding baselines.\n\nIn response, we conducted additional experiments using MLP classifiers with varying levels of regularization and architectural complexity. As shown in the table below, we found that adding **L2 regularization and ReLU activation** significantly improves subject classification performance using raw fMRI voxel inputs. Notably, even a simple two-layer MLP with ReLU achieves near-perfect accuracy and MCC. We interpret this result as strong **empirical support** for our assumption that subject information in fMRI signals can be linearly disentangled from the joint latent space. We will elaborate on this point in our response to the **linearity assumption**.\n\n|**Layer(s)**|**Regulation**|**Activation**|**ACC**|**MCC**|\n|:--------------:|:----------------:|:----------------:|:-----------:|:-----------:|\n|1|—|—|.2833|.1812|\n||L2|—|.3767|.2899|\n||L2|ReLu|.5726|.5262|\n|2|L2|ReLu|**.9999**|**.9999**|\n\n# [Q2] Linear Assumption Justification\nWe appreciate your concern regarding the justification of our linear entanglement hypothesis. Due to **space constraints** (10,000 characters), we were unable to include a detailed discussion in this response. We are truly sorry about this. However, **Reviewer T9by** raised the same point, and we have provided a thorough response there. We kindly refer you to our reply to **T9by** for further clarification on the detailed explanation and discussion of our linear entanglement hypothesis.\n\n# [Q4 Q6] Subspace Analysis and Robust Transformation \nWe thank you for raising the insightful questions regarding the interpretation of the object-specific latent dimensionality $d_\\text{obj}$ and its effect on the model’s representation and dynamics. \n\n## Interpretation of $d_\\text{obj}$ and Subspace Analysis\nWhile $d\\_\\text{obj}$ is treated as a user-defined hyperparameter in our formulation, we conducted **an additional analysis** to investigate its representational efficiency using **PCA** on the object-specific features $\\mathbf{Z}\\_\\text{obj}$ with $d_\\text{obj}=700$. We computed the **cumulative variance explained** and found that:\n- The first **10 components** explain approximately **92.27%** of the variance;\n- **30 components** capture over **98.05%**, and\n- Over **99.9%** of the total variance is explained by fewer than **120 dimensions**.\n\nAlong with the performance stability shown in our ablation in Appendix A.6, this suggests that although we allocate 700 dimensions to $\\mathbf{Z}\\_\\text{obj}$, the intrinsic dimensionality of the object-specific subspace is **significantly lower*. The fact that a small number of principal components **dominate** variance implies that the object subspace learned by our model is **highly structured, low-rank, and compressible**, indicating **meaningful latent representations** rather than arbitrary overparameterization. The model itself is **not sensitive** to the exact choice of $d_\\text{obj}$ beyond a minimum threshold. These findings open up exciting potential for further exploration. In future work, we plan to study the **semantic interpretability** of leading PCA axes, examine temporal and **dynamic properties** of the object representations, and probe whether certain principal components correspond to high-level object categories or visual features.\n\n## Robustness of Basis Transformation\nRegarding line 352, we agree that the term **robust** should be more precisely clarified. This robustness suggests that the disentangled space effectively captures semantic content across a range of dimensions in terms of dimensionality. We will revise the wording to **effective** or **consistent** if you feel robust is over-claimed here.\n\n# [Q3] Abstract Revision\nThanks for pointing this out. We will revise our abstract to avoid ambiguity and include the following contents accordingly:\nWhile reconstruction tasks leverage powerful generative models to produce high-fidelity images from neural recordings, they often pay limited attention to the underlying neural representations and rely heavily on pretrained priors. As a result, they provide limited insight into which neural components encode semantic content or how these representations vary across individuals. Our work shifts the focus toward understanding the structure and variability of semantic encoding in the brain, rather than merely replicating visual appearance.\n\nIn addition, we deeply appreciate your recognition of our work's potential to offer deeper insights into visual functionalities. To further strengthen the paper, we would be greatly thankful for a brief clarification on which specific aspects of neural visual processing could be more thoroughly illuminated. Your expert suggestions would be invaluable in guiding us to enhance the biological interpretability of our findings, whether in this revision or future work, while maintaining our core focus on discriminative decoding."}, {"review_id": "rzFRSJmf0o", "reviewer": "Reviewer_1WBp", "summary": "the authors propose iMIND, the insightful Multi-subject Invariant Neural Decoding model, that substantially advances the Neural Transcoding field. it is composed of a VIT based encoder which extracts features that will be decoupled into their object and subject-specific components to tackle the user identification and multi-label object classification. \nresults are promising, and the visualization of the attention overlay obtained from the model on the images is stunning.\na sufficient ablation study has been done, and the supplementary material completes the manuscript.", "questions": "- could you come up with a figure of the architecture?\n- what are the pros of using a ViT-based encoder instead of other architectures?\n- you did the subject identification task. why didn't you benchmark your model on this task on other fMRI datasets?\n- in Tab. 1, what would the results have been if you also used text inputs, like in [50]?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1 - the task is quite novel and interesting, and the paper is well-written\nS2 - there are both quantitative (Tab. 1-2) and qualitative (Fig. 1-6) results shown, which seems promising and sound\nS3 - there is a detailed ablation study\nW1 - a figure of the proposed framework would have ease the flow and clarity of the entire manuscript\nW2 - there is no code associated to the manuscript, hindering reproducibility (!!!)", "was_revised": true, "final_justification": "After careful examination of the brilliant authors' rebuttal, my only concern for this manuscript remains the absence of a figure of the proposed architecture.\nOverall, I confirm my previous rating and I am eager to recommend this paper for acceptance.", "rebuttal": "We sincerely appreciate Reviewer 1WBp for your questions and suggestions. Below are our responses. (W and Q stand for Weakness and Question, respectively.)\n\n# [W1 W2 Q1] Architecture Figure and Code \n\nThanks very much for your great suggestion. At this moment, NeurIPS does not permit the inclusion of **rich media** or **external links** during the rebuttal phase this year. We **have prepared an architectural figure** and will include it in the final version to make the paper easy to follow. In addition, we would be happy to **share relevant portions of our code** in Markdown format during the discussion phase if you have concerns regarding **reproducibility**.\n\n# [Q2] ViT-based Encoder Justification \nOur choice of a ViT-based encoder for fMRI is motivated by both practical and representational considerations:\n- **Global Context Modeling**: Our fMRI inputs are flattened into **1D** vectors, which discards the original **spatial layout** of voxels. Unlike CNNs, which rely on **local receptive fields**, Vision Transformers (ViTs) have **global receptive fields**, enabling them to model **long-range dependencies across brain regions**. This helps **compensate** for the loss of **spatial structure** in our preprocessed inputs.\n- **Modality Alignment with CLIP**: Since the visual features we use are extracted from a pretrained **ViT-B/16 CLIP model**, using a ViT-based encoder for fMRI improves **architectural compatibility** and **facilitates cross-attention fusion**. This design choice reduces the **modality gap** and encourages more effective **alignment** between neural and visual representations.\n\n# [Q3] More Datasets \nWe appreciate your comment on the generalization of the data set. We agree that demonstrating cross-dataset generalizability is an important step toward establishing a more comprehensive and robust framework.\n\nWe introduce subject identification as an **auxiliary task** to support our main motivation: understanding the neural mechanisms of **visual processing in a data-driven manner**. This focus requires datasets that include paired natural visual stimuli and corresponding neural responses across multiple subjects. Given this requirement, only a few datasets are suitable—namely, NSD, HCP Movie, and BOLD5000.\n\nAmong them, NSD is the only one that provides a **reproducible and well-documented benchmark** for **semantic object classification** tasks. In contrast, both **HCP Movie** and **BOLD5000** lack established benchmarks for classification-based decoding, and existing studies primarily focus on **reconstruction or resting-state analysis**. Moreover, models trained on these datasets often **do not release pre-trained weights**, and the corresponding **preprocessing pipelines** are either **unavailable or not well-documented**, making fair comparisons infeasible.\n\nWe recognize the value of cross-dataset evaluation and plan to extend our experiments to HCP and BOLD5000 in future work. We will include this clarification in the final version of the manuscript.\n\n# [Q4] Text Modality \nThank you for raising this point. We chose not to include the text modality in our current model for both practical and methodological reasons.\n- **Motivational Misalignment**: Our primary goal is to study semantic decoding from **vision-related brain activity**, using only **visual cortex voxels**. In the inference stage, it is not practical to assume text data **is available** for test samples. Introducing text features **risks semantic leakage**, as object names often appear explicitly in captions (e.g., in COCO descriptions), which could bias the model to **cheat on text** rather than decoding semantic content from fMRI. This would compromise our central objective of probing the brain's visual decoding mechanisms.\n\n- **Implementation Complexity**: The CLIP visual features used in our model are extracted from the image encoder **prior to** text-image alignment, meaning they **do not share a common latent space** with CLIP text embeddings. To incorporate text features, we would need to retrieve aligned image-text pairs and **re-extract features from the jointly aligned CLIP embedding space**. Additionally, the visual embeddings **before and after alignment differ in dimensionality**, which introduces **incompatibility** with our current architecture. It is also important to note that CLIP aligns **global image and text embeddings**, whereas our model operates on **local visual features for fine-grained semantic decoding**. Incorporating text would therefore require **substantial** changes to our input representation and necessitate retraining the entire model—including the cross-attention module and downstream components—along with extensive hyperparameter tuning. Due to time constraints during the review period, we were unable to carry out this significant modification.\n\nHowever, we consider your suggestions to be a **promising direction** for investigating **brain-language functionalities** in which we are very intereste and will certainly explore them in our future work."}]

Author rebuttal:

Meta-review: The paper introduces iMIND, a dual-decoding framework that disentangles subject-specific and object-specific fMRI representations for both biometric and semantic neural decoding. The approach is theoretically well-grounded, strongly motivated, and supported by extensive experiments with state-of-the-art results and insightful neuroscientific analyses. All reviewers were positive and found the rebuttal thorough in addressing their concerns. 
**The main limitation is the lack of broader comparisons to more recent baselines, which somewhat narrows the empirical context,** while this tempers the strength of the contribution, the conceptual novelty and demonstrated effectiveness still justify acceptance.
Decision: Accept (poster)