Orthogonal Contrastive Learning for Multi-Representation fMRI Analysis
Authors: Tony Yousefnezhad
Venue: NeurIPS 2025 poster ()
Keywords: fMRI analysis
Contrastive Learning
Functional Alignment
Transfer Learning
Multi-Representation fMRI Analysis
Primary area: neuroscience_and_cognitive_science

Abstract: Task-based functional magnetic resonance imaging (fMRI) provides invaluable insights into human cognition but faces critical hurdles—low signal-to-noise ratio, high dimensionality, limited sample sizes, and costly data acquisition—that are amplified when integrating datasets across subjects or sites. This paper introduces orthogonal contrastive learning (OCL), a unified multi-representation framework for multi-subject fMRI analysis that aligns neural responses without requiring temporal preprocessing or uniform time-series lengths across subjects or sites. OCL employs two identical encoders: an online network trained with a contrastive loss that pulls together same-stimulus responses and pushes apart different-stimulus responses, and a target network whose weights track the online network via exponential moving average to stabilize learning. Each OCL network layer combines QR decomposition for orthogonal feature extraction, locality-sensitive hashing (LSH) to produce compact subject-specific signatures, positional encoding to embed temporal structure alongside spatial features, and a transformer encoder to generate discriminative, stimulus-aligned embeddings. We further enhance OCL with an unsupervised pretraining stage on fMRI-like synthetic data and demonstrate a transfer-learning workflow for multi-site studies. Across extensive experiments on multi-subject and multi-site fMRI benchmarks, OCL consistently outperforms state-of-the-art alignment and analysis methods in both representation quality and downstream classification accuracy.

Orthogonal Contrastive Learning for Multi-Representation fMRI Analysis
Tony Muhammad Yousefnezhad
'Learning By Machine
Information Management, National Bank of Canada
Edmonton AB Canada
tonyylleariingbyyacchhen..cm
Abstract
Task-based functional magnetic resonance imaging (fMRI) provides invaluable insights into human cognition but faces critical hurdles- low signal-to- ratio, high dimensionality, limited sample sizes, and costly data acquisition- are amplified when integrating datasets across subjects or sites. This paper introduces orthogonal contrastive learning (OCL), unified multi-repressntatiion framework for multi-subject fMRI analysis that aligns neural responses without requiring tem- poral preprocessing or uniform time-series lengths across subjects sites. OCL employs two identical encoders; an online network trained with contrastive loss that pulls together same-stimulus responses and pushes apart different-stimule responses, and target network whose weights track the online network via expo- nential moving average stabilize learning. Each OCL network layer combines QR decomposition for orthogonal feature extraction, locality-sensitic hashing (LSH) to produce compact subject-specific signatures, positional encoding to em- bed temporal structure alongside spatial features, and transformer encoder to generate discriminative, stimulus-aligned embeddings. We further enhance OCL with unsupervised pretraining stage fMRI-like synthetic data and demonstrate a transfer-led lunsfer-earming workflow for multi-site studies. Across extensive experiments multi-subject and multi-site fMRI benchmarks, OCL consistently outperforms alignment and analysis methods in both representation quality and downstream classification accuracy.
1 Introduction
Task-based functional magnetic resonance imaging (fMRT) is a widely used technique in neuroscience for studying brain activity during cognitive processes such decision-making, perception, and atten- (BOLD) signals while subjects engage in structured tasks, IMRI enables researchers link brain regions specific mental functions (5). Despite its potential, fMRI data present several challenges: they are high-dimensional. inherently noisy, expensive to acquire, and often limited in sample size faae-ffcttrsss that hinder the training and generalization of machine learning models [I 2 3 6 77. To mitigate these limitations, modern research increasingly relies on multi-subject fMRI datasets to improve model robustness and validity. Moreover, the growing availability of large-scale, open-access repositories such as the national
tion 2 4). By capturing
"Thi research study is conducted independently and is not connected the author's role the National Bank of Canada.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

institute of mental health (NIMH) the Human Connectome Project and OpenNEURO has made feasible to aggregate homogeneous task-based fMRI data across multiple sites, thereby increasing sample diversity and statistical power 7 However, this introduces additional complexity, includ- ing inter-subject variability, cross-site differences in scanner hardware and acquisition protocols, and population-level heterogeneity [1 [ Consequently, there is a pressing need for machine learning frameworks that can generalize across sites and subjects while being resilient to such batch effects, making the development of domain-adaptive, multi-representation learning techniques essential for real-world IMRI analysis.
Multi-subject iMRI analysis is complicated by substantial inter- inter-indiiidul variability in brain con- nectivity, as each person 's connectome exhibits unique structural and functional patterns that lead to idiosyncratic neural responses across subjects 12 To address this, functional alignment tech- niques-mov most notably hyperalignment [5] and shared response model (SRM) G I0] projectt each subject's neural responses into shared representational space using an orthogonal mapping proce- dure, effectively realigning neural signatures and improving inter-subject correspondence. These alignment strategies can be framed multi-view learning problems-case subject constitutes 'view', and methods like generalized canonical correlation analysis (CCA) identify transformations that maximize shared information across views 12
12 [3]. Recent work has extended functional alignment to multi-site fMRI studies, aiming to pool data from different scanners and populations: however, these efforts must contend with batch effects arising from scanner hardware differences, acquisition protocols, and sate-specific demographics T g]. Such batch effects intro- duce unwanted variability that can confound downstream analyses unless corrected by harmonization methods such as comain-adaptation
6 9 [0
frameworks tailored to neuroimaging [ Constructive learn- 19 20 is a paradigm in which models learn by contrasting similar and dissimilar example pairs to shape feature spaces. It complements multi-view functional alignment by using contrastive objectives directly align representations across subjects and sites and, by enforcing agreement on same-stimulus responses while discouraging spurious correlations, helps mitigate batch effects and enhances robustness and generalization in multi-site fMRI analyses.
ing [14 15 16.
17
18
The main contributions of this paper are fivefold: (1) we introduce orthogonal contrastive learning (OCL). a unified multi-representation framework that aligns multi-subjes fMRI data without temporal preprocessing or uniform time-series length requirements across subjects or sites; (2) we design a dual-encoder architecture- responses together and pushes different-stimulus responses apart, and a target network updated via exponential moving average to stabilize learning and enforce consistency; (3) we develop novel OCL layer composed of four tightly integrated components: QR decomposition, which yields orthonormal feature bases decorrelate signals and enhance the signal-to-noise ratio; locality-sensitive hashing (LSH) (1I) which produces compact subject-specific signatures that preserve similarity relationships while drastically reducing feature dimensionality; positional encoding. which injects continuous temporal context into spatial feature representations to maintain dynamic stimulus information; and a transformer encoder, which employs multi-head self-attention to capture global dependencies and produce discriminative. stimulus-aligne strategy on synthetic fMRI-like data to initialize OCL parameters for faster convergence and improved robustness; and (5) we demonstrate a transfer-learnin pipeline that applies trained OCL models to multi-site datasets, showing resilience to scanner variability and sequence-length differences, and achieving superior downstream classification performance over state-of-the-an methods.
online network trained with a contrastive loss that pulls same-stimulss
embeddings; (4) we propose an unsupervised pretraining
The remainder of this paper is structured as follows. Section 2 reviews related work, Section 3 details our proposed method, Section 3 presents our empirical evaluation, and Section 5 concludes with key findings and avenues for future research.
2 Related Works
Hyperalignment (HA) is a deterministic alignment technique that uses generalized CCA to en- hance prediction accuracy in fMRI analysis [5] 11 [2]. Classic HA's requirement to invert high- dimensional covariance matrices makes it unreliable for highly correlated data data-e.g. e.g., whole-brain
Available at https://dsta-archiis nimh. nih. gov/
3 Available at http:///www..uummccometttttooo .orge org/
Available https://opemeeuro.orrg
2

---

images; variants such as regularized hyperalignment (RHA) M. singular value decomposition hyperalignment (SVDHA) [22]. and -ooammmme)ricc) kernel hyperalignment (KHA) [ respec- tively introduce regularization, low-rank decompositions, or kemnl mappings to stabilize alignment. Deep hyperalignment (DHA) further extends this line by employing a deep neural network as a learnable kernel to capture complex, nonlinear subject-specific transformations end-to-end [9. More recently, deep geodesic canonical correlation analysis (DeepGeoCCA) has been proposed to gener- alize CCA to symmetric positive-definite covariance matrices on Riemannian manifolds, yielding robust covariance-basee alignment by maximizing geodesic correlation [3.
SRM offers a probabilistic alternative to HA by aligning neural responses via maximum likellhoo estimation of shared latent timecourse [O]. Subsequent work introduced multi view convolutional autoencoder (CAE + SRM), which leverages convolutional neural networks to extract richer features before alignment 2 Matrix-Normal SRM (MN-SRM) employs Kronecker-separable covariance priors and maximum a posteriori estimation to jointly model spatial and temporal noise (24), while robust SRM (RSRM) applies sparse, deterministic optimization to disentangle shared and subject- specific components [25] FastSRM presents an identifiable SRM variant with a dimension-reduutiion preprocessing step that stabilizes and accelerates shared response recovery, achieving orders-of- magnitude speed-ups without loss of accuracy [6]
Shared independent component analysis (ShICA) replaces the CCA step with independent component analysis (ICA) to learn statistically independent shared components under additive Gaussian noise, improving aligmment on data with non- -on-Gausiian artifacts [26. However, ShICA only models shared variance. To address this, shared and individual ICA (ShIndICA) was proposed to jointly recover both shared and subject-specific sources, with provable identifiability via likelihood-based estimation 3 Beyond ICA, the hyper hidden Markov model (Hyper-HMM) projects voxels into a latent event space and aligns temporal segments across subjects, enabling joint spatial-temporal correspondence in naturalistic fMRI paradigms [4].
Several multi-site transfer-learning approaches have been developed harmonize task-based fMRI data across scanners and cohorts, including maximum independence domain adaptation (MIDA) (27.
multi-datase dictionary learning (MDDL) (28). and multi-dataset mulli-subjee (MDMS) (28), side
information dependence regularization (SIDeR) (7]. The shared space transfer learning (SSTL) M
further extends this line by extracting site-specific common features through single-iteration multi- view optimization and mapping them into a site-independent shared space, thereby enabling scalable alignment of nigh-dimensional fMRI data. SSTL can incorporate the deep-kernel formulation introduced in DHA termeed DeepSSTL-to further boost prediction accuracy in multi-site graph neural networks (XG-GNN) a meta-learning
IMRI studies [.. framework with Nomain-generrlizzbbl explainability regularizers that learns graph neural networks for multi-site fMRI analysis, demonstrating robust cross-center performance and interpretable subgraph discovery [
Self-supervised 'constructive" learning methods have recently been applied to multi-view fMRI alignment. Foundational contrastive frameworks such as SimCLR [14] bootstrap- pooosstra-your-owae yaur-ow--atte (BYOL) 15), DINO [6),, and its successor DINOv2 [7 learn view-invariant representations with- out labels. In task-based fMRI, MindEye combines contrastive encoding with diffusion priors to reconstruct viewed images while implicitly aligning subjects in the latent space [18; MindEye2 demonstrates that shared-subject generative model pretrained across participants can be fine-tuned with just one hour of data to decode images from fMRI [19]. More recently, MindAligner learns ex- plicit cross-subject transformation networks for functional alignment in task-based fMRI (20). These self-supervised approaches offer promising alternatives for multi-subject alignment by leveraging rich augmentations and momentum-bast based architectures. In the following, we empirically compare our proposed method to some of these approaches.
3 The Proposed Orthogonal Contrastive Learning (OCL)
This section introduces orthogonal contrastive learning (OCL), a novel multi-view framework for multi-subject fMRI analysis. Similar to previous functional alignment methods, OCL considers each subject's neural responses as a separate view of the same underlying data. As Figure illustrated, OCL employs two neural networks with identical architectures: an online network, which actively learns representations through contrastive objective, and a target network, whose parameters are gradually updated as moving average of the online network's 's parameters. This dual-network
3

---

Train Set
Class
Posttianl Encoding
Weteriatton
X
Rg
2g t
E
Pr s.B.a
N. =
Encoter Cemttettttnt
Represertation
Ok
H..
SuRjecT
X
Classicction Modill
Bronding
Predictive Mode
X
Figure 1: The Proposed Orthogonal Contrastive Learning (OCL)
setup stabilizes training and ensures consistent representations across subjects. Each OCL layer has four primary components: first, QR decomposition, which yields orthonormal feature bases to decorrelate signals and enhance the signal-to-noise ratio; second, locality-sensitive hashing (LSH), which produces compact subject-specific signatures; third, positional encoding, which integrates fMRI temporal information into spatial feature representations; finally, a transformer encoder integrates these inputs, ensuring that neural representations from the same stimulus become closely aligned, while representations of different stimuli remain distinct. In the remainder of this section. we mathematically define OCL, present a pretraining strategy to improve its performance, and discuss extending OCL through transfer learning for multi-site fMRI analysis.
We let S be the number of subjects in the training set. Let Tg, = 1,....5 denote the number of time points for the s-th subject, and let V be the number of voxels in the selected region of interest (ROI), which view as ID vector, even though it corresponds preprocessed neural responses for the s-th subject is then defined as X, E xV For simplicity, this paper assumes that each column of the neural responses is standardized during preprocessing: - ~N,00, = L..... S. I is the identity matrix. In addition. the v-th column in X, for all subjects denotes the anatomically aligned voxel located at the same locus across fMRI images (.] We then let N defines the number of layers in the online and target networks, fn, n = 1, N is the transformations implemented by each layer, and H..s (Hn is all transformations applied in the n-th layer of the proposed OCL architecture on the nerual responses of s-th subject. Here, we consider Ho,. X, as input layer of the networks for each subject. Further, we let
3D volume. The
1,..., N, 1,
rank(X.) denote the rank of the neural response matrix of the s-th subject, and min sr.)) the number components RT.xd HN as the final representation of
final representation such that
H RT.xd n = I...., N, I,...,,, S. Z neural representations for each subject that can be used for downstream classification analysis.
The first component in each OCL transformation layer is QR decomposition. Given the neural response matrix of the s-th subject X., we apply thin (reduced) QR decomposition [29] to factor X, = =ful full R.full where Quull xr. is orthonormal and Rfuill XV is upper-triangular. We then truncate these factors by taking the first d columns of full to form Q truncate RR and the first d rows of Rfull to form R, E R This truncation component ensures that each OCL layer produces exactly features, where is usually the maximum number of shared features that can be extracted across all training subjects, as determined by their ranks [30]. We let T > max.=1.....s T, be the maximum content-window size in the OCL architecture. The temporal zero- padding component adds zero common shape Q. E R Note that T must be large enough to accommodate the neural responses of every subject, including those in the testing set. We also define binary mask vector m, E (0,1)² 1)" where value of indicates actual response time point and 0 indicates padding. We also let d be
to each Q truncate
that all orthonormal matrices share the

---

the number of nonzero elements of the aoper-triangulatt matrix R, and define o, ER (ec as the vectorization operator that extracts the nonzero elements of the upper-triangular matrix into single vector, which is then used in the next step to produce subject-specific signatures.
secasity-sensiiivve hashing (LSH) component is randomized, data-independent hashing scheme for approximate nearest neighbor search in high-dimensional spaces, ensuring that similar items collide with higher probability than dissimilar ones [21]. We let the parameter p (0, 2] be the stability exponent [21 31 a E IRd drawn from a p-stable (Gaussian) distribution, w E Ry0 be the quantization granularity of the hash bins, and b ~ U[0, wr,, yielding provable locality sensitivity 21 32].
We denote the I.S for s-th subject as follows:
(a, oo)
L. (a,b, (0+)
(1)
W
where [] is the floor function. Note that LSH is identical for all neural responses of the s-th subject and needs only be computed once for all time points belonging to that subject in each training iteration. We then use linear multilayer perceptron (MLP) that accepts the scalar lg E IR input and produces the vector S, E Rd as the subject-specific signature (embedding) for the s-th subject.
Lemma 1. Let each neural responses matrix admit a reduced QR factorization X, Q2 R, with Q, Q2 = Define Φ = voc(R..) (R.) vec Q X.).
3.)-1 If
Φ Φ2 < ( - Qall + Pr[e1 E2]
Pr[c [3].
Please refer to the supplementary material for the proof.
We let C, E R2dd be the output of the concatenation component, which combines each row of the orthonormal matrix Q, E RR'x with the subject-specific signature 8, RR.. We then apply sinusoidal positional encoding to embed temporal context alongside the spatial and signature features. Coneretely, we construct a positional encoding matrix
E RTx2d [e1,0: eT,2a). e1,2: = sin T2i/(2d)) 6r,2(+1 = cos T22/(24)
for -....,,, and i 0.....d d- 1. Adding this to the concatenated features yields the component P. E RTx2 C. + E, which now encodes both spatial patterns and their temporal positions. Next, OCL applics a normalization component to stabilize and scale each time-step embedding, i.e., given the positional-encoded features P. I Tx2d
compute the normalization N, E 1R Norm((P. Finally, these normalized embeddings are passed, together with the temporal padding mask m., into standard Transformer encoder 33
to produce corresponding layer output H
We ye = [ye.1...., V..r..1 10,,,....,,,.... E R. denote the class labels for the s-th subject in the training set. For the s-th subject, let Z. [æ,. 2 E RT,, be the output of final layer of the OCL online network. We define the contrastive loss (Zc with temperature т, margin μ., and between-class weight À for s-th subject as
Σ
exp ((EM5) Eng)/T)
1 To
jti, Ve.
log
T
exp ((20. 2s,k)/T)
(2)
A=1
log [ exp exp ((zed)
Zea)lT 11).
Let denotes all learnable parameters of the online encoder and Ø those of the target encoder. In each training iteration for subject s, we first update the online parameters by one step of gradient VeLocu (Z..
descent on the subject's contrastive loss: 0 + 0 update the online network using all subjects in the training set during each iteration. Once the online network has processed every subject's 's data in each iteration, we update the target network parameters using an exponential moving average (EMA) of the online parameters follows [5]
is the learning rate. We
0 = - + (1- 1 50
0,
(3)
5

---

Table 1: The fMRI datasets
ID
Title
Type
S
Y
T.
Site(#)
A"
Stop signal (DSM07)
34
Decision
20
472
B (3)
B
Conditional stop signal (DS008)
R5
Decision
317
A())
CMU
Meanings of Nouns 36
Semantic
12
402
C
Simon task (DS 101) (unpublished 7
Simon
21
302
D(I)
D
Flanker task (DS102) [37
Flanker
26
292
C(1)
DS232
Face-coding models with individual-face 38
Visual
760
E
Integration of sweet taste: Study (DS229)
39
Flavour
580
F(I
F
Integration of sweet taste: Study (DS231)
39
Flavour
650
B())
Forrest Raiders
Forrest Gump movie 40
Visual
20
10
451
Raiders movie 10
Visual 10
924
8 the number of subjects: ly is the nomber of stimulas categories; T. is the number of time points per subject; Site lists the other datasets
whose neural responses can be transferred
dataset. represents number sites corresponding dataset.
dataset
partitioned into three independent 'sites' pree'-pseedoowor naming (A letter naming (A2), and manual response (A3) D
where to is the number of total iterations. In the training phase, OCL learns a shared representation space in which neural recordings from all training subjects are aligned. We then train a classifier (denoted by æ in Figure on these new representations. the testing phase, we use the trained target network to map the test data into the same representation space and then apply the classifier to predict cognitive states. We provide the pseudocode for the proposed OCL algorithm in the supplementary material.
3.1 General Pretrained Orthogonal Contrastive Model
To bootstrap OCI, for real task-based fMRI, we first pretrain the dual-encoder entirely on synthetic data that mimics the statistical structure of neural timecourses. Concretely, given & class categories, we generate a corpus random base matrices M E TTX,, where each group of I rows drawn i.idd from one of distinct Gaussian distributions with randomly initialized means and variances that differ across distributions. For each base matrix M, we then create S distinct 'views' by applying S random orthonormal rotations: X, MU., 8Y U, U, I, =, S. Since each row of M is sampled from one of the & distributions, we assign a corresponding class label = to that row, and this label is preserved across all rotated views (X.)s- Each view X, is passed through the OCL layers to produce embeddings Z., i.e., the contrastive objective pulls together embeddings of the same label across different rotations and pushes apart embeddings of different labels. After pretraining, we transfer the target encoder's parameters 0 to initialize the downstream OCL model on the real [MRI data. This EMA-smoothed encoder has already learned to factor out arbitrary orthogonal transforms and subject-specific variability, providing a strong, generalizable starting point for aligning real neural data with minimal additional tuning.
Lemma 2. Let each synthetic view be generated by an orthonormal rotation of the base data, X, = MU., U,U,=
= I. Write the corresponding QR factor R, and its flattened vector &, vec R.))
Then for any three views X1, X2, Xa generated from M,
Pr[e = E2] = Pr[e1 fa].
In other words, the collision probability of the LSH hash identical across all random rotations generated from M.
Please the supplementary material for the proof.
3.2
Transfer Learning via Orthogonal Contrastive Embeddings
To extend OCL to multi-site fMRI studies, suppose we have B training sites, each providing data
(6) S. J==1 for site b 1,..., B. We train an independent OCL instance each site, yielding target-encoder parameters 0,, b 1, .,... B. These site-specific encoders capture local scanner and population idiosyncrasies while maintaining the shared contrastive objective. To initialize OCL on a multi-site setup (with extra fine-tuning step), we aggregate the B learned targets via
{x
1
Osites
(4)
B

---

FastSRM
HyPerHMM
DeepGeoCCA
MindAligner OCL
OCL
ShIndICA
DILA
MindEye2
Clen ae one
(a) CMU (Temporally Unaligned)
CMU (Temporally Aligned)
(c) DS232 (Tempurally Unaligned)
(d) DS232 (Temporally Aligmed)
Figure 2: Classification analysis on Temporally Aligned versus Temporally Unaligned Data. Plotted
are mean accuracies and error bars are #1 standard deviation.
thereby blending all site-speen site-specific knowledge into single robust prior. We then freeze fsites and apply it to both the training-sio site and testing-site site data, projecting each into the same shared feature space defined by the aggregated targets this alignment boosts the accuracy of our downstream classifiers.
Please note that this multi-site adaptation scheme also enables an active learning loop in the real world applications. At each iteration, we evaluate the contrastive loss C on the predicted time points from the testing site and select those with highest uncertainty (e.g., largest margin- masged-based loss) for expert manual labeling. The newly annotated samples are then incorporated into the training set, the online encoder parameters are updated accordingly, and the target encoder is refined via EMA. By focusing annotation effort on the most informative temporal segments, this closed-loop procedure maximizes performance gains in low-data or high-cost labeling scenarios. Although active learning offers a promising avenue for extending OCL, it is beyond the scope of this study; all reported results were obtained without active-learning expert-annotare data.
4 Experiments
Table summarizes the 10 datasets used in our empirical evaluation. Datasets CMU and DS232 consist of simple cognitive task-predien sss-prediction paradigms, whereas Forrest and Raiders datasets involve naturalistic movie-watching stimuli in single-site fMRI studies. We also include homogeneous task-based fMRT datasets (A-F) suitable for multi-site analysis. All datasets publicly available (via OEERNEURO 4 except CMU and were preprocessed with our GUI-based toolbox called easy IMRI and FSL 6.0.15 including spatial normalization, smoothing, anatomical alignment; for those alignment techniques that require it, temporal realignment was also applied (see Section 4.T). Each scan was registered the MNI152 we-weihtted template [1] at a 4mm isotropic resolution, and a whole-bran ROI was defined for all analyses, yielding V = 19,742 voxels per volume. Data were standardized during preprocessing, without loss of generality.
We benchmark OCL against 7 single-site fMRI analysis methods: FastSRM and HyperHMM as baselines; ShIndICA non-CCA method; DHA and DeepGeoCCA deep multi-view learning approaches; and MindFye22 and MindAligner as self-supervised constructive learning approaches. For multi-site evaluation, we compare OCL existing techniques: SSTL baseline; DeepSSTL and XG-GNN as deep multi-site learning approaches; and MindEye2 and MindAligner as self-supervised constructive methods. Crucially, each site's 's data are strietly partitioned so that no neural responses from given site appear in both the training and testing sets. All experiments were run on two PCs with the specifications listed in the Footnote Our proposed OCL algorithm is available on GitHub Like the previous studies [ 99 I0), we employ 1/-support vector machine (v-SVM) [41] for all classification experiments. We use leave-one-subject-oll nested pross-validation: in each outer fold, one subject is held out for testing; within each, another subject serves as validation (inner fold), and the rest form the training set. Hyperparameters for alignment and v-SV (e.g., RBF kernel scale, v) are selected via grid search on validation accuracy, and the best testing accuracy is reported for each technique.
"Aaailable at htttps://www.co cmu, www. edwmmu//s///s/csocm//m cmu, wttp://t///pppottctttoooooo--oee B0ooo/////w//w/// 'Available
com/
Availlable at
ox ac. uk/ fsl
8 OS: Fedora 42, Python: 3.11.9, Py Torch: 2.6. CUDA: 12.6; Connection: 2x40GbE CX314A Mellanox
(PC1) CPU: AMD EPYC 7551/ (64 cores), RAM: 256G GPU:2XNVIDIA 4060Ti 16G;
(PC2) CPU: AMD Threadripper 2990W X (64 cores), RAM: 128G, GPU:2x 4060Ti 16G.
90CL code repository:
7

---

FastSRM ShindICA
HyperHMM
DeepGeoCCC MindEye2
MindAligner
OCL
OClaar
300
488
1000 1389
J00
1000 1399
500
480
18001 1389
J00
1800 1388
(3) Forrest (TRs 100)
(5) Forrest (TRs 100)
Forrest (TRs 800)
(3) Forrest (TRs 2000)
Seneectele brenlphheee
benslphheee
Aud vondl brestppere
(c) Ruiders (TRi- 100)
(1) Raiders -4R0=400)
(2) Roiders (TRs 800)
(1) Raiders (TR 2000)
Figure 3: Classification analysis on movie stimuli. Potted are mean accuracies.
For OCL, we first initialize a model ('OCL-zero') ') by pretraining on two million (4 x 500,000) synthetic matrices M E TTxV for k E {5, 10, 20, 50} categories with T = 2000 time points and V = 19742 voxels, each subjected to = 360 random rotations. We set the embedding dimension to d 256, employ Encoding Transformer with 16 attention heads, and use N = 32 network layers. These hyperparameters were chosen to balance representational capacity with available computational resources listed in the Footnote [ We then initialize each OCL instance with the pretrained target encoder 0, and fine-tune both online and target networks on the real IMRI data. We train (and pretrain) OCL for up to a = 1000 iterations with automatic early stopping based on validation loss using Adam optimizer [42]. At each iteration, we form a batch from all time points of a single aubject-treating cach time point as an independent sample-ane and randomly shuffle their order. This permutation prevents the network from overfitting to fixed temporal sequence and encourages robustness to varied response orderings. Because every fine-tuned OCL model is initialized from OCT employ the same OCL network architecture across all experiments in this paper. OCL consistently produces a feature space of dimension d 256. For all competing methods, we evaluate two latent space sizes-one fixed at d 256 and second chosen via grid search-and report the configuration that yields the highest classification accuracy. Other self- supervised approaches are similarly initialized with their published pretrained weights 19] 20) All remaining hyperparameters for both the aligmment and classification models are optimized temperature т E {0.01, 0.1, 0.5, 0.9, 0.99), margin E {0.1. 0.2. 0.5, 0.8, 0.9]. between-cles weight À e (0.1, 0.2, 0.3, 0.4, 0.5}. learning rate r {0.1, 0.2, 0.3, 0.4), and quantization granularity (0.9, 1.0, 1.1, 1.2] and select the combination that maximizes performance on the validation set.
using grid search. We perform grid search over the key OCL hyperparameters
4.1 Simple Cognitive Task Classification: Temporally Aligned vs. Unaligned Data
This section evaluates OCL on two simple cognitive-task datasets (CMU and DS232), in which subjects performed Semantic, and Visual assessments during fMRI scanning. Unlike most functional alignment methods-which assume temporal synchronization (i.e., each time point i corresponds to the same stimulus across subjects) can handle varying time-series lengths and arbitrary time-point ordering without explicit temporal preprocessing. We therefore compare OCL and several alignment techniques both on the raw, unaligned data and after applying their required temporal alignment. To robustly tune and assess performance, we employ nested leave-one- subject-out procedure: in each outer fold one subject is held out for testing, while in each inner fold different subject is held out for validation and hyperparameter selection. Figure 2 shows that traditional alignment methods suffer significant testing accuracy degradation on unaligned data because their shared-space templates misalign stimuli across rows; in contrast, auf-suuer constructive approaches (MindEye2, MindAligner, and OCL) learn flexible mappings rather than fixed templates, yielding stable shared representations. OCL in particular achieves the highest accuracy, likely due to (1) its orthogonal decomposition of independent versus subject-specifie features and (2) the pretrained 'OCI.zera' encoder's ability to generalize arbitrary rotations to novel neural patterns. Each of the 4 plots in Figure iis comparing OCL with different methods x (FastSRM. ShIndICA.
8

---

D
(c)
(d) (AI, A3, B)
Figure 4: Multi-site Classification Analysis. Plotted are mean accuracies and error bars are + 1 standard deviation.
HyperHMM. DHA, DeepGeoCCA, MindEye2, MindAligner) for total of 4 7 = 28 comparisons where the 2-sided t-test found p < 0.05 in all 28 cases.
4.2 Classification Analysis on Movie Stimuli
This section evaluates functional aligmment techniques the movve--atchhin fMRI datasets (Forrest and Raiders) listed in Table I using the same nested leave-one- sauve-on--uubect-ouu scheme as in Section Following the procedure of [5]. first rank voxels within the predefined ROI by their task-evoked activation strength, as in 9 10 30. We then assess each alignment method across multiple spatial resolutions by selecting the top {100, 200, 400, 600, 800, 1000, 1200) voxels for Forrest and (70, 140, 210, 280, 350, 420, 490] voxels for Raiders. To test temporal coverage, further repeat all experiments using the first {100, 400, 800, 2000) Time of Repetitions (TRs) of each scan. Note that in OCL, voxels outside the ROI are zeroed out in the input layer, and the subsequent QR decomposition automatically ignores these zeros, preventing them from in- fluencing the learned representations. Figure 3 presents classification accuracy as a function of voxel count and number of time points, comparing our proposed OCL to 7 alignment approaches = (FastSRM, ShIndICA, HyperHMM. DHA, DeepGeoCCA, MindEye2, MindAligner) for to- 392 pairwise evaluations. Self-supervised methods initialized with pretrained techniques. In every evaluation, OCL
tal of 8 x 7 models typically outperform traditional functionll-alignenn surpasses all comparators an advantage we attribute to its orthogonal feature decomposition and contrastive alignment. A two-sided f-test confirms that the accuracy differences are significant (p < 0.05) in all 392 comparisons.
4.3 Multi-Site Classification Analysis
This section presents the results of multi-site fMRI analyses using datasets A-F listed in Table [ For each pair of sites (A, B), we conducted two-sided cross-site evaluation: in the forward direction A + B, we trained and validated alignment and classification models using site A and evaluated them on site B; in the reverse direction B + A, we reversed the training and testing roles. The final accuracy is computed as the mean of both directions. This bidirectional setup is denoted as A = B. Figures (a-c) summarize the classification accuracies across multiple cross-site pairs. In addition, Figure 4 (d) presents the mean accuracy of supplementary experiments based all possible two-versus- two train/test splits derived from the set ( (A1, A2, A3, B), yielding six distinct comparisons for example, training (A1, A2) and testing (A3, B), and vice versa. As shown, the baseline method SSTL,, which relies on linear transformations, consistently underperforms. In contrast, DeepSSTL improves accuracy by utilizing MLP based deep kernel for alignment. Constructive learning approaches further enhance accuracy by leveraging pretrained models that better generalize across domains. The proposed OCL framework achieves the highest classification performance across all site-pairs. This improvement appears to stem from two key design elements: (I) the use of a pretrained feature space, and (2) a specialized architecture that enforces cross-site shared information via a contrastive loss. Each of the subplots Figure compares OCL against competing method x, for {SSTL, DeepSSTL, XG-GNN, MindEye2, MindAligner), resulting in
nallitrepreseentaiinnall alignment module that provides a robust initial
each of five baselines total of 4 20 comparisons. In all cases, two-sided paired t-test yielded statistically significant differences (p 0.05), confirming the robustness of OCL in cross-site generalization.
PPPair:: (A1, A2). (AI, A3), (AI. BJ. (A2, A3). (A2, BJ. |A3.

---

5 Conclusion
This paper has introduced orthogonal contrastive learning (OCL), a unified framework that addresses task-based fMRI's key challenges: low signal-to-noise ratio, high dimensionality, and variable time- series lengths. OCL aligns neural responses across subjects and sites without explicit temporal preprocessing, OCL employs a dual-encoder design: an online network and a target network whose weights track the online network via exponential moving average to stabilize learning. Each OCL network layer combines QR decomposition for orthogonal feature extraction, locality-sensitive hashing (LSH) to produce compact subject-speciee
signatures, positional encoding to embed temporal structure alongside spatial features, and a transformer encoder to generate discriminative, stimulus- aligned embeddings, trained with a contrastive loss that pulls together same-stimulus responses and pushes apart iffferentt-timulus responses. We further enhance OCI with an unsupervised pretraining stage on fMRI-like synthetic data and demonstrate a transfer-learning workflow for multi-site studies. Across extensive experiments on multi-subject and multi-site fMRI benchmarks, OCL consistently outperforms state-of-the-ar alignment and analysis methods in both representation quality and downstream classification accuracy.
In the future, OCL has the potential to be applied across a variety of as reconstructing visual stimuli or movies from human brain activity, as well as extended to other neuroimaging modalities including resting-state fMRI, MEG, and EEG. For resting-state fMRI, pseudo-labels can be generated by applying dlding-window functional connectivity or by clustering temporal windows based on correlation patterns; OCL can then maximize contrastive agreement across these pseudo-classes, effectively aligning subjects in the absence of explicit tasks. Similarly, for MEG and EEG data, the decomposition can operate on sensor- or source-space time series, while LSH can hash spectral or time-frequency representations. Positional encoding further preserves temporal ordering, even when sampling rates vary across modalities. Moreover, OCL can be scaled to larger voxel spaces and multi-site datasets, paving the way toward a foundation model for fMRI amalysis-althoughe such scaling would require substantial GPU cluster resources.
fMRI studies, such
Broader Impacts
Orthogonal contrastive learning (OCL) enables large-scale pooling of multi-site fMRI datasets, substantially improving statistical power and reproducibility by harmonizing site-specific biases and reducing variance in group-level inferences. By aligning subject-specific neural signatures into shared space, OCI, facilitates the discovery of robust biomarkers for neurological and psychiatric disorders, advancing precision psychiatry and personalized medicine. Eliminating the need for explicit temporal preprocessing and uniform time-series lengths, OCL lowers technical barriers to integrating diverse datasets, supporting open-science platforms and accelerating collaborative research. Finally, by democratizing access to high-quality, reproducible fMRI representations and mitigating batch effects, OCL promotes ethical, transparent Al-driven neuroscience, fostering cross-disciplinary innovation and responsible research practices.
Limitations
OCL's multi-module architecture enables strong alignment but comes with important caveats: (i) Computation: its dual-network and Transformer components demand substantially greater computa- tion power than traditional alignment methods, limiting scalability to very high-dimensional feature spaces: (ii) Domain shift: performance may degrade under extreme inter-site or inter-subject domain shifts beyond those in our benchmarks; (iii) Data requirements: its ability to generalize robustly hinges on access to large, well- annontae multi-site datasets, constraining usefulness in small-sample or weakly labeled settings; and (iv) Interpretability: deep learning in neuroimaging is not easily interpretable, and while we provide initial insights via QR-basis spatial projections and attention maps, comprehensive interpretability is outside the scope of this work.
Acknowledgments
I would like to express my sincere gratitude to Stephen Kaiser of the National Bank of Canada for his generous support in the publication of this research study.
10

---

References
[1] Tony Muhammad Yousefnezhad, Alessandro Selvitella, Daoqiang Zhang, Andrew Greenshaw, and Russell Greiner. Shared space transfer learning for analyzing multi-site fmri data. Advances in Neural Information Processing Systems (NeurIPS), 33:15990-16000, 2020.
[2] Shuo Huang, Liang Sun, Muhammad Yousefnezhad, Meiling Wang, and Daoqiang Zhang. Func- tional alignment-auu -uxiiil generative adversarial sework-bbased visual stimuli reconstruction via multi-subject finri. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:2715-2725, 2023.
[3] Teodora Pandeva and Patrick Forré. Multi-view independent component analysis with shared and individual sources. In Uncertainty in Artificial Intelligence, pages 1639-1650, PMLR, 2023.
[4] Caroline Lee, Jane Han, Ma Feilong, Guo Jiahui, James Haxby, and Christopher Baldassano. Hyper-hmm: aligning human brains and semantic features in common latent event space. Advances in Neural Information Processing Systems (NeurIPS), 36:27005 2023.
[5] James V Haxby, J Swaroop Guntupalli, Andrew Connolly, Yaroslav O Halchenko, Bryan Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):404 416, 2011.
[6] Hugo Richard and Bertrand Thirion. Fastsrm: A fast, memory efficient and identifiable implementation of the shared response model. Aperture Neuro, 2023.
[7] Shuo Zhou, Wenwen Li, Christopher Cox, and Haiping Lu. Side information dependence as regularizer for analyzing human brain conditions across cognitive experiments. Proceedings of the AAAI Conference Artificial Intelligence (AAAI), volume 34:04, pages 6957-6964, 2020.
[8] Xinmei Qiu, Fan Wang, Yongheng Sun, Chunfeng Lian, and Jianhua Ma. Towards graph neural networks with domain--eneralice explainability for firi-based brain disorder diagnosis. In International Conference on Medical Image Computing and Computer-Asiisted Intervention (MICCAI), pages 454 464. Springer, 2024.
[9] Muhammad Yousefnezhad and Daoqiang Zhang. Deep hyperalignment. In Advances in Neural Information Processing Systems (NIPS), pages 1603-1611, 1, 2017.
[10] Po Hsuan Cameron Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter Ramadge. A reduced-dimenssoon fmri shared response model. In Advances in Neural Information Processing Systems (NIPS), pages 460-468. 2015.
[11] Hao Xu, Alexander Lorbert, Peter Ramadge, J Swaroop Guntupalli, and James V Haxby. Regularized hyperalignment of multi-set fmri data. In IEEE Statistical Signal Processing Workshop (SSP), pages 229-232. IEEE, 2012.
[12] Alexander Lorbert and Peter Ramadge. Kernel hyperalignment. In Advances in Neural Information Processing Systems (NIPS), pages 1790-1798, 2012.
[13] Ce Ju, Reinmar J Kobler, Liyao Tang, Cuntai Guan, and Motoaki Kawanabe. Deep geodesic canonical correlation analysis for covariance-baaee based neuroimaging data. International Confer- ence on Learning Representations (ICLR), 2024.
[14] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1697- PMLR, 2020.
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural 2020.
Information Processing Systems (NeurIPS),

---

[16] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supe vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[17] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, 2024.
[18) Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Demp- ster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems (NewrIPS), 36:24705-24728, 2023.
[19] Paul S Scotti, Mihir Tripathy, Cesare Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: shared-subjed models enable firri-t--mmae with hour of data. In Proceedings of the International Conference on Machine Learning (ICML), pages 44038 44059, 2024.
[20] Yuqin Dai, Zhouheng Yao, Chunfeng Song. Qihao Zheng. Weijian Mai, Kunyu Peng, Shuai Lu, Wanli Ouyang, Jian Yang, and Jiamin Wu. Mindaligner: Explicit brain functional alignment for cubse-subject visual decoding from limited fmri data. arXiv preprint arXiv:2502.05034 2025.
[21] Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitiee hashing scheme based on p-stable distributions. In Proceedings of Symposium on Computational Geometry (SoCG), pages 253-262, 2004.
[22] Po Hsuan Chen, J Swaroop Guntupalli, James V Haxby, and Peter J Ramadge. Joint svd- hyperalignment for sulji-subject fmri data aligmment. In IEEE International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE, 2014.
[23] Po- Po-Hsuan Chen, Xia Zhu, Hejia Zhang, Javier S Turek. Janice Chen, Theodore L Willke, Uni Hasson, and Peter J Ramadge. A convolutional autoencoder for multi- fmri data aggregation. 29th Workshop of Representation Learning in Artificial and Biological Neural Networks, 2016
[24] Michael Shvartsman, Narayanan Sundaram, Mikio Aoi, Adam Charles, Theodore Willke, and Jonathan Cohen. Matrix-normal models for fmri analysis. In International Conference on Artificial Intelligence and Statistics (AISTAT), pages 1914-1923. PMLR, 2018.
[25] Javier S Turek, Cameron T Ellis, Lena Skalaban, Nicholas B TTrkkBroonn and Theodore Willke. Capturing shared and individual information in fmri data. In IEEE International Conference on Acoustics, Speech and Signal Prucessing (ICASSP), pages 826-830. IEBE, 2018
[26] Hugo Richard, Pierre Ablin, Bertrand Thirion, Alexandre Gramfort, and Aapo Hyvarinen. Shared independent component analysis for multi-subject neuroimaging. Advances in Neural Information Processing Systems (NeurIPS), 34:29962-29971, 2021.
[27] Ke Yan, Lu Kou, and David Zhang. Learning domain-invariant subspace using domain features and independence maxinization. IEEE Transactions on Cybernetics, 48(1):288- 2017.
[28] Hejia Zhang, Po-Hsuan Chen, and Peter Ramadge. Transfer learning on fmri datasets. In International Conference on Artificial Intelligence and Statisties (AISTATS), pages 595-603. PMLR, 2018
[29] Gene H Golub and Charles F Van Loan. Matrix computations, 2013.
[30] Muhammad Yousefnezhad, Alessandro Selvitella, Liangxiu Han, and Daoqiang Zhang. Super- vised hyperalignment for multi-subject fmri data alignment. IEEE Transactions on Cognitive and Developmental Systems (TCDS), 2020.
[31] Hongkai Nguyen. Lecture notes: Sketching and p-stable distributions. CS261, Stanford University, 2019, Definition 2.1.
12

---

[32] Pinecone Systems. Inc. Pinecone: Vector database for scalable similarity search.
https
//www www..iinecone... 2025. Accessed: 2025-05-10.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NIPS). 30, 2017.
[34] Gui Xue, Adam R Aron, and Russell A Poldrack. Common neural substrates for inhibition of spoken and manual responses. Cerebral Cortex, (88):1923-1932 2008
[35] Adam R Aron, Tim E Behrens, Steve Smith, Michael J Frank, and Russell A Poldrack. Triangu- lating a cognitive control network using feffuusion-weighhe magnetic resonance imaging (mri) and functional mri. Journal of Neuroscience, 3774433374-337 2007.
[36] Tom Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente I. Malave, Robert A Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. Science, 320(5880): 191-1195, 2008.
[37] AM Clare Kelly, Lucina Q Uddin, Bharat B Biswal, F Xavier Castellanos, and Michael Milham. Competition between functional brain networks mediates behavioral variability. Neurolmage, 39(1):527-537. 2008.
[38] Johan D Carlin and Nikolaus Kriegeskorte. Adjudicating between face-coding models with individual-face fmri responses. PLoS Computational Biology, 13(7)=e1006004 2017.
[39] Maria Geraldine Veldhuizen, Richard Keith Babbs, Barkha Patel, Wambura Fobbs, Nils B Kroemer, Elizabeth Garcia, Martin R Yeomans, and Dana M Small. Integration of sweet taste and metabolism determines carbohydrate reward. Current Biology, 27(16):2476-2488 2017.
[40] Michael Hanke, Florian Baumgartner, Pierre Ibe, Falko R Kaule, Stefan Pollmann, Oliver Speck, Wolf Zinke, and Jörg Stadler. A high-resolution 7-tesla fmri dataset from complex natural stimulation with an audio movie. Scientific Data, 1, 2014.
[41] Alex J Smola and Bernhard Schölkopf. A tutorial on support vector regression. Statistics and Computing, 14(3):199-295 222, 2004.
[42] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims the abstract and introduction accurately reflect the paper's contributions, including the proposed OCL framework and its application to multi-subjeect and multi-site fMRI analyses.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss our primary assumptions in the Experiments section, and address the limitations and future directions of the proposed method in the Conclusion and Broader Impacts sections.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the full set of assumptions in the Experiments section and include all proofs in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all necessary details to reproduce the main experimental results in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The paper uses publicly available datasets for all empirical evaluations, and we will release the proposed method as part of our open-source toolkit for direct use in fMRI analysis.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co wwwpllllguuiiiieeeeeeeeeeeeeebll httpg//dd/eeeeeubbmmbmmcese www.//ooieeeisiiieeoaa wommissssoooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and testing details are clearly specified in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We report error bars appropriately, define them clearly, and provide relevant information statistical significance in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information about the computational resources used in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httus::/neurips..c
Answer: [Yes].
Justification: The research presented in this paper conforms the NeurTPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the Broader Impacts section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: This paper does not utilize pretrained language models, image generators, or datasets obtained through web scraping.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not use existing assets
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: This paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor direct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor direct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We used large language models solely for writing assistance, editing, and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) hbtte:///eerrrppcccc//ooeeeeeee hctp////eoeeeeeecee///////..1.. for what should or should not be described.
20


Reviews:
[{"review_id": "sfYfFPEXbm", "reviewer": "Reviewer_ThRu", "summary": "The paper introduces a novel method, OCL (Orthogonal Contrastive Learning), designed to enhance multi-subject and multi-site fMRI data analysis. The authors provide a thorough theoretical foundation and empirical validation of their approach, demonstrating improvements in feature representation quality and classification accuracy compared to existing methods.", "questions": "1.Baseline Comparisons: Lacks implementation details and hyperparameter settings for baselines like DeepGeoCCA and MindAligner, raising concerns about the fairness of comparisons.\n2.Generalizability: All experiments are on task-based fMRI; applicability to resting-state data or other modalities (e.g., MEG, EEG) is not evaluated or discussed.\n3.Computational Complexity: OCL uses deep transformers and large-scale synthetic pretraining, but memory usage, scalability, and training efficiency are not analyzed.\n4.LSH Robustness: While LSH is central to OCL, its sensitivity to noise, outliers, or small samples is not empirically validated.\n5.Interpretability: Despite strong classification performance, the paper lacks neuroscientific interpretation or visualization of learned features.\n6.Synthetic Data Realism: Pretraining uses simplistic Gaussian models that may not reflect real BOLD signal complexity, with no assessment of realism or potential bias.", "limitations": "refer to the questions", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.Innovative Framework: OCL combines orthogonal feature extraction, contrastive learning, and transformers to tackle key challenges in multi-subject fMRI analysis.\n2.Well-Designed Architecture: Uses a dual-encoder framework with online and target networks, integrating QR decomposition, LSH, positional encoding, and transformers effectively.\n3.Strong Performance: Demonstrates consistent superiority over existing methods across multiple fMRI datasets in both representation quality and classification accuracy.\n4.Comprehensive Evaluation: Tested on diverse fMRI tasks (task-based and naturalistic), with rigorous metrics, statistical analysis, and comparisons under various conditions.\n\nThis paper presents a promising and original approach for multi-subject and multi-site fMRI analysis. Addressing the points above would greatly strengthen the contribution and increase its relevance to both machine learning and neuroscience audiences.", "was_revised": true, "final_justification": "I am still positive to this paper.", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough and constructive feedback on our OCL manuscript. We appreciate your recognition of the framework’s technical soundness and originality. Below, we address each of your points in detail.\n\n1. Baseline Comparisons\n\nWe have tuned all competing methods (DeepGeoCCA, MindAligner, etc.) through nested leave-one-subject-out cross-validation, performing grid search on hyperparameters (kernel scales, regularization coefficients, learning rates) in the same way as for OCL. Implementations and pretrained weights follow the original papers, and we confirmed parameter choices with the authors where necessary. We evaluated each baseline at both fixed and optimally chosen latent dimensions, reporting the best results.\n\n2. Generalizability\n\nOCL is fundamentally modality-agnostic: the QR decomposition extracts orthogonal bases from any time-series matrix, and the LSH signature encodes these bases into compact, comparable codes. For resting-state fMRI, one can generate pseudo-labels by applying sliding-window functional connectivity or clustering temporal windows based on correlation patterns; OCL then maximizes contrastive agreement across these pseudo-classes, aligning subjects without explicit tasks. Similarly, for MEG/EEG data, the decomposition can operate on sensor- or source-space time-series, and LSH can hash spectral features or time–frequency representations. Positional encoding preserves temporal ordering even when sampling rates vary between modalities. Although detailed experiments on these extensions exceed our current scope, we include a dedicated “Future Extensions” discussion highlighting how to adapt OCL’s preprocessing and contrastive objectives to new neuroimaging modalities.\n\n3. Computational Complexity\n\nExperiments were conducted on two workstations (four GPUs total), each equipped with two NVIDIA GeForce RTX 4060 Ti GPUs (16 GB VRAM each). Supplementary Material Section C reports runtime footprints. For typical datasets, OCL inference requires approximately 14 GB VRAM on a single GPU. In the revised manuscript, we will expand this discussion by analyzing how memory usage and throughput scale with voxel count, time-series length, and model depth; we will also include guidelines for selecting batch sizes and hardware configurations for different dataset scales. The current analyses confirm that OCL’s resource demands are moderate and comparable to other transformer-based neuroimaging approaches.\n\n4. LSH Robustness\n\nOur theoretical analysis (Lemma 1) shows that LSH collision probability decreases smoothly with increasing Euclidean distance between orthogonal codes, providing inherent resilience to noise and minor perturbations. To validate this, we conducted a new empirical study in which we injected Gaussian noise into the synthetic pretraining data at SNR levels of 10 dB, 5 dB, and 0 dB. Across all conditions, classification accuracy dropped by less than 1 %, demonstrating high robustness. We will include these new analyses, along with additional experiments on outlier sensitivity and varying sample sizes, in the final version of the manuscript\n\n5. Interpretability\n\nWe acknowledge that interpretability in deep-learning–based neuroimaging is challenging and not the primary focus of this work, which aims to advance predictive alignment. Nevertheless, to provide initial insights into learned representations, we project the QR bases back into voxel space and visualize the most significant spatial components. Additionally, we extract self‑attention weights from the Transformer encoder and overlay these attention maps onto cortical surfaces to indicate which temporal segments drive alignment. We will include a discussion of these visualizations and their neuroscientific implications in the revised manuscript.\n\n6. Synthetic Data Realism\n\nThe Gaussian model in our pretraining is intended to teach orthogonal invariance rather than fully emulate hemodynamics. On held‑out real data, OCL representations improve both classification and within‑class correlation (ρ_raw=0.763→ρ_aligned=0.915, please see Supplementary Material Section B), indicating no adverse bias. We will acknowledge the value of more realistic simulators (e.g., HRF modelling) in future work.\n\nWe trust these clarifications address your concerns and strengthen our contribution. Thank you again for your valuable feedback. We appreciate your consideration and hope you will endorse our manuscript for acceptance.\n\nSincerely,\n\nThe Author(s)"}, {"review_id": "TBnesVdGte", "reviewer": "Reviewer_aFC5", "summary": "This paper introduces **Orthogonal Contrastive Learning** (OCL). The method aligns fMRI time-series across subjects and scanning sites without any temporal preprocessing. OCL employs a dual-encoder setup and extracts orthogonal features using QR decomposition. It then generates compact signatures via locality-sensitive hashing and injects time information continuously before embedding with a Transformer. The framework is pretrained on synthetic data and uses a cross-site transfer-learning workflow to boost downstream classification accuracy.", "questions": "1. **Computational Complexity:** Could the authors clarify the computational overhead associated with integrating multiple sophisticated modules (QR decomposition, LSH, positional encoding, Transformer)? Providing runtime or memory benchmarks would significantly strengthen the practical applicability of the proposed method.\n\n2. **Ablation Studies:** Could the authors present further ablations to individually highlight each component’s contribution (QR decomposition, LSH, positional encoding, Transformer)? This would greatly help readers understand the relative importance of each design choice.\n\n3. **Generalizability to Other Modalities:** Have the authors explored or considered applying OCL to resting-state fMRI or other neuroimaging modalities? Demonstrating broader applicability would significantly increase the potential impact of the work.", "limitations": "**No.**\n\nWhile the paper has a “Broader Impacts” section that describes its positive outcomes, it does not have a clear discussion of its own limitations. It also does not consider possible negative effects or dual‐use risks.\n\n**Suggestions:**\n\n- **Computational cost & scalability** OCL uses two encoders and deep transformer layers. This may need large GPU resources. Smaller labs may find it hard to run.\n- **Data requirements & generalization** The method needs big, well‐labeled fMRI datasets from many sites. This may not work for under‐represented groups or new tasks.\n- **Assumption sensitivity** The alignment assumes orthogonality and temporal stationarity. The authors should discuss what happens if these assumptions do not hold.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The OCL framework addresses a significant practical challenge—aligning heterogeneous fMRI datasets without requiring temporal preprocessing, potentially reducing overhead and improving flexibility.\n2. Experimental results appear solid, demonstrating improvements in alignment quality and downstream classification accuracy compared to several state-of-the-art methods across multiple multi-site benchmarks.\n3. Design choices, such as combining QR decomposition for orthogonal feature extraction with LSH signatures, seem well-motivated and original. Using Transformers for temporal dependencies also seems intuitive and empirically effective.\n\n**Weaknesses:**\n1. The method integrates several complex modules (QR decomposition, LSH, positional encoding, Transformer layers), which might increase computational complexity. A more detailed discussion on scalability or efficiency would be beneficial.\n2. Ablation studies could be expanded. Currently, the relative contributions of individual components like QR decomposition, LSH, positional encoding, or Transformer layers are not fully clear. Further clarification here would strengthen the paper’s claims.", "was_revised": true, "final_justification": "The rebuttal provided by the authors addresses my major concern. And I'd like to decide my final rating as **Accept**.", "rebuttal": "Dear Reviewer,\n\nThank you for your positive evaluation and for highlighting both the strengths of OCL and areas for clarification. Below, we address each of your points in detail.\n\n1. Computational Complexity & Scalability\n\nWe have quantified OCL’s overall computational overhead both analytically and empirically. As noted in Section 4, experiments were run on two workstations (four NVIDIA GeForce RTX 4060 Ti GPUs total), with end-to-end inference requiring about 14 GB of VRAM on a single GPU. Further, runtime analysis is provided in Supplementary Material Section C. In the revised manuscript, we will extend this discussion by examining how resource usage and throughput scale with voxel count, time-series length, and model depth, and by offering practical guidelines for batch sizes and hardware configurations. These results demonstrate that OCL’s computational demands are moderate and comparable to other transformer-based neuroimaging methods.\n\n2. Ablation Studies\n\nWe agree that isolating each component’s contribution strengthens the interpretability of design choices. In Supplementary Material Section D, we report an ablation on two cognitive-task datasets (CMU and DS232), comparing four variants (removing QR, LSH, positional encoding, or the Transformer) against OCL_zero and random-initialized pipelines. Results (Figure S2) show that removing any component degrades accuracy significantly—confirming that QR, LSH, positional encoding, and the Transformer each play an essential role.\n\n3. Generalizability to Other Modalities\n\nWhile our current evaluation focuses on task-based fMRI, OCL’s core mechanisms are inherently modality-agnostic, and we have actively considered applying it to resting-state and other neuroimaging modalities. For resting-state fMRI, pseudo-labels can be generated via sliding-window functional connectivity or clustering of temporal segments, enabling OCL to learn contrastive alignment without explicit tasks. For MEG and EEG, OCL can process sensor- or source-space time–frequency matrices by extracting orthogonal spectral bases with QR decomposition, hashing them via LSH signatures, and preserving temporal dynamics through positional encoding. Although comprehensive empirical evaluation on these modalities is beyond the scope of this submission, we include a detailed “Future Extensions” subsection in the revised manuscript, outlining protocols and preliminary guidelines for applying and benchmarking OCL on resting-state fMRI, MEG, and other time-series neuroimaging data.\n\n4.  Discussion of Limitations:\n\nIn the Conclusion and Broader Impacts sections, we already address batch-effect mitigation, reproducibility considerations, and computational costs; in the revised manuscript, we will add a dedicated Limitations section that explicitly highlights (i) scalability constraints of OCL’s multi-module architecture on GPU resources, (ii) potential sensitivity to extreme inter-site or inter-subject domain shifts beyond our current benchmarks, and (iii) its reliance on large, well-annotated multi-site datasets for robust performance and generalization.\n\nWe appreciate your support for our work and believe that these clarifications, analyses, and discussions will reinforce OCL’s practical applicability and scientific rigor. We would be grateful for your recommendation to accept our manuscript.\n\nSincerely,\n\nThe Author(s)"}, {"review_id": "nXGKZJk1z6", "reviewer": "Reviewer_xXsd", "summary": "This paper proposes Orthogonal Contrastive Learning (OCL), a novel multi-representation framework designed for multi-subject and multi-site task-based fMRI analysis. The method integrates several components—QR decomposition, locality-sensitive hashing (LSH), positional encoding, and a transformer encoder—within a dual-encoder architecture (online and target networks). The model is trained using a contrastive loss that aligns same-stimulus fMRI responses across subjects while preserving distinctiveness across different stimuli. A pretraining phase on synthetic fMRI-like data provides a robust initialization, followed by fine-tuning on real data. Extensive experiments demonstrate that OCL outperforms various state-of-the-art functional alignment and contrastive learning methods in both representation quality and downstream classification performance.", "questions": "1. How critical are QR decomposition and LSH individually? Could the authors provide ablation studies isolating these components?\n2. This method involves large Transformer networks and QR decomposition, both of which require training. Given the inherently noisy nature of fMRI data, it raises concerns about whether such a complex architecture is well-suited to this domain. Additionally, these components may introduce computational bottlenecks. How does the training time and memory usage compare to other approaches that leverage pretrained models without the need for such heavy computation?\n3. The positional encoding used in this paper has already been widely applied in the neuroimaging field. Could the authors clarify the specific purpose of using positional encoding in their framework? Is there any methodological innovation in how it is applied here, or is it directly borrowed from existing practices without adaptation?", "limitations": "The paper does not include a dedicated Limitations section, and the limitations mentioned in the main text are relatively sparse. It is recommended that the authors provide further discussion. For example, considering that the proposed method may potentially be applied to sensitive tasks such as cognitive state prediction, mental health diagnosis, or multi-subject generation, the authors are encouraged to briefly discuss the potential societal impact of such applications, including risks of misuse, and offer suggestions for mitigation. In addition, the authors should discuss the scalability boundaries of their approach and analyze the trade-offs between accuracy and computational efficiency, especially given the method's high computational cost.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper is clearly written and well-structured, especially the explanation of the OCL architecture.\n2. The mathematical principles proposed in this paper are all rigorously proven and thoroughly presented, reflecting the authors' strong mathematical foundation.\n3. The paper addresses a well-motivated and difficult problem in multi-subject fMRI analysis and demonstrates tangible improvements over strong baselines in challenging benchmarks.\n\nWeaknesses\n\n1. The paper takes a highly mathematical approach in selecting its methodological framework, incorporating techniques such as QR decomposition and locality-sensitive hashing (LSH). Although the authors provide proofs for the alignment procedures and technical details, what is more crucial is validating the effectiveness of these methods in multi-subject fMRI tasks. The current work lacks theoretical justification for their domain-specific applicability—for example, whether similar approaches have been previously applied in this field. If the proposed method is indeed pioneering, more detailed ablation studies should be conducted to verify its effectiveness, rather than relying solely on mathematical derivations.\n2. The paper explores only classification tasks in the experimental section, which is a rather basic evaluation for fMRI decoding. Given that many recent works—especially those using mainstream models like CLIP—have already achieved high classification accuracy, the current evaluation setting appears limited. The authors are encouraged to explore additional tasks such as retrieval and reconstruction to better demonstrate the generality and broader applicability of their proposed method.\n3. The balance between theory and practice in this paper is not well maintained— the theoretical part is disproportionately emphasized, while the experimental section is relatively limited. Moreover, the presentation of results is overly simplistic, relying solely on bar charts without providing the underlying numerical data in tables or other forms. This limits the clarity and reproducibility of the experimental findings.", "was_revised": true, "final_justification": "After considering the authors’ rebuttal, I have decided to raise my score from 3 to 4. The rebuttal effectively addressed most of my major concerns. Although some omissions mentioned in my comments reduce the perceived completeness and generality of the work, I believe the core contribution is technically solid and relevant to the community. I look forward to the authors’ final revised and polished manuscript.", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough evaluation and constructive suggestions. Below, we address each of your concerns in detail.\n\n1. Criticality of QR Decomposition and LSH\n\nIn Supplementary Material Section D, we perform targeted ablations on CMU and DS232 cognitive-task datasets to assess each component’s role. Omitting QR decomposition produces embeddings that reflect overall data structure but lack subject-level discrimination, whereas removing LSH leaves subspace rotations uncorrected, impairing alignment consistency. Only the full OCL model—jointly leveraging QR and LSH—delivers robust, discriminative embeddings across subjects and sites, demonstrating that both modules are indispensable to our framework.\n\n2. Computational Overhead \n\nWe have quantified OCL’s computational demands on two workstations—each outfitted with dual NVIDIA GeForce RTX 4060 Ti GPUs (16 GB VRAM). As detailed in Supplementary Material Section C and shown in Figure S1, OCL’s pretraining and fine-tuning runtimes are on par with leading self-supervised fMRI frameworks (e.g., MindEye2, MindAligner) and mirror the memory footprints of CLIP-based and other Transformer models adapted for neuroimaging. End-to-end inference on typical fMRI datasets requires approximately 14 GB of VRAM on a single GPU, confirming that OCL’s resource requirements are moderate and practical for the broader neuroimaging community. In the revised manuscript, we will extend this discussion by analyzing how memory usage and throughput scale with voxel count, time-series length, and model depth.\n\n3. Suitability for fMRI\n\nOur pretrained synthetic-data approach effectively mitigates sample scarcity in small fMRI datasets, enabling OCL to generalize from limited real data. The architecture is rooted in hyperalignment theory—QR decomposition aligns subject-specific neural subspaces, and LSH incorporates robustness to the low SNR characteristic of fMRI signals. Together, these components are specifically tailored for noisy neuroimaging data, and our empirical results across multiple benchmarks provide compelling evidence of OCL’s suitability and resilience in real-world fMRI analysis.\n\n4. Role of Positional Encoding\n\nWe employ sinusoidal positional encoding—widely adopted in prior fMRI Transformer models—to inject continuous temporal context into spatial feature representations. Although the encoding formula is standard, our contribution lies in integrating it with QR-derived bases and LSH signatures under variable-length padding masks, ensuring that temporal ordering is preserved even when time-series lengths differ. This integration is critical for aligning dynamic stimulus responses without explicit preprocessing and is not a direct copy but an adaptation tailored to OCL’s multi-component layers.\n\n5. Limitations and Dual-Use Considerations\n\nWe will add a dedicated Limitations section in the final manuscript to acknowledge:\n\nSocietal & Dual-Use Risks: OCL could be applied to sensitive tasks (e.g., cognitive state inference or mental health diagnostics). We will recommend ethical safeguards, strict anonymization, and compliance with institutional review boards to mitigate misuse.\n\nScalability Boundaries: OCL’s multi-module architecture demands multi-GPU hardware (≈14 GB VRAM per GPU); we will discuss optional reductions in embedding dimension or Transformer depth to trade off accuracy and resource use.\n\nData Requirements & Generalization: OCL benefits from large, well-annotated multi-site datasets; we will caution about performance degradation under low-data regimes and suggest few-shot adaptation or synthetic augmentation as remedies.\n\n6. Presentation & Reproducibility\n\nWe apologize that presenting results solely with bar charts may limit clarity and reproducibility. While visual summaries provide quick insights, we will include detailed numerical tables in the Supplementary Material to ensure full traceability and precise comparisons.\n\n7. Task Scope & Future Directions\n\nOur primary objective is to deliver a predictive alignment framework with immediate practical utility for classification-based fMRI analyses in clinical and cognitive neuroscience. Although more advanced brain-decoding tasks—such as retrieval or stimulus reconstruction—offer valuable insights, they extend beyond the current scope of this paper. Crucially, OCL already surpasses recent CLIP-based and other Transformer models (e.g., MindEye2 or MindAligner) in classification accuracy across multi-site benchmarks, highlighting its efficacy in real-world predictive scenarios. We plan to investigate retrieval and reconstruction tasks in future work, leveraging OCL-aligned representations as a foundation.\n\nThank you again for your insightful feedback. We hope this revision addresses all your concerns and highlights OCL’s practical benefits and scientific rigor. We appreciate your consideration and kindly ask for your endorsement of our manuscript.\n\n\nSincerely,\n\nThe Author(s)"}, {"review_id": "z9Pd0WU5nG", "reviewer": "Reviewer_xAGi", "summary": "This paper propose a method called orthogonal contrastive learning (OCL) to align fMRI response across multiple subjects without the need for aligned temporal data. The method consists of QR decomposition for orthogonal feature extraction, locality-sensitive hashing to produce subject specific codes, positional encoding for embedding temporal information, and a transformer encoder to generate the embeddings.  The paper also proposes an initialization method using fMRI-like synthetic data and an approach for transfer learning for multi-site studies. Experiments on multiple datasets shows OCL outperformed multiple baselines in classification accuracy of fMRI stimuli.", "questions": "1. How do the different model design components contribute to the overall performance of the model? \n\n2. What are the ROIs used in the experiments? Is it whole brain? How are they defined/decided? Are the same ROIs/inputs used for each of the baseline methods? \n\n3. Does the performance of OCL vary greatly with changing hyper parameter values? \n\n4. I am somewhat confused about the transfer learning section. Did the authors do the extra active learning loop annotating testing site data with greatest uncertainty, as mentioned at the bottom of p.6? Do any other baselines leverage this added information? If not the comparison does not seem quite fair. \n\n5. Also, I'm confused about the multi-site experiment setup. It seems that only 1 site was used for training, then transferred to one for testing, and multiple sites are thus not used for training. Is this correct? Then it is not testing the multi-site approach in 3.2.", "limitations": "Checklist claims authors discuss limitations in conclusions/broader impacts, but I don't see such discussion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed approach does not require preprocessing of the fMRI data such that sequences must be aligned or of the same length. This aids in usability of the model and ease in combining different datasets.\n2. The experiments were performed using 10 datasets (9 open), classifying different types of fMRI stimuli and using fMRI data obtained from multiple different sites.\n3. The experiments involved comparison against up to 7 baselines, with some statistical analysis of results.  \n\nWeaknesses:  \n1. Questions about model design:  \nWhile the proposed approach does not require aligned sequences, I have concerns about the removal of considering the spatiotemporal data as a whole. While temporal encoding is included, this does not include information about the temporal stimulus information that occurs beforehand (as far as I can tell). The neural response likely depends on what information was presented before. Since the analysis is on a time-point basis, one would lose this important information.\n2. Lack of ablation studies / understanding of contribution of model components:   \nThe proposed method involves many different components. However, the experiments only assess the model as a whole, while the authors conclude the improvement is due to different factors, but there is no direct evidence/evaluation of this. The lack of ablation studies / assessment of contribution of different model components makes it so we cannot understand the advantage of different design choices, which may or may not be effective.\n3. Sensitivity to hyperparameters:  \nThe experiments do not present any sensitivity analysis of hyperparameters. Furthermore, it is clear that the many hyperparameters of OCL were tuned, but it is not clear whether the competing methods had as much care in hyper parameter tuning, and therefore there are concerns whether all models were optimized to their best.", "was_revised": true, "final_justification": "Please see longer response below - the authors helped point out missed ablation experiments and provided many clarifications that addressed most of my questions.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful critique and constructive suggestions. We address each point below, citing relevant sections of our manuscript and outlining planned enhancements.\n\n1. Ablation Studies & Component Contributions\n\nIn Supplementary Material Section D, we perform ablation experiments on the CMU and DS232 datasets by systematically removing or substituting each OCL component (QR decomposition, LSH, positional encoding, and Transformer). Across all variants, we observe consistent declines in both alignment consistency and downstream classification performance relative to the full OCL model, confirming that each module contributes meaningfully to our framework.\n\n2. Regions of Interest (ROI)\n\nAs detailed in Section 3.1 and evaluated in Section 4, we employ a whole-brain ROI defined on the MNI152 T1 template (4 mm isotropic resolution), resulting in V = 19,742 voxels per volume. This single mask is used consistently during both pretraining and fine-tuning to ensure identical input dimensions across all subjects, sites, and datasets. We will include a comprehensive description of the ROI selection process in the final manuscript to enhance reproducibility.\n\n3. Hyperparameter Sensitivity & Fair Tuning\n\nWe evaluated OCL and all baseline methods using nested leave-one-subject-out cross-validation with comprehensive grid searches over key hyperparameters (e.g., learning rate, temperature, margin). Across these experiments, OCL’s classification accuracy varied by less than 5% in nearly all configurations, demonstrating stability and robustness to hyperparameter choices. Identical tuning protocols and search spaces were applied to all methods to ensure fair and consistent comparisons. We will provide further details in the final manuscript to enhance reproducibility.\n\n4. Transfer Learning & Active Learning Loop\n\nSection 3.2 (bottom of p. 6) describes an optional active-learning loop: after aggregating multi-site encoder parameters, we identify high-uncertainty TRs for expert annotation and iteratively refine the target encoder. To ensure fair comparison, our current experiments did not employ this loop, and no expert feedback was used. While integrating active learning is a promising direction for future extensions of OCL, it falls outside the scope of this work. We will explicitly clarify in the final manuscript that all reported results are obtained without any active-learning or additional expert annotations.\n\n5. Multi-Site Experimental Setup\n\nWe apologize for any confusion regarding our site definitions. In our primary multi-site evaluation, dataset DS007 is partitioned into three independent ‘sites’—pseudo‑word naming (A1), letter naming (A2), and manual response (A3)—while dataset DS008 (B) serves as a single held-out site, following the experimental design of SSTL (NeurIPS 2020). We first train on {A1 + A2 + A3} (B=3 in Eq. 4) and test on B, then reverse by training on B and testing on each of A1, A2, and A3, averaging these bidirectional accuracies.  To further support the effectiveness of our proposed OCL framework, we report additional results based on an exhaustive set of two-versus-two train/test splits formed from the set {A1, A2, A3, B}, resulting in six unique comparisons (e.g., training on {A1, A2} and testing on {A3, B}, and vice versa). In these experiments, B = 2 in Eq. (4). Across all splits, OCL achieved a mean accuracy of %98.23 ± 0.64, significantly outperforming the second-best baseline, MindAligner (%91.63 ± 1.71). For the remaining datasets (C–F), each comprising only two scanning sites, we perform a similar train/test swap procedure (B = 1 in Eq. 4), which still qualifies as a multi-site analysis. We will provide a detailed explanation of these protocols along with comprehensive results in the final manuscript.\n\nThank you once again for your thoughtful and constructive feedback. We hope that the revisions satisfactorily address your concerns and effectively underscore the practical value and scientific contributions of OCL. We sincerely appreciate your consideration and respectfully request your support for the publication of our manuscript.\n\n\nSincerely,\n\nThe Author(s)"}]

Author rebuttal:

Meta-review: This paper introduces Orthogonal Contrastive Learning (OCL), a representation framework designed for task-based fMRI analysis involving multiple subjects and sites. This method incorporates QR decomposition, locality-sensitive hashing, positional encoding, and a transformer encoder within a dual-encoder architecture comprising online and target networks. The model is trained using contrastive loss to align same-stimulus fMRI responses across subjects while preserving distinctiveness across different stimuli. Pretraining on synthetic fMRI-like data provides robust initialization, followed by fine-tuning on real data. Extensive experiments demonstrate that OCL outperforms various state-of-the-art functional alignment and contrastive learning methods in terms of both representation quality and downstream classification performance.
The reviewers noted that the paper is more oriented toward the technique than toward applications.
They raised questions about the baseline comparison, the need for ablation studies, the computational complexity, the robustness of the locality-sensitive hashing (LSH) encoding, the realism of the synthetic data, and the generalization to other modalities and interpretability.
These points have been successfully addressed in the rebuttal process. However, this has resulted in a significant discrepancy between the submitted paper and the intended final version. Other concerns remain, such as the lack of consideration of spatiotemporal information in fMRI data. However, the consensus is that the paper is acceptable.
Decision: Accept (poster)