Title: Orthogonal Contrastive Learning for Multi-Representation fMRI Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: fMRI analysis

Contrastive Learning

Functional Alignment

Transfer Learning

Multi-Representation fMRI Analysis
Abstract: Task-based functional magnetic resonance imaging (fMRI) provides invaluable insights into human cognition but faces critical hurdles—low signal-to-noise ratio, high dimensionality, limited sample sizes, and costly data acquisition—that are amplified when integrating datasets across subjects or sites. This paper introduces orthogonal contrastive learning (OCL), a unified multi-representation framework for multi-subject fMRI analysis that aligns neural responses without requiring temporal preprocessing or uniform time-series lengths across subjects or sites. OCL employs two identical encoders: an online network trained with a contrastive loss that pulls together same-stimulus responses and pushes apart different-stimulus responses, and a target network whose weights track the online network via exponential moving average to stabilize learning. Each OCL network layer combines QR decomposition for orthogonal feature extraction, locality-sensitive hashing (LSH) to produce compact subject-specific signatures, positional encoding to embed temporal structure alongside spatial features, and a transformer encoder to generate discriminative, stimulus-aligned embeddings. We further enhance OCL with an unsupervised pretraining stage on fMRI-like synthetic data and demonstrate a transfer-learning workflow for multi-site studies. Across extensive experiments on multi-subject and multi-site fMRI benchmarks, OCL consistently outperforms state-of-the-art alignment and analysis methods in both representation quality and downstream classification accuracy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a method called orthogonal contrastive learning (OCL) to align fMRI response across multiple subjects without the need for aligned temporal data. The method consists of QR decomposition for orthogonal feature extraction, locality-sensitive hashing to produce subject specific codes, positional encoding for embedding temporal information, and a transformer encoder to generate the embeddings.  The paper also proposes an initialization method using fMRI-like synthetic data and an approach for transfer learning for multi-site studies. Experiments on multiple datasets shows OCL outperformed multiple baselines in classification accuracy of fMRI stimuli.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed approach does not require preprocessing of the fMRI data such that sequences must be aligned or of the same length. This aids in usability of the model and ease in combining different datasets.\n2. The experiments were performed using 10 datasets (9 open), classifying different types of fMRI stimuli and using fMRI data obtained from multiple different sites.\n3. The experiments involved comparison against up to 7 baselines, with some statistical analysis of results.  \n\nWeaknesses:  \n1. Questions about model design:  \nWhile the proposed approach does not require aligned sequences, I have concerns about the removal of considering the spatiotemporal data as a whole. While temporal encoding is included, this does not include information about the temporal stimulus information that occurs beforehand (as far as I can tell). The neural response likely depends on what information was presented before. Since the analysis is on a time-point basis, one would lose this important information.\n2. Lack of ablation studies / understanding of contribution of model components:   \nThe proposed method involves many different components. However, the experiments only assess the model as a whole, while the authors conclude the improvement is due to different factors, but there is no direct evidence/evaluation of this. The lack of ablation studies / assessment of contribution of different model components makes it so we cannot understand the advantage of different design choices, which may or may not be effective.\n3. Sensitivity to hyperparameters:  \nThe experiments do not present any sensitivity analysis of hyperparameters. Furthermore, it is clear that the many hyperparameters of OCL were tuned, but it is not clear whether the competing methods had as much care in hyper parameter tuning, and therefore there are concerns whether all models were optimized to their best.\n\nQUESTIONS:\n1. How do the different model design components contribute to the overall performance of the model? \n\n2. What are the ROIs used in the experiments? Is it whole brain? How are they defined/decided? Are the same ROIs/inputs used for each of the baseline methods? \n\n3. Does the performance of OCL vary greatly with changing hyper parameter values? \n\n4. I am somewhat confused about the transfer learning section. Did the authors do the extra active learning loop annotating testing site data with greatest uncertainty, as mentioned at the bottom of p.6? Do any other baselines leverage this added information? If not the comparison does not seem quite fair. \n\n5. Also, I'm confused about the multi-site experiment setup. It seems that only 1 site was used for training, then transferred to one for testing, and multiple sites are thus not used for training. Is this correct? Then it is not testing the multi-site approach in 3.2.\n\nLIMITATIONS:\nChecklist claims authors discuss limitations in conclusions/broader impacts, but I don't see such discussion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Orthogonal Contrastive Learning (OCL), a novel multi-representation framework designed for multi-subject and multi-site task-based fMRI analysis. The method integrates several components—QR decomposition, locality-sensitive hashing (LSH), positional encoding, and a transformer encoder—within a dual-encoder architecture (online and target networks). The model is trained using a contrastive loss that aligns same-stimulus fMRI responses across subjects while preserving distinctiveness across different stimuli. A pretraining phase on synthetic fMRI-like data provides a robust initialization, followed by fine-tuning on real data. Extensive experiments demonstrate that OCL outperforms various state-of-the-art functional alignment and contrastive learning methods in both representation quality and downstream classification performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper is clearly written and well-structured, especially the explanation of the OCL architecture.\n2. The mathematical principles proposed in this paper are all rigorously proven and thoroughly presented, reflecting the authors' strong mathematical foundation.\n3. The paper addresses a well-motivated and difficult problem in multi-subject fMRI analysis and demonstrates tangible improvements over strong baselines in challenging benchmarks.\n\nWeaknesses\n\n1. The paper takes a highly mathematical approach in selecting its methodological framework, incorporating techniques such as QR decomposition and locality-sensitive hashing (LSH). Although the authors provide proofs for the alignment procedures and technical details, what is more crucial is validating the effectiveness of these methods in multi-subject fMRI tasks. The current work lacks theoretical justification for their domain-specific applicability—for example, whether similar approaches have been previously applied in this field. If the proposed method is indeed pioneering, more detailed ablation studies should be conducted to verify its effectiveness, rather than relying solely on mathematical derivations.\n2. The paper explores only classification tasks in the experimental section, which is a rather basic evaluation for fMRI decoding. Given that many recent works—especially those using mainstream models like CLIP—have already achieved high classification accuracy, the current evaluation setting appears limited. The authors are encouraged to explore additional tasks such as retrieval and reconstruction to better demonstrate the generality and broader applicability of their proposed method.\n3. The balance between theory and practice in this paper is not well maintained— the theoretical part is disproportionately emphasized, while the experimental section is relatively limited. Moreover, the presentation of results is overly simplistic, relying solely on bar charts without providing the underlying numerical data in tables or other forms. This limits the clarity and reproducibility of the experimental findings.\n\nQUESTIONS:\n1. How critical are QR decomposition and LSH individually? Could the authors provide ablation studies isolating these components?\n2. This method involves large Transformer networks and QR decomposition, both of which require training. Given the inherently noisy nature of fMRI data, it raises concerns about whether such a complex architecture is well-suited to this domain. Additionally, these components may introduce computational bottlenecks. How does the training time and memory usage compare to other approaches that leverage pretrained models without the need for such heavy computation?\n3. The positional encoding used in this paper has already been widely applied in the neuroimaging field. Could the authors clarify the specific purpose of using positional encoding in their framework? Is there any methodological innovation in how it is applied here, or is it directly borrowed from existing practices without adaptation?\n\nLIMITATIONS:\nThe paper does not include a dedicated Limitations section, and the limitations mentioned in the main text are relatively sparse. It is recommended that the authors provide further discussion. For example, considering that the proposed method may potentially be applied to sensitive tasks such as cognitive state prediction, mental health diagnosis, or multi-subject generation, the authors are encouraged to briefly discuss the potential societal impact of such applications, including risks of misuse, and offer suggestions for mitigation. In addition, the authors should discuss the scalability boundaries of their approach and analyze the trade-offs between accuracy and computational efficiency, especially given the method's high computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **Orthogonal Contrastive Learning** (OCL). The method aligns fMRI time-series across subjects and scanning sites without any temporal preprocessing. OCL employs a dual-encoder setup and extracts orthogonal features using QR decomposition. It then generates compact signatures via locality-sensitive hashing and injects time information continuously before embedding with a Transformer. The framework is pretrained on synthetic data and uses a cross-site transfer-learning workflow to boost downstream classification accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The OCL framework addresses a significant practical challenge—aligning heterogeneous fMRI datasets without requiring temporal preprocessing, potentially reducing overhead and improving flexibility.\n2. Experimental results appear solid, demonstrating improvements in alignment quality and downstream classification accuracy compared to several state-of-the-art methods across multiple multi-site benchmarks.\n3. Design choices, such as combining QR decomposition for orthogonal feature extraction with LSH signatures, seem well-motivated and original. Using Transformers for temporal dependencies also seems intuitive and empirically effective.\n\n**Weaknesses:**\n1. The method integrates several complex modules (QR decomposition, LSH, positional encoding, Transformer layers), which might increase computational complexity. A more detailed discussion on scalability or efficiency would be beneficial.\n2. Ablation studies could be expanded. Currently, the relative contributions of individual components like QR decomposition, LSH, positional encoding, or Transformer layers are not fully clear. Further clarification here would strengthen the paper’s claims.\n\nQUESTIONS:\n1. **Computational Complexity:** Could the authors clarify the computational overhead associated with integrating multiple sophisticated modules (QR decomposition, LSH, positional encoding, Transformer)? Providing runtime or memory benchmarks would significantly strengthen the practical applicability of the proposed method.\n\n2. **Ablation Studies:** Could the authors present further ablations to individually highlight each component’s contribution (QR decomposition, LSH, positional encoding, Transformer)? This would greatly help readers understand the relative importance of each design choice.\n\n3. **Generalizability to Other Modalities:** Have the authors explored or considered applying OCL to resting-state fMRI or other neuroimaging modalities? Demonstrating broader applicability would significantly increase the potential impact of the work.\n\nLIMITATIONS:\n**No.**\n\nWhile the paper has a “Broader Impacts” section that describes its positive outcomes, it does not have a clear discussion of its own limitations. It also does not consider possible negative effects or dual‐use risks.\n\n**Suggestions:**\n\n- **Computational cost & scalability** OCL uses two encoders and deep transformer layers. This may need large GPU resources. Smaller labs may find it hard to run.\n- **Data requirements & generalization** The method needs big, well‐labeled fMRI datasets from many sites. This may not work for under‐represented groups or new tasks.\n- **Assumption sensitivity** The alignment assumes orthogonality and temporal stationarity. The authors should discuss what happens if these assumptions do not hold.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method, OCL (Orthogonal Contrastive Learning), designed to enhance multi-subject and multi-site fMRI data analysis. The authors provide a thorough theoretical foundation and empirical validation of their approach, demonstrating improvements in feature representation quality and classification accuracy compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.Innovative Framework: OCL combines orthogonal feature extraction, contrastive learning, and transformers to tackle key challenges in multi-subject fMRI analysis.\n2.Well-Designed Architecture: Uses a dual-encoder framework with online and target networks, integrating QR decomposition, LSH, positional encoding, and transformers effectively.\n3.Strong Performance: Demonstrates consistent superiority over existing methods across multiple fMRI datasets in both representation quality and classification accuracy.\n4.Comprehensive Evaluation: Tested on diverse fMRI tasks (task-based and naturalistic), with rigorous metrics, statistical analysis, and comparisons under various conditions.\n\nThis paper presents a promising and original approach for multi-subject and multi-site fMRI analysis. Addressing the points above would greatly strengthen the contribution and increase its relevance to both machine learning and neuroscience audiences.\n\nQUESTIONS:\n1.Baseline Comparisons: Lacks implementation details and hyperparameter settings for baselines like DeepGeoCCA and MindAligner, raising concerns about the fairness of comparisons.\n2.Generalizability: All experiments are on task-based fMRI; applicability to resting-state data or other modalities (e.g., MEG, EEG) is not evaluated or discussed.\n3.Computational Complexity: OCL uses deep transformers and large-scale synthetic pretraining, but memory usage, scalability, and training efficiency are not analyzed.\n4.LSH Robustness: While LSH is central to OCL, its sensitivity to noise, outliers, or small samples is not empirically validated.\n5.Interpretability: Despite strong classification performance, the paper lacks neuroscientific interpretation or visualization of learned features.\n6.Synthetic Data Realism: Pretraining uses simplistic Gaussian models that may not reflect real BOLD signal complexity, with no assessment of realism or potential bias.\n\nLIMITATIONS:\nrefer to the questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}