Title: Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Pathology Foundation Models

Whole Slide Images

Mutation Detection

Vision Transformer

Multiple Instance Learning
Abstract: Pathology foundation models (PFMs) have emerged as powerful tools for analyzing whole slide images (WSIs). However, adapting these pretrained PFMs for specific clinical tasks presents considerable challenges, primarily due to the availability of only weak (WSI-level) labels for gigapixel images, necessitating multiple instance learning (MIL) paradigm for effective WSI analysis. This paper proposes a novel approach for single-GPU \textbf{T}ask \textbf{A}daptation of \textbf{PFM}s (TAPFM) that uses vision transformer (\vit) attention for MIL aggregation while optimizing both for feature representations and attention weights. The proposed approach maintains separate computational graphs for MIL aggregator and the PFM to create stable training dynamics that align with downstream task objectives during end-to-end adaptation. Evaluated on mutation prediction tasks for bladder cancer and lung adenocarcinoma across institutional and The Cancer Genome Atlas (TCGA) cohorts, TAPFM consistently outperforms conventional approaches, with H-Optimus-0 (TAPFM) outperforming the benchmarks. TAPFM effectively handles multi-label classification of actionable mutations as well. Thus, TAPFM makes adaptation of powerful pre-trained PFMs practical on standard hardware for various clinical applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the classification of whole slide pathology images. It proposes a method for adapting pretrained pathology foundation models (PFMs) to specific downstream tasks by decoupling and alternating the optimization of feature representations and attention weights. Experimental evaluations on bladder and lung cancer datasets, for both binary malignancy classification and multi-label mutation prediction, demonstrate improved performance over existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n*Presentation and Technical Contributions:*\nThe paper is generally well written and clearly organized. The proposed approach appears reproducible, and the authors provide code to support replication. However, the technical description in Section 3 lacks clarity. Concretely, the core contribution seems to boil down to disabling gradients and alternating between two optimization steps. While this may still be a reasonable and useful strategy, its novelty and significance are somewhat limited and would benefit from a clearer, more direct, exposition. The justification provided in Proposition 1 feels superficial and does not convincingly support the conclusion that decoupling leads to “more stable parameter trajectories” than joint optimization.\nLastly, the title emphasizes “single GPU,” but the manuscript does not clearly explain whether existing approaches are incompatible with single-GPU training, or whether this limitation is a motivation for the proposed method. It remains unclear if the method is specifically designed to reduce computational requirements, or if the focus is instead on improving training dynamics and predictive performance. Clarifying this point would help align the title with the actual contributions and goals of the paper.\n\n*Experimental Evaluation:*\nThe experimental results are promising and demonstrate improvements over baseline methods. However, the scope of the evaluation is somewhat limited:\n(a) It is confined to two tasks (bladder and lung cancer), both using what appears to be a very similar setup (e.g., the same set of classes for multi-label classification).\n(b) Some experimental details are unclear; for example, how the locally collected data are integrated with publicly available datasets.\n(c) Relying solely on AUC as the evaluation metric reduces the clinical relevance and impact, as it does not capture all aspects of model performance in comparison to other methods.\n\n*Additional Comments:*\n* Line 12: Define \"TCGA\" at first mention.\n* Line 13: Define \"H-Optimus-0\" at first mention.\n* Lines 21–24: Please indicate the resolution of a whole-slide image (WSI), in addition to the individual pixel size.\n* Line 39: The term \"dual-loss mechanism\" is unclear and not elaborated on later in the paper. Please clarify.\n* Section 2 (especially 2.1): The section is dense with abbreviations, which makes it difficult to follow. Consider simplifying or expanding key terms.\n* Line 91: What is a \"learnable CLS token\"? Please provide an explanation or reference.\n* Section 3.2: Please clarify which components (e.g., Equation 1) are based on existing methods and which represent novel contributions.\n* Equation 5: Should the notation be $g_{z_i}$ instead?\n* Equations 5 and 6: Why are inconsistent notations ($\\times$ and $\\cdot$) used? Is there a distinction that should be clarified?\n* Lines 142–144: Please justify the claim made in this sentence; it currently lacks sufficient support.\n* Proposition 1: The statement and proof are vague; please see comments above regarding the need for a clearer and more rigorous justification.\n* Line 178: Define \"EGFR, KRAS, MET, and ALK\" and provide appropriate references.\n* Line 189: What is \"Otsu thresholding\"? A brief explanation or citation would help readers unfamiliar with the term.\n* Line 197: Shouldn't the number of tiles used during inference match the number used during training? \n* Section 4.2: How does the training/inference runtime compare to that of other (non-decoupled) methods? \n* Line 244: Since the loss depends on $\\lambda$, does a lower numerical loss value actually reflect better performance? \n* Table 2: Why is there no comparison to methods without TAPFM?\n\nQUESTIONS:\n1) Improve the clarity and precision of the technical contributions described in Section 3.\n2) Include additional evaluation metrics beyond AUC for the existing experiments on BLCA and LUAD. This would provide a better picture of model performance, capturing not only statistical power but also the utility of predictions on test data. Metrics such as accuracy, precision, recall, F1 score, or calibration measures could demonstrate the robustness and clinical relevance of the method compared to other approaches.\n3) Additional minor issues mentioned above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to adapt pathology foundation models for whole slide tissue image (WSI) classification. The proposed method makes use of the attention mechanism of the vision transformer architecture for feature aggregation and employs separate computation graphs for parameter updates.\n\nSTRENGTHS AND WEAKNESSES:\nThis study introduces a different approach to integrating and adapting foundation models and multi-instance learning (MIL) based classification models for WSI classification tasks. The experiments with several pathology foundation models (PFMs) and MIL approaches show that for certain PFM and MIL integrations, the proposed method achieves good performance improvement (e.g, UNI+DSMIL vs UNI (TAPFM) on TCGA LUAD dataset). However, when the best performance of the proposed method (H-Optimus-0 TAPFM) is compared with that of the best alternative approach (H-Optimus-O+ABMIL FT), the improvements are marginal and may not have much impact in a clinical research setting.  \n\nAdditional investigation is needed to better assess the benefits and limitations of the proposed method. The WSI classification task may not be the best analysis task to evaluate this method. Other analysis tasks such as survival analysis may benefit more. \nMoreover, the authors use relatively large institutional datasets to train their models. One of the main benefits of using foundation models is to be able to use relatively limited volume of training data. The authors should consider training their models with TCGA data and use the institutional datasets for testing. \n\nSome recent works have demonstrated high performances (e.g. [1,2]; [1] shows AUC scores of about 0.97). There should be a comparison with those works as well. While adaptation of PFMs is a reasonable approach, clinical research studies will want to use the best methods available regardless of the underlying methodology. \n\n[1] Dang, Thao M., Yuzhi Guo, Hehuan Ma, Qifeng Zhou, Saiyang Na, Jean Gao, and Junzhou Huang. \"MFMF: Multiple Foundation Model Fusion Networks for Whole Slide Image Classification.\" In Proceedings of the 15th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1-8. 2024.\n[2] Li, Hao, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. \"Generalizable whole slide image classification with fine-grained visual-semantic interaction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11398-11407. 2024.\n\nQUESTIONS:\nAdditional investigation is needed to better assess the benefits and limitations of the proposed method. The WSI classification task may not be the best analysis task to evaluate this method. Other analysis tasks such as survival analysis may benefit more. \n\nMoreover, the authors use relatively large institutional datasets to train their models. One of the main benefits of using foundation models is to be able to use relatively limited volume of training data. The authors should consider training their models with TCGA data and use the institutional datasets for testing.\n\nLIMITATIONS:\nSome recent works have demonstrated high performances (e.g. [1,2]; [1] shows AUC scores of about 0.97). There should be a comparison with those works as well. While adaptation of PFMs is a reasonable approach, clinical research studies will want to use the best methods available regardless of the underlying methodology. \n\n[1] Dang, Thao M., Yuzhi Guo, Hehuan Ma, Qifeng Zhou, Saiyang Na, Jean Gao, and Junzhou Huang. \"MFMF: Multiple Foundation Model Fusion Networks for Whole Slide Image Classification.\" In Proceedings of the 15th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1-8. 2024.\n[2] Li, Hao, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. \"Generalizable whole slide image classification with fine-grained visual-semantic interaction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11398-11407. 2024.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n(1) The paper proposes TAPFM, a two-stage optimization scheme that lets Vision-Transformer pathology foundation models be fine-tuned end-to-end on a single GPU.\n\n (2) In the forward pass, the ViT yields tile features and CLS-token attention scores; the latter are soft-maxed to weights and used to build a permutation-invariant slide representation.\n\n (3) Stage-1 back-prop updates only a lightweight linear classifier with a standard cross-entropy loss, while Stage-2 back-prop uses detached gradients from the first stage to craft a second loss that jointly pushes the ViT’s features and attentions, updating the backbone but not the classifier and thereby avoiding unstable gradient feedback.\n\n (4) The authors supply proofs of permutation invariance, an $O(K C_{ViT}​)$ time bound (K is tiles per slide, $C_{ViT}​$ the cost of one forward + backward through the backbone), and an argument that the detached update breaks circular dependencies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Existing adaptation workflows keep the PFM frozen and add a separate MIL head, or perform a self-supervised intermediate step before a second-stage supervised training; truly end-to-end slide-level back-prop has been deemed too memory-hungry. TAPFM shows that full, single-stage fine-tuning of billion-parameter PFMs is feasible on one high-memory GPU, eliminating the complexity of multi-stage procedures while delivering better accuracy\n\n(2) Leveraging the ViT’s CLS-token attention scores as tile weights eliminates the need for a separate aggregation network and yields a permutation-invariant bag representation with virtually no additional computation.\n\n(3) Across two cancer types, four actionable mutations, three PFMs (UNI, GigaPath, H-Optimus-0), and an external TCGA validation, TAPFM delivers 1–4 percentage points AUC improvements over both fixed-feature and naïve end-to-end baselines.\n\n(4) The same algorithm works unchanged on ViT-L, ViT-H, and 1 B-parameter ViT-G backbones, and scales to the simultaneous prediction of four mutations, including rare MET and ALK events.\n\n(5) A formal derivation shows that detaching the first-stage gradients eliminates the circular dependency between backbone and aggregator updates; empirically, this corresponds to smoother, faster convergence.\n\n(6) The paper proves permutation invariance of the bag representation and shows that total compute per slide grows linearly with the number of tiles times the cost of a single ViT pass.\n\n(7) Algorithm 1 is explicit, hyperparameters and memory budgets are reported, ablations on λ and tile count are included.\n\nWeaknesses:\n\n(1) The evaluation compares TAPFM only to fixed-feature MIL heads and naïve end-to-end training. It omits any self-supervised adaptation pipelines, making it harder to gauge the relative benefit of the proposed approach.\n\n(2) Performance is given as single AUC values with no confidence intervals or other measures of statistical significance, making it unclear whether the 1–4-point improvements are reliable.\n\n(3) All experiments are limited to mutation prediction in two tumour types. Because no results are shown for other high-value slide-level objectives—such as grading, subtype detection, prognosis, or treatment-response—the work’s broader significance to computational pathology remains uncertain; demonstrating gains on a more varied task set would better substantiate TAPFM’s impact.\n\n(4) The experiments are indeed run on a single device, but that device is an 80 GB H100. Readers should keep in mind that “single GPU” in this work specifically means a high-memory, top-tier card.\n\n(5) The methodology is designed for ViT-based models. Though most PFMs are ViT-based today, it could be a limitation for future PFM.\n\nQUESTIONS:\n(1) Could you provide formal measures of statistical significance—such as confidence intervals or p-values—for the reported AUC improvements to verify that the 1–4-point gains are reliable?\n\n(2) How does TAPFM’s accuracy and runtime change when the tile budget is reduced to fit GPUs with 24 GB or 48 GB memory? Is it even possible to fit in those GPUs? An ablation would clarify hardware requirements.\n\n(3) Can you supply empirical evidence (e.g., gradient-norm or loss-variance plots) that demonstrates the claimed stabilisation from the detached two-stage update compared with joint optimisation?\n\n(4) Do the learned CLS-attention weights correspond to histologically meaningful regions? A few slide visualisations or a brief expert evaluation would bolster confidence in the weighting mechanism.\n\n(5) Have you evaluated TAPFM on other clinically important slide-level tasks (grading, subtype classification, prognosis, treatment response, or datasets beyond BLCA and LUAD)? Additional results would help establish the method’s significance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe size of WSIs is typically extremely huge. Existing WSI models are unable to model them in an end-to-end manner. As a result, the paradigm of combining pretrained models with MIL has become a mainstream approach in the field of WSI analysis. This paper addresses the common challenge in existing methods where features extracted by frozen pretrained models are not well adapted to downstream tasks, while end-to-end training remains infeasible. It proposes a task-adaptive method for PFMs, which leverages the internal attention mechanism of ViTs for MIL feature aggregation, and separately optimizes the PFM and MIL components via detached computation graphs. The proposed approach achieves state-of-the-art performance across multiple datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper achieves the optimization of both the foundation model for image patch feature extraction and the MIL model for feature aggregation on a single GPU.\n2. It proposes a novel method for constructing separate computation graphs.\n3. It cleverly leverages internal attention information from the foundation model.\n\nWeaknesses:\n1. The paper lacks an overall illustrative figure that summarizes or visually explains the proposed method.\n2. While the method performs well on datasets beyond LUAD EGFR-TCGA, its performance on this particular dataset is relatively average. The authors should provide a detailed explanation for this and summarize the potential limitations of the proposed method.\n3. The proposed attention loss focuses on image patches that are important for the downstream task. On datasets with a large number of positive samples, could this lead to the attention mechanism eventually concentrating on only a single image patch?\n\nQUESTIONS:\nFor the experiments in Table 2, although the paper demonstrates in Table 1 that the proposed method achieves the best results, at least one top-performing baseline should be included in Table 2 as a comparison. This would make the contribution of the paper clearer to readers.\n\nLIMITATIONS:\nplease refer to Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}