Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
Authors: Neeraj Kumar
Chad Vanderbilt
Venue: NeurIPS 2025 poster ()
Keywords: Pathology Foundation Models
Whole Slide Images
Mutation Detection
Vision Transformer
Multiple Instance Learning
Primary area: machine_learning_for_sciences

Abstract: Pathology foundation models (PFMs) have emerged as powerful tools for analyzing whole slide images (WSIs). However, adapting these pretrained PFMs for specific clinical tasks presents considerable challenges, primarily due to the availability of only weak (WSI-level) labels for gigapixel images, necessitating multiple instance learning (MIL) paradigm for effective WSI analysis. This paper proposes a novel approach for single-GPU \textbf{T}ask \textbf{A}daptation of \textbf{PFM}s (TAPFM) that uses vision transformer (\vit) attention for MIL aggregation while optimizing both for feature representations and attention weights. The proposed approach maintains separate computational graphs for MIL aggregator and the PFM to create stable training dynamics that align with downstream task objectives during end-to-end adaptation. Evaluated on mutation prediction tasks for bladder cancer and lung adenocarcinoma across institutional and The Cancer Genome Atlas (TCGA) cohorts, TAPFM consistently outperforms conventional approaches, with H-Optimus-0 (TAPFM) outperforming the benchmarks. TAPFM effectively handles multi-label classification of actionable mutations as well. Thus, TAPFM makes adaptation of powerful pre-trained PFMs practical on standard hardware for various clinical applications.

Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
Neeraj Kumar
Chad Vanderbilt
Memorial Sloan Kettering Cancer Center kumarn6mmskcc.com org
Memorial Sloan Kettering Cancer Center
vanderbe@mskcc..org
Abstract
Pathology foundation models (PFMs) have emerged as powerful tools for analyzing whole slide images (WSIs). However, adapting these pretrained PFMs for specific clinical tasks presents considerable challenges, primarily due to the availability only weak (WSI-level) labels for gigapixel images, necessitating multiple instance learning (MIL) paradigm for effective WSI analysis. This paper proposes a novel approach for single-GPU Task Adaptation of PFMs (TAPFM) that uses vision transformer (ViT) attention for MII. aggregation while optimizing both for feature representations and attention weights. The proposed approach maintains separate computational graphs for MII. aggregator and the PFM to create stable training dynamics that align with downstream task objectives during end-to-end adaptation. Evaluated on mutation prediction tasks for bladder cancer and lung adenocarci- noma across institutional and The Cancer Genome Atlas (TCGA) cohorts, TAPFM consistently outperforms conventional approaches, with H-Optimus-O (TAPFM) outperforming the benchmarks. TAPFM effectively handles multi-label classifica- tion of actionable mutations as well. Thus, TAPFM makes adaptation of powerful pre-trained PFMs practical on standard hardware for various clinical applications.
Introduction
Hematoxylin and Eosin (H&E) staining the most common slide preparation method in pathology, used for visualizing tissue architecture and cellular details for cancer diagnosis. Whole slide images these tissue slides, commonly scanned
(WSIs) serve high-resolution digital representations either x or 40x optical resolution that captures 0.50y²² or 0.25p²² of tissue per pixel, respectively. WSIs form the basis of computational pathology that employs machine learning (ML) and computer vision techniques for digital cancer assessment []]. Due to memory constraints preventing direct processing of gigapixel WSIs and the availability
only slide-level labels for clinical tasks, WSI processing adopts a multiple instance learning (MIL) approach. In the MIL. framework, each WSI is represented as a bag smaller tiles (c.g., 224 x 224 x 3); a neural network encodes each tile into a feature embedding, and the tile embeddings are then aggregated into slide-level bag representation to compute predictions using only bag-level labels during training 2 3 as illustrated in Figure !
Computational pathology has experienced a paradigm shift with the introduction of pathology foundation models (PFMs), which learn powerful representations from large collections of WSIs through suff-super pre-training of vision transformers (ViTs) 4. 5 6 downstream applications such as gene mutation prediction, survival analysis, and treatment response estimation, existing methods typically use these PFMs as fixed feature extractors and train separate MII. aggregators to generate slide-level predictions 2 10]. The fixed-feature approach fails to adapt PFM parameters to the specific downstream task, potentially limiting performance [II [2 [3]. To address this limitation, we propose a novel Task Adaptation of Pathology Foundation Models (TAPFM) approach that: (I) leverages ViT's internal attention mechanism for MIL aggregation,
7
8 9.. For specific
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(2) maintains separate computational graphs for PFM and MIL parameter updates with a dual-loss mechanism on single GPU, and (3) seamlessly integrates with popular PFMs to improve their performance on clinically relevant tasks.
Input WSI
Tile Extraction
Bag of liles
Fenture Extraction
Bag Feature
Pathology l'oumiation
MIL Aggrigattion
Model (PFM)
All...ggggatii
Clussification
Figure 1: WSI Processing Pipeline- A representative H&E stained WSI scanned at 40x optical resolution with 100, 000 125, 000 RGB pixels. For efficient processing, manageable sized tiles (e.g. 224 x 224 3) shown in black boxes usually extracted from the tissue region. Bag of tiles obtained from a WSI are then passed through pathology foundation model to obtain feature representations (one feature vector per tile) which are then combined using multiple instance learning methods to compute a bag feature vector (one per WSI) that is used for downstream tasks such as binary classification (e.g. gene mutation prediction: O for absence and 1 for presence of mutation).
2 Related Work
2.1 Pathology Foundation Models (PFMs)
CTransPath [1] established an early benchmark by training a hybrid convolutional-trassooi
architecture on 32,220 WSIs across 25 anatomic sites. HIPT [15] and REMEDIS [66] explored different architectural approaches with ViT-S (DINO (17]) and ResNet-50 (SimCLR) respectively. Phikon 18 demonstrated the efficacy of ViT-L trained with iBOT on TCGA slides, while UNI significantly expanded scale with its VIT-H architecture trained via DINOv2 [19] on 100,000 slides across 20 tissue types. Subsequent models pushed boundaries further with Virchow (6] exploring a ViT-huge model trained on 1.5 million WSIs. This trend toward increased scale continued with Prov- GigaPath [8] processing 1.3 billion tiles from 171,189 WSIs spanning 31 tissue types and Virchow2 7 scaling to 1.7 billion tiles from 3.1 million slides across multiple magnifications. H-optimus-0 [)] leveraged ViT-giant architecture trained on hundreds of millions of tiles from over 500,000 WSIs. Several approaches have explored multimodality, including vision-language models (CONCH 20 PRISM [1], MUSK (22) and vision- genomics integration (Orpheus (23).. expanding PFMs beyond visual representation learning. Despite this architectural diversity, vision-only models including UNI 5 GigaPath g and H-optimus-0 [ have demonstrated superior performance on clinically relevant tasks such cancer diagnosis, mutation prediction, and treatment response estimation [ The dominant architecture across these PFMs remains ViTs 44] trained through self-supervised learning, predominantly using DINOv2 [19 on diverse WSI data. This paper specifically focuses on ViT based PFMs that utilize only pathology images as input.
2.2 Multiple Instance Learning (MIL) in Computational Pathology
MIL. methods for WSI analysis have evolved from attention-based mechanisms spatial-aware architectures. Early MII. approaches used simple aggregation operations such as mean or max pooling combine tile-level features 3. A major advance was drtentioo-based MIL (ABMIL) (2). which learns attention weights to prioritize diagnostically relevant tiles. CLAM [10] extended attention based MIL for multi-class classification. DSMIL (24] introduced dual-stream approach coupling max-pooling with attention scoring. VarMIL (25) incorporated variance modeling to capture tissue heterogeneity while maintaining computational efficiency. Spatially aware MIL methods have also emerged to capture relationships between WSI tiles. TransMIL [26] leveraged transformer architectures with positional encoding, while graph-based approaches like PatchGCN [S]] represent tiles nodes in graph structure based on physical adjacency. Graph transformer processing (GTP) (27 further refined this approach by combining graph structures with attention mechanisms. Despite architectural advances, benchmarking studies reveal that performance depends heavily

---

the specific clinical task and the quality of input embeddings, with no single aggregation method consistently outperforming others across all applications (2
2.3 Task Adaptation of PFMs
MIL methods usually rely on PFMs as fixed feature extractors, creating a disconnect between representation learning and task-specific adaptation for WSI analysis. Li al. (29] proposed an Information Bottleneck (IB) based fine-tuning approach that addresses computational constraints through instance sparsification on smaller backbone models (ResNet-50). The multiple forward passes required by the IB approach make it computationally infeasible for modern large-scale PFMs on single GPU systems due to memory constraints. While recent approaches have attempted to avoid multiple forward passes through end-to-end fine-tuning of large-scale PFMs [3] 3 31, these methods typically require substantial computational resources spanning tens of GPUs. Most recently, Campanella et al. [32 demonstrated the clinical utility of PFM adaptation for EGFR mutation prediction Lung cancer patients, but required distributed training across 24 NVIDIA H100 GPUs. Such computational requirements place PFM adaptation beyond the reach of most research groups and clinical institutions.
To the best of our knowledge, no existing approach has leveraged transformer's s self-attention mech- anism for MIL aggregation and enabled task adaptation of large-scale PFMs single GPU for downstream clinical applications, while addressing the optimization challenges that arise when jointly training foundation models and MIL aggregators.
3 Methodology
3.1 Problem Formulation
F2,, I2..... Egk } represent H&E stained WSI composed of K non-overlapping tiles, where each tile I; E HR Xx corresponds to the tissue region extracted from the WSI. For MII., a cancer patient's WSI has bag-level label y E {0, 1} for binary classification tasks (e.g., presence or absence of a gene mutation in a patient), but no tile-level annotations. Our goal is to adapt a pretrained PFM fe. parameterized by to extract tile-level features that are relevant for downstream task. A ViT [4] based PFM maps each tile to feature vector 2; fo(az) E R 55 g 9 by first diving each input tile into a grid of V non-overlapping patches (tokens) of size P x P. with an additional learnable CLS token prepended to the sequence. These tokens are linearly projected and embedded with position information before being processed through multiple self-attention layers to compute tile feature representations.
Let X {x),
3.2
Attention-Based Aggregation
The CLS token attends to all other tokens, compute attention weights that indicate the importance of each token for the feature representation. We propose to leverage these attention weights for MIL aggregation. For WSI with K tiles, let Z = |=f 2 ..... =X|" E k denote the feature matrix, where each row 2 R 2 is the feature vector (CLS token embedding) for tile i. Similarly, let a = [41, a2...., ax|" E R denote the vector of attention weights derived from the VIT. For each tile 34, the attention weight a; is computed as:
H
N
a4
H
N
cta.j
3=1
where A is the attention weight from the CLS token to the jth token in the 4 attention head while и and N are the numbers of attention heads and tokens, respectively. The proposed approach maintains a separate computation graph for the aggregator by detaching tile features and attention weights from the PFM's computation graph. The detached attention weights undergo min-max scaling to the range [0.1] followed by softmax normalization, ensuring they form a proper probability distribution for tile importance scoring to compute the bag representation Z as:
K
Z Zz a
(2)
i=1
3

---

where both Z and are detached from the PFM's computation graph. This ensures that the gradients from the classification loss only flow through the aggregator parameters while keeping the PFM's parameters fixed during this stage of optimization. The bag representation Z is then passed through linear classifier to predict the bag-level label:
(WZ
where a is the sigmoid activation while W E RIxD and E are aggregator parameters Bagy {W, 6). that are learned during backpropagation with weighted cross-entropy loss:
Lagg (v, 9) = - log(ū) + (1 y) log(1 - 9)]
where wy is the weight assigned to class y to handle class imbalance.
3.3
PFM Adaptation
Instead of employing conventional end-to-end backpropagation, we propose to detach gradients from the aggregator's 's computation graph to formulate dedicated loss function for PFM adaptation.
Feature Alignment Loss: Let us define G = [ ......... I' KR as the feature gradient
matrix where tt row contains the gradient the aggregator loss (equation 4 with respect to the corresponding tile's feature vector. During backpropagation through the aggregator, the gradients with respect to each feature vector are automatically computed (by chain rule) based on their contribution
to the bag representation fz, Uzz from the aggregator's ' computation graph to compute the feature alignment loss as:
0Lage aLass
az We propose to detach the feature gradients
K ΣΣ f=1 d=1
Deeature = -tr( Σ Fc 0z.))
2, x Iz.dd
This loss guides feature vectors to move in the direction that reduces the classification loss and can be interpreted as a first-order approximation of the effect of feature changes on the aggregator loss.
Attention Loss: Let us define the gradient of the aggregator loss with respect to attention weights as
ga = [9a) Ioz. 9ax I R Similar to the feature gradients, the gradient of the aggregator loss
with respect to each attention weight is automatically computed by the chain rule as Ja; &h
da,
(zi, Goze) ). We propose to compute the attention loss using the detached attention gradient as:
Lattention a
ga ag.
Σ
04 X 9o:
This loss encourages attention weights to adjust based on the informativeness of each tile for the downstream task, increasing (or decreasing) weights for informative (or uninformative) tiles.
Task Adaptation Loss (TAL): For PFM updates, TAL combines the feature and the attention loss:
CPFM
Creature >Lattention
(7)
where À is the hyperparameter that controls the relative importance of the attention loss. The PFM parameters, OPFMs are then updated with the backpropagation using its own loss: LPFM. The training procedure for the proposed approach is presented in Algorithm [ with illustraion shown in Figure and its implementation available at http://gghhhbbccom////maaaaaaaot
3.4 Theoretical Analysis
The key innovation in our approach lies in the decoupling the optimization process. In conventional end-to-end training with a unified computational graph, gradients flow through both the PFM and aggregator simultancously:
V Vogg.OPFMt C= (aCoss
0ū
0Lagg
0ù. ( 0z oL
OZ da
+
(8)
8ÿ
00ags'
0g
az
oz P0 da 00PPM

---

Algorithm Task Adaptation of Pathology Foundation Models (TAPFM)
1: Input: Dataset D of WSIs and labels (6X,,y))6= pretrained pathology foundation model
feprar MIL aggregator fe...- learning rates (Ngg and прFм)
2: for epoch
num _asm_eeheess_____eee do 3: for each (X6., }b) in D do
4:
Extract tiles {2 22,, from X3 as tensor X B
5:
// PFM forward pass
6:
(Z, A)
(X) (Extract last layer CLS feature and attention matrices)
Compute a = [a1, 02.,02.....00 ax]" E R where, a; = π Σ N EE= Aliss (Average
7:
CLS attention across heads and tokens)
8:
// Detach features and attention from PFM computation graph detocced = detach Z)) (Detach feature matrix]
9:
10:
Adetached detach a) (Detach attention vector
11:
I/ Aggregator parameter update
12:
Adetached softmax (amaaaxmmiimmmmn (minmax (Adotachhdd) )) (Scale to [0,1] range and normalize)
13:
= Z" a (Compute Bag representation)
Compute 9 = f (z) and Legg (equation
15:
Backpropagate: Gogg f 0agg Taaga V VaCogg
I/ Detach gradients from aggregator computation graph
17:
G detached = detach ( (Detach feature gradient matrix
18:
detached = detach(g.. (Detach attention gradient vector) Sa
19:
PFM fine-tuning with detached gradients
20:
Compute fine-tuning loss LPFM (Section 3.3
Backpropagate: 0PFM 0PFM TPFMM VOPFM CPFM
22:
end for
23:
end for
24:
return Fine-tuned PFM (ferru) and trained MIL aggregator (fengg)
TAPFM instead implements the following two-stage optimization:
&ogg
0g
Vango Lage
V
(Stage 1: Aggregator Update)
(9)
00.g
(aLag
0z
0z
az
NeppM CpFm detach
(Stage 2: PFM Update)
(10)
az
da
00
OZ
00rFM
This design resolves the circular dependency challenge in jointly optimizing PFM and MIL aggregator parameters by detaching gradients between optimization stages, that would otherwise create unstable training dynamics.
Proposition (Gradient Stabilization). The TAPFM approach breaks the circular dependency at each teration by enforcing:
dhagg | x g(0pFM Oug and 0CPFM | x f(0PFM.-19
(11)
00rrM resulting in more stable parameter trajectories than joint optimization.
00agg
Proof. In joint optimization, the parameter updates create an implicit feedback loop:
APFM. 8pFMM-1 ПPFM Vor (PPPM 0ager-1)
(12)
Gaggs Ouggr-1 Tagg V Vengo Luge (0pF MM Ougg:-1)
(13)
Note that 0 Oagn: depends on OPFM, which itself depends on Oaggi 1- This creates circular depen- dency where each parameter set is chasing moving target. However, the proposed approach breaks this loop by detaching the gradient computation graphs:

---

Input WSI
Tile Extraction
Bag of Tiles
Pathology Foundation Model
(Vision Transformer)
Task Adappatioannnno
Gradiants
K
ÿ = o (WZ + b)
Z = z'
a aj
Detach
Loss
Classifier
Bag Feature
i=1
CLS Features and Attention Weights
MIL Aggregator
Figure 2: TAPFM Algorithm Overview - The proposed approach passes WSI tiles through PFM to extract (last layer) CLS features Z and attention weights (solid arrows). These are detached from the PFM's computation graph and used by the MIL aggregator to compute bag-level predictions and aggregator loss Lagy (solid arrows). During backpropagation (dashed arrows), gradients from Lagg are detached to formulate the task adaptation loss LPFMS (equation 7 for fine-tuning PFM parameters, while aggregator parameters are updated using Lagg. This dual-optimization approach with separate computational graphs enables task adaptation on a single GPU.
Bagsr 0agge-1 Maas Bago Lago (dctach(000xxx...)) 0,
(14)
8PPm, = 0PFM. - NPFM V e CRFM (0PMM deach
(15)
This detaching operation ensures that during aggregator optimization, 0PFM is treated as a constant, and during PFM optimization, the updated Dagy influences the loss but does not receive gradient updates. This effectively eliminates the circular dependency and stabilizes training.
The stabilization effect of TAPFM's detached-gradien optimization illustrated by the training-loss variance plots in Fn Appendix Fi shows that when training with unified compu- tational graph, the loss oscillates without convergence, whereas TAPFM's srtached-gradices opti- mization exhibits smooth exponential decay- decay-confirming the theoretical stability analysis presented in this section. Additional properties of the proposed Algorithm I are elaborated in the appendix: permutation invariance (Appendix I. computational and space complexity malysis(Appeenae) A.2
and A. and prevention of catastrophic forgetting during task adaptation (Appendix A..4.
3.5 Multi-label Classification
The proposed Algorith T can be easily adapted for multi-label classification problems where each patient can be associated with multiple binary labels. Let [y', .y. .,?...,y' represent the ground truth vector where yd {0, }} indicates the presence or absence of the j-th mutation. The aggregator loss for each class is computed using the weighted cross-entropy loss as in Equation 4 Note that the each output node in MIL aggregator considered as class-specific binary predictor for multi-label classification. The total aggregator loss is then formulated as a weighted sum Lagy = Σÿ= 0j F where aj is the weight assigned to class j. calculated as the inverse of the ratio positive examples of that class to the total number of training instances.
Noticeably, multi-label classification differs fundamentally from both binary and multi-class ap- proaches. While multi-class classification assumes mutual exclusivity among classes (one-hot encoding), multi-label classification allows multiple concurrent positive labels, better reflecting clinical reality where tumors harbor multiple actionable mutations simultancously (33]. This

---

is particularly relevant in scenarios such as collision tumors [4] and resistance driven secondary mutations [35, where combinations of mutations have distinct prognostic implications [6].
While the methodology presented in this paper primarily focuses on classification tasks, TAPFM's core approach generalizes to other clinically relevant objectives as well. By replacing the classifi- cation head (Equation (4) with task-specific layers (e.g.. Multi-task logistic regression for survival prediction (37]) and corresponding loss functions, TAPFM can adapt PFMs for survival analysis tasks while maintaining single-GPU training efficiency. Additional survival prediction experiments illustrating this generalization are provided in Appendix D.2
4
Experiments
The proposed TAPM approach is evaluated on clinically relevant mutation prediction tasks using institutional and public cohorts of bladder cancer (BLCA) and lung adenocarcinoma (LUAD) patients.
Datasets: The institutional dataset consists of H&E WSIs of 2, 030 BLCA and 8, 820 LUAD patients collected during routine clinical care at Memorial Sloan Kettering Cancer Center (MSKCC), including both 20x and 40x (~~ 30% of all WSIs in each cohort) magnifications to reflect real-world data acquisition variability. WSIs from The Cancer Genome Atlas (TCGA) cohorts TCGA-BLCA (260 patients) and TCGA-LUAD (438 patients), all scanned at 40x resolution, are exclusively used for external validation assess generalizability of the proposed approach. Only one WSI per patient is used in all training and validation cohorts.
For the binary classification task, TAPM is evaluated on two clinically relevant mutation prediction tasks: FGFR3 in BLCA and EGPR in LUAD. For multi-label classification, TAPM's ability to simultaneously predict four actionable mutations in LUAD patients is assessed: EGFR, KRAS, MET, and ALK. The prevalence rate of these mutations across institutional (TCGA) cohorts are: 16% (14%) for FGFR3 in BLCA, and in LUAD: 26% (14%) for EGFR, 27% (35%) for KRAS, 4% (2%) for MET, and 3% (1%) for ALK.
Benchmarks: We evaluate state-of-the-art PFMs such as UNI [3), GigaPath 88. and H-Optimus- [ Motivated by prior findings that lightweight MIL models can attain clinical performance comparable to computationally expensive aggregators (28). this work employs meforryefffcient MIL
methods on single-GPU systems: ABMIL [2], DSMIL 224, CLAM [1O. and VarMIL (25).
Implementation Details: All reported experiments in this paper are conducted using Py 2.5.1 on a single NVIDIA H100 (80GB memory). Each WSI is processed at native resolution (20x or 40x) to extract non overlaapag tiles in a sliding window manner after filtering out the non- non-tissue regions with Otsu thresholding (38] For 20x WSIs, tiles of size 224 x 224 x 3 pixels are extracted directly, while for 40x images, tiles of size 448 x 448 x 3 pixels are extracted and resized to 224 x 224 3 pixels to maintain consistent spatial context. As detailed the space complexity analysis (Appendix (A.3). the memory requirements of the proposed TAPFM method scale quadratically with the number of tokens (patches) processed by ViTs. Thus, the 224 x 224 tilc size is selected as the optimal dimension that prevents out-of-memory errors while maximizing the contextual information captured per tile. During training. 300, 100, and 75 tiles per WSI per epoch are randomly sample without replacement for UNI, Gigapath, and H- H-Optimus-A respectively the maximum number of tiles processable for each PFM on a single H100 GPU while maintaining end-to-end one-tuning capability. At inference, all tiles obtained from given WSI are used for downstream mutation prediction tasks.
For the proposed TAPFM method À = 1.0 used for all experiments. The institutional cohort is evaluated under f-fold cross- cross- validation at the patient level. In each fold, patients are stratified by label and scan resolution and split into 80% training, 10% validation, and 10% testing, with no patient overlap across splits. Area Under the receiver operating characteristies Curve (AUC) is the primary metric reported (binary: AUC; multi-label: macro-average AUC). Model selection is performed on the fold-specific validation set. For external validation, each of the 5 fold-specific models trained on the institutional data is evaluated on the corresponding TCGA cohort.
For training, AdamW (39 with weight decay of le-4 is used as opitimzer, applying differential learning rates of le-6 and le-5 for PFM and aggregator parameters, respectively. Training data augmentation included random horizontal flips, random rotations (of 90", 180", or 270"), and Gaussian blur. A cosine annealing scheduler with warm restarts (To = 10, Tmaat = 2) is also used for better convergence 40). Each batch contained WSI, and TAPFM is trained for 20 epochs while
7

---

all other benchmarks are trained for 50 epochs, with the institutional validation set used to select the best-performing model for all evaluations on the insititutional and TCGA testing sets.
4.1 Results
Table shows the performance comparison of three sets of models on the testing data: (1) fixed-PFM with trained MIL aggregators, (2) fine-tuned PFM and MIL aggregators (equivalent to setting in equation 7 with external MIL methods), and (3) proposed TAPFM. It is evident that the proposed TAPFM approach outperforms the other benchmarks across both binary mutation prediction tasks. H-Optimus-O (TAPFM) consistently achieves the best performance across both institutional and external TCGA testing cohorts, followed by Gigapath (TAPFM), indicates generalizability of the proposed approach. Teble_e TAPFM evaluation to the more challenging task of simultancously predicting four actionable mutations LUAD. H-Optimus-O (TAPFM) consistently outperforms GigaPath (TAPEM) across all mutations, even for the rate MET and ALK mutations. Area Under Precision-Recd Recall Curve (AUPRC) values are reported in Appendix D.I
4.2 Runtime Performance
Training times for BLCA are 12 hours for UNI, 21 hours for Gigapath, and 24 hours for H-Optimus-O. For LUAD cases, training required 2 days 4 hours for UNI, days hours for Gigapath, and days hours for Hoptimus. Inference times per WSI are 4.85 minutes for UNI, 6.38 minutes for Gigapath, and 7.15 minutes for H-Oppimus-O. These results confirm that TAPFM enables efficient PFM task adaptation standard hardware making it suitable for clinical implementation.
4.3 Convergence
For binary FGFR3 classification in BLCA, Figure 3a shows UNI reaching maximum validation performance by epoch 6 (AUC 0.8542), Gigapath by epoch (AUC 0.8764), and H-Optimus-O by epoch 7 (AUC 0.8960). Additionally, experiments on LUAD datasets (not shown) demonstrated that all PFMs converge within 4 epochs for binary classification tasks. For multi-label LUAD classification, convergence occurred at epoch 8 for UNI, epoch 10 for Gigapath and epoch 11 for H- Optimus-O. Loss-variance comparisons between joint optimization and TAPFM's estached-gradies training are shown in Appendix 55 illustrating markedly smoother convergence and confirming the theoretical stability analysis presented in Section 3.4
22
(a)
(b)
(c)
Figure 3: For bianry FGFR3 prediction task in BLCA - (a) validation AUC trajectories of various TAPFM models, and the variation of log- trgtrransforeed training loss of the H- H-Optimus-O (TAPFM) model with (b) attention loss weighting (A) and (c) tile sampling density.
4.4 Ablation Studies
Systematic ablation studies investigate the influence of key hyperparameters- - attention loss weighting (A in equation 7 and number of tiles sampled per WSI per epoch on the performance of H- H-Optimus- 0 (TAPFM) model for the binary FGFR3 prediction task in BLCA patients.
Lambda: To evaluate the impact of the attention loss term in TAL (Equation 77 log-linear conver-
gence fits of the form log La a + k were computed over epochs 2 to 20. Log-transforned loss fits are shown in Figure 3b with raw curves in Appendix 4a All values of À yielded consistent exponential decay, with convergence rates of -0.0701 (R² 0.9737), -0.0718 (R² = 0.9808), -0.0706 (R²2 0.9765), and 00.0100 2R² 0.9693) for À {0.25, 0.5, 0.75, 1.0), respectively.
8

---

Table Performance comparison of different PFM and MIL aggregation methods for binary classifi- cation tasks in terms of AUC (mean + 95% CI from 5-fold cross- voossvalidation). N indicates the number of patients in each cohort. Best and second best models are in bold and underlined text, respectively.
BLCA FGER3
LUAD EGFR
Model
Institutional Cohort
TCGA
Institutional Cohort
TCGA (N 438)
(N 194)
(N = 260)
(N = 876)
Fixed- PFM with Trained MIL Aggregators
UNI + DSMIL
0.7876 + 0.024
0.7928 0.022
0.7352 + 0.021
0.7624 + 0.025
UNI + CLAM
0,7893 1 0.023
0.7912 0.024
0.7389 1 0.022
0.7645 1 0.024
UNI + VarMIL
0.7912 + 0.022
0.7948 0.021
0.7414 0.020
0.7663 + 0.023
UNI + ABMIL
0.7904 + 0.023
0.7962 0.023
0.7396 0.021
0.7718 + 0.022
GigaPath DSMIL
0,8232 + 0.021
0.8645 + 0.019
0.7657 + 0.020
0.8153 + 0.022
GigaPath CLAM
0,8246 1 0.020
0.018 0.8673 + 0.018
0.7672 + 0.019
0,8164 1 0.021
GigaPath + VarMIL
0.8274 + 0.019
0.8689 + 0.018
0.7691 0.018
0.8192 + 0.020
GigaPath ABMIL
0.8294 + 0.019
0 + 0.017
0.7711 + 0.019
0.8205 + 0.019
H-Optimus-O+ + DSMIL
0,8365 + 0.020
0.8731 t 0.018
0.7694 + 0.019
0.8237 + 0.021
H-Optimus-O+ + CLAM
0,8373 + 0.019
0.8752 + 0.017 0.017
+.018
0,8253 = 0.020
H-Optimus-O+ + VarMIL
0,8401 + 0.018
0.8774 + 0.016
0,7735 0.018
0.8282 + 0.019
H-Optimus-O + ABMIL
0,8412 + 0.018
0.8786 + 0.017
0.7742 0.017
0.8295 + 0.018
Fine-tuned (FT) PFM with MIL Aggregators
UNI + DSMIL (FT)
0,8132 + 0.023
0.8209 0.021
0.7526 1 0.022
0.7837 + 0.024
UNI + CLAM (FT)
0,8147 + 0.022
0.8223 4 0.020
0.7542 + 0.021
0.7865 + 0.023
UNI + VarMIL (FT)
0.8176 t 0.021
0.8252 0.020
0.7568 + 0.020
0.7894 + 0.022
UNI + ABMIL (FT)
0,8193 1 0.021
0.8237 1 0.021
0,7581 1 0.021
0.7922 1 0.021
GigaPath DSMIL (FT)
0,8372 1 0.020
0.8831 +: 0.017
0.7986 + 0.019
0.8351 + 0.020
GigaPath CLAM (FT)
0.8381 + 0.019
0.8847 + 0.016
0.8012 + 0.018
0.8377 + 0.019
GigaPath VarMIL (FT)
0.8407 1 0.018
0.8873 + 0.015
0.8043 + 0.017
0.8408 + 0.018
GigaPath ABMIL (FT)
0.8393 + 0.019
0.8858 + 0.016
0.8074 + 0.018
0.8421 + 0.017
H-OOptimusO- + DSMIL (FT)
0,8478 1 0.019
0.8843 0.016
0.8121 0.018
0.8479 + 0.019
H-Optimus-O+ + CLAM (FT)
0,8491 t 0.018
0.8859 + 0.015
0.8143 + 0.017
0.8512 + 0.018
H-Optimus-O+ + VarMIL (FT)
0.8526 1 0.017
0.8889 1 0.014
0.8167 1 0.016
0.8543 1 0.017
H-Optimus-O+ + ABMIL (FT)
0,8512 + 0.018
0.8874 + 0.015
0.8189 + 0.017
0.8529 = 0.016
Proposed TAPFM method
UNI (TAPFM)
0,8415 + 0.021
0.8536 + 0.019
0.8175 + 0.018
0.8309 + 0.020
Gigapath (TAPFM)
0.8624 1 0.016
0.8881 1 0.013
0.8376 1 0.015
0.8482 1 0.009
H-OOptius-O (TAPFM)
0,8837 1 0.018
0,8956 1 0.011
0.8503 1 0.013
0.8590 1 0.010
Table 2: Performance comparison on multi-label classification of actionable mutations in LUAD. Results are reported as AUC (mean +95% CI from 5-fold vooss-valiationn. Based on binary classification results (Table only the top performers are shown, as performance varied primarily by foundation model type.
Institutional Cohort (N = 876)
Model
Macro Average
EGFR
KRAS
MET
ALK
UNI (TAPFM)
0.8287 1 0.019
0.7814 1 0.021
0.8095 1 0.028
0.8341 1 0.031
0.8134 1 0.018
Gigapath (TAPFM)
0.8595 1 0.015
0.8075 1 0.018
0,8350 1 0.024
0.8632 1 0.026
0.8413 1 0.014
H-Optimus-O (TAPFM)
0.8665 + 0.018
0.8153 + 0.019
0.8420 + 0.025
0,8702 + 0.024
0.8485 + 0.017
TCGA (N
438)
(TAPFM)
0,8342 0.022
0.7896 1 0.024
0.7913 1 0.035
0.8287 1 0.038
0.8110 + 0.021
GigaPath (TAPFM)
0.8603 0.018
0.8147 0.020
0.8179 + 0.029
0.8540 0.032
0.8367 + 0.021
H-Optimus-0 (TAPFM)
0.8629 0.017
0.8226 + 0.019
0.8241 + 0.030
0.8563 0.034
0,8415 = 0.016
Although convergence behavior remained stable across this range, increasing À produced steeper decay and lower final training loss, with the best performance observed at À = 1.0. These results suggest that stronger weighting of the TAL attention supervision improves training efficiency without compromising stability.

---

Number of tiles: Figure shows that increasing the number of tiles per WSI consistently improved the performance of H-Optimus-O (TAPFM) model (raw loss curves in Appendix 4b). Another exponential decay model, log Lx = + bk, was fit to training loss curves over epochs A = 2 to 20, with estimated convergence rates of -0.0532 (R² = 0.9731), -0.0569 (R² 0.9811), 1), and -0.0725 (R² 0.9752) for models trained with 25, 50, and 75 tiles per WSI, respectively. The 75-tile model converged by epoch with a validation AUC of 0.8960, while the 50-tile model reached 0.8764 by the same epoch. The 25-tile variant (TAPFM H-Optimus-O) converged earlier-by epoch 5-but plateaued at lower AUC of 0.8527. These findings indicate that denser tile sampling both accelerates convergence and improves performance on the institutional validation set.
Experimental analysis of incorporating cosine regularization term (see Appendix AA.5) for feature alignment loss in equation revealeed no significant variation in FGFR3 prediction performance. Con- sequently, À = 1.0 and the maximum number of tiles per WSI per epoch that could be accommodated on single GPU for each PFM are used in all LUAD experiments
rrevelee
5 Clinical Impact
Because LUAD harbors a large number of drug-targetable mutations, molecular diagnostics are especially valuable for guiding its targeted therapies [41 42 LUAD was the first tumor type to develop official guidelines recommending universal EGFR and ALK testing [43. which were subsequently expanded in 2018 to include additional actionable genes when sufficient tissue is available [4]. One of the principal challenges remains obtaining adequate DNA from small core biopsies to perform this extensive molecular workup 45). Although this paradigm has been embraced in LUAD, other malignancies such as BLCA have been sequenced far less frequently outside major therapies [46]. The ability to predict actionable
centers despite the availability of FGF mutations directly from H&E-stained WSIs promises to streamline and economize current diagnostic workflows. By triaging or potentially obviating the need for costly and tissue-requiring molecular assays, TAPFM can help overcome the infrastructure and financial barriers that preclude precision oncology in lesource-limited settings. Even when resources are available, TAPFM offers inference that can be performed as soon as the digital image is scanned. Therefore, clinicians can receive molecularly informed results within hours instead of days/weeks. This study demonstrates that single model (TAPFM) can simultaneously detect multiple targetable mutations in LUAD, and that its performance generalizes from the originating institution an independent TCGA cohort.
6 Conclusion
This work introduces TAPFM, a novel approach for adapting PFMs to specific clinical tasks by leveraging ViT's attention mechanism for MIL aggregation and detached dual-gradient approach for updating PFM parameters on a single GPU. TAPFM bridges the gap between self-supervised pretraining and supervised downstream adaptation in computational pathology, enabling more ef- fective use of PFMs for clinical applications. The experimental results establish its effectiveness for clinically relevant mutation prediction tasks for BLCA (FGFR3) and LUAD (EGFR, KRAS, MET, ALK) patients while maintaining computational efficiency. Notably, TAPFM successfully tackles the challenging task of simultancous prediction of four actionable mutations in LUAD patients, maintain- ing reasonable performance even for rare mutations like MET and ALK. Despite promising results, certain limitations of this work can be addressed in future studies. The approach shows potential for extension to additional clinical endpoints including survival analysis, recurrence prediction, and treatment response estimation. Investigations into which specific transformer layers benefit most from task adaptation could optimize the approach by selectively updating only those parameters. Additional external validation across multi-institutional cohorts with diverse scanning protocols and expanded biomarker panels would strengthen the clinical utility of the proposed approach. Implementations that scale to distributed training across multiple GPUs to increase the number of tiles processed per WSI during training may enhance TAPFM's generalization performance.
Acknowledgments
We thank the research funding provided by a Cancer Center Support Grant from the NIH/NCI (grant number P30CA008748) and the Warren Alpert Foundation through the Warren Alpert Center for
10

---

Digital and Computational Pathology at Memorial Sloan Kettering Cancer Center. We also thank Swaraj Nanda, Siddharth Singi, Jamal Benhamida, David Kim, Jie-Fu Chen, Amir Momeni -orooiin.....eni.. Gregory M. Goldgof, and Gabriele Campanella for their contributions to data curation, experiment design, and manuscript preparation.
References
[1] Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood. Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering, 1(12)930 949, 2023.
[2] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International Conference on Machine Learning, pages 2127-2136. PMLR, 2018.
[3] Gabriele Campanella, Matthew G. Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus Busam, Edi Brogi Victor E. Reuter, David S. Klimstra, and Thomas Fuchs. Clinical -rrdde computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine. (118):30111309, 2019.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, In International Conference on Learning Representations, 2021.
[5] Richard J Chen, Tong Ding, Ming Y Lu. Drew FK Williamson, Guillaume Jaume. Andrew I Song, Bowen Chen, Andrew Zhang. Daniel Shao, Muhammad Shaban, et al. Towards general purposs foundation model for computational pathology. Nature Medicine, 30(3):500888 862, 2024.
[6] Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et A foundation model for clinical-grad gradde computational pathology and rare cancers detection. Nature medicine, 30(10):2924-29 -9355, 2024.
[7] Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, et al. Virchow2: Scaling self- supervieed mixed magnification models in pathology. arXiv preprint 40X1::2408.00738 2024.
[8] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang. Rajesh Rao, Tristan Naumann, Cliff' Wong. Zelalem Gero, Javier González, Yu Gu, al. A whole-slide foundation model for digital pathology from real-world data. Nature, 130880)))):81118888 2024.
Bioptimus. H-optimus-O:: foundational model for histopathology. htttpa://github. com/ bioptimus/ 2024. Accessed: April 2025.
[10] Lu, D. F. K. Williamson, Y. Chen, R. Chen, M. Barbieri, and F. Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering. 5(6):555-570, 2021.
[11] Faisal Mahmood, benchmarking crisis in biomedical machine learning. Nature Medicine, pages 1-1, 2025.
[12] Saghir Alfasly, Ghazal Alabtah, Sobhan Hemati, Krishna Rani Kalari, Joaquin Garcia, and HR Tizhoosh. Validation of histopathology foundation models through whole slide image retrieval. Scientific Reports, 15(1))3990, 2025.
[13] Gabriele Campanella, Shengjia Chen, Manbir Singh, Ruchika Verma, Silke Muchistedt, Jennifer Zeng. Arych Stock, Matt Croken, Brandon Veremis, Abdulkadir Elmas, et al. A clinical benchmark of public self self-supervised pathology foundation models. Nature Communications, 16(1):3440, 2025.
[14] Xiyue Wang. Sen Yang. Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81: 102559, 2022.
[15] Richard Chen, Chengkuan Chen, Yicong Li. Tiffany Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self supervied learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16144 16155, 2022.

---

[16] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur. Simon Kornblith, Ting Chen, Nenad Tomasev, Jovana Mitrović, Patricia Strachan, al. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nature Biomedical Engineering, 7(6)7756-77. 2023.
[17] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[18] Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Axel Camara, Alice Mac Kain, Charlie Saillard, and Jean Saan-Bapiiste Schiratti. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv, pages 2023-07, 2023.
[19] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA,, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu. Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification.
[20] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang. Tong Ding, Guillaume Jaume, Igor Odintsov. Long Phi Le, Georg Gerber, et al. A visual-languaae foundation model for computational pathology. Nature Medicine, 30(3):863-88 874, 2024.
[21] George Shaikovski, Adam Casson, Kristen Severson, Eric Zimmermann, Yi Kan Wang, Jeremy D Kunz, Juan A Retamero, Gerard Oakley, David Klimstra, Christopher Kanan, al. Prism: A multi-modal generative foundation model for slide-Ievel histopathology. arXiv preprint arXiv:2405. 10254, 2024.
[22] Jinxi Xiang, Xiyue Wang. Xiaoming Zhang, Yinghua Xi, Feyisope Eweje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, al. A vision-language foundation model for precision oncology. Nature, pages 1-10, 2025.
[23] Kevin M Bochm, Omar SM EI Nahhas, Antonio Marra, Michele Waters, Justin Jee, Lior Braunstein, Nikolaus Schultz, Pier Selenica, Hannah Y Wen, Britta Weigelt, et al. Multimodal histopathologic models stratify hormone receptor- posittve early breast cancer. Nature Communications, 16(1):2106, 2025.
[24] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with supersupe contrastive learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 14318-14328, 2021.
[25] Yoni Schirris, Efstratios Gavves, Iris Nederlof, Hugo Mark Horlings, and Jonas Teuwen. Deepsmile: contrastive self-supervised pre-training henefits msi and hrd classification directly from h&e whole-slide images in colorectal and breast cancer. Medical Image Analysis, 79:102464, 2022.
[26] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147, 2021.
[27] Yi Zheng, Rushin H. Gindra, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, and Vijaya B. Kolachalama. A graph-transforme for whole slide image classification. IEEE Transactions on Medical Imaging, 41111):30333311 2022.
[28] Shengjia Chen, Gabriele Campanella, Abdulkadir Elmas, Aryeh Stock, Jennifer Zeng, Alexandros D. Polydorides, Adam Schoenfeld, Kuan lin Huang, Jane Houldsworth, Chad Vanderbilt, and Thomas J. Fuchs. Benchmarking embedding aggregation methods in computational pathology: A clinical data perspective. In MICCA/ Workshop on Computational Pathology with Multimodal Data (COMPAYL), 2024.
[29] Honglin Li. Chenglu Zhu. Yunlong Zhang, Yuxuan Sun. Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottleneck for weakly-supervise pathology whole slide image classification. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7454-7463, 2023.
[30] Gabriele Campanella, Eugene Fluder, Jennifer Zeng, Chad Vanderbilt, and Thomas J Fuchs. Beyond multiple instance lcarning: Full resolution all-in-memory end-to-end pathology slide modeling. arXiv preprint 24X3::2403.048880
2024.
[31] Neeraj Kumar, Swaraj Nanda, Siddharth Singi, Gabriele Campanella, Thomas Fuchs, and Chad Vanderbilt. DEMO: A scalable artificial intelligence framework for rapid EGFR mutation screening in lung cancer. In GenAl for Health: Potential, Trust and Policy Compliance, 2024.
12

---

[32] Gabriele Campanella, Neeraj Kumar, Swaraj Nanda. Siddharth Singi, Eugene Fluder, Ricky Kwan, Silke Muchlstedt, Nicole Pfarr, Peter J Schüffler, Ida Häggström, al. Real-world deployment of fine-tuned pathology foundation model for lung cancer biomarker detection. Nature Medicine, pages 1-9, 2025.
[33] Min-Ling Zhang and Zhi-Hua Zhou, A review on multi-lahel learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819-188) 1837, 2014.
[34] Mary M Torrez Khalil Sheibani, and Mohammad A Vasef. Mutually exclusive driver mutations identifies
2 separate primaries in a collision tumor initially interpreted a solitary lung adenocarcinoma with tumor heterogeneity. Respiratory Medicine Case Reports, 47:101986, 2024.
[35] Xinyu von Buttlar, Joshua E Reuss, Stephen V Liu, and Chul Kim. Eml4-alk rearrangement as mechanism of resistance to osimertinib in metastatic lung adenocarcinoma: case report. JTO Clinical and Research Reports, 206)) 2021.
[36] Michelle Shiller, Mclissa Johnson, Robert Auber, and Sandip Pravin Patel. Clinical perspectives on the value of testing for s I and keapl mutations in advanced nscle. Frontiers in Oncology. 14: 14:1459737, 2024.
[37] Chun-Nam Yu, Russell Greiner, Hsiu-Chin Lin, and Vickie Baracos. Learning patient-specific cancer survival distributions as a sequence of dependent regressors. Advances in neural information processing systems, 24, 2011.
[38] Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, (1)xx22-6, 1979.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[40] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[41] Alex Friedlaender, Maurice Perol, Giuseppe Luigi Banna, Kaushal Parikh, and Alfredo Addeo. Oncogenic alterations in advanced nscle: molecular super-highway, Biomarker Research, 12(1):24, 2024.
[42] Mathieu Chevallier, Maxime Borgeaud, Alfredo Addeo, and Alex Friedlaender. Oncogenic driver mutations in non-small cell lung cancer: Past, present and future. World Journal of Clinical Oncology, 12(4):217,
2021.
[43] Neal Lindeman, Philiip T Cagle, Mary Beth Beasley, Dhananjay Arun Chitale, Sanja Dacic, Giuseppe Giaccone, Robert Brian Jenkins, David J Kwiatkowski, Juan-Sebastian Saldivar, Jeremy Squire, et al. Molecular testing guideline for selection of lung cancer patients for egfi and alk tyrosine kinase inhibitors: guideline from the college of american pathologists, international association for the study of lung cancer, and association for molecular pathology. Journal of Thoracic Oncology. 8(7):823 859, 2013.
[44] Neal Lindeman. Philip T Cagle, Dara L Aisner, Maria E Arcila, Mary Beth Beasley, Eric H Bernicker. Carol Colasacco, Sanja Dacic, Fred R Hirsch, Keith Kerr, et al. Updated molecular testing guideline for the selection of lung cancer patients for treatment with targeted tyrosine kinase inhibitors: guideline from the college of american pathologists, the international association for the study of lung cancer, and the association for molecular pathology. Archives of pathology & laboratory medicine, 142(3):321-3) -3661113,,,11 2018.
[45] Dara Aisner and Carrie B Marshall. Molecular pathology of non-smail small cell lung cancer: a practical guide. American journal of clinical pathology, 138(3):332- 346, 2012.
[46] Claudia Maria Ascione, Fabiana Napolitano, Daniela Esposito, Alberto Servetto, Stefania Belli, Antonio Santanicllo, Sarah Scagliarini, Felice Crocetto, Roberto Bianco, and Luigi Formisano. Role of fgfr3 in bladder cancer: Treatment landscape and future challenges. Cancer Treatment Reviews, 115:102530, 2023.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
13

---

The paper is focused on task adaptation of pathology foundation model on single gpu for clinical applications. This is reflected accurately in the abstract and throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The authors have stated in conclusion how the proposed approach can be extended beyond the analysis presented in this study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theorems and properties of the proposed method are provide in the main text and appendix with proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
14

---

All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All method details, (hyper-)parameter settings, and other details (such as data) are included in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
15

---

The authors provide link to code repository that contains implementation of the algorithm. Although the institutional data can not be released with the paper, the TCGA data that is used for external validation in the paper is publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c ec/
Sellic//uides. haaiii/ggiide//deeubiiissee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ htzpp.ccc/ubllc//uiic.. wwwpc///wiieeeeeee Sttp:///eeeSSeeeeeeeeeeeee wswssnssiccgiiooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All method details, (hyper-)parameter settings, and other details (such as data) are included in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The proposed method is rigorously evaluated on both internal and external testing datasets maintaining patient separation from the training and checkpoint selection (validation) datasets. For creating error bars, we need to train and evaluate the proposed method on multiple sets of training, validation, and testing splits obtained from institutional data, which is computationally demanding task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: is mentioned that the authors used a single NVIDIA H100 GPU for experi- ments reported in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aaaaaar.eeeeo/ccccccccaa htt.aa/eeeehhcscciiellll.le.ee
Answer: [Yes]
Justification: This paper adheres to every aspect of the NeurlPS code of ethics, specifically using de-identified institutional and TCGA data for experiments.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes section clinical implications of the proposed method to highlight how this work can potentially impact cancer patients.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper private institute data and public TCGA data in the experiments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appropriate credits provided to original publications (such as for TCGA datasets), wherever required.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code to implement the proposed method is provided in anonymized repository with appropriate license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects are directly involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects are directly involved in this work. However, appropriate approval for working with deidentified healthcare data from the institutional review board was obtained for this study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: LLMs are not the core components of the proposed method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Theoretical Analysis Details
A.1 Permutation Invariance
A crucial property for MIL. models is permutation invariance, which ensures that the order of instances bag does not affect the final prediction and below it is shown that the proposed TAPFM methods hold the permutation invariance property.
Theorem (Permutation Invariance). Let X f21, FKk be bag of instances and π be permutation of indices {1, ..., K}. The bag representation Z computed by TAPFM invariant to permulation, i.e., ((X) (1==(11) "..))....... .
Proof. Let X f21, .... 22.,, JK} be bag of instances and π be a permutation of indices (1, K}. Let X (In(1): x2))); ... In(K) be the permuted bag.
Using matrix notation, let Z ,
Kx be the feature matrix and a
[01, ax]]" E IRK be the attention weight vector for the original ordering.
Let P. R be the permutation matrix corresponding to Then, the feature matrix and attention vector for the permuted bag can be written as:
Z P ag P
(16)
(17)
The bag representation for the original ordering is:
Z(X) = Z' 0:51
(18)
1=1
The bag representation for the permuted ordering is:
Z(X²) Z ar
(19)
'
(20)
Z''p"Paa
(21)
= =Z"aa
(22)
Z(X)
(23)
The equality PTP,, = I holds because permutation matrices are orthogonal. Therefore, Z(X) = Z (X,),, which proves that the bag representation is permutation invariant.
In component form, this equivalent to:
K
K
Z(X²) Σ (x(1) = Σ
aラ=ニ = Z(X)
(24)
1=1
where the change of variables j (2) is used.
A.2 Computational Complexity
Theorem 2 (Computational Complexity). The computational complexity of TAPFM for processing single bag of K instances is O(K Cwr). where Cwr is the complexity of a a.forward and backward pass through the specific VIT of PFM for single instance.
Proof. The computational complexity of TAPPM is analyzed by examining each step of the proposed Algorithm in detail:
* Feature Extraction (Forward Pass):
21

---

Each of the K tiles in the bag is passed through the ViT model to extract features. The complexity of processing a single tile depends on the ViT architecture:
Patch embedding: O(P2. C. D), where P is the patch size, C' is the number of channels, and D is the embedding dimension. Self-attention layers: O N D), where L. is the number of layers, N is the number of tokens (patches). MLP blocks: (L N DD)).
The total complexity for single tile forward pass is O(CViT_forward) forward) O(p2. C.D+L N L- N. D²).
For K tiles, the total forward pass complexity is O(K Cvir_forward)
Attention Weight Computation:
For each tile i, the average attention weight is computed by aggregating over H attention heads and N tokens:
H
0:
da.j
(25)
This requires O(H N) operations per tile, resulting in total complexity of O(K H NN) for all tiles.
Bag Representation Computation:
Computing the weighted sum of feature vectors:
K
= Z a
(26)
This requires O(K D) operations, where D is the feature dimension.
Aggregator Forward and Backward Pass:
The aggregator applies a linear transformation followed by sigmoid activation:
ý <oW oWZ
(27)
The forward pass has complexity O(D) for the matrix-vecto multiplication.
Computing the aggregator loss and its gradient with respect to the bag representation has complexity O(D).
Updating the aggregator parameters has complexity O(D).
Computing Gradients for the PFM:
The feature loss is:
K
fenturee trGG
Σ
(zi (x.)
(28)
i=1
Computing this loss has complexity O(K- D). The attention loss is:
K Σ
Lattention = T ga
a. 9o:
(29)
Computing this loss has complexity O(K).
PFM Backward Pass:
The backward pass through the ViT for all K tiles has complexity of O(K OvT backward) )
which is typically on the same order as the forward pass: OCVVi backwart) 0 .2 D + L 2 D D L NLL-N-DD D2).
22

---

Combining all steps, the total computational complexity is:
O(K CVii_fforwad forward + K H -
+ K + K backward)
(30)
Since GVI = CViT_ forward + CviT backward and typically Cur >> H N and Cvir >> D, the overall complexity is dominated by the ViT forward and backward passes, resulting in O(K -vi)).
This complexity analysis shows that the proposed TAPFM approach scales linearly with the number of instances in the bag. The constant factor Ovi depends on the specific architecture of the Vision Transformer but is independent of the bag size.
A.3 Space Complexity Analysis
In addition to time complexity, space complexity is critical consideration for practical deployment of TAPFM, especially for gigapixel WSIs processed using transformer-based models.
Theorem 3 (Space Complexity). The space complexity of TAPFM for processing single bag of K instances (tiles) is P[(0PMM] + K -Suett + Sgrod)). where [0 is the size of the PFM parameters, and Sact and Sgrad are the memory requirements for activations and gradients per instance, respectively.
Proof. The space complexity of TAPFM is analyzed by examining the memory requirements of each component in the proposed approach as follows:
Model Parameter Storage: The memory required to store the VIT parameters is O(0pras]), which can be further decomposed as ( N D²), where is the num- ber of transformer layers, N is the number of tokens, and D is the embedding dimension.
Feature Matrix Storage: For bag of K instances, the feature matrix Z R requires O(K D) memory.
Attention Vector Storage: The attention weight vector a E R requires O(K) memory.
During the forward pass, each instance requires storing intermediate activations for all transformer layers. For single instance, this requires O( N D) memory for hidden states and O(L Il N²) for attention maps, where H is the number of attention heads. Let's denote the total activation memory per instance Sact O(L N- D D L H N
Activation Memory:
Gradient Memory: During backpropagation, gradients for both activations and parameters must be stored. The memory requirement for gradients per instance denoted as Syred- which is typically on the same order as Sact-
Aggregator Memory:
The aggregator requires O(D) memory for parameters and ((1) for the output, which is negligible compared the PFM memory requirements.
Gradient Matrices for Feature and Attention: The feature gradient matrix G3 R requires O(K D) memory, and the attention gradient vector E RR requires O(K) memory.
Combining all components and identifying the dominant terms, the total space complexity is:
0(0rFMl + K D K + K Sact + K Syrad K D K)
(31)
Since Sact and Sgrad are typically much larger than D, and combining like terms produces:
O(0PFM| + K (Suct + Syrad))
(32)
Above equation indicates that the memory requirements TAPFM scale linearly with the number of instances (tiles) K. However, the constant factors Sact and S grrad can be substantial for large ViT models, potentially limiting the number of instances that can be processed simultaneously on single GPU.
D
23

---

A.4 Task-Specific Adaptation without Catastrophic Forgetting
Proposition 2 (Preservation of Pretrained Knowledge). The proposed TAPFM approach effectively implements specialized form of continual learning that prevents catastrophic forgetting. The Fisher information matrix for pretrained parameters is implicitly preserved:
FOFFM ())())))) VepFM log (2:12::0000 V V8pF (ogp(s)s;0FFN)
(33)
Proof. Catastrophic forgetting occurs when fine-tuning pre-trained model on a new task destroys the knowledge acquired during pretraining. In iitoommtion-thhoreee terms, this happens when parameter updates move away from regions of high Fisher information with respect to the pretraining task.
The Fisher information matrix FDrrM characterizes the curvature of the loss landscape around the pretrained parameters. quantifies how sensitive the model's output to small changes in each parameter. Parameters with high Fisher information are crucial for the model's performance on the pretraining task.
In standard fine-tuning, the gradient update is: A0PFM -1 Vpr Ctask
(34)
This update does account for the importance of parameters for the pretraining task, potentially leading to catastrophic forgetting. In contrast, optimal updates for preserving pretrained knowledge while adapting new task would be of the form:
A0PFM = -nFFrrm OpPM Verru Crask
(35)
This is form of natural gradient descent, which adapts the update direction based on the curvature of the loss landscape.
The proposed TAPFM approach implicitly approximates this behavior through gradient detachment. By separating the aggregator and PFM optimization, TAPFM allows the PFM adapt more conser- vatively. The detached gradients from the aggregator act as filtered signal that guides the PFM to adapt while respecting its pretrained structure.
Specifically, the proposed PFM update is:
A0PFM -1 CPFM
(36)
where LPFM is defined using detached gradients from the aggregator. This approach resembles regularized optimization problem:
min Liass((PPM) + X R(0PFM - OPPM OPFM
Opretrained
(37)
where R is an implicit regularization term that penalizes deviations from the pretrained parameters. The solution to this regularized problem can be approximated as:
A0PFM -n(I + X Fepruc - Lrask
(38)
As À increases, this approaches the natural gradient update:
A0PFM ~ - Forrss Veres Crast Re -nj Koppas Vorexe Crask
(39)
where 1/ = 1 1 is an effective learning rate.
The detached gradient approach of TAPFM serves as an implicit regularization mechanism that enables task-specific adaptation while preserving the foundational knowledge embedded in the pretrained model. This balance between adaptation and preservation is especially important for PFMs, where pretraining captures general tissue morphology and fine-tuning adapts to specific clinically relevant tasks.
24

---

A.5 Cosine Regularization for Feature Alignment Loss
The feature alignment loss in equation 5 encourages feature vectors move in the direction that reduces the classification loss. However, in certain scenarios, direct optimization of this loss may lead to updates where feature vectors and gradients point in opposing directions, potentially causing unstable training dynamics. To mitigate this issue, a cosine regularization term is considered. The cosine regularization term is defined as: Freg Σ (1 cos( 2i, 9=, where cos 9z,) represents the cosine similarity between the feature vector 2; and gradient fz,, calculated as:
(Zis 92,
(os((zi99z))
(40)
2 |22:22 11
This regularization term penalizes scenarios where feature vectors and their gradients are oriented in opposite directions (negative cosine similarity). When incorporated into the task adaptation loss, the complete objective becomes:
LPFM = Deeature + XCtttention + BBreeg
(41)
where hyperparameter controlling the strength of the regularization. The purpose of this regularization is to promote more stable optimization by encouraging gradual updates to the learned representations while still allowing adaptation the downstream task. Theoretically, this prevents drastic changes in feature space that might compromise the pretrained knowledge while ensuring that updates remain aligned with the classification objective.
B Training Loss Curves
37
2.5
12.3
20.2
(a) Training loss variation with respect to lambda in TAL (equation 7 for H-Optimus-O (TAPFM) model.
[pech
(b) Training loss variation with the number of tiles for H-Optimus-O (TAPFM) model.
Figure 4: Ablations studies for FGFR3 prediction in BLCA patients.
C TAPFM Optimization Stability
Figure empirically demonstrates the stabilization benefit of TAPFM's detached gradient approach compared to unified computational graph training. When attempting joint optimization with a unified graph (left panel), the training loss exhibits unstable oscillations around 0.85 without convergence over 20 epochs. indicating optimization conflicts between the PFM and aggregator parameters. In contrast, TAPFM's detached gradient approach (right panel) shows smooth exponential decay from 0.89 to below 0.08, confirming the theoretical analysis in Section 3.4. This stable convergence enables effective single-GPU training where joint optimization fails due to competing gradient signals through shared parameters.
25

---

Unified computational graph (unstable)
TAPFM detached gradients (stable)
0.94
0.53
0.75
0.70
0.90
0 0.50
0.
Training
0.80
0.45
0.86
F 0.35
0.84
0.30
0.15
0.80
0.05
101112 19145 19 Epoch
10 121314 1617 19 Epach
(a) Unified computational graph (unstable)
TAPFM detached gradients (stable)
Figure 5: Training loss comparison demonstrating optimization stability. Unified computational graph training fails converge due gradient conflicts, while TAPFM's detached approach enables smooth convergence.
D Additional Results
D.1 Mutation Prediction
To provide a comprehensive assessment accounting for class imbalance, we report Area Under the Precision-Reed Curve (AUPRC) values for the mutation prediction experiments. Tables 3 and present AUPRC results for the binary and multi-label mutation prediction tasks, respectively.
D.2 Survival Prediction
To demonstrate TAPFM's versatility beyond mutation prediction, we evaluated its performance on overall survival prediction using TCGA-BLCA and TCGA-LUAD cohorts. For this task, we replaced the classification head with a multi-task logistic regression (MTLR) layer and used partial log- likelihood loss for optimization [37]. Results 5 that TAPFM's ' task adaptation approach extends effectively to time-to-event prediction tasks well.
26

---

Table Performance comparison of different PFM and MIL aggregation methods for binary classi- fication tasks in terms of AUPRC (mean + 95% CI from 5-fold cross- -ross-validation), N indicates the number of patients in the testing cohort. Best and second best models are in bold and underlined text, respectively.
Model
BLCA FGPR3
LUAD EGER
Institutional
TCGA
Institutional (N = 8768
TCGA (N 438)
194)
(N = 260)
Fixed-PFM with Trained MIL. Aggregators
UNI + DSMIL
0.4523 1 0.047
0.4687 + 0,048
0.4785 1 0.038
0,5142 1 0.042
UNI + CLAM
0,4571 1 0.046
0.4712 + 0.047
0.4831 1 0.037
0.5189 1 0.041
UNI + VarMIL
0.4618 1 0.045
0.4763 - 0.046
0.4893 1 0.036
0.5247 1 0.040
UNI + ABMIL
0.4597 = 0.046
0.4789 = 0.046
0.4862 = 0.037
0.5312 + 0.039
GigaPath DSMIL
0.5634 +: 0.041
0.6247 + 0.039
0.5473 + 0.034
0.6189 + 0.037
GigaPath CLAM
0.5683 + 0.040
0.6294 + 0.038
0.5521 +: 0.033
0.6235 0.036
GigaPath VarMIL
0.5741 + 0.039
0.6352 + 0.037
0.5584 + 0.032
0.6298 + 0.035
GigaPath ABMIL
0.5789 + 0.039
0.6397 = 0.036
0.5627 = 0.032
0.6341 + 0.034
H H-Optimus-0 0 + DSMIL
0.5891 + 0.038
0.6453 = 0.036
0.5692 = 0.031
0.6429 + 0.036
H-Optimus-O- + CLAM
0.5924 + 0.038
0.6512 = 0.035
0.5748 + 0.030
0.6481 + 0.035
H-Optimus-O- VarMII.
0.5987 + 0.037
0.6583 + 0.034
0.5809 + 0.029
0.6547 + 0.034
H H-Oppimus-O0 0 + ABMII
0.6023 + 0.037
0.6621 + 0.034
0.5841 + 0.029
0.6583 + 0.033
Fine-tuned (FT) PFM with MIL Aggregators
UNI + DSMIL (FT)
0.5247 + 0.044
0.5389 = 0.045
0.5134 + 0.036
0.5497 0.040
UNI + CLAM (FT)
0,5293 + 0.043
0.5427 = 0.044
0,5189 - 0.035
0.5558 + 0.039
UNI + VarMIL (FT)
0.5361 1 0.042
0.5487 1 0.043
0.5258 1 0.034
0.5631 1 0.038
UNI + ABMIL (FT)
0.5412 1 0.042
0.5463 1 0.044
0.5297 1 0.034
0.5689 1 0.037
GigaPath DSMIL (FT)
0.5923 + 0.039
0.6687 = 0.034
0.6142 # 0.031
0.6547 + 0.035
GigaPath CLAM (FT)
0.5971 = 0.038
0.6734 = 0.033
0.6213 = 0.030
0.6608 + 0.034
GigaPath VarMIL (FT)
0.6047 + 0.037
0.6812 + 0.032
0.6301 + 0.029
0.6689 + 0.033
GigaPath ABMIL (FT)
0.6021 + 0.038
0.6781 + 0.033
0.6358 + 0.029
0.6723 + 0.032
H-Optimus-00 DSMIL (FT)
0.6187 = 0.036
0.6794 = 0.033
0.6473 = 0.028
0.6831 0.034
H H-Optimus-00 0 + CLAM (FT)
0.6234 + 0.035
0.6847 = 0.032
0.6541 = 0.027
0.6912 = 0.033
H-Optimus-O- + VarMII. (FT)
0.6318 + 0.034
0.6931 + 0.031
0.6624 + 0.026
0.6989 + 0.032
H-Optimus-O- + ABMII (FT)
0.6289 + 0.035
0.6897 = 0.032
0.6671 + 0.027
0.6958 + 0.031
Proposed TAPFM method
UNI (TAPFM)
0.6142 + 0.041
0.6389 + 0.039
0.6542 + 0.030
0.6647 + 0.036
Gigapath (TAPFM)
0.7235 + 0.032
0.7251 = 0.028
0.7089 + 0.025
0.7143 + 0.021
H-Optimus-4 O (TAPFM)
0.7163 + 0.035
0.7164 = 0.024
0.7287 + 0.023
0.7319 + 0.017
Table 4: Performance comparison on multi-label classification of actionable mutations in LUAD in terms of AUPRC (mean + 95% CI from 5-fold cross-validation) Based on binary classification results (Table D. only the top performers are shown, as performance varied primarily by foundation model type.
Institutional Cohort (N 876)
Model
Macro Average
EGFR
KRAS
MET
ALK
UNI (TAPFM)
0.6589 0.032
0.6047 + 0.036
0.1673 + 0.061
0.1429
0.068
0.3934 + 0.031
Gigapath (TAPFM)
0.6872 0.027
0.6396 0.031
0.2483 1 0.055
0.1507
0.062
0.4315 1 0.026
H-Optimus-O (TAPFM)
0.6773 0.025
0.6177 + 0.029
0.3457 + 0.051
0.1951 0.058
0.4590 + 0.024
TCGA (N 438)
UNI (TAPFM)
0.6631 + 0.038
0.6095 + 0.042
0.1547 + 0.072
0.1392 + 0.081
0.3916 + 0.036
GigaPath (TAPFM)
0.6850 + 0.032
0.6413 + 0.036
0.2374 + 0.063
0.1431 + 0.074
0.4269 + 0.031
H-Optimus-O (TAPFM)
0.6942 0.030
0.6284 + 0.034
0.3104 1 0.059
0.1862 1 0.069
0.4548 : 0.029
27

---

Table 5: Overall survival prediction performance comparison on TCGA cohorts. Results are reported as concordance index (C-index) with 95% confidence intervals from 5-fold -ross-validation. Higher values indicate better survival prediction.
TCGA-BLCA
TCGA-LUAD
method
C-index
95% CI
C-index
95% CI
UNI (TAPFM)
0.6218
(0,5759, 0.6677)
0.6687
(0.6042, 0.7332)
GigaPath (TAPFM)
0.6341
(0.5935, 0.6747)
0.6825
(0.6214, 0.7436)
H-Optimus-O (TAPFM)
0.6408
(0.6045, 0.6771)
0.6953
(0.6371, 0.7535)
28


Reviews:
[{"review_id": "u5978376en", "reviewer": "Reviewer_cPrU", "summary": "The size of WSIs is typically extremely huge. Existing WSI models are unable to model them in an end-to-end manner. As a result, the paradigm of combining pretrained models with MIL has become a mainstream approach in the field of WSI analysis. This paper addresses the common challenge in existing methods where features extracted by frozen pretrained models are not well adapted to downstream tasks, while end-to-end training remains infeasible. It proposes a task-adaptive method for PFMs, which leverages the internal attention mechanism of ViTs for MIL feature aggregation, and separately optimizes the PFM and MIL components via detached computation graphs. The proposed approach achieves state-of-the-art performance across multiple datasets.", "questions": "For the experiments in Table 2, although the paper demonstrates in Table 1 that the proposed method achieves the best results, at least one top-performing baseline should be included in Table 2 as a comparison. This would make the contribution of the paper clearer to readers.", "limitations": "please refer to Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper achieves the optimization of both the foundation model for image patch feature extraction and the MIL model for feature aggregation on a single GPU.\n2. It proposes a novel method for constructing separate computation graphs.\n3. It cleverly leverages internal attention information from the foundation model.\n\nWeaknesses:\n1. The paper lacks an overall illustrative figure that summarizes or visually explains the proposed method.\n2. While the method performs well on datasets beyond LUAD EGFR-TCGA, its performance on this particular dataset is relatively average. The authors should provide a detailed explanation for this and summarize the potential limitations of the proposed method.\n3. The proposed attention loss focuses on image patches that are important for the downstream task. On datasets with a large number of positive samples, could this lead to the attention mechanism eventually concentrating on only a single image patch?", "was_revised": true, "final_justification": "The authors’ responses have addressed most of my concerns. I will maintain my score.", "rebuttal": "**Note:** For brevity PMID refers to the PubMed identifier of the papers referenced in the rebuttal. We have provided full paper details for those articles that do not have a PMID or are not referenced in the manuscript.\n\n---\n\n### **Reviewer Comment 1**  \n*The paper lacks an overall illustrative figure that summarizes or visually explains the proposed method.*\n\n**Author Response:**  \nWe appreciate the reviewer’s understanding of the key contributions of the manuscript. We have created an illustration for the proposed TAPFM method and can include it in the final version of the manuscript, if accepted.\n\n### **Reviewer Comment 2**  \n*While the method performs well on datasets beyond LUAD EGFR-TCGA, its performance on this particular dataset is relatively average. The authors should provide a detailed explanation for this and summarize the potential limitations of the proposed method.*\n\n**Author Response:**  \nThe recently published results by Campanella et al. (Nature Medicine, July 2025, PMID: 40634781) show the most advanced *EGFR* prediction model (to date) trained using 24 Nvidia H100 GPUs simultaneously yielded AUC of 0.847 on internal datasets and 0.860 on TCGA. The proposed TAPFM method outperforms this most recent state-of-the art by yielding 0.867 on institutional and provides comparable AUC of 0.863 on TCGA samples while using only one Nvidia H100 GPU for training. We will include the recent Nature Medicine paper as reference to provide additional context in the final version of the manuscript, if accepted.  \n\nIt is noteworthy that the proposed TAPFM also works for detecting multiple mutations under the multi-label classification setting, as shown by results in Table 2. This goes beyond conventional binary and multi-class classification paradigms. While binary classification handles one mutation at a time, multi-class classification assumes mutual exclusivity among classes, each tumor is assigned to a single class (e.g., *EGFR* or *KRAS*), which inherently prevents modeling co-occurring mutations as labels are usually represented as one-hot-coded vectors (e.g. `[0,1,0,0]`) for multi-class classification.  \n\nHowever, tumors in the real-world occasionally harbor multiple actionable mutations simultaneously, in scenarios like collision tumors (PMID: 38304116) and when a second driver mutation is the resistance mechanism (PMID: 34590027). In our large institutional dataset of LUAD we have many examples where multiple *EGFR*, *KRAS*, *MET*, and *ALK* driver alterations coexist in the same tumor. There are also common scenarios where combinations of driver alterations plus some additional mutations are prognostically important, such as the combination of *KRAS* with another mutation have worse prognosis than *KRAS* alone and may be less likely to benefit from immunotherapy (PMID: 39703851). Thus, TAPFM can be utilized for these situations where we want to simultaneously detect multiple mutations in a given tumor.  \n\nMulti-label classification offers a more biologically faithful framework by allowing the model to predict a vector of multiple binary mutations (e.g., `[1, 0, 1, 0]` for concurrent *EGFR* and *MET* mutations). Importantly, TAPFM enables this by training a single model that simultaneously predicts the presence or absence of multiple co-occurring mutations from H&E-stained whole slide images. \n\nDespite its clinical relevance, multi-label classification remains underexplored in the context of pathology-based mutation prediction, likely due to its increased complexity and data imbalance challenges. The strong performance of TAPFM across both common and rare mutations in LUAD, including *MET* and *ALK*, demonstrates its robustness in tackling this more difficult yet clinically important problem. For background, multi-label classification has been formally studied in machine learning literature as a distinct and complex learning problem (see Zhang and Zhou, 2014).\n\n> **Zhang, M.-L., & Zhou, Z.-H.**  \n> *A review on multi-label learning algorithms.*  \n> IEEE Transactions on Knowledge and Data Engineering, 26(8), 1819–1837. 2014.  \n\nWe can include the above reference in the manuscript that readers can use to understand the complexity of multi-label mutation prediction in comparison to more common multi-class classification. The limitations of the proposed TAPFM method are listed in 286–294 in the manuscript.\n\n### **Reviewer Comment 3**  \n*The proposed attention loss focuses on image patches that are important for the downstream task. On datasets with a large number of positive samples, could this lead to the attention mechanism eventually concentrating on only a single image patch?*\n\n\n**Author Response:**  \nIn the worst-case scenario, the attention score distribution could be uniform across all patches obtained from a WSI. However, having a single important patch seems less plausible and would require specific, controlled experiments that are beyond the scope of this paper. In our experience, high attention is consistently spread across tumor regions within the slide.\n\nIn scenarios where tumor comprises a small percentage of the tissue area, it is actually advantageous that the model concentrates attention on a relatively small number of tiles, since any tile that does not contain tumor cannot meaningfully inform the mutation status. This is precisely why attention is such a powerful and necessary mechanism in computational pathology: the model must not only learn the classification task but also learn what is relevant within a large pool of uninformative or irrelevant data.\n\nTo our knowledge, this manuscript presents the first approach that utilizes attention *from within the ViT itself*, without relying on an external model (e.g., ABMIL, reference 2) for slide-level aggregation of tile features. Our focus in this work is on demonstrating the efficacy of this integrated architecture. Visualization of attention maps, while visually compelling, offers limited insight for the mutation prediction task, where the morphological correlates of tumor mutations remain poorly established. In other words, pathologists cannot reliably infer mutation status based solely on morphologic features. As such, saliency maps are unlikely to yield actionable information beyond confirming high model attention on tumor regions, a prerequisite that underlies TAPFM’s well-documented high performance.\n\n### **Reviewer Comment 4**  \n*For the experiments in Table 2, although the paper demonstrates in Table 1 that the proposed method achieves the best results, at least one top-performing baseline should be included in Table 2 as a comparison. This would make the contribution of the paper clearer to readers.*\n\n**Author Response:**  \nThe proposed TAPFM method addresses the multi-label classification problem instead of the more common multi-class classification problem as described in section 3.5 and shown by results in Table 2. See response to comment 2 for more details on the multi-label classification problem.  \n\nTo the best of our knowledge, there are no parallel methods where multi-label classification has been implemented in the techniques for which we are benchmarking against, and the methods would require separate development, which is outside the scope of the manuscript. Thus, there are no available external methods for which are comparable to include in Table 2.  \n\nWe will however include UNI (TAPFM) results for multi-label classification in Table 2 as well in the final manuscript for completeness."}, {"review_id": "2eFGvk9JbX", "reviewer": "Reviewer_ggKh", "summary": "(1) The paper proposes TAPFM, a two-stage optimization scheme that lets Vision-Transformer pathology foundation models be fine-tuned end-to-end on a single GPU.\n\n (2) In the forward pass, the ViT yields tile features and CLS-token attention scores; the latter are soft-maxed to weights and used to build a permutation-invariant slide representation.\n\n (3) Stage-1 back-prop updates only a lightweight linear classifier with a standard cross-entropy loss, while Stage-2 back-prop uses detached gradients from the first stage to craft a second loss that jointly pushes the ViT’s features and attentions, updating the backbone but not the classifier and thereby avoiding unstable gradient feedback.\n\n (4) The authors supply proofs of permutation invariance, an $O(K C_{ViT}​)$ time bound (K is tiles per slide, $C_{ViT}​$ the cost of one forward + backward through the backbone), and an argument that the detached update breaks circular dependencies.", "questions": "(1) Could you provide formal measures of statistical significance—such as confidence intervals or p-values—for the reported AUC improvements to verify that the 1–4-point gains are reliable?\n\n(2) How does TAPFM’s accuracy and runtime change when the tile budget is reduced to fit GPUs with 24 GB or 48 GB memory? Is it even possible to fit in those GPUs? An ablation would clarify hardware requirements.\n\n(3) Can you supply empirical evidence (e.g., gradient-norm or loss-variance plots) that demonstrates the claimed stabilisation from the detached two-stage update compared with joint optimisation?\n\n(4) Do the learned CLS-attention weights correspond to histologically meaningful regions? A few slide visualisations or a brief expert evaluation would bolster confidence in the weighting mechanism.\n\n(5) Have you evaluated TAPFM on other clinically important slide-level tasks (grading, subtype classification, prognosis, treatment response, or datasets beyond BLCA and LUAD)? Additional results would help establish the method’s significance.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Existing adaptation workflows keep the PFM frozen and add a separate MIL head, or perform a self-supervised intermediate step before a second-stage supervised training; truly end-to-end slide-level back-prop has been deemed too memory-hungry. TAPFM shows that full, single-stage fine-tuning of billion-parameter PFMs is feasible on one high-memory GPU, eliminating the complexity of multi-stage procedures while delivering better accuracy\n\n(2) Leveraging the ViT’s CLS-token attention scores as tile weights eliminates the need for a separate aggregation network and yields a permutation-invariant bag representation with virtually no additional computation.\n\n(3) Across two cancer types, four actionable mutations, three PFMs (UNI, GigaPath, H-Optimus-0), and an external TCGA validation, TAPFM delivers 1–4 percentage points AUC improvements over both fixed-feature and naïve end-to-end baselines.\n\n(4) The same algorithm works unchanged on ViT-L, ViT-H, and 1 B-parameter ViT-G backbones, and scales to the simultaneous prediction of four mutations, including rare MET and ALK events.\n\n(5) A formal derivation shows that detaching the first-stage gradients eliminates the circular dependency between backbone and aggregator updates; empirically, this corresponds to smoother, faster convergence.\n\n(6) The paper proves permutation invariance of the bag representation and shows that total compute per slide grows linearly with the number of tiles times the cost of a single ViT pass.\n\n(7) Algorithm 1 is explicit, hyperparameters and memory budgets are reported, ablations on λ and tile count are included.\n\nWeaknesses:\n\n(1) The evaluation compares TAPFM only to fixed-feature MIL heads and naïve end-to-end training. It omits any self-supervised adaptation pipelines, making it harder to gauge the relative benefit of the proposed approach.\n\n(2) Performance is given as single AUC values with no confidence intervals or other measures of statistical significance, making it unclear whether the 1–4-point improvements are reliable.\n\n(3) All experiments are limited to mutation prediction in two tumour types. Because no results are shown for other high-value slide-level objectives—such as grading, subtype detection, prognosis, or treatment-response—the work’s broader significance to computational pathology remains uncertain; demonstrating gains on a more varied task set would better substantiate TAPFM’s impact.\n\n(4) The experiments are indeed run on a single device, but that device is an 80 GB H100. Readers should keep in mind that “single GPU” in this work specifically means a high-memory, top-tier card.\n\n(5) The methodology is designed for ViT-based models. Though most PFMs are ViT-based today, it could be a limitation for future PFM.", "was_revised": true, "final_justification": "I’m keeping a borderline accept rather than pushing to accept because, while TAPFM is practical and well-executed, single-GPU end-to-end adaptation via a detached two-stage update, the measured gains over the strongest fine-tuned baselines are modest (≈+0.01–0.03 AUC across cohorts), and the “single-GPU” result depends on an 80-GB H100. The work has novelty and clear contributions, but its broader significance remains unclear without a more comprehensive evaluation across diverse datasets, data types, and PFMs to demonstrate generalizable impact.\n\nResolved via rebuttal/discussion:\nStatistics: authors committed to 5-fold CV with 95% CIs (example CIs provided) and to add AUPRC.\n\nUnresolved:\nSignificance: Evaluation focuses on BLCA/LUAD mutation prediction (plus a survival add-on in rebuttal), lacks diverse datasets, slide-level objectives, and PFMs to demonstrate robustness.", "rebuttal": "**Note:** For brevity PMID refers to the PubMed identifier of the papers referenced in the rebuttal. We have provided full paper details for those articles that do not have a PMID or are not referenced in the manuscript.\n\n---\n### **Reviewer Comment 1**  \n*The evaluation compares TAPFM only to fixed-feature MIL heads and naïve end-to-end training. It omits any self-supervised adaptation pipelines, making it harder to gauge the relative benefit of the proposed approach.*\n\n**Author Response:**  \nWe appreciate the reviewer’s comment. The pathology foundation models we evaluate have already undergone extensive self-supervised pretraining (e.g., DINO v1/v2), and our focus is on task adaptation for mutation prediction using clinical labels within the MIL setting, which self-supervised, by definition, methods do not leverage. The most advanced *EGFR* prediction model to date, reported in Campanella et al. (*Nature Medicine*, 2025, PMID: 40634781), achieved AUCs of 0.847 (internal) and 0.860 (TCGA) using **24 H100 GPUs**. TAPFM achieves comparable or better results (0.867 internal, 0.863 TCGA) with a **single H100**, demonstrating the efficiency and impact of our adaptation strategy.\n\n### **Reviewer Comment 2**  \n*Performance is given as single AUC values with no confidence intervals or other measures of statistical significance, making it unclear whether the 1–4-point improvements are reliable.*\n\n**Author Response:**  \nWe have run 5-fold cross validation (CV) for all tasks mentioned in Table 1 and 2, on institutional data while keeping TCGA datasets as independent test sets for model evaluation within each CV fold, that produces the recommended confidence intervals (CI). We will include the complete CV with CIs results in the final manuscript.  \n\nSpace constraints limit including all CIs for all experiments in rebuttal but as an example 5-fold CV results for best binary predictions are:\n\n- *EGFR* TAPFM (H-Optimus-0): AUC 0.8503 (CI: 0.8385 – 0.8643) on institutional data and AUC 0.8590 (CI: 0.8496 – 0.8612) on TCGA.  \n- *FGFR3* TAPFM (H-Optimus-0): AUC 0.8837 (CI: 0.8609 – 0.8974) on institutional data and AUC 0.8956 (CI: 0.8890 – 0.9108) on TCGA.\n\n### **Reviewer Comment 3**  \n*All experiments are limited to mutation prediction in two tumour types. Because no results are shown for other high-value slide-level objectives—such as grading, subtype detection, prognosis, or treatment-response—the work’s broader significance to computational pathology remains uncertain; demonstrating gains on a more varied task set would better substantiate TAPFM’s impact.*\n\n**Author Response:**  \nWe appreciate the reviewer’s comment and the opportunity to clarify. While we recognize that tasks such as grading, subtype detection, prognosis, and treatment-response prediction are commonly cited as valuable applications in computational pathology, we would note that many of these objectives are best formulated using alternative modeling approaches. For example, subtyping and grading can indeed be approached as slide-level tasks, but are more effectively addressed via pixel-level segmentation or patch-level classification models, which provide region-specific interpretability and quantification (e.g., PMID: 37544502; 37491086; 39448755).\n\nRegarding the significance of our selected tasks: both bladder (BLCA) and lung adenocarcinoma (LUAD) are diseases where mutational status, specifically of *FGFR3*, *EGFR*, *MET*, *ALK*, and *KRAS*, is directly linked to targeted therapy eligibility (see Section 5). As such, the mutation prediction tasks in our study are not only clinically actionable but also prognostic and predictive of treatment response, fulfilling the reviewer’s criteria.\n\nFurther, pathologists’ training is centered around visual assessment of tumor presence, grade, stage, histological subtypes, but no reliable visual markers exist for detecting genetic mutations from H&E stained whole slide images. Prediction of mutations cannot be accomplished by visual inspection and the features must be learned by the model, making this a particularly challenging task for which out-of-the-box pathology foundation models (PFMs) have poor performance.\n\nTo further assess the utility of TAPFM beyond mutation prediction, we conducted additional experiments on overall survival prediction (from whole slide images) using the TCGA-BLCA and TCGA-LUAD cohorts. TAPFM achieved concordance indices of **0.6408 ± 0.0185** on BLCA and **0.6953 ± 0.0297** on LUAD, outperforming the previous best results reported in *Tang et al.* (CVPR 2024), which were **0.6398 ± 0.0263** and **0.6719 ± 0.0402**, respectively, on the same datasets.\n\nWe also note that out-of-the-box foundation model performance has been reported in Figure 4 of *Yang et al.* (*Nature Communications*, April 2025, PMID: 40064883). This figure does not report all confidence intervals but states that their optimized method achieves best performance of **0.6039** (95% CI: 0.5093–0.6986).\n\nIn the final manuscript we will report in the appendix a table with our results along with out-of-the-box foundation model performance for the survival prediction task.\n\n> **Tang, Wenhao, et al.**  \n> *Feature re-embedding: Towards foundation model-level performance in computational pathology.*  \n> Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n### **Reviewer Comment 4**  \n*The experiments are indeed run on a single device, but that device is an 80 GB H100. Readers should keep in mind that “single GPU” in this work specifically means a high-memory, top-tier card.*\n\n\n**Author Response:**  \nYes, the H100 is indeed a high-memory GPU. However, TAPFM avoids the complexity of multi-GPU setups, which improves accessibility. We also outline future directions in *lines 289–290* to further reduce memory requirements. Specifically, we believe that if we inspect which layers of the PFM receive most of the updates during model training we can reduce the GPU memory requirements even further by only fine tuning those layers for downstream tasks. \n\n### **Reviewer Comment 5**  \n*The methodology is designed for ViT-based models. Though most PFMs are ViT-based today, it could be a limitation for future PFM.*\n\n**Author Response:**  \nWe explicitly acknowledge that the proposed method is specifically for ViT based PFMs in *line 60* of the manuscript.\n\n### **Reviewer Comment 6**  \n*Could you provide formal measures of statistical significance—such as confidence intervals or p-values—for the reported AUC improvements to verify that the 1–4-point gains are reliable?*\n\n**Author Response:**  \nPlease see response to comment 2.\n\n### **Reviewer Comment 7**  \n*How does TAPFM’s accuracy and runtime change when the tile budget is reduced to fit GPUs with 24 GB or 48 GB memory? Is it even possible to fit in those GPUs? An ablation would clarify hardware requirements.*\n\n**Author Response:**  \nThank you for raising this point. It’s not entirely clear whether the question is about training or inference, so we’ll briefly address both.\n\n**Inference:** The TAPFM model, though trained on an Nvidia H100, can be run on GPUs with much less memory. This is done by loading tiles in batches from the WSI during inference. Tile features and attention scores are accumulated incrementally before a final pass through the classification head. As a result, the GPU memory requirement is modest, and the total number of tiles processed remains the same regardless of batch size. Performance is unaffected; inference time scales with the number of batches. We routinely run inference with these models on consumer grade GPUs.\n\n**Training:** It is noteworthy that training is memory intensive because the GPU memory needs to store the massive ViT based PFM models plus image tensors while maintaining gigantic computation graphs to keep track of model parameters and their gradients during forward and backwards passes through the encoder and aggregator. Thus, GPU cards with reasonable memory is required to build such models. We received out of memory errors when we tried training on Nvidia A40 GPUs (48GB). We will include this as a limitation, if the reviewer insists.\n\n### **Reviewer Comment 8**  \n*Can you supply empirical evidence (e.g., gradient-norm or loss-variance plots) that demonstrates the claimed stabilisation from the detached two-stage update compared with joint optimisation?*\n\n**Author Response:**  \nYes, we can include loss-variance plots that show the unified computation graph does not converge during training, while the proposed detached gradient approach converges, in the Appendix to show this.\n\n### **Reviewer Comment 9**  \n*Do the learned CLS-attention weights correspond to histologically meaningful regions? A few slide visualisations or a brief expert evaluation would bolster confidence in the weighting mechanism.*\n\n**Author Response:**  \nWe have not included such visualizations because:\n\na. It has never been shown in a blinded way that pathologists can ascertain genomic mutations based on visual inspection alone. \nb. Such visualizations (in others as well as our experiments) have shown that the model assigns high attention scores on tiles that contain tumor within a WSI. This is expected but not insightful as somatic mutations are present only within cancerous tissue.  \nc. There could be multiple mutations for which the same tumor regions (tiles) receive high attention scores. It is unclear at this time whether those underlying visual patterns are unique or common across various mutations. The proposed multi-label TAPFM can be used to explore this, but we have not yet embarked on this line of inquiry.\n\n### **Reviewer Comment 10**  \n*Have you evaluated TAPFM on other clinically important slide-level tasks (grading, subtype classification, prognosis, treatment response, or datasets beyond BLCA and LUAD)? Additional results would help establish the method’s significance.*\n\n**Author Response:**  \nPlease see response to comment 3."}, {"review_id": "Jl8uxFIoie", "reviewer": "Reviewer_njqY", "summary": "This paper proposes a method to adapt pathology foundation models for whole slide tissue image (WSI) classification. The proposed method makes use of the attention mechanism of the vision transformer architecture for feature aggregation and employs separate computation graphs for parameter updates.", "questions": "Additional investigation is needed to better assess the benefits and limitations of the proposed method. The WSI classification task may not be the best analysis task to evaluate this method. Other analysis tasks such as survival analysis may benefit more. \n\nMoreover, the authors use relatively large institutional datasets to train their models. One of the main benefits of using foundation models is to be able to use relatively limited volume of training data. The authors should consider training their models with TCGA data and use the institutional datasets for testing.", "limitations": "Some recent works have demonstrated high performances (e.g. [1,2]; [1] shows AUC scores of about 0.97). There should be a comparison with those works as well. While adaptation of PFMs is a reasonable approach, clinical research studies will want to use the best methods available regardless of the underlying methodology. \n\n[1] Dang, Thao M., Yuzhi Guo, Hehuan Ma, Qifeng Zhou, Saiyang Na, Jean Gao, and Junzhou Huang. \"MFMF: Multiple Foundation Model Fusion Networks for Whole Slide Image Classification.\" In Proceedings of the 15th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1-8. 2024.\n[2] Li, Hao, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. \"Generalizable whole slide image classification with fine-grained visual-semantic interaction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11398-11407. 2024.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This study introduces a different approach to integrating and adapting foundation models and multi-instance learning (MIL) based classification models for WSI classification tasks. The experiments with several pathology foundation models (PFMs) and MIL approaches show that for certain PFM and MIL integrations, the proposed method achieves good performance improvement (e.g, UNI+DSMIL vs UNI (TAPFM) on TCGA LUAD dataset). However, when the best performance of the proposed method (H-Optimus-0 TAPFM) is compared with that of the best alternative approach (H-Optimus-O+ABMIL FT), the improvements are marginal and may not have much impact in a clinical research setting.  \n\nAdditional investigation is needed to better assess the benefits and limitations of the proposed method. The WSI classification task may not be the best analysis task to evaluate this method. Other analysis tasks such as survival analysis may benefit more. \nMoreover, the authors use relatively large institutional datasets to train their models. One of the main benefits of using foundation models is to be able to use relatively limited volume of training data. The authors should consider training their models with TCGA data and use the institutional datasets for testing. \n\nSome recent works have demonstrated high performances (e.g. [1,2]; [1] shows AUC scores of about 0.97). There should be a comparison with those works as well. While adaptation of PFMs is a reasonable approach, clinical research studies will want to use the best methods available regardless of the underlying methodology. \n\n[1] Dang, Thao M., Yuzhi Guo, Hehuan Ma, Qifeng Zhou, Saiyang Na, Jean Gao, and Junzhou Huang. \"MFMF: Multiple Foundation Model Fusion Networks for Whole Slide Image Classification.\" In Proceedings of the 15th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1-8. 2024.\n[2] Li, Hao, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. \"Generalizable whole slide image classification with fine-grained visual-semantic interaction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11398-11407. 2024.", "was_revised": true, "final_justification": "I have updated my rating after reviewing the authors' responses to my questions and comments. The paper has technical merit and contributions but relatively low accuracy improvements over other existing methods, hence, my rating of borderline accept.", "rebuttal": "**Note:** For brevity PMID refers to the PubMed identifier of the papers referenced in the rebuttal. We have provided full paper details for those articles that do not have a PMID or are not referenced in the manuscript.\n\n---\n\n### **Reviewer Comment 1**  \n*This paper proposes a method to adapt pathology foundation models for whole slide tissue image (WSI) classification.*\n\n**Author Response:**  \nThe core TAPFM methodology presented in this paper can be used for various types of downstream pathology tasks including classification, regression, survival prediction, etc. If one replaces the last layer with the task specific layer (such as the Cox proportional hazard layer for survival prediction instead of Equation 3) and uses an appropriate loss function (such as the partial log-likelihood loss for survival prediction instead of Equation 4), the core TAPMF method can be used directly to update the pathology foundation models (PFM) weights to adapt to a variety of clinical tasks. We will mention this additional detail in the final version of the paper as we did not specifically make this point in the original manuscript.  \n\nTo further assess the utility of TAPFM beyond mutation prediction, we conducted additional experiments on overall survival prediction from whole slide images using the TCGA-BLCA and TCGA-LUAD cohorts. TAPFM achieved concordance indices of **0.6408 ± 0.0185** on BLCA and **0.6953 ± 0.0297** on LUAD, outperforming the previous best results reported in *Tang et al.* (CVPR 2024), which were **0.6398 ± 0.0263** and **0.6719 ± 0.0402**, respectively, on the same datasets.  All of these experiments are done with 5-fold cross validation (CV).\n\nWe also note that out-of-the-box foundation model performance has been reported in Figure 4 of *Yang et al.* (*Nature Communications*, April 2025, PMID: 40064883). This figure does not report all confidence intervals but states that their optimized method achieves best performance of **0.6039** (95% CI: 0.5093–0.6986).  \n\nPlease note that the above survival prediction performances are from image only experiments.  Multimodel survival predictions have been proposed but that is not comparable to our current work.\n\nIn the final manuscript, we will report in the appendix a table with our results along with out-of-the-box foundation model performance for the survival prediction task. TCGA is of limited use in real-world scenarios (see response to Comment 5). We chose to highlight tasks that have clinical utility with achievable performance in the main manuscript.\n\n> **Tang, Wenhao, et al.**  \n> *Feature re-embedding: Towards foundation model-level performance in computational pathology.*  \n> Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n---\n\n### **Reviewer Comment 2**  \n*However, when the best performance of the proposed method (H-Optimus-0 TAPFM) is compared with that of the best alternative approach (H-Optimus-O+ABMIL FT), the improvements are marginal and may not have much impact in a clinical research setting.*\n\n**Author Response:**  \nWe thank the reviewer and respectfully disagree that the observed gains are marginal. H-Optimus-0 TAPFM outperforms H-Optimus-0 + ABMIL FT across all tasks and cohorts (e.g., **0.9021 vs. 0.8874 AUC** for FGFR3, **0.8553 vs. 0.8529 AUC** for EGFR; see Table 1). These differences are clinically meaningful when thresholds are tuned for deployment, as demonstrated in the prospective silent trial *Campanella et al. (Nature Medicine*, July 2025, PMID: 40634781), where a model with comparable AUC reduced EGFR testing by 43%.  \n\nUnlike that work, which required 24 Nvidia H100 GPUs, TAPFM achieves this performance using a single GPU. Additionally, TAPFM introduces a novel mechanism leveraging ViT self-attention scores for MIL aggregation and gradient routing, an approach not described in prior literature (see lines 95–97).  We will include reference to *Campanella et al.*'s *Nature Medicine* paper in the final version of the paper to provide additional clinical context.\n\n---\n\n### **Reviewer Comment 3**  \n*The WSI classification task may not be the best analysis task to evaluate this method. Other analysis tasks such as survival analysis may benefit more.*\n\n**Author Response:**  \nIt should be noted that not all WSI classification tasks are a monolith.  Different tasks have unique challenges so comparing AUC performance of tumor detection/classification to mutation prediction is inappropriate, see response comment 6.\n\nSee response to comment 1 regarding survival analysis.\n\n\n---\n\n### **Reviewer Comment 4**  \n*Moreover, the authors use relatively large institutional datasets to train their models. One of the main benefits of using foundation models is to be able to use a relatively limited volume of training data. *\n\n**Author Response:**  \nWhile perhaps the first interpretation of the use of PFMs was to demonstrate superior performance to other encoders on small datasets, the limitations of such models have become apparent (see references 11, 12, 29, and 30 in the manuscript), and this is precisely what we intend to address with this work.  \n\nResults from Table 1 (for fixed PFMs with trained MIL aggregators) do not yield satisfactory performance for the challenging task of mutation prediction across BLCA and LUAD when the foundation models are used as feature extractors with frozen weights.  \n\nThus, task adaptation of PFMs for clinically relevant mutation prediction becomes necessary and TAPFM provides a method for task adaptation of ViT-based PFMs on a single GPU, which has never been done before to the best of our knowledge.\n\n---\n\n### **Reviewer Comment 5**  \n*The authors should consider training their models with TCGA data and use the institutional datasets for testing.*\n\n**Author Response:**  \nTCGA is a relatively idealized dataset that has large tissue resections that are predominantly tumor as opposed to most material available for sequencing, which are often small biopsies. Further, in the real world, like with our institutional dataset, the tumor slides might have regions of tumor that make up <10% of the tissue area on the slide.  \n\nThese datasets with lower signals stress test performance and allow for better generalization. Models that are trained exclusively on TCGA are unlikely to generalize to real-world tasks (references 11 and 12 in the manuscript).  \n\nAs an example, we trained ABMIL models using features obtained from state-of-the-art PFMs on TCGA. When these models were tested on our internal cohort, the mean macro-AUC across PFMs dropped from **0.778 to 0.725** for EGFR prediction in LUAD. In contrast, models trained on our internal cohort and evaluated on TCGA perform better (Table 1, results for fixed PFMs with trained MIL aggregators).  \n\nWhile it is possible to adapt an existing PFM with TAPFM on TCGA data and test on institutional data, the advantages of TAPFM lie in exposing the foundation model to the most diverse dataset available.\n\n---\n\n### **Reviewer Comment 6**  \n*Some recent works have demonstrated high performances (e.g., [1,2]; [1] shows AUC scores of about 0.97). There should be a comparison with those works as well.*\n\n**[1]** Dang, Thao M., et al. *MFMF: Multiple Foundation Model Fusion Networks for Whole Slide Image Classification.* ACM-BCB 2024.  \n**[2]** Li, Hao, et al. *Generalizable whole slide image classification with fine-grained visual-semantic interaction.* CVPR 2024.\n\n**Author Response:**  \nThe reference works achieved ~0.97 AUC on tumor detection (CAMELYON16) and tumor type classification (TCGA NSCLC: LUAD vs. LUSC) tasks. However, these tasks are fundamentally different from the mutation prediction task addressed in our submission.  \n\nPathologists are trained to visually identify tumors and distinguish between tumor subtypes like LUAD versus LUSC (ref: *WHO Classification of Tumours of the Lung, Pleura, Thymus and Heart*, 4th Ed., 2015), but no visual markers exist for detecting genetic mutations from H&E-stained WSIs.  \n\nMoreover, the referenced works focus on building better slide-level classifiers on top of ResNet-based feature extractors. In contrast, modern ViT-based PFMs (refs 5 and 9 in our manuscript) achieve nearly perfect performance on simpler WSI classification tasks right out of the box.  \n\nUsing the H-Optimus-0 PFM as a fixed feature extractor with ABMIL, we achieved:\n\n- **AUC 0.9824 ± 0.0187** on CAMELYON16  \n- **AUC 0.9871 ± 0.0115** on TCGA NSCLC under 5-fold CV\n\nThese tasks appear to be effectively “solved” by existing PFMs and do not require adaptation.  \n\nIn contrast, Table 1 shows that out-of-the-box PFMs perform poorly for clinically relevant mutations (e.g., FGFR3 in BLCA and EGFR in LUAD), as noted in references 11, 12, 29, and 30.  \n\nTAPFM adapts ViT-based PFMs to improve performance on these harder tasks. It shows improvements on internal datasets and when run inference-only on TCGA (Table 1).  \n\nA recent study by *Campanella et al.* (*Nature Medicine*, July 2025, PMID: 40634781) confirms the benefit of adapting PFMs for EGFR prediction in LUAD. We will include this as a reference in the final version.  \n\nThat study used 24 Nvidia H100 GPUs to fine-tune Prov-Gigapath, achieving AUC of **0.847** (internal) and **0.860** (TCGA). TAPFM achieves **0.867** (internal) and **0.863** (TCGA) using just one H100 GPU.  \n\nTAPFM also supports multi-label classification for co-occurring mutations (Section 3.5, Table 2).\n\n**Clinical Relevance:** NCCN guidelines recommend comprehensive genomic testing for LUAD and BLCA, but these tests are expensive, slow, not universally available, and consume tissue. TAPFM provides a fast, cost-effective, tissue sparing tool as an adjunct to sequencing (see Section 5 of the manuscript).\n\n> **Travis, William D., et al.**  \n> *WHO Classification of Tumours of the Lung, Pleura, Thymus and Heart.*  \n> 4th ed., vol. 7. Lyon, France: International Agency for Research on Cancer, 2015. ISBN: 978-92-832-2436-5.\n---"}, {"review_id": "lACf7M87Hi", "reviewer": "Reviewer_JNqh", "summary": "This paper addresses the classification of whole slide pathology images. It proposes a method for adapting pretrained pathology foundation models (PFMs) to specific downstream tasks by decoupling and alternating the optimization of feature representations and attention weights. Experimental evaluations on bladder and lung cancer datasets, for both binary malignancy classification and multi-label mutation prediction, demonstrate improved performance over existing approaches.", "questions": "1) Improve the clarity and precision of the technical contributions described in Section 3.\n2) Include additional evaluation metrics beyond AUC for the existing experiments on BLCA and LUAD. This would provide a better picture of model performance, capturing not only statistical power but also the utility of predictions on test data. Metrics such as accuracy, precision, recall, F1 score, or calibration measures could demonstrate the robustness and clinical relevance of the method compared to other approaches.\n3) Additional minor issues mentioned above.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Presentation and Technical Contributions:*\nThe paper is generally well written and clearly organized. The proposed approach appears reproducible, and the authors provide code to support replication. However, the technical description in Section 3 lacks clarity. Concretely, the core contribution seems to boil down to disabling gradients and alternating between two optimization steps. While this may still be a reasonable and useful strategy, its novelty and significance are somewhat limited and would benefit from a clearer, more direct, exposition. The justification provided in Proposition 1 feels superficial and does not convincingly support the conclusion that decoupling leads to “more stable parameter trajectories” than joint optimization.\nLastly, the title emphasizes “single GPU,” but the manuscript does not clearly explain whether existing approaches are incompatible with single-GPU training, or whether this limitation is a motivation for the proposed method. It remains unclear if the method is specifically designed to reduce computational requirements, or if the focus is instead on improving training dynamics and predictive performance. Clarifying this point would help align the title with the actual contributions and goals of the paper.\n\n*Experimental Evaluation:*\nThe experimental results are promising and demonstrate improvements over baseline methods. However, the scope of the evaluation is somewhat limited:\n(a) It is confined to two tasks (bladder and lung cancer), both using what appears to be a very similar setup (e.g., the same set of classes for multi-label classification).\n(b) Some experimental details are unclear; for example, how the locally collected data are integrated with publicly available datasets.\n(c) Relying solely on AUC as the evaluation metric reduces the clinical relevance and impact, as it does not capture all aspects of model performance in comparison to other methods.\n\n*Additional Comments:*\n* Line 12: Define \"TCGA\" at first mention.\n* Line 13: Define \"H-Optimus-0\" at first mention.\n* Lines 21–24: Please indicate the resolution of a whole-slide image (WSI), in addition to the individual pixel size.\n* Line 39: The term \"dual-loss mechanism\" is unclear and not elaborated on later in the paper. Please clarify.\n* Section 2 (especially 2.1): The section is dense with abbreviations, which makes it difficult to follow. Consider simplifying or expanding key terms.\n* Line 91: What is a \"learnable CLS token\"? Please provide an explanation or reference.\n* Section 3.2: Please clarify which components (e.g., Equation 1) are based on existing methods and which represent novel contributions.\n* Equation 5: Should the notation be $g_{z_i}$ instead?\n* Equations 5 and 6: Why are inconsistent notations ($\\times$ and $\\cdot$) used? Is there a distinction that should be clarified?\n* Lines 142–144: Please justify the claim made in this sentence; it currently lacks sufficient support.\n* Proposition 1: The statement and proof are vague; please see comments above regarding the need for a clearer and more rigorous justification.\n* Line 178: Define \"EGFR, KRAS, MET, and ALK\" and provide appropriate references.\n* Line 189: What is \"Otsu thresholding\"? A brief explanation or citation would help readers unfamiliar with the term.\n* Line 197: Shouldn't the number of tiles used during inference match the number used during training? \n* Section 4.2: How does the training/inference runtime compare to that of other (non-decoupled) methods? \n* Line 244: Since the loss depends on $\\lambda$, does a lower numerical loss value actually reflect better performance? \n* Table 2: Why is there no comparison to methods without TAPFM?", "was_revised": true, "final_justification": "I’m leaning toward acceptance, though I don’t feel strongly either way. The paper reads well, but it’s not particularly clear on the advantages of the proposed method (aside from the 24×H100 vs. 1×H100 comparison mentioned only in the rebuttal), why it performs well, or shows results that are especially strong. The authors have addressed my comments and questions, but I wasn’t convinced to raise my score.", "rebuttal": "**Note:** For brevity PMID refers to the PubMed identifier of the papers referenced in the rebuttal. The non-PMID-indexed reference is fully cited. Original reviewer comments are truncated to allow for complete responses.\n\n### **Reviewer Comment 1**  \n*The technical description in Section 3 lacks clarity. … incompatible with single-GPU training.*\n\n**Author Response:**  \nWe would like to clarify a technical distinction: TAPFM employs detached gradients to strategically direct gradient flow, not to disable it. This is a deliberate design choice to address the instability that arises when fine-tuning pathology foundation models (PFMs) in the MIL setting for gigapixel pathology images. The detachment breaks the computation graph between the PFM and the aggregator module, allowing proper gradient flow during backpropagation (Equations 8–10). This design makes training feasible on a single GPU, where memory constraints prohibit full end-to-end joint optimization of large PFMs with transformer backbones under a MIL framework.\n\nTAPFM also uniquely uses self-attention scores from the ViT backbone to compute slide-level MIL aggregation. This internal use of ViT attention for MIL has not been described before (see lines 95–97).\n\nConventional approaches freeze the PFM and train only the MIL aggregator, resulting in suboptimal performance (see references 11–13 and Table 1). Campanella et al. (*Nature Medicine*, July 2025, PMID: 40634781) demonstrated that adapting foundation models to mutation prediction substantially improves performance, achieving AUCs of 0.847 (institutional) and 0.860 (TCGA) for *EGFR* prediction using distributed training on 24 H100 GPUs. However, this distributed strategy is unavailable for single-GPU systems where computations cannot be parallelized across devices.\n\nAn alternative is encoder-aggregator training in a unified computational graph on a single GPU, but this suffers from optimization conflicts. In the MIL framework, the bag representation $Z =$ **Z**$^T$**a** requires the PFM to simultaneously optimize feature extraction (**Z**) and attention weighting (**a**) through shared parameters $θ_{PFM}$. Gradients from both pathways ($∂Z/∂$**Z** · $∂$**Z**$/∂θ_{PFM}$ and $∂Z/∂$**a** · $∂$**a**$/∂θ_{PFM}$) create conflicting optimization signals. TAPFM's detached-gradient formulation decouples this: aggregator learns from feature / attention combinations using fixed PFM outputs, then PFM adapts without competing aggregator gradients.\n\nThis enables fine-tuning PFMs on a single GPU without compromising performance: TAPFM achieves AUCs of 0.867 (institutional) and 0.863 (TCGA), comparable to the 24-GPU baseline. We can include training loss curves comparing unified vs. detached optimization to empirically demonstrate this effect, and will reference Campanella et al.’s work in the final manuscript to provide context for our single-GPU results.\n\n\n### **Reviewer Comment 2**  \n*Evaluation scope is limited:*\n\n>*(a) Confined to two tasks...*\n\n>*(b) Some details are unclear...*\n\n>*(c) Sole reliance on AUC...*\n\n**Author Response:**  \nWe appreciate the reviewer's thoughtful feedback and address each concern below.\n\n**(a) Evaluation Scope**  \nTAPFM's evaluation demonstrates its effectiveness across clinically actionable biomarkers with direct therapeutic implications, as detailed in Section 5. *FGFR3* mutations in bladder cancer guide treatment with erdafitinib. In LUAD, *EGFR*, *ALK*, *KRAS*, and *MET* mutations inform treatment with personalized therapies (e.g. sotorasib and tyrosine kinase inhibitors).  \n\nThe multi-label framework addresses a fundamental clinical need that conventional classification cannot. As shown in Table 2, TAPFM enables simultaneous prediction of co-occurring mutations. While binary classification handles one mutation at a time and multi-class frameworks assume mutual exclusivity, multi-label classification models concurrent mutations, a reality in LUAD with coexisting *EGFR*, *KRAS*, *MET*, and *ALK* alterations (please see response to cPrU for more details).\n\n**(b) Data Integration Methodology**  \nNo complex integration is required. Both institutional and TCGA datasets contain WSIs matched with mutation labels derived from molecular testing. Variations in slide preparation and scanning are expected and useful for robust evaluation.\n\n**(c) Evaluation Metrics**  \nWe will include confidence intervals from 5-fold cross validation (CV) using institutional data and TCGA as independent test sets. We will also report AUPRC values to support performance interpretation in appendix. We avoid threshold-based metrics (e.g., F1, accuracy) due to their sensitivity to cutoffs, which limits their clinical interpretability.\n\n### **Reviewer Comment 3**  \n*TCGA*\n\n**Author Response:**  \nWe will define TCGA as *The Cancer Genome Atlas* at first mention.\n\n---\n\n### **Reviewer Comment 4**  \n*\"H-Optimus-0\"*\n\n**Author Response:**  \nWe use this as a model name (ref 9).\n\n---\n\n### **Reviewer Comment 5**  \n*resolution of a whole-slide image*\n\n**Author Response:**  \nWe will replace “magnification” with “resolution” in line 21 for clarity.\n\n---\n\n### **Reviewer Comment 6**  \n*\"dual-loss mechanism\"*\n\n**Author Response:**  \nWe will clarify that the term refers to the use of two loss function: classification (Equation 4) and task adaptation (Equation 7).\n\n---\n\n### **Reviewer Comment 7**  \n*Section 2 (especially 2.1) ... Consider simplifying or expanding key terms.*\n\n**Author Response:**  \nThe names of the algorithms, not abbreviations, as named in the original cited papers are mentioned in Section 2.\n\n---\n\n### **Reviewer Comment 8**  \n*Line 91: \"learnable CLS token\"*\n\n**Author Response:**  \nWe refer to the CLS token from ViT models and have cited the original ViT manuscript in *line 89*.\n\n---\n\n### **Reviewer Comment 9**  \n*Section 3.2: Please clarify ... which represent novel contributions.*\n\n**Author Response:**  \nThe following are novel contributions:\n\na. MIL aggregation using internal attention weights (averaged across heads and tokens) for ViT-based PFMs (lines 79–81, 96–97).  \nb. Separate aggregator and encoder optimization in distinct computation graphs on a single GPU (lines 103–104, 115–116).  \nc. Scaling of attention weights for use in MIL aggregation (lines 104–109).  \nd. Task adaptation loss combining novel feature alignment and attention loss components (Section 3.3).  \ne. Theoretical analysis of TAPFM and its properties (Section 3.4, Appendix A.1–A.4).\n\n---\n\n### **Reviewer Comment 10**  \n*Equation 5: Should the notation be $g_{z_{i}}$ instead?*\n\n**Author Response:**  \nYes, we will correct this.\n\n---\n\n### **Reviewer Comment 11**  \n*Equations 5 and 6: Why are inconsistent notations ( x and .) used? ...*\n\n**Author Response:**  \nBoth are scalar multiplications. We will use “x” consistently.\n\n### **Reviewer Comment 12**  \n*Lines 142–144: Please justify the claim made in this sentence; it currently lacks sufficient support.*\n\n**Author Response:**  \nPlease see response to comment 1.\n\n### **Reviewer Comment 13**  \n*Define \"EGFR, KRAS, MET, and ALK\" ...*\n\n**Author Response:**  \nThese are names of common proto-oncogenes that are extensively studied in clinical oncology and molecular pathology. We have provided appropriate clinical references in Section 5 of the manuscript. We will ensure all gene names are italicized to indicate that they refer to genes. The names no longer reflect their original derivations; for example, *KRAS* was originally derived from “Kirsten Rat Sarcoma Viral Oncogene Homolog” but is now formally named “KRAS Proto-Oncogene, GTPase”.\n\n### **Reviewer Comment 14**  \n*\"Otsu thresholding\"*\n\n**Author Response:**  \nWe will include the following reference:\n\n> **Otsu, Nobuyuki.**  \n> *A Threshold Selection Method from Gray-Level Histograms.*  \n> IEEE Transactions on Systems, Man, and Cybernetics, Jan. 1979. \n\n### **Reviewer Comment 15**  \n*... tiles used during inference match the number used during training?*\n\n**Author Response:**  \nTraining is memory intensive as it involves gradient tracking through massive PFM and MIL aggregator parameters, limiting the number of tiles per slide on a single GPU. Inference, however does not have this constraint (see response to comment 7 for reviewer ggKh). Slide-level predictions require that all tiles be evaluated. \n\n### **Reviewer Comment 16**  \n*How does the training/inference runtime compare to that of other (non-decoupled) methods?*\n\n**Author Response:**  \nTo the best of our knowledge, there are no decoupled methods for PFM task adaptation for ViT-based PFMs in the MIL setting. Contemporary approaches rely on distributed training across multiple GPUs (lines 77–81), with Campanella et al. (*Nature Medicine*, July 2025, PMID: 40634781) using 24 H100 GPUs.  \n\nWe identified calculation errors in the training times reported in Section 4.2 and will update them in the revised manuscript. The correct TAPFM training durations are: BLCA – 12h (UNI), 21h (Gigapath), 24h (H-Optimus-0); LUAD – 2d 4h (UNI), 4d 2h (Gigapath), 4d 6h (H-Optimus-0). Average inference time per WSI is 4.85–7.15 minutes (lines 225–226).\n\n### **Reviewer Comment 16**  \n*... lambda, does a lower numerical loss value actually reflect better performance?*\n\n**Author Response:**  \nThe Task Adaptation Loss (TAL, Equation 7) is designed as a surrogate objective that encourages feature alignment with the classification task. Lower TAL values may correlate with better performance by design and as supported by the ablation results in Figure 2b and Section 4.4. \n\n### **Reviewer Comment 17**  \n*Table 2: Why is there no comparison to methods without TAPFM?*\n\n**Author Response:**  \nPlease see response to Comment 4 for reviewer cPrU.\n\n### **Reviewer Comment 18**  \n*Include additional evaluation metrics beyond AUC for BLCA and LUAD... Metrics like accuracy, precision, recall, F1, or calibration could show robustness and clinical relevance.*\n\n**Author Response:**  \nSee response to comment 2(c)."}]

Author rebuttal:

Meta-review: After rebuttal all reviewers agreed that the technical contributions were significant and novel enough (esp. the combination of foundational models and MIL) even though the gains in performance were on the modest side.
Decision: Accept (poster)