Revisiting End-to-End Learning with Slide-level Supervision in Computational Pathology
Authors: Wenhao Tang
Rong Qin
Heng Fang
Fengtao Zhou
Hao Chen
Xiang Li
Ming-Ming Cheng
Venue: NeurIPS 2025 poster ()
Keywords: Computational Pathology
Multiple Instance Learning
End-to-End Learning
Primary area: machine_learning_for_sciences

Abstract: Pre-trained encoders for offline feature extraction followed by multiple instance learning (MIL) aggregators have become the dominant paradigm in computational pathology (CPath), benefiting cancer diagnosis and prognosis.
However, performance limitations arise from the absence of encoder fine-tuning for downstream tasks and disjoint optimization with MIL. While slide-level supervised end-to-end (E2E) learning 
is an intuitive solution to this issue, it faces challenges such as high computational demands and suboptimal results.
These limitations motivate us to revisit E2E learning. 
We argue that prior work neglects inherent E2E optimization challenges, leading to performance disparities compared to traditional two-stage methods.
In this paper, we pioneer the elucidation of optimization challenge caused by sparse-attention MIL and propose a novel MIL called ABMILX. 
ABMILX mitigates this problem through global correlation-based attention refinement and multi-head mechanisms.
With the efficient multi-scale random patch sampling strategy, an E2E trained ResNet with ABMILX surpasses SOTA foundation models under the two-stage paradigm across multiple challenging benchmarks,
while remaining computationally efficient ($<$ 10 RTX3090 GPU hours).
We demonstrate the potential of E2E learning in CPath and 
calls for greater research focus in this area.
The code is https://github.com/DearCaat/E2E-WSI-ABMILX.

Revisiting End-to-End Learning with Slide-level Supervision in Computational Pathology
Wenhao Tang" 1.2 1.2
Rong Qin +2
Heng Fang
Fengtao Zhou
Hao Chen
Xiang LifL2
Ming-Ming Cheng
Nankai International Advanced Research Institute (Shenzhen Futian)
VCIP, School of Computer Science, Nankai University
HHuzzong University of Science and Technology
4 'The Hong Kong University of Science and Technology
"Equal Contribution.
*Corresponding author.
Abstract
Pre-trained encoders for offline feature extraction followed by multiple instance learning (MIL) aggregators have become the dominant paradigm in computational pathology (CPath), benefiting cancer diagnosis and prognosis. However, perfor- mance limitations arise from the absence of encoder fine-tuning for downstream tasks and disjoint optimization with MIL. While slide-level supervised end-to-end (E2E) learning is an intuitive solution to this issue, it faces challenges such as high computational demands and suboptimal results. These limitations motivate us revisit E2E learning. We argue that prior work neglects inherent E2E optimization challenges, leading to performance disparities compared to traditional two-stage methods. In this paper, we pioneer the elucidation of optimization challenge caused by sparse-attention MIL and propose a novel MIL called ABMILX. ABMILX mitigates this problem through global correlation-base attention refinement and multi-head mechanisms. With the efficient multi-scale random patch sampling strategy, an F2E trained ResNet with ABMIIXX surpasses SOTA foundation mod- els under the two-stage paradigm across multiple challenging benchmarks, while remaining computationally efficient (< 10 RTX3090 GPU hours). We demonstrate the potential E2E learning in CPath and calls for greater research focus in this area. The code is here.
1 Introduction
Computational pathology [15, 53, 14] (CPath) is an interdisciplinary field that combines pathology, gigapixel image analysis, and computer science to develop computational methods for analyzing and interpreting pathological images (whole slide images, WSIs or slides). This field leverages advanced algorithms, machine learning, and artificial intelligence techniques to assist pathologists in tasks such as cancer sub-typing [25, 66), grading [6], and prognosis [63, 65]. Due to clinical demands and the challenge of pixel-level annotation in gigapixel pathological images, CPath typically focuses slide-level learning. However, analyzing such gigapixel images in slide-level presents significant challenges in terms of efficiency and performance.
To address these challenges, Campanella al. [7] proposed a two-stage paradigm based on multiple instance learning (MIL) [41], allowing efficient WSI analysis without fine-grained annotations. This
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

PANDA(Graddon)
Paraseter(())
NSCLC
BLCA (Survival)
(Subiyping)
20
Epoch Time
SOTA FMs
00-35
M0,35
77: :000
38
307
10000
Low AUC Goin
High Cost
Attention Sampling
26
LUBC
1900
BRCA (Subtyping)
(Survival)
52.00
Portrained Data(M)
TransMIL
ABMILX
100
Randon Sampling
1900
ABMIL
High AUG Gain
64.00
100
LOM Cost
GIGAPATH
15
128-ttenlart
85
87
89
91
93
95
97
BRCA(SurVival) (UMD(Survival)
500
2000
1500
AUC (8)
(a) Compare end-to-end trained ResNet with various foundation models
(b) Paraneter and
(c) Performance of different strategies in CPath EZE
Pretrained Data
Figure (a We compare E2E trained ResNet with various foundation models using two-stage paradigm in terms of performance, model size, and pretraining data. This demonstrates the perfor- mance potential of E2E learning for computational pathology under low computational budget. (c)
Compared to sampling strategies, different MILs have more significant impact and lower cost on E2E learning.
approach first divides each WSI (a bag) into thousands of image patches (instances). Pretrained encoders extract offline instance features, which are then aggregated into bag features by a sparse- attention MIL model, ultimately leading slide prediction. By operating in the latent space rather than images, this paradigm enables slide-level supervised training within reasonable memory constraints. However, its performance heavily depends on the quality of offline features [10]. To improve offline feature quality, a series of pathology foundation models [11, 62, 24, 64] (FMs) like UNI [11] and GigaPath [64] have been developed. As shown in Figure 1, despite scaling data volume to 170K WSIs (>200TB) and model size over IB, these approaches still perform unsatisfactorily on specific tasks. We attribute this to the lack of unified optimization in the two-stage paradigm, resulting in encoders with insufficient adaptation of downstream task and disjoint optimization with MIL models.
End-to-end supervised learning with joint encoder and MIL at the slide level (E2E learning) offers a fundamental solution, enabling efficient downstream data utilization and task-specific encoder learning. However, due prohibitive computational costs and suboptimal performance, this area remains underexplored. Existing works [50, 9, 61] typically employ patch sampling to maintain a reasonable computational budget, focusing on improving sampling quality enhance performance. However, previous work overlooked the optimization challenges infroduced by MIL in E2E learning, resulting limited performance improvements. The results in Figure 1(c) show that complex sampling strategies incur significant time costs with minimal performance gains, And different MILs significantly impact E2E training. Specifically, E2E learning with aparse-attention MIL performs poorly, falling below SOTA MIL methods using offline features extracted by ResNet-50 (R50) and significantly underperforming SOTA FMs. As shown in Figure 2, sparse attention is crucial for CPath, enabling models to focus on key regions from thousands of patches and performs increasingly well with superior features. However, we suggest that it can also disrupt the encoder E2E learning due to its insufficient consideration of discriminative regions and potential extreme focus redundant ones. Poor features further affect the accuracy of attention the next iteration, leading to deteriorating iterations and compromising the entire optimization process.
To retain the benefits of sparse attention while mitigating its induced optimization challenges in E2E learning, we propose ABMILX, a novel MIL model based on the widely used ABMIL [25]. ABMILX incorporates multi-head attention mechanism to capture diverse local attention from different feature subspaces, and introduces global attention plus module that leverages patch correlations to refine local attention. Both modules help the encoder learn more discriminative regions and avoid excessive focus on redundant areas. Furthermore, we adopt simple but effective multi-scale random patch sampling to incorporate multi-scale information while reducing E2E learning computational costs. Our E2E learning framework achieves significant performance improvements (e.g.. +20% accuracy on PANDA) while maintaining computationally efficient (< 10 RTX3090 GPU hours on TCGA- BRCA). The main contributions can be summarized as follows:

---

Oiscrininative Patches
Redendant Patches
Farly Trašndng
Early Training
Cffikee Enceder
Unstuble Encoder
D6000
Worse
00000
Horse Feetures
Worse
Bether
Q0000
Pestures
Bether
Extrene: Sporse MEL
2
2
Eatrome
Sparee
Sparee
FC
FC
Converged
Converged
Baclvord
Loss
(o) MELL benufit Chuth Teak Le Two Tho-Stage Parudiign
(8) ES-NIL excountered more screrv ESE optinizution prosium Cheth
(c) ARMELE shows better adaptabbiiity Chath
Figure 2: In E2E learning. MIL can be viewed soft instance selector that iteratively optimizes with the encoder. The encoder outputs instance features to MIL for attention-based aggregation and receives the instance gradients for optimization. The attention from MIL affects the gradients of different instance features, leading to selective learning of patches by the encoder. In contrast to two-stage learning approaches, the commonly used excessively sparse attention makes the encoder optimization overfitted on limited discriminative regions and vulnerable to redundant ones. Worse features further affect the accuracy of selection, compromising the optimization loop.
We revisit slide-level supervised E2E learning for CPath and pioneer the identification of optimization challenges. We show that E2E learning with slide-level supervision and its optimization collapse risks from the sparse attention MII. deserve more attention.
To address E2E learning optimization challenges while maintaining sparse attention, we propose the ABMILX model. By incorporating multi-head attention mechanisms and global correlation based attention plus modules, it significantly improves performance.
We propose a slide-level supervised E2E learning pipeline based on multi-scale random patch sampling. It keeps a reasonable computational budget and introduces multi-scale information. Within this pipeline, an E2E trained ResNet with ABMILX surpasses the SOTA FMs under two-stage frameworks across multiple challenging benchmarks. This pioneerly demonstrates the potential of E2E learning in CPath.
Related Works
Computational Pathology. The advent of WSI in computational pathology (CPath) has revolution- ized approaches to cancer diagnosis and prognosis by furnishing a comprehensive, high-resolution view of tissue specimens [15, 53, 14]. Due processing gigapixel images is computationally in- tensive, traditional CPath methods have adopted a two-stage paradigm to prioritize efficiency [38]. In the first stage, an offline feature extractor. typically pre-trained in general datasets [38, 21] or pathology datasets [11, 62, 24, 64], is employed to encode tissue patches into features. In the subsequent stage, MIL are applied to aggregate these features for slide-level prediction. Most re- search [26, 31, 51, 55, 58, 33, 54, 68, 32, 57] has focused on refining MIL stage with advanced aggregation mechanisms, notably the use of sparse attention. MIL model computes attention scores for each patch and aggregates only the most informative ones [25], thereby reducing noise and en- hancing slide-level prediction accuracy WSIs with scattered key histological features. Some studies have sought to better exploit information contained in WSIs by directly extracting supplementary visual cues from the entire slide [52, 19]. Others have refined the extracted features to better match the dataset through either multi-stage feature extractor fine-tuning [30] or online instance feature re-embedding [56] to more precisely tailor the extracted features to the dataset. However, current two-stage approaches rely pretrained offline feature extractors that are not jointly optimized with the MIL model, potentially resulting in features that inadequately capture the complex nuances of pathology data in WSIs [30]. In this context, E2E approaches have emerged
promising paradigm.
E2E Learning in Computational Pathology. E2E learning, which jointly optimizes feature extrac- tion and prediction from WSIs in CPath, offers a more adaptive encoder that enhances the discrim- inability of representations. However, high computational costs and unsatisfactory performance have hindered systematic research in this area, with existing E2E CPath methods falling into instance-level and slide-level supervised approaches. Instance-level supervised methods [13, 39, 46, 47] simplify processing by training encoders with pseudo-labels for individual patches rather than slide labels.

---

Gigapixel Whole Slide Images
1824 X. 1824 patches
Gpath Tasks
Segsentation
Multi-Scalee
Online Feature Extractor
Sub- Sub-typing
ABMILX
Gradinng
Patching
Randon Sampling
Survival Analysis
256 x 250 patches
ABMIL:
Global Concelation
ABMILX: (Ours)
Global Cornekation
(Previous work)
Sparse
Exhrame Sparse
Features
Global Attention Plus
10
Bag Features
Refined
Global
Multi-head
Sofftrax
Feature
Attention
Connelation
Local Attention
Features
Concat
MatMull
Mulei-head Features
Softmax
Multi-
Instance
Feature
Attention
Instance
Bag Feature
Local Attention
Figure Overview of the proposed E2E training pipeline and ABMILX. ABMILX introduces multi-head local attention to address the extreme sparsity issue in ABMIL [25], which hinders E2E optimization. Furthermore, ABMILX refines the local attention using global feature correlations via the attention plus. This encourages the model to focus on task-specific regions during E2E learning.
They neglect the crucial inter- aiter-patch context required for clinical analysis [7, 40, 23], making them compromise rather than fully end-to-end solution. In contrast, slide-level supervised methods analyze entire slide preserve contextual interdependencies and deliver comprehensive, clini- cally relevant analysis. It is broadly classified into two main categories. The first group leverages memory-efficient architectures and model-parallel techniques to enable E2E learning on gigapixel images [43, 8, 17, 59, 34, 60), yet it still demands substantial computational resources (e.g.. 64 V100 GPUs [60]) and fails to deliver satisfactory outcomes [42]. Alternatively, another group alleviates the efficiency challenges with data sampling to train selected subsets [50, 9, 61]. This approach [50] and iitention-based [9, 61, 12] methods, to identify key regions from slide. These complex sampling also give rise to iterative [50, 9] and multi-stage [30] pipelines. In summary, although current E2E methods are complex and com- putationally intensive, they still underperform
focuses on diverse sampling mechanisms such as clustering
two-stage methods. This performance gap can be attributed to overlooked optimization challenges caused by MIL. In this paper, we pioneer the proposition that the optimization challenges the performance bottleneck of E2E methods.
3 Methodology
3.1 Slide-level Supervised End-to-End Learning
The lack of encoder adaptation in the two-stage paradigm limits the feature specificity on CPath tasks, thereby calling for slide-level supervised E2E training to jointly optimize the MIL. model and the encoder. The upper of Figure shows the overall E2E learning pipeline of our method, which consists of multi-scale random instance sampling, instance feature encoder, ABMILX, and task head. Specifically, given the target number of sampled instance s and slide X, instance subset could be collected through our multi-scale random instance sampling strategy V(-) as encoder input to avoid massive training cost, L ((s, X). The i-th instance is embedded into an instance feature e, E Rº by encoder, e; = Fo (L.) F()) denotes the mapping functions of any encoder and the denotes the corresponding learnable parameters. After that, the features all sampled instances E == (ei,... , ,,, e.} will be aggregated through our proposed ABMILX, Z To (E)E). The T(.) denotes the mapping functions of ABMILX and the ø denotes the corresponding learnable parameters. Then slide features Z are inputted into task head H., for the slide-level prediction

---

ÿ (2). Finally, we only utilize the slide-level ground truth y and the ÿ to joint optimize the aforementioned modules through task loss function L:
(0, 20 (i) + arg min 6,0.0
Clye, 9s),
where denotes the number of slides in train set, while 0, o. and ij are the final parameters of encoder, MIL, and task head, respectively. Considering that E2E learning allows the attention from MIL to affect the instance gradients backpropagatd to encoder, the key insight of our method is to guide the encoder to learn task-specific discriminative regions through our proposed ABMILX.
Multi-scale Random Instance Sampling. The sampling stage aims to take a subset from massive instances for training, thereby reducing the cost of E2E learning. The sampling methods generally fall into random and selective sampling [50, 9, 61, 27]. The latter focuses on traversing the slide to obtain high-value instance samples. which significantly increases training time and heavily relies on the evaluation model [49, 30, 12J. In this paper, we introduce a multi-scale random instance sampling (MRIS) method to maintain low training cost while leveraging multi-scale instances to capture information at different granularities. Specifically, given a multi--cale set {II." -i.... .r). we adopt sampling ratio set (01, ,.J'' ,0 obtain the number 83 of ig for sampling:
ł, Vs Vs([j, Sj, X), 83 [s aj}:
(2)
where V-(() denotes the function of vanilla random sampling. is notable that we set Σj= We resize the sampled instances of different scales and context extents {É..... Ly.... Le} to a unified resolution and merge them as the final sampling set L. On the one hand, multi-scall sampling simulates the multi-scale perspective of pathologists during diagnosis and improves the CPath performance of our method. On the other hand, the unified resolution for different context avoids the additional cost and maintains parallel training. while remaining the different scale perspectives of original instance. Appendix C.4 give more details about sampling.
to ensure that Σ}= (8,)
3.2 ABMILX for Effective End-to-End Learning
Sparse-attention MIL. that relies on local instance features, such the most representative AB- MII. [25], could avoid key regions being overwhelmed by redundant instances and performs increas- ingly well with superior features. However, demonstrate that the sparse attention will introduce interference risks in E2E learning and bring suboptimal performance. The risks primarily stem from the insufficient consideration of discriminative instances and excessive focus on redundant ones. In this paper, we propose ABMILX, which consists of a multi-head local attention and global attention plus module to mitigate the optimization risks from both local and global perspectives. also maintains the sparse characteristic to effectively collaborate with the fine-tuned encoder.
Multi-head Local Attention. Considering that the false attention from under-converged MILL usually exhibit a random distribution, we propose a multi-head local attention module (MHLA) to directly suppress the excessive focus on redundant instances while improving the attention for the discriminative ones. Specifically, we divide the features of all sampled instances E into m head features { ... .i..... Hmm), where H E Rxx(D/m) Within each head, the head features are input into shared MLP compute the corresponding attention, A MLP (H). The A R denotes the local attention vector of the j-th head, which possesses sparse characteristic important in CPath tasks. In the E2E learning, the separate voting from multiple heads allows reduce the excessive focus on redundant instances, while the attention from different feature subspaces helps to provide more comprehensive view discriminative instances. Finally, aggregate the features within each head through A to obtain the head-level slide features (Z Z.... ,."). are then concatenated the final slide feature
zm),, which
Z E GRRM Ix Concat(Z*
Z
Z' = Softmax (T(AA))TH')
(3)
where ((-) denotes the mapping function of our global attention plus module. It aims at further refining A through propagating sparse attentions from discriminative instances to their similar instances for better feature aggregation and optimization. Compared to directly averaging the head attention and aggregating the whole instance features, head-level aggregation enables MII. obtain more diverse representations from different feature subspaces.
5

---

Global Attention Plus Module. Tissues with similar pathological characteristics typically exhibit highly similar morphology, leading higher correlation among corresponding instance features. Therefore, besides directly enhancing attention from the local instance perspective, we propose an global attention plus module (A+) to leverage the global correlations for attention refinement, which could indirectly improve the focus for the discriminative instances while suppressing the redundant ones. propagates A' between similar instances to obtain a the global sparse attention and then combines it with A', thereby correcting the local sparse attention from MHLA. When integrating the MHLA with the A+ module, we first share A+ module across different heads to obtain the refined -tte-tttntioon by computing similarity matrix US, respectively, and then perform feature aggregation within each head for the refined head-level slide features as mentioned in Eq 3:
Q K VDD /
((A')= = A + A = A' + a Softmax
A
(4)
where Q: H K H The W E [RDD/m]x[D///// and W E [[D/m]x[D'// are both the linear transforms. To preserve the sparsity, we introduce shorteut branch with a learnable scaling factor a that adaptively combines global sparse attention U A and the original local one.
The propagation weight of the i-th instance, denoted as P(i). is defined as the sum of its influence on all instances. In classic transformer- based methods, the propagation weights Pirans determined by only the similarity matrix Us, However, for the global spurse atlention introduced in ABMILX, the weights Pata is also significantly affected by the original sparse head attention value A
Perans (i) = Σ Uh A Pater (1) A ΣU
U
(5)
Therefore, ABMILX utilizes the A' as prior distribution to grant sparse discriminative instances with higher propagation weights to find more potential instances while suppressing the normal ones. More theoretical analysis about ABMILX is available in Appendix A.
3.3 Sparse Attention Analysis in E2E Learning
To intuitively analyze the effect sparse atten- tion on E2E training, we quantitative the spar- sity of different MILs by the proportion ac- tivated patches. Sparsity is statistically derived from the CAMELYON dataset [3]. Moreover, the right figure visualizes attention scores (bot- tom) and corresponding distribution (middle) of MILS during training. We demonstrate the fol- lowing: (1) In E2E optimization, extreme spar- sity causes ABMIL to overlook discriminative regions while over-foc on redundant ones, leading to worst performance. (2) Although the global attention TransMIL eliminates this ex- treme sparsity and covers some discriminative regions, it is also largely distracted by the redun- dant ones, which also brings limited accuracy gains. (3) In contrast, both MHIAA and A+ make ABMILX maintains reasonably sparse attention, which considers most of the discriminative re- gions while maintaining low attention to nor- mal patches. Benefited from them, ABMILX achieves the best performance different C- Path tasks. Besides, learnable also helps adap- tively adjusting the sparsity and brings more accuracy gains. More experiments and analysis about the affect of different MILs E2E learn- ing available in Sec. 4.3 and Appendix C.2.
Different MILs in E2E
Sparsity
Sub.1
Surv.
ABMIL
80
89.23
62.70
TransMIL.
13
91.44
63.42
MHLA (a
61
91.58
63.80
MHLA&A+ (a
29
92.84
65.49
MHLA&A+ (learnable a))
36
93.97
67.78
Attention Distribution
ABMIL ABMILX
TransMIL
Arterdion Score
1003
2006
3000
4260
5000
Sortad Patcc
Tumor Area
High-Attention
Patch
ABMIL
ABMILX
TransMIL

---

Table 1: Sub-typing results main datasets and training cost of different CPath methods.
Encoder
Method
Pretraining Data
#Parameter
FLOPs
TCGA-BRCA
TCGA-NSCLC
ABMIL [25]
26M+0.66M
83.80+
92.32++2.8
CLAM [38]
26M+0.79M
85 83.86119
92.28
TranMILL [48]
ImageNet- 1K [16]
26M+2.67M
28.52 44
92.49.2..66
ResNet-S0
DSMIL [29]
26M+0.87M
~2.12T
85.68
91.12+3.04
1M Data
WIKG [33]
26M+1.71M
88 +-5.80
92.57-+2.58
RRTMIL [56]
26M+2.70M
89.35
94,43.
2DMamba [67]
26M+2.27M
87. 5.20
95.21 +2.00
ABMIL [25]
27M+0.66M
01.09944.77
96222+1.07
TrasMILL [+8]]
27M+2.67M
91.41 + 3.96
96.39+11.78
CUIE [62]
CHIEF [62]
27M+1.05M
~2224T
91.43 1.4
96.84
RRTMIL [56]
307M-22.70M
92.49 4.
97.00.+1.41
2DMamba
27M+2:27M
91.88-1 +-111
96.93++1.00
ABMIL [25]
307M+0.66M
94.05 4: 4.440
97.04++1.00
UN]
TranMILL [48]
Mass- 100K [1]]
307M+22.7M
93.33 g.3.
97.27.11
RRTMIL [56]
~ 100M Data
307M--2.70M
94.61 3.1.8
97.88 1.18
2DMamba [67]
307M-2.27M
93.00 +-4224220
97. 97.14+41.48
ABMIL [25]
1134M+0.66M
94.39 3.3. 4B
96.54+11.66
TransILL [48]
Slide-170K
1134M+2.67M
93.97 1:3
97.61
GIGAP [64]
GIGAP [64]
1134M+86M
~114T
93.72 3.
97.53 1.10
~1133 Data
RRTMIL. [56]
1334M+..7MM
94.82 6.8
97.63++1.20
2DMamba [67]
1134M+2.27M
93.84 1.3
96.87 1.66
ResNet-50
Best stage
26M+2.
2.12T
45.45
4:2.07
ResNet-18
C2C [49]
ImageNet-IK [16]
12M+0.79M
0.93T
91.13 +1.78
95.92 +0.71
ResNet-50
FT [30]
26M+0.
2.12T
86.48
-2.87
94.67 -0.54
~ 1M Data
ResNet-18
ABMILX (ours)
12M+0.80M
0.93T
93.97
+4.62
97.09 +1.88
ResNet-50
ABMILX (ours)
16M+0.80M
2.12T
95.17 +5.82
97,06-1 -1.85
4 Experiment
4.1 Datasets and Evaluation Metrics
We use PANDA [6]. TCGA-BRCA, and TCGA-NSCLC to evaluate the performance in cancer grading and sub-typing tasks. For cancer prognosis, we use TCGA-LUAD, TCGA-BRCA. TCGA- BLCA to evaluate performance on the survival analysis task. For external validation, we use CPTAC-LUAD, CPTAC-LUSC
evaluate the generalization ability. For cancer grading. we evaluate model performance using top-1 accuracy (Acc.). And area under the ROC curve (AUC) used for sub-typing. For survival analysis, we employ the concordance index (C-index) [20]. To ensure robust statistical evaluation, we conducted 1000-time bootstrapping evaluation and report the mean and 95% confidence interval. Please refer to Appendix for more details.
4.2 Main Results
Comparison Methods. We compare several classical and latest MIL aggregators based on ResNet encoders [25, 38, 48, 29, 33, 56, 67]. Furthermore, we evaluate against three SOTA pathology FMs: UNI [11], CHIEF [62], and GigaPath (GIGAP) [64]. Following their settings, we employ ABMIL and TransMIL as aggregators. We also compare pre-trained aggregators of CHIEF and GIGAP. C2C [49] and FT [30] are E2E methods that adopt clustering- based and aitention-basee samplings, respectively.
Foundation Model Dominate Two-Stage but Cost More. Two-stage algorithms are limited by offline feature. Specifically, in grading task (Table 2), the best-performing MII. with the R50 shows 12% accuracy gap compared to UNI with ABMIL. This performance difference also observed in other tasks (Table 1), with gaps of 5% and 2% on BRCA-subiyping and BRCA-survival respectively, With FM features, the superior performance of ABMIL compared with advanced methods further highlights the importance of sparse attention CPath. However, these significant improvements come at considerable cost. Pretraining pathology FMs demands vast amounts of data, which are difficult to acquire and share publicly. For example. UNI uses 100 million patches from approximately 100,000 slides for pretraining, while publicly available datasets typically contain fewer than 1,000 slides. The resources required by large models (e.g., GIGAP uses 3,072 1000 GPU hours) are also huge. Furthermore, the performance of FMs does not scale proportionally with increasing data and model size. Specifically, the most expensive GIGAP lags behind UNI by 3% on the PANDA. Large FMs have not achieved the same impressive performance PANDA and BRCA as they did on
7

---

Table 2: Performance comparison across ISUP grading (PANDA) and survival analysis. OOM
denotes Out-of- Out-of-Memory in 24GB-3090.
Encoder
Method
B2B
PANDA (Aco:1)
LUAD
BRCA
BLCA
ABMIL [25]
58.89 -8.89-0.80
s9.56
64.93
58 55.01 1.9
CLAM [38]
59.45 29.18
59.79
62.9019.1
$5.78
TransMIL [48]
56.42_2..14
64.15
59. 19.15
56.96.9..
ResNet-50
DSMIL [29]
x
61.24-2.20
61.70
61 61.96499.6
56.22.+..2
WIKG [33]
62.72
00M
0OM
0OM
61.17
8..4
63.03
60.78
2DMamba [67]
61.56-2.38
61.41+7.0
61 61.94+8.9
54.98
ARMII. [25]
65.66-2.13
62.09
64.02+0.0
60.78
TransMLL [48]
60.89 60.89-2.29
65.55+a..
61.46+9.4
58 8..83+
[62]
CHIEF [62]
64.24-2.12
60.29
67.95
99.63
RRTMIL [56]
69.73
63.82+8.0
67.3 67.30+0.1
61.39+7.9
2DMamba [67]
72.49
60.57
64.30
59.8422..4
ABMIL |25|
74.69 74.69-2.31
59,65
67.05 10.2
57.29
TransMIL. [4S]
68.06_2..0
60.43.9.4
12.76
60.45
UNI [11]
RRTMIT. [56]
x
74.93
61.6448.5
66.91
61.077+8.3
2DMamba [67]
76.37 -2.07
ss.0
64.69
60.94.
ABMIL [25]
71.85 22.08
60.36+..6
63.81490.3
$9.85
TransMLL [4S]
65.45_9.04
60.401...
62.901n.2
60.12....5
GIGAP [64]
GIGAP [64]
X
65.86
62.99
62.6440.3
57.63-5.4
RRTMIL [56]
72.46-1.74
62 +
66.43+a.8
57.81
2DMamba [67]
75.72
64.49.7.0
65.35
57.58
ResNet-50
62.72_9.15
64.15
64.91...1
60.78
ResNet- 18
C2C [49]
62,91 +0.19
ResNet-50
[30]
+3.34
ResNet-18
ABMILX (ours)
78.34 +15.6
64.91 40.76
67 +2.85
61.20 10 40.42
ResNet-50
ABMILX (ours)
78.83 +16.1
64.72 +0.57
67.20 +2.27
60.78 +0.00
We suggest that the two-stage method based on FMs has saturated performance classical tasks and is bottlenecked by the lack of encoder adaptation in challenging tasks.
NSCLC.
ABMILX Shows E2E Potential. Through E2E learning with ABMILX and downstream data, we achieve FMs-level performance using ResNet models that were mr-itrained in ImageNet- kk. outperforms FMs multiple challenging datasets (+4% Acc. on PANDA, +0.8% AUC on BRCA). Moreover, the E2E learning cost of ABMILX substantially lower than the pretraining cost of FMs, approaching the cost of training second-stage aggregators, with more details provided in next section. Additionally, we show that fine-tuning upstream pre-trained aggregators, like CHIEF and GIGAP, did not yield the desired results. This further underscores the necessity of E2E training of encoders and aggregators with slide supervision. In particular, we demonstrate the scalability of the proposed method with respect to the model size. Except for survival analysis influenced by the sampling numbers (Table 2), R50 shows a general improvement compared to R18. Most critically, we validated the generalization ability through external validation the CPTAC dataset (Table 3). A ResNet-50 encoder trained on TCGA using our E2E framework not only shows superior generalization but also outperforms UNI, ViT-L pre-trained over one billion pathology patches. This result validates that our E2E learning approach fosters robust transferability that can overcome cross-dataset domain shifts, rivaling the benefits of massive-scale pre-training. In conclusion, empowered by ABMILX, present the impact and enormous potential of E2E learning in CPath. We also present more discussion Appendix C.I.
Table 3: Performance of different methods on external validation from TCGA to CPTAC datasets.
Encoder
Method
E2E
CPTAC-NSCLC (AUC) CPTAC-LUAD (C-indext)
ABMIL [25]
66.42
46.34
ResNet-50
TransMII. [48]
74.59
48.24
WIKG [33]
X
64.04
00M
ABMIL [25]
83.73
53,59
UNI [1I]
TransMIL [48]
X
85.24
51.36
WIKG [33]
X
83.56
0OM
ResNet-50
ABMILX (ours)
V
85.19
54.00

---

Table 4: Top: Comparison of computational cost. TTime (3090 GPU hours) denotes Train Time on BuCA-subyyping We add the features extraction time for two-stage methods. IT (s / slide) is the Inference Time. Memory (GB) is the GPU memory evaluated with batch size and fp16 during training. Bottom: Abalation of ABMILX in E2E training.
Encoder
Method
EZE
Pretrain Cost.
TTimel
Memory J
ITU
Grad.1
Sub."
Surv.
CHIEF [62]
ARMIL [25]
X
32GBx h
1+2h
2G
6.2s
65.66
91.09
64.02
UNI [1]]
TransMIL [48]
80GB x
1-7h
8G
68.06
93.33
60:45
GIGAP[6H [64]
GIGAP [69]
X
80GB x3.72hh
6+23h
7G
83s
65.86
93.72
62.64
ResNet-18
ABMILX- MRIS (ours)
9h
78.34
93.97
67
5OTA FM
75.46
77.56
78.34 78.83
SOTA FMs
91.58
93.97
95.47.
5OTA PMs
SOTA R58
89. 23
SOTA R58
633
62:78
61.84
65 36
60
64"
85
R19+ABATL
+Pultt -Head
<Attn. Plus
Freeze Eacoder
+250
RIBEL *Multi head +Attn. Plus
Fronze Encoder
+559
R38-
Multt feedd AAtin. Plus
Freeze Encoder
+850
(a) Survival
(b) Grading
(c) Sub typing
4.3 Ablation Study
In this subsection, we systematically investigate the impact of MIL in E2E training and ablate the ABMILX. Unless otherwise specified, all ablation experiments use ResNet-18 as the encoder. For the survival analysis task, we utilize the larger BRCA dataset. All efficiency experiments on the BuCA-sub benchmark. To evaluate model inference speed, we use an input size of 1x 10000x x224xx242 representing the average data volume processed in clinical scenarios.
conducted
MIL Matters in End-to- End Trainings. Right Table shows the impact of the sam- pling and aggregation modules on E2E learning. We observe that different MILs &
Method
TTimel
Grad.1
Sub.1
Surv.
Different MIL Models with MRIS
ABMIL [25]
9h
75.46
89.23
62.70
DSMIL [29]
9h
76.28
91.09
64.32
have a significant effect on E2E learning performance. Specifically, ABMIL ex- hibits unsatisfactory performance in E2E training. except the PANDA dataset. We attribute this to its excessive spar- sity hindering E2E optimization. The
TransMIL [48]
10h
75.08
91.44
63.42
RRTMII [56]
9h
17.99
61.82
53.42
ABMILX
9h
78.34
93.97
67.78
Different Sampling Strategies with ABMILX
Attention Sampling
68h
77.43
93.14
66.53
Random Sampling 9h
76.77
92.72
67.24
PANDA dataset contains fewer patches per slide (500 vs. 10,000 for TCGA-BRCA), enabling MIL to focus on discriminative regions more easily, thus suffering minimal impact on E2E optimization. RRTMIL exacerbates this problem, leading to optimization collapse. This complex MIL method, with a serial feature re-embedding module preceding ABMIL. makes E2E training more fragile. It further impairs the representation of features already affected by sparse attention, accelerating the collapse of the optimization loop. methods, partially mitigate this issue. However, relying
TransMIL and DSMIL, the transformer-bast solely on global attention struggles to focus on key regions within the numerous redundant patches in training. resulting in considerable performance gap compared to FMs. ABMILX, while maintaining desirable sparsity, alleviates optimization issues and achieves significant performance improvements. Furthermore, complex sampling strategies, such as attention-based sampling, offer only limited performance gains compared to vanilla random sampling. Such strategies require patch evaluation and incur substantial training time (TTime). Multi-scale random instance sampling (MRIS) shows better performance. Appendix C.2 provide further discussion.
Validity of Our ABMILX. Table 4 (bottom) ablates key components of ABMILX. E2E training with ABMIL performs poorly except for PANDA due to optimization challenges. It performs below SOTA MIL with R50 features and significantly underperforming FMs. After introducing multi-head mechanisms, the extreme focus on redundant instances caused by sparse attention effectively mitigated, thus achieving consistent improvements. More importantly, by refining attention using global patch correlations in the attention plus module, optimization issues are further alleviated. This improvement helps ABMILX achieve FMs-level performance. Furthermore, the sharp performance degradation when freezing the encoder demonstrates the necessity of E2F learning. We also validate ABMILX under two-stage paradigm in Appendix C.3.

---

Computational Cost Analysis. Table 4 (top) shows that the significant computational cost of FMs attributed to pre-training and inference. The resource consumption FM pre-training increases rapidly with model size. Large models also severely impact their clinical application, with FMs taking up to seconds to process a single slide, excluding data pre-processing. Although feature input reduces the cost of the second-stage training, increasingly complex aggregators continue to increase training time and memory consumption. In contrast, our E2E training pipeline maintains lower computational cost. Specifically, we do not require additional pre-training, and the overall training time and memory consumption are comparable to traditional second-stage feature-based training. Benefiting from the effectiveness of E2E learning, pipeline offers significant advantages for clinical applications. achieves competitive performance with only 1/50 of the inference time.
4.4 Qualitative Results
Feature Visualization. To validate that the performance gains of E2E training stem from task-specific encoder fine-tuning, we visualize instance features from the PANDA dataset using UMAP [22] in right figure. Features extracted offline by ResNet pre-trained ImageNet exhibit dispersed distribution the feature space, with poor separation between tumor and normal instances. Pre-training helps UNI pro- vide preliminary separation of instance types, but instances with the same annotations are not densely clustered. In contrast, after E2E learn- ing with our proposed ABMILX, the ResNet- extracted features demonstrate improved inter- class separability and intra-class compactness.
Ours E2E ResNet
Offline UNI
Offline ResNet
background
benign tissue
cancerous tissue
5
Conclusion
The lack of well-adapted offline features and disjointly optimized models has become performance bottleneck in CPath. While slide-level supervised E2E learning presents a fundamental solution, remains underexplored due to efficiency and performance challenges. Our work revisits slide-level supervised E2E learning in CPath from the MIL perspective. We demonstrate the impact sparse- attention MIL on E2E optimization. After addressing optimization challenges through the proposed ABMILX, we show that E2E-trained ResNet achieves comparable performance to foundation models with lower computational costs. We believe E2F learning has the potential to benefit upstream pre- nee-training and achieve further breakthroughs with increased computational resources. Revisiting the role of MIL in E2E learning may be key to realizing its potential.
Acknowledgements
This work supported by Shenzhen Science and Technology Program ( "Science and Technology Yongjiang 2035" key technology breakthrough plan project (2024Z120) 20), Chinese government-guiie guideed local science and technology development fund projects (scientific and technological achievement transfer and transformation projects) (25470102G), Supercomputing Center of Nankai University (NKSC).
References
[1] Jimmy Lci Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 1rx.v:1607.06450 2016.
arXiv preprint
[2] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen. Babak Ehteshami Bejnordi. Byungjae Lee, Kyunghyun Paeng. Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon challenge. IEEE transactions medical imaging, 38(2):550-56 560, 2018
10

---

[3] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer. Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22):2199 22000 2017.
[4] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast 2017.
cancer. Jama, 318((2):2199
[5] Benjamin Bergner: Christoph Lippert. and Aravindh Mahendran. Iterative patch selection for high- resolution image recognition. arXiv preprint arXiv:2210.13000 2022.
[6] Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David Steiner, Hester Van Boven, Robert Vink, al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine, 28(1):154-166 2022.
[7] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor. Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E. Reuter, David S Klimstra, and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8): 1301-1199,,, 2019.
[8] Gabriele Campanella, Eugene Fluder, Jennifer Zeng, Chad Vanderbilt, and Thomas J Fuchs. Beyond multiple instance learning: Full resolution all-in-memory end-to-end pathology slide modeling. arXiv preprint arXiv:2403.cm 240.040500088 2024.
[9] Lei Cao. Jie Wang. Yuanyuan Zhang, Zhiwei Rong, Meng Wang. Liuying Wang. Jianxin Ji. Youhui Qian. Liuchao Zhang, Hao Wu, et al. E2efp-mil: End-to-end and high- gegeraalllli weakly supervised deep convolutional network for lung cancer classification from whole slide image. Medical Image Analysis, 88: 102837, 2023.
[10] Richard Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144-16155. 2022.
[11] Richard J Chen, Tong Ding, Ming Y Lu. Drew FK Williamson, Guillaume Jaume, Andrew Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general purposs foundation model for computational pathology. Nature Medicine, 30(3):8508861 2024.
[12] Yuqi Chen, Juan Liu, Zhiqun Zuo, Peng Jiang, Yu Jin, and Guangsheng Wu. Classifying pathological images based on multi-instanes learning and end-to-end attention pooling. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[13] Philip Chikontwe, Meejeong Kim, Soo Jeong Nam, Heounjeong Go, and Sang Hyun Park. Multiple instance learning with center embeddings for histopathology classification. In MICCAI, pages 519-528. Springer, 2020.
[14] Didem Cifci, Gregory P Veldhuizen, Sebastian Foersch, and Jakob Nikolas Kather. Ai in computational pathology of cancer: Improving diagnostic workflows and clinical outcomes? Annual Review of Cancer Biology. 7:57-71, 2023.
[15] Miao Cui and David Y Zhang. Artificial intelligence and computational pathology. Laboratory Investiga- tion, 101(4):412 422- 2021.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[17] Stephan Dooper, Hans Pinckaers, Witali Aswolinskiy, Konnie Hebeda, Sofia Jarkman, Jeroen van der Laak, Geert Litjens, BIGPICTURE Consortium, et al. Gigapixel end-to-end training using streaming and attention. Medical Image Analysis, 88:102881, 2023.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint 2r11:22000.1199, 2020.
11

---

[19] Heng Fang, Sheng Huang, Wenhao Tang, Luwen Huangfu, and Bo Liu. Sam-mil: A spatial contextual aware multiple instance learning approach for whole slide image classification. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6083-6092, 2024.
[20] Frank E Harrell Jr. Kerry L Lec, and Danicl B Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, 15(4): 361-387 1996.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[22] John Healy and Leland Mclnnes. Uniform manifold approximation and projection. Nature Reviews Methods Primers, 4(1):82, 2024.
[23] Julius Hense, Mina Jamshidi Idaji, Oliver Eberle, Thomas Schnake, Jonas Dippel, Laure Ciernik, Oliver Buchstab, Andreas Mock, Frederick Klauschen, and Klaus-Rober Müller. Xmil: Insightful explanations for multiple instance learning in histopathology. Advances in Neural Information Processing Systems, 37:8300-8328 2025.
[24] Zhi Huang, Federico Bianchi. Mert Yuksekgonul, Thomas Montine, and James Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature Medicine, pages 1-10, 2023.
[25] Maximilian llse, Jakub Tomezak, and Max Welling. Attention- based deep multiple instance learning. In ICML, pages 2127-2136. PMLR, 2018.
[26] Guillaume Jaume, I Lukas Oldenburg, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Thomas Pecters, Andrew Song, and Faisal Mahmood. Transcrintomics-guidee slide representation learning in computational pathology. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 9632-9644, 2024.
[27] Hassan Keshvarikhojasteh, Josien PW Pluim, and Mitko Veta. Multiple instance learning with random sampling for whole slide image classification. In Medical Imaging 2024: Digital and Computational Pathology. pages 372-376. SPIE, 2024.
[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[29] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In CVPR, pages 14318-14328, 2021.
[30] Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottleneck for weakly-supervised pathology whole slide image classification. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 7454-7463, 2023.
[31] Hao Li, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang. Liansheng Wang, Bowen Ding, and Yuchen Han. Generalizable whole slide image classification with fine grained visual-semantic interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11398-11407. 2024.
[32] Honglin Li, Yunlong Zhang, Pingyi Chen, Zhongyi Shui, Chenglu Zhu, and Lin Yang. Rethinking transformer for long contextual histopathology whole slide image analysis. arXiv preprint arXi::2410.14195, 2024.
[33] Jiawen Li, Yuxuan Chen, Hongbo Chu, Qiehe Sun, Tian Guan, Anjia Han, and Yonghong He. Dynamic graph representation with awwleede-wwwaee attention for histopathology whole slide image analysis. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 11323-11332. 2024.
[34] Jiawen Li, Tian Guan, Qingxin Xia, Yizhi Wang, Xitong Ling, Jing Li, Qiang Huang, Zihan Wang, Zhiyuan Shen, Yifei Ma, et al. Unlocking adaptive digital pathology through dynamic feature learning, arXiv preprint 2412:22412.204300 2024.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10012-10022 2021.
12

---

[36] Zhuang Liu, Hanzi Mao. Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition. pages 11976- 1986, 2022.
[37] Tlya Loshchilov and Frank Hutter. arXiv 2017.
Decoupled
weight decay regularization. arXiv preprint
[38] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard Chen, Mattco Barbieri, and Faisal Mahmood. Data-cfficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6):555-5700 2021.
[39] Xiaoyuan Luo, Linhao Qu. Qinhao Guo, Zhijian Song, and Manning Wang. Negative instance guided self-distillation framework for whole slide image analysis. IEEE Journal of Biomedicul and Health Informatics. 2023.
[40] Yingfan Ma, Mingzhi Yuan, Ao Shen, Xiaoyuan Luo, Bohan An, Xinrong Chen, and Manning Wang. Sela-mil: Developing instance-level classifier via weakly-supervised self-training for whole slide image classification. Computer Methods and Programs in Biomedicine, page 108614, 2025.
[41] Oded Maron and Tomás Lozano-Pérez. A framework for multiple-instance learning. Advances in neural information processing systems, 10, 1997.
[42] Hans Pinckaers, Bram Van Ginneken, and Geert Litjens. Streaming convolutional neural networks for end- to-end learning with multi-megapixe images. IEEE transactions on pattern analysis and machine intelligence. 1483)11581-1500 2020.
[43] Hans Pinckaers, Bram Van Ginneken, and Geert Litjens. Streaming convolutional neural networks for end- lo-end learning with multi-megapixel images. IEEE transactions pattern analysis and machine intelligence, 44(3):1581-154) 2020.
[44] Rong Qin, Xin Liu, Xingyu Liu, Jiaxuan Liu, Jinglei Shi, Liang Lin, and Jufeng Yang. No pains, more gains: Recycling sub-salient patches for efficient high-resolution image recognition. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14965-14975, 2025.
[45] Rong Qin, Xingyu Liu, Jinglei Shi, Liang Lin, and Jufeng Yang. Boosting the dual-stream architecture ultra-high resolution segmentation with resolution-biaset uncertainty estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25960-25970, 2025.
[46] Linhao Qu, Manning Wang, Zhijian Song, et al. Bi-directional weakly supervised knowledge distillation for whole slide image classification. Advances in Neural Information Processing Systems, 35:15368-1555 2022.
[47] Linhao Qu, Yingfan Ma, Xiaoyuan Luo, Qinhao Guo, Manning Wang, and Zhijian Song. Rethinking multiple instance learning for whole slide image classification: A good instance classifier is all you need. IEEE Transactions on Circuits and Systems for Video Technology. 2024.
[48] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. NewrlPS, 34, 2021.
[49] Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A Moskaluk. Sana Syed, and Donald Brown. Cluster-to-conuuer. A framework for end-to-end multi-instance learning for whole slide image classification. In Medical Imaging with Deep Learning, pages 682-698 698. PMLR, 2021.
[50] Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A Moskaluk, Sana Syed, and Donald E Brown. Cluster-to-conquer conquuer A framework for end-to-end multi-instance learning for whole slide image classifi- cation. arXiv preprint arXiv:2103. 2021.
[51] Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu, Vila-mil: Dual-scale vision-language multiple instance learning for whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11248-12255 2024.
[52] Jun Shi, Dongdong Sun, Kun Wu, Zhiguo Jiang, Xue Kong, Wei Wang, Haibo Wu, and Yushan Zheng. Positional imcodinggguide transformer- based multiple instance learning for histopathology whole slide images classification. Computer Methods and Programs in Biomedicine, 258: 108491, 2025.
[53] Andrew H Song. Guillaume Jaume, Drew FK Williamson, Ming Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood, Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering. pages 1-20, 2023.
13

---

[54] Andrew HI Song, Richard J Chen, Tong Ding, Drew FK Williamson, Guillaume Jaume. and Faisal Mahmood. Morphological prototyping for unsupervised slide representation learning in computational pathology. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11566-11578, 2024.
[55] Wenhao Tang, Sheng Huang, Xiaoxian Zhang, Fengtao Zhou, Yi Zhang, and Liu. Multiple instance learning framework with masked hard instance mining for whole slide image classification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4078-4087, 2023.
[56] Wenhao Tang, Fengtao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang. and Bo Liu. Feature re-embedding: Towards foundation model-level performance in computational pathology. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11343 11352, 2024.
[57] Wenhao Tang, Heng Fang, Ge Wu, Xiang Li, and Ming-Ming Cheng. Revisiting data challenges of computational pathology: A pack-based multiple instance learning framework. arXiv preprint 2r:1v:5509.2092. 2025.
[58] Wenhao Tang, Sheng Huang, Heng Fang, Fengtao Zhou, Bo Liu, and Qingshan Liu. Multiplc instance learning framework with masked hard instance mining for gigapixel histopathology image analysis. arXiv preprint arXiv:2509. /1526, 2025.
[59] Yucheng Tang, Yufan He, Vishwesh Nath, Pengfeig Guo, Ruining Deng. Tianyuan Yao, Quan Liu, Can Cui, Mengmeng Yin, Ziyue Xu, et al. Holohisto: end-to-end gigapixel wsi segmentation with 4k resolution sequential tokenization. arXiv preprint arXiv:2407.03307 2024.
[60] Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, and Furu Wei. When an image is worth 1,024 x 1,024 words: A case study in computational pathology. arXiv preprint arXiv:2312.03558. 2023.
[61] Xuenian Wang, Shanshan Shi, Renao Yan, Qiche Sun, Lianghui Zhu, Tian Guan, and Yonghong He. Task- oriented embedding counts: Heuristic clustering-driven feature finc-tuning for whole slide image classification. arXiv preprint arXiv:2406.00672 2024.
[62] Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang. Yu Li, et al. pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 634(8035):970-97)) 2024.
[63] Zhuoyu Wen, Shidan Wang. Donghan M Yang, Yang Xie, Mingyi Chen, Justin Bishop, and Guanghua Xiao. Deep learning digital pathology for personalized treatment plans cancer patients. Seminars in Diagnostic Pathology, pages 109-119. Elsevier, 2023.
[64] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang. Rajesh Rao, Tristan Naumann, Cliff Wong. Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pages 1-8, 2024.
[65] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
[66] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802-188122 18812, 2022.
[67] Jingwei Zhang, Anh Tien Nguyen, Xi Han, Vincent Quoc-Huy Trinh, Hong Qin, Dimitris Samaras, and Mahdi S Hosseini. 2dmamba: Efficient state space model for image representation with applications on giga-pixel whole slide image classification. arXiv preprint arXiv:2412.00678 2024.
[68] Yunlong Zhang, Honglin Li, Yunxuan Sun, Sunyi Zheng, Chenglu Zhu, and Lin Yang. Attention- challenging multiple instance learning for whole slide image classification. In European Conference on Computer Vision, pages 125-143. Springer, 2024.
14

---

Appendix
Table of Contents
A Theoretical Analysis of Optimization Risk in End-to-End Training
16
A.1 Definition of Optimization Risk
16
A.2 Multi-head Local Attention Mechanism
16
A.3 Attention Plus Propagation
A.4 Empirical Validation of Theoretical Analysis
B Datasets and Implementation Details
19
B.1 Datasets
19
B.2 Preprocess
20
B.3 Implementation Details
21
Additional Quantitative Results
22
More about E2E Methods
22
C.2 More about MILs in E2E "e._............................
23
C.3 ABMILX in Two-Stage Framework
23
C.4 Ablation
24
D Additional Qualitative Results
24
Additional Related Works
26
F Limitation & Broader Impacts
27
15

---

A Theoretical Analysis of Optimization Risk in End-to-End Training
A.1 Definition of Optimization Risk
Definition 1. (Optimization Risk) Let N denote the set of noisy instances, D denote discriminative instances, and (()) is measure of the contribution of attention à to the final bag feature Z. O(-) is monotone increasing. We define the optimization risk R as the impact of maximum attention value among noisy instances bag feature:
cxp(a;)
R = -mmax @z), TEN
where 0 Σx=1 exp(ax))
(6)
where is the number of sampled instances, and a, is derived from the instance feature e, after MLP,
a, MLP(e;), A = {a....., Cj. ....))
Rationale for Max Operator in Optimization Risk. The of the maximum operator in Definition 1, as opposed to a summation, is crucial for accurately modeling the optimization risk. Under effective learning conditions, the model is expected assign significantly higher attention to discriminative instances (D) compared to noisy instances (A) with high probability. Let us assume that for the set of noisy instances N, there exists an attention value âN such that the summed contribution of noisy instances can be approximated by Oāāxr M ~ Σiev O(ā,). Similarly, let âp represent a typical attention value for a discriminative instance. well-behaved scenarios, âp âv. implying that the collective contribution of noisy instances to the bag feature Z is relatively small, and thus the optimization risk is low.
However, the primary concern for optimization risk arises when, even with low probability, the MIL. model assigns typically large attention value single noisy instance. If this single noisy instance significantly influences Z, the backpropagation process can adversely affect the optimization of the feature encoder, leading to suboptimal instance features e;. This, turn, can result in poorer attention scores a,, creating a detrimental feedback loop. The max operator specifically targets this scenario by focusing on the worst-case contribution from a single noisy instance. larger variance in attention values within N, particularly the presence of outliers with high attention, directly translates higher optimization risk as measured by the max operator.
ABMILX is designed to mitigate this optimization risk by intervening when noisy instances receive unduly high attention, thereby preventing their disproportionate impact O(maxieN âi), and breaking the aforementioned vicious cycle.
A.2 Multi-head Local Attention Mechanism
For ABMILX with heads, each head generates independent attention distributions ÂMHLA {Ā(1) ....y AO) ...., Á(m) through distinct feature subspaces. The correlation of different heads can be defined by CorT(AO) A()) E
[0, 1]. We can obtain the following derivation:
Corr (a.0) a (1) -
1, 3 RMILLA = RABMIL
(7)
multi-head ottention collapses full corrolation
Corr (a. a. (k)
O, Vi,j.
(8)
multi-head attention fully independent
m
RMHLA Σ R ( < m max R 15
m
RABMIL
(9)
m
upper bound of multi-head risk
impact
Under the diversity assumption of head specialization, the optimization risk satisfies:
RMHLA < RABMIL
(10)
16

---

The (9) can be proved by:
z
(1) (1) 0 c
Z = Z(1)
Z(m) J. Z E EERID, z E I
(11)
max RV) m
max 15
( (1) 1 VEN
RABMIL.
(12)
the risk of each head only affects the 1 dimension in Z. O
1 O.
mul-head
A.3 Attention Plus Propagation
In order to simplify the analysis, this subsection does not deal with the multi-head mechanisms. Let U be the normalized feature similarity matrix where Uiz sim(e,, ey) [0, 1]. The attention refinement:
Unaa.
(13)
For noisy instance with highest original attention à' maxieN 0i = arg maxieN à, when the propagation term between discriminative instances EnED ΣXED Untag are higher that between the noise ones ΣJEN ΣKEN Ug the post-softmax effect yields:
exp(a')
exp(a')
(14)
Σk=rexp(an exp(ax)
E exp(a:)
exp(ă')
exp(a')
S 0
= RA S RABMIL
(15)
kml exp(ās)
E1 exp(ax)
(Proof of Eq.(14)) Substituting Eq.(13) into the softmax operation:
cxp(@')
exp (a' + En Uvnan)
(16)
Ei= exp(ax)
Σi=l exp (ax + a En, Uknan)
Let A = a En Ukn an represent the attention modulation term. We decompose the attention:
exp(a' + +
exp(a')
exp(A') k= exp( AE)
(17)
Ei exp(ax Ax)
1 exp(as)
Original risk
Modulation factor A
The critical inequality < 1 holds when:
(xp(A') < Σ exp(Ax)@x
(18)
k=1
By Jensen's 's inequality:
Σ ď exp(At) 2 exp
Σ au
(19)
Σ
a A Σ
àn
U Unkate A, = Σ
a. Σ
Upar (JEN)
(20)
h 1=1
km
1=1
kmm
17

---

Through the subtraction operation Σx az Ak - As we derive:
a - A; =a
Σ nED
àn
(Unk - UsN)ax - Σ Σ
On (unk
Uyk)ak
(21)
k=1
nEN
k=
Intru-class Modulation Difference Q
a
Σ
ān
Σ
Un nk - a E
ân
EUsaa + +2+a Σ an Σ Unbay - a Σ àn Σ
UjAa
RED
RED
nED
REN
RED
KEN
nED
RED
Cross-class Propugation
(22)
Q
Σ an Σ
nk - a Σ ân
EUNa + 1Q+
(23)
RED
RED
nED
REN
mm Unk
Σ REN
(24)
RED
RED
As discriminative instances i E D and noise ones j E N typically share weak cross-class correlations Uty/Uzi with each other but have stronger correlations within classes, the values of and T could be ignored, T < a EnED ân ERED Un a EnED ân ExEx Uyaaz. As mentioned before, A is sparse attention vector. Therefore, is more than that in N:
holds when the number of high value instances in D
àz 4 20
(25)
km
àg A 2 A,
(26)
k=
āR exp AR) 2 exp
Σ â Ax
2 exp(A')
(27)
In this case, the highest attention á' will be suppressed, yielding lower risks and higher benefits: exp(a')
exp(a')
x A
< á'
(28)
Σ1 k- exp(āx)
Ei= exp(ax)
exp(a')
exp(a')
Ra+ 0
x A
RABMIL
(29)
-E= exp(ās)
Ei= exp(ax)
(Proof of Eq.()
exp(a: + Aa) exp exp(Ax)
(30)
With post-softmax attention definition, àg E7 exp(a.))
exp(as)
we can obtain:
exp(ax) = àΦ x
exp(an)
(31)
==1
Substituting E31( into (30):
exp(a: (a + AR)
Σ à
exp(an)
exp(Ax)
k=1
(32)
Σ
exp(an)
x
ån exp(An)
Thus, the original softmax expression can be decomposed as: exp(a' A')
exp(a')
xxp(A))
(33)
Σi=1 exp(ax + Ax) Σiz exp(an)
Ei= âi exp(Axx Ax)
18

---

A.4 Empirical Validation of Theoretical Analysis
Our theoretical analysis posits that the maximum attention score of noisy instances serves as proxy for optimization risk. The theory further suggests that ABMILX can mitigate this risk while maintaining reasonable sparsity. To empirically validate this connection, we introduce the MAX-N metric, defined as the product of the maximum attention score of noisy instances for each slide and the total number of instances.
We measured MAX-N on the CAMELYON dataset during the early E2E training stage, with the results reported in Table 5. The experimental results align closely with our theoretical analysis. ABMILX drastically reduces the MAX-N score (our risk proxy) from 21.2162 (for ABMIL) to 2.6557, demonstrating its effectiveness in mitigating optimization risk. Concurrently, it maintains functional sparsity level (36) compared to the baseline's 's (80). achieving the "reasonable sparsity" predicted by our theory. This effective risk mitigation and balanced sparsity directly translate to substantial performance improvement (95,88% vs. 91.78%). These findings validate our theoretical framework, demonstrating a clear alignment between our analysis and the experimental results.
Metric/Method
ABMIL
ABMILX
MAX-N
21.2162
2.6557
Sparsity
80
36
Performance 91.78
95.88
Table 5: Empirical validation of ABMILX's risk mitigation on the CAMELYON dataset. ABMILX significantly reduces the optimization risk proxy (MAX-N) and balances sparsity, aligning with our theoretical analysis and leading to superior performance over the ABMIL baseline.
B
Datasets and Implementation Details
B.1
Datasets
We validate our E2E training ABMILX on various computational pathology tasks, including cancer grading (PANDA [6]), subtyping (TCGA-NSCLC, TCGA-BRCA), survival analysis (TCGA-LUAD, TCGA-LUSC, TCGA-BLCA), and diagnosis (CAMELYON [3, 2)).
PANDA [6] (CC-BY-4.0) is a large-scale, multi-center dataset dedicated to prostate cancer detection and grading. It comprises 10,202 digitized H&E-stained whole-slide images, making it one of the most extensive public resources for prostate cancer histopathology. Each slide is annotated according to the Gleason grading system and subsequently assigned International Society of Urological Pathology (ISUP) grade, enabling both cancer detection and severity assessment. The dataset includes a diverse distribution of ISUP grades, with 2,724 slides classified as grade 0 (benign), 2.602 as grade 1. 1,321 as grade 2, 1,205 as grade 3, 1,187 as grade 4, and 1,163 as grade 5. Spanning multiple clinical centers, PANDA ensures a broad range of samples, mitigating center-specifie biases.
The Non- SmnlSmll Cell Lung Cancer (NSCLC) project of The Cancer Genome Atlas (TCGA) by the National Cancer Institute is the primary dataset for the cancer sub-typing task. TCGA-NSCLCH is the most common type of lung cancer, accounting for approximately 85% of all lung cancer cases. This classification includes several subtypes, primarily Lung Adenocarcinoma (LUAD) and Lung Squamous Cell Carcinoma (LUSC). The dataset contains 541 slides from 478 LUAD cases and 512 slides from 478 LUSC cases, with only image-level labels provided.
The Breast Invasive Carcinoma (TCGA-BRCA) project is another sub-typing dataset we used. TCGA-BRCA includes two subtypes: Invasive Ductal Careinoma (IDC) and Invasive Lobular Carcinoma (ILC). contains 787 IDC slides and 198 ILC slides from 985 cases. To mitigate the impact of class imbalance on E2E optimization, we employed oversampling of the ILC class, resulting in a training set with an IDC:ILC ratio of 2:1.
The primary goal of survival analysis is to estimate the survival probability or survival time of patients over a specific period. Therefore, we used the TCGA-LUAD, TCGA-BRCA, and TCGA-BICA projects to evaluate the model performance for survival analysis tasks. Unlike the diagnosis and
19

---

Table Comparison between E2E methods and two-stage methods. We add the features extraction time on two-stage methods. "FM denotes the best performance achieved among foundation models (CHIEF [62], UNI [11], GIGAP [64]), with only the highest value reported. Existing E2E methods devote excessive resources to sampling, incurring long training times yet offering only marginal gains over two-stage ResNet50, and incorporating FM features further diminishes E2E's advantage. In contrast, our ABMILX drastically shortens training while improving performance, remaining competitive even against latest two-stage methods using FM features.
Encoder
Method
TTime
Grad.
Sub.
Surv.
Latest Two-stage Methods
WIKG [33]
3h
62.72
88.37
60.65
R50
RRT [56]
3h
60.42
89.35
63.03
WIKG [33]
24h
74.97
94.76
66.97
FM
RRT [56]
24h
74.00
94.84
67.30
E2E Training
R18
C2C [49]
84h
62.91
91.13
R50
FT [61]
45h
66.06
86.48
R18
ABMILX
9h
78.34
93.97
67.78
R50
ABMILX
22h
78.83
95.17
67.20
sub-typing tasks, the survival analysis datasets are case-based rather than WSI-based. The WSIs of TCGA-LUAD and TCGA-BRCA are identical to those used in the sub-typing task but with different annotations. The TCGA-BICA dataset includes 376 cases of bladder urothelial carcinoma.
We supplemented the CAMELYON dataset (CC-BY-4.0) to evaluate qualitative and quantitative results of different methods. The dataset comprises CAMELYON-16 [3] and CAMELYON-I7 [2), which are among the largest publicly available datasets for breast cancer lymph node metastasis diagnosis, each providing binary labels (metastasis or not). The CAMELYON dataset contains 899 WSIs (591 negative and 308 positive) from 370 cases. Additionally, 159 slides have complete pixel-level annotations, making this dataset particularly suitable for qualitative analysis.
We randomly split the PANDA dataset into training, validation, and testing sets with a ratio of 7:1:2. Due to the limited data size, the remaining datasets divided into training and testing sets with ratio of 7:3.
B.2 Preprocess
End-to-End Training, To efficiently process gigapixel slides in our E2E pipeline, we first crop each WSI into a series of non-overlapping patches of size (256 256) and discard background regions, including holes, as in CLAM [38]. Except for PANDA, we perform patching at 10x magnification, with an average of approximately 3,000 patches per slide. For the PANDA dataset, we process patches at 40x magnification, resulting in an average of 505 patches. Additionally, we extract patches at 5x and 20x magnifications to support our multi-scale random sampling strategy. After patch extraction, we store all data using the LMDB (Lightning Memory-Mapped Database) format. Ultimately, each dataset contains an average of 4~~6 million 256x256 patches.
Two-Stage Framework. Following prior works [38, 48, 66, 56], we crop each WSI into a series of non-overlapping patches of size (256 256) at 20x magnification and discard the background regions, including holes, as in CLAM [38]. The average number of patches per dataset is around 10.000. To efficiently handle the large number of patches, we follow the traditional two-stage paradigm, using pre-trained offline model to extract patch features. This includes a ResNet-50 [21] pre-trained on ImageNet- [16]. Specifically, the last convolutional module of the ResNet-50 removed, and a global average pooling is applied to the final feature maps to generate the initial feature vector. Additionally, we also use otate-of-the-art foundation models pre-trained on WSIs, such CHIEF [62], GigaPath [64] and UNI [11].
20

---

Table 7: More MIL aggregators in E2E training. We categorize these aggregators to two type: Sparse-Attentle (S.A.) and Transformer-He (Trans.). We categorize these aggregators into two types: Sparse-Attention (S.A.) and Transformer-like (Trans.). Existing S.A. methods, primarily focused on the two-stage paradigm, face challenges in E2E optimization. While Transformer-basee methods partially alleviate the optimization challenges caused by extreme sparsity, they struggle to focus on key regions within the numerous redundant patches in the E2E training. It leads to noticeable performance gap compared to two-stage methods under FM features (marked in gray).
Aggregator
Aggr. Type
Grad.
Sub.
Surv.
Best in FMs
74.97
94.84
67.30
ABMIL [25]
S.A.
75.46
89.23
62.70
RRTMIL [56]
S.A.
17.99
61.82
53.42
QAMIL
Trans.
75.12
90.65
64.29
TransMIL [48]
Trans.
75.08
91.44
63.42
DSMIL [29]
Trans.
76.28
91.09
64.32
VITMIL
Trans.
76.98
92.61
63.67
ABMILX
S.A.
78.34
93.97
67.78
B.3 Implementation Details
End-to-End Training. To maintain consistency with traditional ResNet-based two-stage methods, we removed the last stage module of ResNet. For MIL, we added a LayerNorm [1] layer at its input to better optimize the Encoder. We also disabled the bias of all fully connected layers in MIL, which we found beneficial for E2E optimization. For training, we used diffcrent hyperparameters for different tasks. For cancer grading (PANDA), we employed an Adam [28] optimizer with a learning rate of 2 x 10-4 and a weight decay of x 10-5, training for 200 epochs. For sub-typing (NSCLC, BRCA), we used an AdamW [37] optimizer with learning rate of 8 10-5 and weight decay, training for 75 epochs. For survival analysis (LUAD, BLCA, BRCA), we utilized an AdamW optimizer with a learning rate of 8 10-5 and a weight decay of x 10-2, training for 30 epochs. The learning rate was adjusted using the Cosine annealing strategy. During training, we applied simple geometric data augmentations such as flipping and RandomResizeeCropp Since slides are typically H&EEstained, we found that color-related data augmentations could significantly impact performance. All experiments are conducted on 3090 GPUs. We adjusted the batch size based on the 24GB memory limit and the number of samples in different datasets. Unless otherwise specified, all ablation experiments use ResNet- as the encoder. All efficiency experiments are conducted on the BRCA- subt-ingg benchmark. We calculate the FLOPs for all models using an input size of 1 xx12233x x24 x 224, which simulates the sampling of 512 patches during E2E training. To evaluate model inference speed, we use input size of x x 224, representing the average data volume processed in clinical scenarios.
Two-Stage Framework. Following [38, 48, 56), the offline feature is projected to a 512-diimnsioonal feature vector using a fully-connected layer. For features extracted by ResNet-50, an AdamW optimizer [37] with a learning rate of 2 x 10- and no weight decay is used for model training. For features extracted by foundation models, the learning rate is changed rate is adjusted using the Cosine annealing strategy. All models are trained for 200 epochs for cancer grading. For sub-typing and survival analysis tasks, the number of epochs is reduced to 75. Notably, due to the training of the GIGAP [64] aggregator exceeding the memory limit of a 3090 GPU, we sampled the number of patches to 1024 during its training. For all two-stage methods except GIGAP aggregator, following [38, 48, 56]. we used the complete patch sequence for training. Due to the variable sequence length, we conventionally set the batch size to 1. We employed unified hyperparameters to train all methods.
10 The learning
21

---

Table 8: CPath performance of different methods. Our methods marked in gray OOM denotes Out-of-Memory. Two-stage paradigms based on FMs have achieved saturated performance on classical tasks (CAMELYON and NSCLC) leveraging advantages of over 100K pre-trained pathology slides. However, these approaches are bottlenecked by the lack of encoder adaptation in challenging benchmarks (BRCA-sublypee and Survival Analysis). We further show the results of ABMILX in two-stage paradigms, demonstrating its versatility as a pure MIL. architecture in CPath tasks without requiring hyperparameter tuning.
Diagnosis
Grading
Suh-typing
Survival Analysis
Encoder
Agyregator
CAMELYON
PANDA
BRCA
NSCLC
LUAD
BRCA
BLCA
ARMIL [25]
91.8 9..84
58.89 $8.89
83.80 83.80
92.32
59.56
64.93
55.01
CLAM [38]
91.85+4.0
29.45+2.2
86+6.4
92.28-2.7
59.79-s.7
62.90-9.4
55.78+8.0
TranMIL [48]
4..59
56.42222.1
88. 28.52=5.4
92.49-2.7
64.15
19.15
$6.96
R50
DSMIL |291
12.63
61.24
85.68
91.1223.0
61.70_
61,9629.5
56.22
(26M Para.)
(InageNet-1K)
DTFD [66]
91.88.4.0
60.62.1.1
83.46.7.2
92.36=2.7
59,47=5.6
61.76=10.3
58.60
WIKG [35]
91.42+4.0
62.72+2.2
88.37
92.57-2.6
0OM
60.65
OOM
RRTMIL
9..19
61.97
89.35
94.43=2.2
62.19
63.03
60.78
ARMILX
9..37
61.05+2.2
87.45
93.28=224
61.25=s.6
63.79
58.64 88.6448..6
ABMIL
90.04
65 05.661311
94.09
96.22-1.7
62.09
64.02.2..0
60.78..8.c
CHIEF [62]
TrasMIL
95.30
60.89+2.2
91.4++4.0
96,39=1.7
65.55
61.46=9.4
88.83+
(27M Para.)
(Slide 60K)
CHIEF
69.61
64.24
4..43
96.8411.5
60.29
27.95
59.63
ABMILX
4..5
65.82
92.
1.
62.71
66.08
60.09
UNI [1]]]
ABMIL
96.58
74.69
94.05
97.04_.1.0
59.65
67.05
57.29
(307M Para.)
TrasMIL
2..63
68.06:::.11
93.33
97.27=1.6
60.4=-0.4
62.76=10.5
60.45
(Mas-1000)
ABMILX
96.77
18.54
93.34.3..9
97.81
59.39
66.47.9.9
60.83
GIGAP [64]
ABMIL
9..43
71885
94.39
96.54
60,56
63.81
89.85
TranSML
96.59+3.2
65.45+22.0
93.97+3.9
97.61=1.2
60.40=s.8
62,90=9.2
60.12 60.12++8.6
(1134M Para.)
(Slide 170K)
GIGAP
95.53
2..6642222
93.72++..4
97.53-1.2
62.99-s.77
62.64-9.3
57.63++5.4
ABMILX
96.7443.0
73.01
94.83
96.09=2.0
59.69.2..7
66.34.2..
57.81
E2F Approaches
18 ResNet-50
ABMILX
2
78.34
93.97+2.9
97.09-1.4
64.91
67.78+s.s
61.20+
ABMILX
96.06
78.83
95.17
97.06
64.72
67.20
60.78
C Additional Quantitative Results
C.1 More about E2E Methods
Slide-level supervised E2E methods process entire slides in a unified manner, preserving the critical interdependencies necessary for robust clinical interpretation. These methods exploit the complete spatial context of gigapixel images, thereby providing more comprehensive and clinically pertinent analysis. Building on these merits, several researchers have introduced innovative slide-level super- vised E2E methods to further enhance WSI analysis. Sharma et al. [50] proposed an E2E framework (C2C) that clusters patch representations and employs adaptive attention with KL--divrgeece regu- larization to robustly classify whole slide images. Li et al. [30] proposed a task-specific fine-tuning framework (FT) that employs a variational information bottleneck to distill patches into sparse subset via Monte Carlo-sampled Bernoulli masks, thereby enabling E2E backbone fine-tuning.
In this section, we compare and analyze these E2E methods. As shown in Table 6, existing E2E pipelines often allocate substantial computational resources to patch or region-level sampling strate- gies for processing gigapixel WSIs, resulting in prolonged training times (e.g., 84h for C2C [49] and 45h for FT [6]]), yet yielding only marginal performance gains compared to two-stage approaches with R50 features. Once FMs are integrated into two-stage frameworks, the FM-based approach outperforms previous E2E methods by significant margin, effectively diminishing their advantage. In contrast, our ABMILX approach drastically shortens training time (9h with ResNet18, which is comparable to SOTA two-stage ResNet50) and simultaneously delivers substantial performance improvements over both prior E2E and two-stage ResNet50- methods. Moreover, even when compared with latest two-stage methods under FM features, ABMILX X-R50 maintains competitive performance in both training efficiency and final performance, highlighting the effectiveness of our approach.
22

---

Grad.
Sah
Surv:
Grad.
Sah
Surv.
Grad.
Suh.
Surv.
77.04
93.97
76.22
FFN
78.34
93.03
65.23
256
76.15
93.97
67.78
77.32
90,91
65.61
384
76.01
90.21
64.13
78.34
91.65
62.83
512
78.34
91.51
63.20
76.48
93.97
67.78
16
76.50
92.59
64.62
(b) Projection Dim for input of ABMILX.
(c) Head Number Multi-head Local Attention of ABMILX.
(a) Feed Forward Network.
MH. denotes multi-heads.
Grad.
Sub.
Surv:
Grad.
Sub.
Surv.
Grad.
Sub.
Surv.
w/o MS.
76.77
92.72
67.24
64
76.22
93.07
63.52
Rand.
78.34
93.97
67.78
2 74.02
92.74
64,92
128
78.34
93.86
63.73
4
78.34
91.97
66.22
384
74.32
92.52
64.12
Regional Rand. Sampling
77.21
92.35
67.46
512
75.62
93.97
65.59
67.28
10
76.09
92.25
67.78
768
93.66
67.78
76.11
92.67
67.23
1280
92.14
67.00
(d) Multi-scale Ratio Multi- scale Random Sampling. MS. de- notes multi-scale
(e) Sampling Number in Multi-
(f) Sampling Strategy. Rand. de- notes naive random sampling.
scale Random Sampling.
Table 9: Ablation studies on various components of our method. Default settings are marked in gray.
C.2 More about MILs in E2E Learning
Table presents the performance various MIL aggregators in the E2E learning. Besides the commonly used DSMIL [29] and TransMIL [48], we implemented ViTMIL, based on two-layer multi-head self-attention (MSA) structure, and QAMII,, using a single-layer multi-head query attention. Comparing existing sparse attention methods and Transformer-like methods, we observe: 1) The E2E optimization challenge posed by sparse attention is significant. Specifically, although RRTMIL [56]. TransMIL, and VITMIL share similar MSA front-end structure, the difference in their final aggregation methods leads to substantial performance variations. RRTMIL directly employs ABMIL as the aggregator, while the other two utilize the [CLS] token from the MSA. This demonstrates the impact of sparse attention on encoder optimization. Furthermore, the feature re-embedding module (MSA layer) in RRTMIL further impairs the representation of the affected features, accelerating the collapse of the optimization loop. 2) Maintaining sparsity beneficial for E2E optimization. Although experimented with various Transformer-liks methods in the E2E setting, they still underperformed compared to FM. We attribute this to the fact that in the E2E training, relying solely global attention struggles to focus on learning key regions within the numerous redundant patches. Our proposed ABMILX mitigates the E2E optimization challenges while preserving sparsity, thus achieving superior performance.
C.3 ABMILX in Two-Stage Framework
We supplement quantitative results the CAMELYON [4, 3] dataset in Table 8 and demonstrate ABMILX's performance in a two-stage framework. The results reveal the following insights: 1) The quality of offline features determines the performance of two-stage methods. Although different MII. aggregators show performance variations, these differences are significantly smaller under FMs compared ResNet-50 (R50). Under FM features, classical ABMIL often outperforms advanced MIL approaches, highlighting the importance of sparse attention in CPath tasks. 2) Two-stage methods based on FM features have achieved saturated performance on traditional tasks (CAMELYON and NSCLC), leveraging massive pre-training data and large models. Further performance gains from increased data volume and model size are limited. Conversely, large FMs also encounter performance bottlenecks in challenging tasks (BRCA-subtyping and Survival Analysis). We attribute this to the encoder's lack of downstream task adaptation. Our proposed E2E method surpasses performance on these challenging benchmarks, demonstrating the effectiveness and potential of E2E approaches in CPath tasks. 3) We further validate the effectiveness ABMILXX two-stage paradigms, demonstrating its versatility as pure MIL architecture in CPath tasks. ABMILX performs
23

---

ResNet ABMILX (E2E)
ResNet ABMIL (E2E)
UNI ABMIL (2-Stage)
Figure 4: Attention visualization the PANDA dataset [6]. All slides from the Karolinska Center, with annotations limited to three types: background, benign tissue, and cancerous tissue. We highlight cancerous tissue in blue and display high-attention patches as bright patches for comparison.
more exceptionally under FM features, further confirming the significance of maintaining sparse characteristics for CPath tasks.
C.4
Ablation
We conduct ablation studies on the hyperparameters related to our method Table 9 and provide the following analysis.
Feed Forward Network. Feed Forward Network (FFN) common component in modern models [35, 36, 18], which we explore in ABMILX's 's design. We add FFN after the bag feature aggregation module further refine bag features before input to the task head. We find that due to FFN's large parameter count, requires a larger training dataset. It performs poorly conventional datasets with smaller data scales, except for PANDA. PANDA, with its 10,000 slides, allows FFN to demonstrate performance improvements.
Sampling Number. Sampling number is critical hyperparameter sampling strategies, closely related to downstream tasks. We observe that larger sampling number does not consistently improve performance. We attribute this the fact that an appropriate sampling number helps the model eliminate redundant instances that interfere with optimization. For PANDA, with an average of around 500 patches, lower sampling numbers perform better. contrast, survival analysis tasks often require a larger sampling number for more comprehensive analysis.
Sampling Strategy. Beyond naive random sampling, we explore region-based random sampling. We divide the entire slide into N sub-regions and perform random sampling within each sub-region. The results in Table 9f demonstrate that the diversity brought by naive random sampling is beneficial for optimization. However, more region division can compromise diversity and lead to performance degradation.
D Additional Qualitative Results
PANDA. To further demonstrate the effectiveness of ABMILX and E2E learning, we visualize the attention scores (bright patches) different methods in Figure We demonstrate the following: (1) Compared to the offline FM approach, the encoder trained with E2E (column and 2) produces more cohesive instance features. This enables the MII. attention to more comprehensively cover cancerous tissue (blue areas). (2) Compared to ABMIL, ABMILX reduces attention to normal patches. This
24

---

Slide Annotation
ResNet ABMILX (E2E)
UNI & ABMIL (Offiline)
Low
High
Figure 5: Heatmap visualization on the PANDA dataset [6]. The top row shows original slide and its annotation (with cancerous tissue red). The middle and bottom rows present attention maps generated by ResNel & ABMILX (E2E) and UNI & ABMIL (Offline) respectively. Color intensity ranges from blue (low attention) to red (high attention), illustrating how each approach prioritizes different tissue regions. Notably, model yields more uniform attention distribution while effectively highlighting cancerous areas.
indicates that ABMILX allows the encoder to learn more effectively from discriminative patches in the E2E training. It leads more accurate representations of normal patches and improved differentiation from tumor patches. (3) Beyond improved feature, ABMILX also benefits from the proposed global attention plus module. This module refines the attention map based on feature correlations, mitigating the issues of excessive attention to redundant regions and insufficient attention discriminative regions.
Training
Converged
CAMELYON. We provide additional visual- izations of different MIL models during E2E training and convergence stage in right Figure. During E2E training, extreme sparsity causes ABMIL [25] to overlook discriminative regions while overly focusing redundant areas. Al- though TransMIL [48] covers a small number of discriminative regions, it is distracted by large amount of attention on redundant ones. This prevents the encoder from adequately learning diseriminative regions, causing ABMIL to fail in correctly localizing target areas. While the converged TransMIL can localize it, its training process struggles consistently focus on dis- criminative areas, resulting in incomplete iden- tification of the overall tumor region. In con- trast, AB ABMILX benefits from more effectively enabling the encoder to learn from discrimina- tive regions during training. Consequently, it achieves stronger discriminative capabilities in con- verged stage, simultaneously enhancing focus on tumor regions while reducing interference from redundant areas.
ABMIL
E2E Training
ABMILX (ours)
in E2E Training
TronsMIL
in E2E Training
25

---

Ours E2E ResNet
Offline UNI
Offline ResNet
background
benign tissue
cancerous tissue
Figure 6: More UMAP [22] visualization on PANDA dataset [6].
More Visualization. To provide more comprehensive quantitative analysis of the proposed method, we present heatmaps and additional UMAP [22] visualizations in Figures and 6, respectively.
E
Additional Related Works
While E2E high-resolution image analysis relatively mature in general computer vision [45, 44, 5], its application to CPath presents significant challenges due to their unique gigapixel scale. As elaborated the Related Work section, current F2E methods for WSI analysis are broadly categorized into instance-lewe -evell supervised and slide-level supervised approaches. As elaborated in the Related
26

---

Work section, current E2E methods can be divided into instance-level supervised approaches and slide-level supervised approaches. Instance-level supervised methods [13, 39, 46, 47] adopt a pseudo E2E: paradigm, in which the encoder is trained using instance-level pseudo-labels rather than genuine slide-level supervision. This strategy simplifies the problem by processing patches independently; however, it neglects the essential inter-patch contextual relationships required for robust clinical interpretation [7, 40] and creates a disconnect between training and downstream clinical applications. Moreover, the performance of these methods fundamentally depends on the quality of the pseudo- labels, indicating that they constitute a compromise rather than a fully E2E solution. For example, Qu et al. [47] introduce novel instance-level MIL framework that leverages weakly supervised contrastive learning and prototype- based pseudo label generation to markedly improve both instance and bag-level classification in WSI analysis. Luo et al. [39] propose negative instance guided self-distillation framework that leverages true negative samples and a prediction bank to constrain pseudo-label distributions, enabling E2E instance-level classifier.
F
Limitation & Broader Impacts
This work pioneered the exploration of end-to-end (E2E) optimization challenges in computational pathology and effectively mitigated them. It demonstrated the potential and advantages of E2E learning in this domain. However, our current full-training approach makes direct fine-tuning of large foundation models challenging under limited computational resources. Investigating the effectiveness of our proposed method for fine-tuning foundation models is a direction for future work. Furthermore, as this work focuses on computational pathology, is directly relevant to tasks such as multi-cancer diagnosis and prognosis. This work has the potential to inspire and facilitate the deployment of more accurate and efficient clinical diagnosis and prognosis algorithms.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the optimization challenge in E2E learning in CPath and propose ABMILX and an efficient training framework to address the problem. And we call for more community attention to E2E learning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the supplementary material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [No]
Justification: We provide a theoretical analysis of the E2E optimization challenge CPath and how ABMILX mitigates it in the supplemental material. However, providing complete rigorous mathematical proof is extremely challenging.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the supplemental material and code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
See the supplemental material and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttpssccccpbccccccceececcccccccc /ttpg/eeeeeeeeeeeeiiieeeeee /Cooeeeooiissseeeeee cesssonnooccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See the supplemental material and code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: intervals.
We perform 1,000 bootstrap experiments, and report the 95% confidence
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the experiment part and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttpr//eerupseeeeee/eee/eeeeeeer oc httpppl/tttiisasuiiiiiiieap
Answer: [Yes]
Justification:
We preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This work has the potential support cancer diagnosis and prognosis.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Null.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: Hard to find licenses of TCGA datasets, but the PANDA and CAMELYON is under CC-BY-4.0 licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Null.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Null.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Null.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
33

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Null.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https / 220 for what should or should not be described.
34


Reviews:
[{"review_id": "Af2kvrhhsv", "reviewer": "Reviewer_AkkT", "summary": "This paper revisits end-to-end (E2E) learning with slide-level supervision for computational pathology, aiming to overcome the limitations of the traditional two-stage paradigm. The authors identify a key optimization challenge caused by sparse-attention MIL modules and introduce ABMILX, a novel MIL method that combines multi-head attention and global correlation refinement. Their proposed E2E pipeline with ABMILX achieves state-of-the-art performance while being significantly more computationally efficient than existing foundation models. This work demonstrates the potential of E2E learning in pathology and encourages further research in this direction.", "questions": "1. How sensitive is ABMILX to choices like the number of attention heads and the global attention scaling factor, and how do these impact its performance?\n\n2. Can the proposed E2E pipeline handle real-world variations such as slide size, noise, and rare classes not present in benchmark datasets?\n\n3. Are there any noticeable gaps between the theoretical analysis and the experimental results, and if so, what further work is planned to address them?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Comprehensive Analysis of Optimization Challenges\nThe paper offers a detailed analysis of why E2E learning underperforms in CPath due to MIL-induced attention sparsity and clearly identifies the core bottleneck.\n\n2. Novel and Effective MIL Architecture\nThe proposed ABMILX module successfully mitigates the limitations of sparse attention, leading to clear performance improvements in E2E learning.\n\n3. Efficiency and Clinical Relevance\nThe E2E pipeline provides competitive or superior accuracy to state-of-the-art models at a fraction of the computational cost, making it more practical for real-world clinical use.\n\n\n### Weaknesses\n1. Limited Theoretical Justification\nThe theoretical foundations of ABMILX are only partially provided, with formal proofs or rigorous analysis not fully developed in the main text.\n\n2. Generality Not Fully Demonstrated\nExperiments are focused on a few public cancer datasets, so it is unclear how well the method generalizes to other pathology domains or more diverse clinical cases.\n\n3. Limited Discussion on Generalization of Large FMs\nThe comparison with large foundation models is mostly centered on accuracy and efficiency, with little discussion of their generalization or transferability strengths.", "was_revised": true, "final_justification": "Thank you for the detailed rebuttal and the added experiments/clarifications. I will keep my score. I have also considered the other reviewers’ comments and your responses during the discussion. In brief: strong empirical gains and practical efficiency; architectural novelty feels moderate.", "rebuttal": "Thanks for providing feedback and taking the time to review our work! We have provided point-by-point clarifications on the issues and supplemented the experiments as suggested.\n\n> **W1: Clarification on Theoretical Justification**\n\n**R1:** As stated in Section 3.3 of the main text, we have provided a comprehensive and rigorous mathematical analysis of ABMILX in the Supplemental Material (Sec.A). This includes the definition of optimization risk, as well as complete proofs demonstrating how the proposed Multi-head Local Attention Mechanism and Attention Plus Propagation reduce optimization risk. It will be included in the main text in the revised version.\n\n> **W2&Q2: The results of more medical tasks and clarification on real-world data**\n\n**R2:** \n\n- **【Clarification on dataset used】** First, we would like to clarify that computational pathology is closely linked to cancer analysis, as the **analysis of pathological images is typically regarded as the gold standard for cancer diagnosis and prognosis in clinical practic**e. Therefore, cancer datasets are commonly used in the field of computational pathology. Due to ethical concerns and the inaccessibility of private data, we utilized public datasets. However, the public datasets we used are **diverse in sources** (TCGA, CAMLYON, PANDA), **covering various cancers across multiple organs** (prostate, breast, lung, bladder) with **rich and challenging task types** (cancer subtyping, cancer grading, cancer patient survival time analysis, cancer metastasis detection). These datasets are **derived from real-world clinical scenarios**. For instance, TCGA, the largest cancer database in the United States, is not a benchmark specifically designed for model evaluation. Furthermore, **these datasets vary in size** (PANDA images typically have dozen of millions of pixels, while TCGA images usually exceed 1 billion pixels) and **noise ratios** (CAMLYON has less than 10% cancer cell area, whereas other datasets generally have over 40%). In addition, we **supplemented external validation experiments using CPTAC data (another large cancer data source)** to demonstrate the performance of our method across more clinical cases. The response to **R3** provides specific results and details.\n\n- **【Diabetic retinopathy grading】** Furthermore, we further verify the generalizability of our method to other high-resolution E2E medical tasks through diabetic retinopathy grading (DR) [1] and existing frameworks [2]. The results are reported in the table below, which demonstrate the advantages of the proposed ABMILX in optimizing high-resolution end-to-end tasks.\n\n  - |          | ABMIL | TransMIL | ABMILX |\n    | -------- | ----- | -------- | ------ |\n    | DR (Acc) | 75.3  | 74.3     | 78.1   |\n\n> **W3: The generalization compared foundation models**\n\n**R3:** We have supplemented the following external validation on CPTAC to demonstrate the generalization ability of the encoder trained on TCGA via E2E learning:\n\n| TCGA->CPTAC          | R50+ABMIL | R50+TransMIL | R50+WIKG | UNI+ABMIL | UNI+TransMIL | UNI+WIKG | R50+ABMILX (E2E) |\n| -------------------- | --------- | ------------ | -------- | --------- | ------------ | -------- | ---------------- |\n| CPTAC-NSCLC (AUC)    | 66.42     | 74.59        | 64.04    | 83.73     | 85.24        | 83.56    | **85.19**        |\n| CPTAC-LUAD (C-index) | 46.34     | 48.24        | OOM      | 53.59     | 51.36        | OOM      | **54.00**        |\n\n*OOM = Out-of-Memory on 24GB-RTX-3090*\n\nThe experimental results demonstrate that the R50 optimized via an E2E approach on TCGA not only exhibits significant improvements in generalization but even outperforms UNI (a ViT-L model pre-trained on 1B-scale pathological data). This validates the strengths of both E2E learning and our proposed ABMILX in terms of generalization ability and transferability across real-world variations.\n\n> **Q1: Clarification on hyperparameters**\n\n**R4:** We analyzed the impacts of different global attention scaling factors and numbers of attention heads in **Section 3.3 of the main text** and **Section C.4 of the Supplemental Material**, respectively. We have reorganized these experiments in the table below:\n\n| Global Attention Scaling Factors | Subtyping | Survival |\n| -------------------------------- | --------- | -------- |\n| $\\alpha = 0$                     | 91.58     | 63.80    |\n| $\\alpha = 1$                     | 92.84     | 65.49    |\n| Learnable $\\alpha$               | 93.97     | 67.78    |\n\n| Numbers of Attention Heads | Grading | Subtyping | Survival |\n| -------------------------- | ------- | --------- | -------- |\n| 4                          | 78.34   | 91.65     | 62.83    |\n| 8                          | 76.48   | 93.97     | 67.78    |\n| 16                         | 76.50   | 92.59     | 64.62    |\n\nWe use a learnable global attention scaling factor by default, while the number of heads has an impact on model performance. However, selecting 8 or 4 by default can achieve sufficiently strong performance without the need for additional hyperparameter tuning.\n\n> **Q3: Gaps between the Theoretical Analysis and the Experimental Results**\n\n**R5:** According to our theoretical analysis in the Supplemental Material, the maximum attention score of noisy instances can serve as a representative of optimization risks. And our theoretical analysis suggests that ABMILX can mitigate this risk while maintaining reasonable sparsity. **Therefore,** **we use the product of the maximum attention score of noisy instances for each slide and the total number of instances (MAX-N)** **to validate the gap between theoretical analysis and the experimental results****.** And we have measured MAX-N on the CAMELYON dataset in the early E2E training stage. The results are reported in the table below. As consistent with the theoretical analysis, ABMILX indeed mitigates optimization risks effectively while maintaining reasonable sparsity. **This proves that there is no significant gap between our theoretical analysis and experimental results. These experiments and analyses will also be added to our revised version.** A minor gap is that our experimental results have shown that the value of $\\alpha$ has a certain impact on the final performance, but this was not fully discussed in our theoretical analysis. We suggest that the influence of the value of $\\alpha$ stems from its role in controlling the attention sparsity of MIL. In the future, we will conduct corresponding theoretical analyses and experiments to design a MIL architecture with better performance in terms of sparsity.\n\n| Metric/Method | ABMIL   | ABMILX |\n| ------------- | ------- | ------ |\n| MAX-N         | 21.2162 | 2.6557 |\n| Sparsity      | 80      | 36     |\n| Performance   | 91.78   | 95.88  |\n\n[1] Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences.\n\n[2] No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition. CVPR 2025."}, {"review_id": "yBEB63P4sQ", "reviewer": "Reviewer_tziM", "summary": "The paper argues that the traditional two-stage learning paradigm lacks joint optimization. It suggests that this process is limited by an optimization collapse, which is caused by the interaction between the encoder and the MIL module. To solve this optimization collapse problem, a new MIL aggregator is proposed, composed of multi-head local attention and global correlation-based attention. At the same time, to improve the effectiveness of the implementation, a multi-scale random sampling strategy is also applied in the model.", "questions": "1. What are the separate contributions of the ABMILX module and the multi-scale sampling strategy to the performance\n2. The paper's baselines are somewhat old. Please find more recent and novel baselines for comparison.\n3. The authors are requested to provide a more rigorous mathematical analysis explaining how sparse attention causes optimization collapse in the E2E setup. Additionally, a novel metric to quantify optimization collapse, along with a visual representation of the separate contributions of the sampling strategy and ABMILX in addressing this collapse, would be beneficial.", "limitations": "see Questions and Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.The primary contribution is not just the proposed model (ABMILX), but the pioneering identification and clear elucidation of the optimization challenge caused by sparse-attention MIL in an E2E setting. Figure 2 provides an excellent conceptual illustration of this \"deteriorating iteration\" problem.\n2.A MIL aggregator is proposed, based on multi-head local attention and global correlation-based attention , which improves the model's performance by attempting to solve the problem of optimization collapse in joint optimization.\nWeaknesses:\n1. This paper's main contribution is proposing a new MIL aggregator that utilizes different attention mechanisms; however, the innovation is very limited and the contribution is insufficient. Additionally, the design of the attention mechanisms used is very common and lacks theoretical support.\n2. The authors use a combination of a sampling strategy and ABMILX to improve the model's performance. The ablation study does not specify the respective contributions of the sampling strategy and the ABMILX module to the model. Meanwhile, while the ablation study compares two different sampling strategies, the comparison is not convincing.", "was_revised": true, "final_justification": "Thank you for the author's detailed explanation. All my questions about the innovation and experimental part have been answered.", "rebuttal": "> **W1: Novelty and contribution**\n\n**R1:** \n\n【**Innovation in research direction**】We would like to clarify that the main contribution of this paper lies in **Revisiting slide-level E2E work in computational pathology** and proposing that the **overlooked sparse MIL models in E2E are actually the primary bottleneck hindering the performance improvement**. We define this as the E2E optimization challenge, with **its** **mathematical definition provided in the Supplemental Material**. This differs entirely from the focus of previous E2E papers, which mainly concentrated on improving sampling strategies and reducing memory usage, yet still showed significant performance gaps compared to foundation model-based two-stage schemes.\n\n【**Architectural novelty**】Based on this E2E optimization challenge, we propose a concise yet effective MIL model, ABMILX. Therefore, **the design motivation of this model differs from the common attention-based aggregators mentioned by the reviewer**, which focus on the two-stage paradigm with offline feature inputs. Additionally, **the design of the model is novel, and Reviewer igAe has expressed strong interest in the ABMILX structure**. Specifically, we introduce a multi-head mechanism to the classic sparse architecture ABMIL to alleviate the extreme sparsity of the original architecture. More importantly, **we utilize global feature correlation to refine local attention scores**. **This utilization of global correlation is pioneering—existing methods typically** **utilize global correlations to** **refine** **instance** **features rather than** **instance** **attention scores**. Furthermore, we suggest that compared to existing works, which often compute local attention and global attention independently, our idea of using global attention to refine the local one is also a quit ecreative attention combination manner.  For instance, as shown in our experimental results, RRT-MIL, which uses global attention and local attention independently, achieves significantly worse performance than our ABMILX in E2E learning. It is precisely the optimization challenge caused by sparse attention scores that motivated our innovation. **We provide rigorous and comprehensive proofs in the Supplemental Material regarding how these two mechanisms reduce optimization risk.**\n\n【**Experimental and community contributions**】We have conducted a comprehensive and systematic evaluation of E2E and the proposed ABMILX, demonstrating the potential of E2E in computational pathology. Comparisons were performed across **7 datasets**, **4 different tasks**, **3 state-of-the-art foundation models** of varying sizes, and **9 different MIL aggregators**. **Without any pre-training on pathological data**, with a training cost of only **10 3090 GPUHours**, our method outperforms foundation models on multiple challenging benchmarks while maintaining a significant inference efficiency advantage. Furthermore, we supplemented results showing its **generalization comparable to foundation models**, as well as the **excellent performance of ABMILX on non-pathological datasets or other application settings**. More importantly, to demonstrate the potential of E2E, we supplemented **experiments on ABMILX and the proposed E2E framework in UNI PEFT**. Due to space constraints, details are provided in responses **R1 and R3 to Reviewer igAe**.\n\n> **W2&Q1: Ablation of ABMILX and MRIS**\n\n**R2:** In fact, we have already provided comprehensive and detailed ablation experiments in our paper (**Sec.3.3, 4.3, and Table 3 in main text**) regarding the performance contributions of ABMILX and MRIS. Here, we have reorganized and summarized a clearer version, as shown in the table below. Moreover, it should be clarified that we did not claim in the original paper that MRIS could address optimization risks. It is merely a sampling strategy that efficiently incorporates multi-scale information while keeping low time cost. And ABMILX is designed to mitigate the optimization risks in the E2E process.\n\n| Task/Method | ABMIL + RS | ABMIL + MRIS | ABMILX + RS | ABMIL+ MHLA+MRIS | ABMIL+ MHLA+ A+(ABMILX) + MRIS |\n| ----------- | ---------- | ------------ | ----------- | ---------------- | ------------------------------ |\n| Survival    | 62.64      | 62.70        | 67.24       | 63.80            | 67.78                          |\n| Subtyping   | 87.99      | 89.23        | 92.72       | 91.58            | 93.97                          |\n| Grading     | 73.51      | 75.46        | 76.77       | 77.56            | 78.34                          |\n\n> **W2: The comparison with sampling strategies**\n\n**R3:** In fact, our paper has already presented sufficient sampling strategies for comparison. These include naive **random sampling (RS)** and our MRIS, as well as **cluster sampling (C2C)**, **naive attention sampling (Attention)**, and **variational information bottleneck-based attention sampling (FT)** (which fall under selective sampling strategies). Here, we have reorganized and summarized a clearer version, as shown in the table below. We suggest these experiments validate one of our insights: in the E2E process, the role of MIL in optimization is often more pronounced than that of sampling strategies.\n\n| Task/Method | ABMILX + Attention | ABMILX + MRIS | ABMILX + RS | ABMIL + RS | ABMIL + MRIS | ABMIL + Attention | C2C   | FT    |\n| ----------- | ------------------ | ------------- | ----------- | ---------- | ------------ | ----------------- | ----- | ----- |\n| Survival    | 66.53              | 67.78         | 67.24       | 62.64      | 62.70        | 62.74             | -     | -     |\n| Subtyping   | 93.14              | 93.97         | 92.72       | 87.99      | 89.23        | -                 | 91.13 | 86.48 |\n| Grading     | 77.43              | 78.34         | 76.77       | 73.51      | 75.46        | -                 | 62.91 | 66.06 |\n\n> **Q2: Latest Baselines**\n\n**R4:** In fact, **we compared the performance of WIKG and RRT under FM features (UNI, CHIEF, GIGAP) in Table 1 of the Supplemental Material.** In the table below, we have reorganized these results and **supplemented 2DMamba [1]**. In the revised version, this content will be included in the main text. \n\n| Encoder   | Method  | Grad.     | Sub.      | Surv.     |\n| --------- | ------- | --------- | --------- | --------- |\n| R18 (E2E) | ABMILX  | 78.34     | 93.97     | **67.78** |\n| R50 (E2E) | ABMILX  | **78.83** | **95.17** | 67.20     |\n| UNI       | RRT-MIL | 74.00     | 94.61     | 66.91     |\n| UNI       | WIKG    | 74.97     | 94.19     | 65.28     |\n| UNI       | 2DMamba | 74.97     | 93.59     | OOM       |\n| CHIEF     | RRT-MIL | 69.73     | 92.49     | 67.30     |\n| CHIEF     | WIKG    | 71.22     | 93.13     | 65.92     |\n| CHIEF     | 2DMamba | 71.59     | 90.96     | OOM       |\n| GIGAP     | RRT-MIL | 72.46     | 94.84     | 65.40     |\n| GIGAP     | WIKG    | 72.17     | 94.76     | 66.97     |\n| GIGAP     | 2DMamba | 75.36     | 93.33     | OOM       |\n\n *OOM = Out-of-Memory on 24GB-RTX-3090*\n\n> **Q3: optimization collapse**\n\n**R5:** First, as stated in Section 3.3 of the main text, **we have provided additional mathematical analysis on optimization collapse in the supplemental material (Sec.A)**, which will be incorporated into the main text in revised revisions. A concise mathematical explanation is that during the E2E process, the gradients received by the encoder $\\theta$ for $i$-th instances consists of the gradients from the instance embedding $e_i$ and the gradients from the corresponding normalized attention score $\\hat{a}_i$:\n\n$\\theta \\leftarrow \\frac{\\partial \\mathcal{L}}{\\partial e_i}+\\frac{\\partial \\mathcal{L}}{\\partial \\hat{a}_i}=\\frac{\\partial \\mathcal{Z}}{\\partial e_i}\\cdot\\frac{\\partial \\mathcal{L}}{\\partial Z}+\\frac{\\partial \\mathcal{L}}{\\partial \\hat{a}_i}= \\frac{\\partial \\mathcal{\\sum\\hat{a}_j e_j}}{\\partial e_i}\\cdot\\frac{\\partial \\mathcal{L}}{\\partial Z}+\\frac{\\partial \\mathcal{L}}{\\partial \\hat{a}_i}=\\hat{a}_i\\frac{\\partial \\mathcal{L}}{\\partial Z}+\\frac{\\partial \\mathcal{L}}{\\partial \\hat{a}_i}$,\n\nwhere $ \\mathcal{L}$ denotes the loss value and $Z=\\sum\\hat{a}_je_j$ denotes the aggregation of all instance embeddings. It could be concluded that the gradients from $i$-th instance embedding $\\frac{\\partial \\mathcal{L}}{\\partial e_i}=\\hat{a}_i\\frac{\\partial \\mathcal{L}}{\\partial Z}$ are scaled by corresponding normalized attention score $\\hat{a}_i$. And sparse attention tends to assign high scores to noisy instances $\\mathcal{N}$, leading to optimization collapse. Therefore, we suggest using the maximum attention score of noisy instances as a metric to quantify optimization collapse $\\mathcal{R}$:\n\n$\\mathcal{R} = O(\\max_{i \\in \\mathcal{N}} \\hat{a}_i).$\n\nTo better demonstrate optimization collapse, we use the product of the maximum attention score of noisy instances for each slide and the total number of instances as the final metric (**MAX-N**). Furthermore, we have measured MAX-N on the CAMELYON dataset in the early training stage. The results are reported in the table below, demonstrating that ABMILX effectively mitigates optimization collapse while maintaining reasonable sparsity. These experiments and analyses will also be added to the revised version. **Finally, in page 6 of the main text, we have demonstrated how ABMILX mitigates optimization collapse through visualization.** \n\n| Metric/Method | ABMIL   | ABMILX |\n| ------------- | ------- | ------ |\n| MAX-N         | 21.2162 | 2.6557 |\n| Sparsity      | 80      | 36     |\n| Performance   | 91.78   | 95.88  |\n\n*Sparsity is a metric proposed in the main text to measure model sparsity, and it can also be regarded as an indirect metric of optimization collapse.*\n\n[1] 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification. CVPR 2025."}, {"review_id": "dMfgHujWGX", "reviewer": "Reviewer_igAe", "summary": "This paper revisits slide-level supervised end-to-end (E2E) learning in computational pathology,\nand proposes a novel MIL model, ABMILX, to address the optimization challenge caused by sparse-attention MIL.\nThe proposed ABMILX integrates multi-head attention and global correlation-based refinement, \ncombined with multi-scale random sampling. \nThe framework proposed in the paper achieves superior performance across multiple benchmark datasets.", "questions": "1. From Table 1,\nthe performance gap between UNI + ABMIL and E2E + ResNet50 + ABMILX does not appear to be very large. \nHowever, according to Fig. 1(a), E2E + ResNet shows a clear improvement over models like UNI. \nDoes this suggest that the performance difference between ABMILX and ABMIL is actually small?\nI believe the authors need to provide more comparisons under fair conditions to further evaluate the method.\n\n2. In addition to being used as a standalone prediction model, ABMIL has also been widely applied in other frameworks, \nfor example, as a pooling tool after GNN message passing (e.g. PatchGCN[1]). \nHow does ABMILX perform when replacing ABMIL in these settings?\n\n3. In Table 3, regarding the results of models under the two-stage paradigm, were patches also randomly sampled? I would like to see further comparisons of ABMILX with more recent state-of-the-art models under two-stage settings.\n\n\nAlthough I do not fully agree with the paper’s position on E2E, I really like the design of ABMILX. I think ABMILX is an interesting and easily generalizable model, and I am therefore particularly interested in seeing more comprehensive experiments on ABMILX. \nI am willing to increase my ratings based on the rebuttal.\n\n\n[1] Chen, Richard J., et al. \"Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks\", MICCAI 2021", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThis paper proposes ABMILX to mitigate the optimization risks from both local and global perspectives. \nI find this an interesting and easily generalizable module that could inspire future work. \nABMILX computes patch attention scores using self-attention, \nallowing the model to assign patch importance with global awareness. \nIt also leverages multiple replicas and multi-head attention to capture diverse features. \nI think this design very intuitive, and I have not seen a similar approach before.\n\nThe overall idea is easy to follow, and the method is clearly explored. \nThe figures and discussions are well done.\n\n**Weaknesses**\n\nI have some concerns regarding the E2E part.\n\nI do not fully agree with the paper's description of E2E. \nPathology foundation models have already been widely developed. \nAlthough pretraining these models on large-scale datasets comes at a high cost, \nbut once trained, they can be applied directly to most datasets and tasks without fine-tuning, \nand have been proven to perform well across various cancer types and tasks. \nWhile E2E encoders might be able to extract more task-specialized features, \nthe need to train separate models for different datasets and tasks could actually increase overall training cost.\n\nIn addition, I feel that implementing E2E by using a smaller encoder (only ResNet) and randomly sampling a small number of patches is rather crude. Since E2E requires random patch sampling, it becomes difficult to properly model relationships between patches (such as spatial positioning or multi-scale structures), which many studies have shown to be crucial. This also limits the scalability of the E2E framework, making it challenging to design models that can capture richer contextual information.\n\nAlthough I really like the design of ABMILX, I think the exploration of ABMILX in this paper is insufficient. The comparative methods chosen are relatively outdated. I would like to see combinations of advanced backbone encoders with recent aggregation strategies, such as WIKG + UNI. It is unclear why these newer models were only tested in a two-stage setting with a ResNet50 encoder. I hope to see more comparisons with state-of-the-art models from the past two years, especially under settings that use strong foundation model encoders.", "was_revised": true, "final_justification": "While I still have concerns regarding the scalability of the proposed end-to-end approach, I acknowledge the contribution of ABMILX. The additional experiments presented in the rebuttal suggest that ABMILX is promising.", "rebuttal": "Thanks for providing feedback and taking the time to review our work! We have provided point-by-point clarifications on the issues and supplemented the experiments as suggested.\n\n> **W1&W2: Clarification on E2E and Foundation Model**\n\n**R1:** \n\n- First, we would like to clarify that E2E and Pathology Foundation Models (FM) are not opposing concepts. We acknowledge that FMs have made extensive applications and prominent contributions in pathology due to their effectiveness and generalization. However, we argue that the performance of traditional patch-based FMs is approaching saturation [1], and an increasing number of FMs are adopting E2E-based self-supervised pre-training [2, 3]. **We believe that E2E with slide-level supervision primarily advantages in its training process being more aligned with clinical tasks** (rather than merely learning from medical images themselves) and **its more efficient data utilization**. A concurrent work holds a similar view [4]. We have shown that small ResNet models pre-trained only on ImageNet-1k can achieve performance comparable to or even superior to that of two-stage schemes based on computationally expensive pre-trained FMs, with **E2E training costing <10 3090 GPU-hours (a cost similar to, or even lower than, the training cost of some latest large-scale slide encoders** [3]).\n\n- More importantly, we supplement the following **external validation** to demonstrate the generalization ability of encoders trained via E2E:\n\n  - | TCGA->CPTAC          | R50+ABMIL | R50+TransMIL | R50+WIKG | UNI+ABMIL | UNI+TransMIL | UNI+WIKG | R50+ABMILX (E2E) |\n    | -------------------- | --------- | ------------ | -------- | --------- | ------------ | -------- | ---------------- |\n    | CPTAC-NSCLC (AUC)    | 66.42     | 74.59        | 64.04    | 83.73     | 85.24        | 83.56    | 85.19            |\n    | CPTAC-LUAD (C-index) | 46.34     | 48.24        | OOM      | 53.59     | 51.36        | OOM      | 54.00            |\n\n*OOM = Out-of-Memory on 24GB-RTX-3090*\n\nWithout any fine-tuning, the model trained on TCGA was directly tested on CPTAC-sourced data. **The R50 trained via E2E not only showed significantly improved generalization but even outperformed UNI.**\n\n- Furthermore, constrained by resources, this work is limited to small-scale models and datasets. However, we believe that E2E, after addressing optimization challenges, has greater potential in pathology. **We conducted a preliminary attempt at UNI by applying ABMILX for E2E Parameter-Efficient Fine-Tuning (PEFT) on UNI.** The following results demonstrate the advantages of ABMILX in E2E tasks and preliminarily verify the potential of E2E to further enhance FMs. We note that work [5] adopted a similar approach but overlooked the impact of optimization challenges caused by MIL on E2E training.\n\n  - |       | UNI   | R50 (E2E) | UNI+PEFT (E2E) | UNI+PEFT (E2E) | UNI+PEFT (E2E) |\n    | ----- | ----- | --------- | -------------- | -------------- | -------------- |\n    |       | ABMIL | ABMILX    | ABMIL          | TransMIL       | ABMILX         |\n    | PANDA | 74.69 | 78.83     | 74.11          | 76.61          | 80.99          |\n\n- 【**Patch Sampling**】 In fact, we **only perform patch sampling during the training phase, with sampling varying across iterations to ensure diversity.** This approach is relatively common not only in general domains [6] but also in Pathology [7]. We have conducted comprehensive ablation studies on the sampling strategy in both the main text and supplementary materials, and found that it not only achieves optimal efficiency but also helps eliminate data redundancy and enhance data diversity.\n\n> **W3&Q3: Clarification on Latest Baselines and Two-stage Details**\n\n**R2:** In fact, **we compared the performance of WIKG and RRT under FM features (UNI, CHIEF, GIGAP) in Table 1 of the Supplemental Material.** In the table below, we have reorganized these results and **supplemented 2DMamba [8]**. In the revised version, this content will be included in the main text. Furthermore, except for the training of the GIGAP aggregator (due to memory constraints), all other two-stage experiments did not employ any random sampling strategy. Additional details are provided in Supplemental Material B.3.\n\n| Encoder   | Method  | Grading   | Subtyping | Survival  |\n| --------- | ------- | --------- | --------- | --------- |\n| R18 (E2E) | ABMILX  | 78.34     | 93.97     | **67.78** |\n| R50 (E2E) | ABMILX  | **78.83** | **95.17** | 67.20     |\n| UNI       | RRT-MIL | 74.00     | 94.61     | 66.91     |\n| UNI       | WIKG    | 74.97     | 94.19     | 65.28     |\n| UNI       | 2DMamba | 74.97     | 93.59     | OOM       |\n| CHIEF     | RRT-MIL | 69.73     | 92.49     | 67.30     |\n| CHIEF     | WIKG    | 71.22     | 93.13     | 65.92     |\n| CHIEF     | 2DMamba | 71.59     | 90.96     | OOM       |\n| GIGAP     | RRT-MIL | 72.46     | 94.84     | 65.40     |\n| GIGAP     | WIKG    | 72.17     | 94.76     | 66.97     |\n| GIGAP     | 2DMamba | 75.36     | 93.33     | OOM       |\n\n*OOM = Out-of-Memory on 24GB-RTX-3090*\n\n> **Q1&Q2: ABMIL vs. ABMILX on more applications**\n\n**R3:** **Multiple fair ablation studies comparing ABMILX and ABMIL are presented in both the main text (Figure 1(c), Sec.3.3, Sec.4.3) and supplemental material (Table 2, Table 3, Sec.D)**. Here, we reorganize and summarize their comparisons in E2E and Two-Stage settings, and **supplement the comparisons in more scenarios** (Pavement Distress Classification, Diabetic Retinopathy Grading, Path GCN framework).\n\n- **【Clarification of details】** The data in Figure 1(a) are identical to those in Table 1. *E2E+ResNet* in Figure 1(a) corresponds to *E2E+ResNet-50+ABMILX* in Table 1. The possible reason for the minor discrepancy is that Table 1 only includes one challenging benchmark, while Figure 1 is more comprehensive.\n\n- **【E2E & Two-Stage】** We have summarized the table below from the main text and supplemental material. All comparisons in the table ensure that all other settings are identical except for the MIL model.\n\n  - |        | E2E     | E2E       | E2E      | Two-Stage | Two-Stage | Two-Stage |\n    | ------ | ------- | --------- | -------- | --------- | --------- | --------- |\n    |        | Grading | Subtyping | Survival | Grading   | Subtyping | Survival  |\n    | ABMIL  | 75.46   | 89.23     | 62.70    | 71.85     | 94.39     | 63.81     |\n    | ABMILX | 78.34   | 93.97     | 67.78    | 73.01     | 94.83     | 66.34     |\n\n- **【More Tasks】** Following [9], we supplemented experiments on E2E recognition of high-resolution images, such as Pavement Distress Classification (CQU-BPDD [10]) and Diabetic Retinopathy Grading [11], to more thoroughly compare ABMIL and ABMILX.\n\n  - | Task\\Model | ABMIL | TransMIL | ABMILX |\n    | ---------- | ----- | -------- | ------ |\n    | CQU-BPDD   | 81.3  | 82.0     | 82.5   |\n    | DRG        | 75.3  | 74.3     | 78.1   |\n\n- **【PathGCN】** We supplemented comparisons with PathGCN, demonstrating the effectiveness of ABMILX when applied to other frameworks.\n\n  - | PathGCN+\\Task | Grading | Subtyping | Survival |\n    | ------------- | ------- | --------- | -------- |\n    | ABMIL         | 76.90   | 93.64     | 64.97    |\n    | ABMILX        | 77.92   | 94.23     | 65.96    |\n\n[1] PathBench: A comprehensive comparison benchmark for pathology foundation models towards precision oncology.\n\n[2] A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model.\n\n[3] A whole-slide foundation model for digital pathology from real-world data. Nature.\n\n[4] Foundation Models - A Panacea for Artificial Intelligence in Pathology?\n\n[5] Unlocking adaptive digital pathology through dynamic feature learning.\n\n[6] Masked Autoencoders Are Scalable Vision Learners. CVPR 2022.\n\n[7] Multiple instance learning framework with masked hard instance mining for whole slide image classification. ICCV 2023.\n\n[8] 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification. CVPR 2025.\n\n[9] No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition. CVPR 2025.\n\n[10] An iteratively optimized patch label inference network for automatic pavement distress detection. T-ITS.\n\n[11] Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences."}]

Author rebuttal:

Meta-review: The paper identifies that extreme sparsity in MIL attention causes optimization failures in end-to-end (E2E) pathology training. It introduces ABMILX, combining multi-head local attention and a global correlation refinement, and shows that with multi-scale random sampling, their method can match or surpass strong two-stage foundation model baselines while being far more computationally efficient.

Strengths: They effectively describe a plausible failure mode in E2E MIL and an effective fix that is easy to integrate. 

Weaknesses: Comparisons should also clarify that FMs are typically SSL-pretrained on massive unlabeled slides (a key OOD-generalization advantage, which isn't really assessed here) and that your E2E uses ImageNet-pretrained ResNets (at least in experiments, although I don't see why it couldn't be used with models that are randomly initialized as is common practice with foundation models). The ImageNet initialization could be a potential confounding factor for some of the claims, especially regarding the amount of time required. 

Discussion & Rebuttal: During the rebuttal, the authors strengthened the paper with additional baselines and added results. These additions largely resolved concerns about evaluation rigor.

Decision and rationale: Accept (poster). The paper provides a valuable insight into E2E MIL instability and demonstrates a simple yet impactful solution. The caveats mentioned in the weaknesses ought to be explicitly mentioned in the paper. 

Note: The two-stage paradigm was NOT introduced by Lu et al. [30]; earlier work (e.g., Campanella et al., 2019 [3]) predates this claim. They used MIL to learn an encoder for tile embeddings, and then used an RNN as an aggregator for those tile embeddings. This ought to be corrected in the paper (there may be an even earlier reference).
Decision: Accept (poster)