Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis
Authors: Honglin Li
Yunlong Zhang
Pingyi Chen
Zhongyi Shui
Chenglu Zhu
Lin Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Digital Pathology
Efficient Transformer
Primary area: machine_learning_for_healthcare
TL;DR: We analyze why Transformer and linear attention do not work well on WSI modelling in performance and efficiency and propose our solution.

Abstract: Histopathology Whole Slide Image (WSI) analysis serves as the gold standard for clinical cancer diagnosis in the daily routines of doctors. To develop computer-aided diagnosis model for histopathology WSIs, previous methods typically employ Multi-Instance Learning to enable slide-level prediction given only slide-level labels.
Among these models, vanilla attention mechanisms without pairwise interactions have traditionally been employed but are unable to model contextual information. More recently, self-attention models have been utilized to address this issue. To alleviate the computational complexity of long sequences in large WSIs, methods like HIPT use region-slicing, and TransMIL employs Nystr\"{o}mformer as an approximation of full self-attention. Both approaches suffer from suboptimal performance due to the loss of key information. Moreover, their use of absolute positional embedding struggles to effectively handle long contextual dependencies in shape-varying WSIs.
In this paper, we first analyze how the low-rank nature of the long-sequence attention matrix constrains the representation ability of WSI modelling. Then, we demonstrate that the rank of attention matrix can be improved by focusing on local interactions via a local attention mask. Our analysis shows that the local mask aligns with the attention patterns in the lower layers of the Transformer. Furthermore, the local attention mask can be implemented during chunked attention calculation, reducing the quadratic computational complexity to linear with a small local bandwidth. Additionally, this locality helps the model generalize to unseen or under-fitted positions more easily.
Building on this, we propose a local-global hybrid Transformer for both computational acceleration and local-global information interactions modelling. Our method, Long-contextual MIL (LongMIL), is evaluated through extensive experiments on various WSI tasks to validate its superiority in: 1) overall performance, 2) memory usage and speed, and 3) extrapolation ability compared to previous methods.

Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis
Honglin Li Yunlong Zhang Pingyi Chen 4.3 Zhongyi Shui
Chenglu Zhu Lin Yang".
Zhejiang University
2 Research Center for Industries of the Future and School of Engineering, Westlake University [lihonglin in, wawchengll..angnnnnnlnaaaaa yanglainnuesttllnee.. edu.
Abstract
Histopathology Whole Slide Image (WST) analysis serves as the gold standard for clinical cancer diagmosis in the daily routines of doctors. To develop computer- aided diagnosis model for histopathology WSIs, previous methods typically em- ploy Multi-Instance Learning to enable slide-level prediction given only slide-level labels. Among these models, vanilla attention mechanisms without pairwise in- teractions have traditionally been employed but are unable to model contextual information. More recently, self-attention models have been utilized to address this issue. To alleviate the computational complexity of long sequences in large WSIs, methods like HIPT use region-slicing, and TransMIL employs Nyströmformer as approximation of full self-attention. Both approaches suffer from suboptimal performance due to the loss of key information. Moreover, their use of absolute positional embedding struggles to effectively handle long contextual dependencies in shape-varying WSIs. In this paper, we first analyze how the low-rank nature of the lenueee attention matrix constrains the representation ability of WSI modelling. Then, we demonstrate that the rank of attention matrix can be improved by focusing on local interactions via local attention mask. Our analysis shows that the local mask aligns with the attention patterns in the lower layers the Transformer. Furthermore, the local attention mask can be implemented during chunked attention calculation, reducing the quadratic computational complexity linear with small local bandwidth. Additionally, this locality helps the model generalize to unseen or under-fitted positions more easily. Building on this, we propose local-global hybrid Transformer for both computational acceleration and local-global information interactions modelling. Our method, Long-contextun MIL (LongMIL), is evaluated through extensive experiments on various WSI tasks to validate its superiority in: 1) overall performance, 2) memory usage and speed, and 3) extrapolation ability compared to previous methods. Our code will be available at https://gthub..om/mmokkeee LLnnggggr-LLLLL--MMMMM
Introduction
Though digital pathology images have been widely used for Cancer diagnosis [50 65 89 94
45
and prognosis [9] [2 71] and gene expression [82] via automatic computer-assisted analysis, the Giga-pixels of resolution, as large as 150, 000 x 150, 000 pixels [50] 12 60) of Whole Slide Image (WSI), still poses great challenges on both annotation labelling and efficient computation for model training 422 Thus, previous methods
92 focus on developing annotation-
*Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

& computational efficient learning to cope with those problems by employing Multiple Instance Learning (MIL) 32) with only WSI-level supervision.
Currently, there are mainly three steps (or mainstream genres) of WSI analysis framework: 1) access
better instance-level patch embedding via Self-supervised Learning 30
40 77
2) design WSI head architectures [50 65 89 and
train the head with frozen instance embedding. 3)
1-4 patch sequence histogram of WSIs
fine-tune patch embedding with WSI level weak la- bel for better task-specific results 190, 42]. Here
TCGA-BCCA
SPACS
shape (003
in this paper, we focus on the step-2 2 and uncover- ing that there are still some room for improvement: Firstly, the vanilla attention used in AB-MII,, DS-
tranffomer complexiyy
-20
MII., CLAM, etc. [32 40 50 89], despite its com- putational efficiency (compared to self-attention), is unable model contextual or interaction informa- tion across instances within a WSI. These interac- tions, which play crucial role in prediction decision- making M 665 indeed, can be modelled via self- attention mechanism. However, the long sequence of WSI instances pose ((n²) computation complexity with self-attention (Fig. D. Although this complex- ity can be alleviated by self-attention approximation methods like Nyströmformer [83] 75
2500 5000 7500 15000 12500 15000 17520
sequence length: (nstance tum)
Figure 1:
Handling an extremely long se- quence with a magnification of 20x (or qua- drupling to 40x) poses a significant challenge. The computational complexity of transform- ers, denoted as On")), becomes prohibitive in such cases, leading to computational explo- sion.
used in Trans-
MIL 65, this approximation only get sub-optimal
performance compared to self-attention as pointed out in 20 29]. Authors in HIPT 99 mitigates the complexity by non-overlap large region slicing, but the interactions of instances from different region slicing are highly ignored (e.g. adjacent patches may be separated into two regions).
The above issues highlight strong need for an effective and efficient Transformer for WSI modelling. To begin, we discuss the performance bottleneck of basic Transformer [72) for WSI. Different to Vision Transformer [23) for natural image modelling where the number of patched tokens are smaller than patch embedding size (e.g. ViT-Small-p16 with embedding size 384 attend on 196+ tokens), the Transformer-bant based WSI model suffers severe low-rank bottleneck of attention matrix 3 22 given the long-sequee segg-seeence (n >> 1024) of WSI but limited embedding size (d S 1024). We reveal this problem in WSI theoretically, thus finding that one self-attention layer with limited embedding size can not model local contexts and global interactions at the same time. By stacking multiple self-attention layers, we notice that the low layer focus more on local context (Fig. 2h) after training while high layer focus on global. However, the rank of the attention matrix still limited (Fig. 2 resulting in constrained performance.
We assert that the low-rank bottleneck causes the attention mechanism become confused between local and global interactions, even after training. In other words, using Q and KT with only 2dn points, it's hard model n X n interactions comprehensively in the context of WSI where n >> d. We believe that it would be better focusing on less interactions in one layer. Motivated by the low layers of Transformer showing highly sparse attention pattern (Fig. (2)+) with locality, we propose a local attention mask to learn local interaction more directly. This local mask, more importantly, can highly improve the rank of the attention matrix, showing better representation ability. Furthermore, the local attention mask can be implemented during chunked attention calculation, reducing the quadratic computational complexity to linear with a small local bandwidth. In addition, this locality helps the model generalize to unseen under-fitee positions more casily (where absolute position embedding used in methods like TransMIL may fail, see Appendix A.2 for more illustration).
Building on this, we propose a local-global hybrid Transformer for both computational acceleration and -gobbll information modelling.
Our main contributions can be summarized as 3 folds:
1) We firstly theoretically uncover why Transformer model for WSI-MIL analysis fails, based on the low-rank bottleneck of attention matrix for long sequence but limited embedding size. We then further analyze the sparsity and locality pattern of attention matrix empirically to hint our local attention design.
2

---

2) We convert the full self attentienoo into local attention which shows three advantages: higher rank for better representation ability, lighter computational complexity and extrapolation ability for shape-vary WSIs. We further combine the full self-attention for global long-range dependency after stacking layers of local attention.
3) Our WSI-analysis experiments are performed on both diagnosis and prognosis tasks on WSI datasets including Breast, Stomach, Colon and Rectal Carcinoma, which show strong universality of the method and practical potential for real-world applications.
a) low-rank but sparse attention matrix
b) our designed local mask for 2-d image
Attertion Raformer, sparse ratio-0.902
252
c) high-rank and sparse attention (ours)
d) rank and sparsity comparison
Attention Matrix of model, rank-4583, sparse racio=0.879
sparsity
sparsity among
080
2000
1750
070
1210
0.44
100
0.7
752
0.70
360
000
Tonslomer
Figure 2: Rank and sparsity of attention matrix in WSI analysis.
2 Related Work
2.1
Multiple Instance Learning for WSI Analysis
Whole Slide Images (WSIs) contain rich set of visual information that can aid in pathological analysis 6 50). However, accurate annotation of cell-level information within WSIs is labor-intensive
and time-consuming 6 50 9. To address this issue, weakly-supervised methods have gained
popularity in pathology WSI analysis. Attention-based Multi-Instance Learning (AB-MIL)) 32 is
adopted to learss instance adaptive weights, allowing the model to focus informative regions
3

---

within the WSIs. This approach significantly reduces the annotation burden of pathologists while still providing valuable insights for patient-level diagnosis. In the context of
pathology WSI analysis, several innovative approaches, DS-MII,, CLAM, DTFD, etc. [32 40
50 89
35
18 2 have been proposed. However, their utilized vanilla attention with light computational
cost can not model WSI contextual information, which is useful in pathologist diagnosis decision making MT 65). The fine-grained details and global contextual information can also be captured
by multi-scale modelling [9] 40). Graph Network I1 43 28 8 25 is also useful to make model be context-aware. Similar to this, HIPT [9] and TransMIL [65] have explored the advantages of Transformer with pairwise interactions to model this contextual information. Since Transformer can be generalized to Graph Network [24], both modelling the pairwise interaction, in this paper we focus more on Transformer and try to adapt it better to fit the shape varying and long context properties of WSI. Unlike the authors in [80] who focus on the low-rank properties of pathology images, we investigate from the perspective of low-rank in the attention matrix of Transformer.
2.2 Efficient Transformer for Long Sequence
The primary goal of this area is to alleviate the computation and memory complexity of self-attention mechanism on long sequence input. A lot of modifications sparsify the attention matrix [59][]]]]]
with some fixed patterns. Extend to this, some work 773 74 64
using learnable patterns in a data- driven fashion, e.g. Reformer (38) introduces a hash-based similarity measure to efficiently cluster
tokens into chunks. Linformer 75 technique leverage low-rank approximations of the self-attention
matrix, decomposing the N x N matrix to N x k. The kernels also serve as an approximation of the attention matrix, including Performers (37], Linear Transformers
[16]. Another popular method of reducing computation cost is to reduce the resolution of the sequence, hence reducing computation cost by a commensurate factor, e.g. Perceiver [33], Swin Transformer (47. The recent
Nyströmformer [33] used in TransMIL 655 can also be seem as
low-rank approach. Above work mainly focus on a light approximation of self- celf-attention or using sparse attention, which is indeed worse than the full attention [0).. Recent work like FlashAttention (20] and others (62 [4]
using chunked computation scheme and 10-aware mechanism to be memory-efficient and gain full ability like self-attention. Another lines of work try merge RNN and Transformer, e.g. Transformer- XL [19] proposed segment-level recurrence mechanism that connects multiple segments and blocks, and now is widely used in most successful LLMs [52] 91, 69). Recently, linear RNNs (88] [54 [33 27]
and its variants (21 [7] are also proposed, but these recurrent ability is designed for 1-d sequence with causal or auto-regressive property, not fit well for image recognition. To fit longer sequence,
better positional embeddings like RoPE, ALiBi, etc. 666 [58 14 are also proposed. Different to these work focus on NLP task, here in this paper we try to build an efficient Transformer for WSI analysis. which is a unique challenge in vision task.
3 Method
3.1 Preliminary: Attention-based WSI Analysis
Given WSI X as input, the goal is to make slide-level prediction Y by learning classifier fX; 0). X is firstly patched into a long sequence of small instances X = [Z,,.., In,) because of its extremely large resolution, where n is the number of instance. The slide-level supervision Y is given by doctors who consider the latent label y; of all instance T4. Most previous work 6 50 9 try to model this process by Max-pooling operation, so initially, this annotation process is treated as:
(1) Since the end-to-end training from raw image input WSI-level output is infcasible because of large memory cost, conventional approaches convert it into two separate stages: Firstly, convert all small patches into instance embeddings 7 {21, -.-; 2,z } by a pre-trained backbone such ResNet [31] or ViT [79), which refers to general features from public ImageNet, or learned on the related dataset to extract the domain-specific representations 99 36.. Then, aggregate all patches' features within slide and producing the slide-level prediction Y =Z(Z; 0). In this paper, we mainly focus on the latter one, where is an vanilla attention function followed by a linear classifier head as:
Y max maxx(11,-. (y1, Vn
=
a;2)),
(2)
1=1
4

---

where a; is attention weights and o--) is linear head.
However, above vanilla attention method assigning adaptive weight to each instance to make simple summation or pooling can not model the interactions among different instances. Thus, to handle
this problem, Transformer with self-attention is employed in TransMIL. [65] and HIPT [ where the attention sublayer computes the attention scores for the i-th query 4: E RIxd (1 S i S n) in each head, where d is the head dimension. In other words, each instance will compute an attention score list as interactions with all instances. These attention scores are then multiplied by the values to return the output of the attention sub-layer as:
0, = softmax(q, VV,
where the {Q: : qt. K K1 V vz) E R=xd are obtained through linear transform from the input embedding Z, the softmax K") is the attention score and O Raxd is the output. Given O which encodes the interactions among instances, we can further use Equation (2) and input O to replace Z for final prediction, mean-pooling and class token in VIT (79] can also be adopted. Note that here we omit dropout, FFN, residual connection and some detailed blocks in Transformer for simplicity.
Positional Embedding: Since the operation in Equation (3 is position-agnostic, Transformer 722 23 try model contextual interactions by incorporate position information. Absolute positional embedding assigns a positional vector pm to each position m and adds it to the embedding vector the absolute positional embedding (23] for 2-d is employed, while 65 use convolutions as implicit positional embedding [78 but treat data as 1-d sequence.
as: 2; = 2, Pmn.i- In HIPT [
TransMIL.
Relative positional embedding that model the positional difference m- has become popular. Rotary positional embedding (RoPE) [66] encodes the position with rotations: f(gm, = Rm where Ru is a rotation matrix with angles proportional to m. With the rotation's 's property, the query-key product exhibits a positional difference:
flam, m)) n) = qm Rn-mki K
The core idea of RoPE to insert position m, n signal on q, k and reflect the relative position on the newly attention matrix. Though the RoPE is designed for 1-d language sequence, it can also be extended to 2-d paradigm for application on WSI analysis [56].
Computational Complexity: Though above Transformer with self attenttenoo can well model the interactions among instances, its computational cost (n?dd is too heavy for long sequence of WSI due to the interactive attention score calculation (see Appendix A.5.3 for 40x magnification
WSI modelling). Previous WSI Transformer SOTA like TransMIL. 665 and HIPT (9] relieve this problem with different ways: 1) attention approximation: TlansMiL. (65 utilizes Nyströmformer a mechanism employs kernel-based low-rank approximation to approximate full self alffeattention for acceleration. 2) region slicing: HIPT ( utilizes the locality of image by slicing WSI into 4096 4096 squares without overlapping. Given the fixed window size, each square there are fixed 16* 16 256 patches with shape of 256 x 256, thus the computational cost can also be seen as linear complexity.
3.2 Low-rank and Sparsity of Attention Matrix for Long-sequence WSI
Low-rank bottleneck:
Considering all queries in {Q: : 9. the Equation can be seen as:
= softmax (QK
(5)
where the rank of QKT T can be derived as:
r r(Quxa(K)) )axx)
((Kx)))))(((())) min(n, d).
(6)
In the context of WSI analysis, the patched sequence length n of most WSIs is larger than 1024
(Fig. []. while the embedding size d of the pre-trained patch encoder is less than 1024 (e.g. 1024 in Res.Net-50, 384 in ViT-Small, 768 in ViT-Base). Thus, in Equation 6 we have d < 1024 < n, indicating that the rank of the attention matrix in Transformer-bast WSI analysis is constrained by the embedding size d:
r(Qk < min(n, d) d.
(7)
As a result, the representation ability of self-attention is limited by the low-rank bottleneck, thus vanilla Transformer based model in WSI analysis suffers sub-optimal performance. Though the
5

---

non-linear softmax operation can change the rank, but we still observe limited rank of softmax
after training (Fig. Ba+d). This is an extremely different problem compared ViT modelling, e.g. ViT-Small with embedding size of 384 only need to focus on 196 patch tokens (with image size of 224 and patch size of 16), which can model both local contextual and global interactions simultaneously with full-rank attention matrix. We contend that, under this circumstance, Transformer for WSI modelling may become confused when handling local contextual and global interactions with a single layer.
Under this assumption, it also more easy understand the limitation of previous SOTA transformer for WSI: TransMIL (55] with Nyströmformer [83] employs kernel-based low-rank approximation to approximate full self-attention However, it is worth noting that the approximation may produce lower rank of attention matrix compared to basic self-attention, thus resulting lower performance in TransMI... Similar problems also happens in other models like softmax-free linear attention We show lot of experiment of these linear attention model in Appendix [A.5.6]
54
67
85
An intuitive modification handle the low-rank problem is set larger embedding size d. but this makes computational complexity O(n²d) d) more severe, let alone most pathology patch pre-trained foundation models (36]
49 10 carry fixed embedding size. Noting that infeasible to fully represent a matrix with shape of n X n if n >> d given totally 2nd feature points from Qnxd and Knxd, why not focus the attention more important interactions? Thus in the contrast, we alleviate the low-rank bottleneck together with the problem of computational cost by focusing on locality, motivated by the sparse and local pattern in low layers of attention (Fig. 2a).
Sparsity with locality: Here we define the the selection index for retained sparse attention matrix when the softmax probability is greater than a threshold:
I = where
softmax (
(QK') T)
(8)
where the threshold r is normalized by sequence number n, c.g. T = 0.0001/n. Then, the sparsity ratio can be denoted as:
len(I)
n
We note that there is an obvious sparsity and local pattern in lower Transformer layers (Fig. 2h under this protocol. Given the learned locality and sparsity, introduce to learn local contextual information with addable local attention mask for A = softmax(QK") (without loss of generality, here we mainly derive on 1-d sequence for simplicity):
maski.j = -{-inn
iin ii 5l > b otherwise,
(10)
where b is the local band width. Then the attention matrix is converted
( if i- b.
Aij softmax (QKT + mask)
(11)
pp otherwise,
where 0 < <p < is the softmax probability.
We claim that our proposed sparse local attention capture both higher rank and reduced computational complexity, which will work better for WSI modelling:
1) Higher rank: is easy to prove that the band matrix A in Equation I1 is of a lower-bound of rank as n - b. Here we give intuitive verification with a small band matri::
T011
d12
013
014
0
0
0
021
A22
023
425
0
0
0
031
A32
033
034
4g5
036
0
0
Q.11
a.12
043
4.45
Q416
0.17
0
0
km=0,603>>>
a52
053
054
455
056
055
058
(12)
0
063
064
065
066
467
068
(69
074
(17
076
077
078
079
0
0
0
0g5
086
4g7
0.8
039
0
0
096
0g7
0g8
(g9

---

Then, let's consider the lower-left sub-matrix ranging from (6 (6+1,1) to (n, - b):
a(b+1)1
1(++1))
((n+))(m--))
0
((442)2
((6+2)(n-b)
Asub
(13)
an(n-b)
which is apparently a upper triangular matrix with full rank of n b. Thus, we have:
rank(A) 2 rank(Asus)
(14)
Since >> 1024 >>>>>>>>> b practically, our model with higher rank carry stronger representation ability which can focus more on local contextual information.
Given larger n but fixed small b in Equation there will be a lot of zeros upper-right and lower-left areas. We note that these zeros can be omitted during attention matrix calculation by chunking method [20). Since most operations in softmax K can be skipped, the modified complexity O(brd) will linearly related to sequence length and band-width, which is heavily reduced compared to O(nd)).
2) Reduced computational complexity:
3) Extrapolation ability: Despite above advantages, here further show that model tackle WSIs with varying input shape better, in other words, extrapolation ability. RoPE need to be well trained fine-tuned
unseen seldom seen longer length 45 13 81]. Another strategy Attention with Linear Bias (ALiBi) adds pre-defined bias term after the query-key dot-product attention matrix before softmax. For the original 1-d ALiBi [58], the bias static, non-learned matrix softmax(g:k ( We also introduce 2-d ALiBi by 2-d Euclidean distance among token positions, and we find that shows similar pattern (Appendix A.3 compared to our 2-d local attention but it needs to focus on all instances.
computed by the distance between tokens from different positions.
OK"
Stage allention (SA)
palcc
patch embeddings
patches
distance (Kr..1a) (ox.))))
sofumax(QK*
Stage-1: feature and preparation
2d positions
Overall-stage: local- global madelling
SA
Figure 3: LongMIL framework for WSI local- -ooabll spatial contextual information interaction and fusion. 1) Preparing patch feature embedding and 2-d positions of WSIs. 2) Performing pairwise computations among all positions within a WSI by local masking as acceleration. 3) Overall local- global forward of the model, where position information need be feed both local (local masking) and global (positional embedding).
3.3 LongMIL framework and implementation
To realize long contextual MIL modelling and better WSI analysis performance, the overall framework (as depicted in Fig. 3 of our method includes 3 stages:
1) Segment and patch WST into instances, then save its corresponding foreground patch feature embedding and 2-d positions for preparation.

---

2) Calculate the local self-attention matrix by the local window mask given position distance. This process finished by trunk method like FlashAttention 20 to omit non-masking areas.
3) After multi layers (two as default) of local attention focusing on local contextual information interactions, a pooling function with window size 2 2 is employed to reduce token number by 4 times. Then basic self-attention focus on global interactions computed to get final feature and prediction.
4 Experiments
In this section, we present the performance of the proposed method and compare it with various baselines. Ablation experiments are performed to further study the proposed method, for paper length, more experimental results presented in the Appendix (A.5)
Datasets and Tasks. We use four datasets evaluate our method for both tumor subtyping and survival prediction. For data details and pre-processing, please see Appendix A.4
Table 1: Slide-Level Tumor Subtyping on BRACS by using two pre-trained embeddings. Top Rows. Various WSI-MIL architectures with vanilla attention (no interaction among different instances). Bottom Rows. TransMIL (using Nyströmformer and learnable absolute position embedding), full attention (+RoPE) and our LongMIL.
BRACS tumor subtyping
VIT-S Lunit [36
VIT-S DINO (our pre-train)
Method
Fl
AUC
FI
AUC
KNN (Mean)
0.503+0.011
=0.007
0.43011.029
0.649#0.008
KNN (Max)
0.472:100009
0.771 +0.018
0.416:10.019
0.645
Mcan-pooling
0.006
=0.017
0.487+10004
0.717
Max-pooling
0.649+0.032
0.843=t0.01
0.508:
0.81 00000
AB-MIL 322
0.668
0.866=0.016
0..214
0,837
DS-MIL [40]
0.607
0.824
0.622++0063
0.808+0.033
CLAM-SB [50)
0.647 10.020
0.836
0.627 10.032
0.806
DTFD-MIL MaxS
89
0.597 10.025
0.874 0.0026
0.521 100059
0,807 10.016
DTFD-MIL AFS 89
0..08
0.869 100018
0.538
0.824
TrLnsMIL 65
0.648
0.835
0.591 10.049
0.798
Full Attention
0.689
0..100
0.648 +00028
0.83940.00
LIngMIL. (ours)
0.706:10002
0.888
0.65710.026
0..48
Pre-training Patch Encoders. Our work mainly focus on the WSI-head results based on some good pre-trained encoders for histopathology including HIPT (9, Lunit (36] and newly foundation models like UNI [10] and GigaPath [84). We also include ResNet-50 pretrained in ImangeNet- 1k
and ViT-small pretrained BRACS patch data by ourself with DINO 7
Implementation Details. We train our model with PyTorch on RTX-3090 GPU, with a WSI-level batchsize of 1, learning rate of le-4, and weight decay le-2. We add positional encoding into the framework, please check our code for details.
4.1 Slide-level Tumor Subtyping
Evaluation Metrics. For all the experiments, we report the macro-AUC and macro-Fl scores since all these dataset suffering class imbalance. For TCGA-BRCA, perform 10-fold cross-validation with the same data split adopted HIPT [9]. Besides, the dataset BRACS is officially split into training, validation and testing, thus the experiment is conducted 5-times with different random seeds. The mean and standard variance values of performance metrics are reported for multi-runs or cross-validation runs.
Baselines for Comparison. We first show the results of Mean-/Max- pooling and KNN for traditional evaluation. Then we directly evaluate several classical WSI-MIL methods, including AB-MIL (32]

---

DTFD-MIL 89). Then we compare our method with Full Attention (RoFormer) and TransMIL 65). We omit HIPT for BRACS since need WSI larger than threshold and should based on their pre-trained backbone.
DS-MIL AO) CLAM |50,,
Results Analysis: For BRACS 3-categories tumor subtyping, the results reported in Table 1 We can first observe that both Full Attention and our LongMIL show improvement respectively, For Full Attention, attributing to its full self-attention for pairwise interaction ability, it shows better performance compared to all vanilla attention modules [32 40
50 and especially TransMIL 665
which use attention approximation, but is not quite stable to beat DTFD [9)]
For TCGA-BRCA c-attggois tumor subtyping, we show the results in the Appendix A.5.1
4.2 Slide-level Survival Prediction
Table 2: Slide-Level Survival Prediction based HIPT [9] pre-trained embedding with various WSI-MIL architectures including vamilla attention, GCN, TransMIL, self-attention (HIPT with region slicing and absolute embedding), full self-attention and our LongMIL.
Method
COADREAD
STAD
BRCA
AB-MIL
32
0.566=00075
0.562+0.049
0.549
AMISL
86
0.561 +0.088
0.563+0006
0.545=0.071
DS-MIL
40
0.4
0.546#0.047
0.548=0.058
GCN-MIL 43
0.538 0.049
0.513 :0006
HIPT [9]
0,608 +0.088
0.570
TransMIL 65
0.597
0.564
0.587 0.063
Full Attention
0.603
0.568
0,601 00047
LongMII. (ours)
0.624
0.589.1..06
0.619 0.005
Evaluation Metrics. For all the experiments, C-Index scores are reported for the 3 datasets. We follow the data splits and pre-trained patch embedding provided in HIPT [ for fair comparison. The performance results are also reported via the mean and standard variance values of performance
metries by multiple folder cross- voss-validation with the same running setting to HIPT 9
Baselines for Comparison. For this task, we use the survival cross-entropy loss proposed by Zadeh et
al. [87]. The results are summarized in Table where we directly evaluate several survival prediction
WSI-MIL methods, including AB-MIL 32 AMISL [86]. DS-MIL [40] GCN-MIL 43 Then
we compare our method with some state-of-the-art combining position embedding on Transformer: TransMIL (65) and HIPT (9. Though our method show some improvement, the C-index score is still too low daily clinical usage depending on only WSI information. In the near future, we would like to investigate more on this task, e.g. combining multi-modall features as used in [[2] since
Transformer also born with great ability on multi- mudti-modality fusion [9
63
4.3 Evaluation on Pathology Foundation Models
have been emerging as strong patch encoders, we here further provide evaluations based on PathFMS including UNI (TO) and GigaPath (84). The pre-processing procedure the same previous sections. Since the WSI params are pre-trained in GigaPath, we also experiment it using random initialization for fair comparison. For the mismatch of UNI patch encoder and GigaPath WSI head, we add LinLinear layer feature projector. The results is shown in Table 3 we find that our method also show consistency improvement with PathFMs. Furthermore, we find that the pre-training plays a key role to the success of Prov-GigaPath WSI-head, since transformers are much more parameterrize than previous simple attention-based MIL. However, the WSI-level pretrained model relies on patch encoder (e.g. UNI patch encoder + GigaPath WSI do not show competing result). We also provide more difference analysis of efficient attention mechanism compared to GigaPath WSI head in Appendix A.6 In table we also include survival prediction based on PathFMs.
Since recent Pathology Foundation Model (PathFMs) 10
48
84

---

Table 3: Slide-Level Tumor Subtyping BRACS based Pathology Visual Foundation Models.
BRACS tumor subtyping
UNI [10
GigaPath
84
Method
F1
AUC
F1
AUC
AB-MIL 32
0.692
0.875
0.640.1.002
0.837100010
CLAM-SB
[00
0.6401..057
0.844 10,025
0.62410.003
0.826110.04
DTFD-MIL
89
0.655
0.878
0.610
0.843
TransMIL 65
0.502
0.859 =00023
0.599
0..38
Full Attention
0.705
0..04
0.663-10.023
0.850
GigaPath-random init
0..48100044
0.837
0.627
0. 00.08
GigaPath-pretrained
0.668100022
0.861 100030
0.677
0.862.10004
LIn (ours)
0.728
0.887
0.673++0.023
0,856+00015
Table 4: TCGA-BRCA Survival Prediction based on Pathology Visual Foundation Models.
Method
UNI
10
GigaPath
[4
AB-MIL
32
0.630+0.054
0.635=0.033
AMISL
86
0.627 1.0080
0.620.1..040
DS-MIL
440
0.61640.003
0.612
TransMIL
0.598
0.599
Full Attention
0..08
0.61 7.006
LongMII. (ours)
0.656:10.061
0.645
4.4 Further Experiments and Ablations
We also provide abundant ablations in Appendix A..5. to select best setting including: Transformer blocks and multi-head number, dropout ratio, weight decay, learning rate and the local window size. For linear attention result, please check Appendix 6A.5.6 with main findings that the linear attentions show low performance like TransMII. since it get low-rank problem more easily. For linear RNN method like Mamba, the result also relatively lower than Transformer since 2-d WSTs do not hold causal property like 1-d data. For extrapolation validation, please check Appendix A.2 where method show significant performance improvement (p-value 0.1). We also find that our method show consistency improvement when equipped on different magnification (e.g. 40x) and patch size (224- -> 448) as shown in Appendix A.5.4
Conclusions and Limitations
In conclusion, our work introduces advancements in computer-aided diagnosis for histopathology WSI analysis. By analyze the low-rank bottleneck and sparsity property and proposing a local-global hybrid Transformer model, our method, Long-contextual MIL (LongMIL), demonstrates superior performance in handling large and shape-varying WSIs. The evaluations across various tasks highlight its accuracy, extrapolation ability, and efficiency compared previous methods. Our contributions enhance WSI analysis and provide valuable insights for future research. The LongMIL has two limitations: First, its application is restricted to very large image via Transformer modelling. Second, limited embedding size is adopted for practical aim and fair comparison, which is the key to stronger performance based on the low-rank assumption. Future work will aim address these limitations.
Acknowledgements
This study was partially supported by Zhejiang Provincial Natural Science Foundation of China (Grant no. XHD23F0201), the National Natural Science Foundation China (Grant no. 92270108), and the Research Center for Industries of the Future (RCIF) Westlake University.
10

---

References
[1] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- long-document transformer. Proceedings of EMNLP, 2020.
[2] Benjamin Bergner, Christoph Lippert, and Aravindh Mahendran. Iterative patch selection for high-resolution image recognition. In The Eleventh International Conference on Learning Representations, 2023.
[3] Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Low-rank bottleneck multi-head attention models. In International conference on machine learning, pages 864-873. PMLR, 2020.
[4] Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio, Giosuè Scognamiglio, Guillaume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, Maria Gabrani, Florinda Feroce, and Maria Frucci. Bracs: A dataset for breast carcinoma subtyping in h&e histology images, 2021.
[5] Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester van Boven, Robert Vink, al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine, 28(1):154-11 163, 2022.
[6] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, (010):101-1109 2019.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. CoRR, 0os/2104.14220 2021.
[8] Tsai Hor Chan, Fernando Julio Cendra, Lan Ma, Guosheng Yin, and Lequan Yu. Histopathology whole slide image analysis with heterogeneous graph representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15661-156771 2023.
[9] Richard J. Chen and et al. Scaling vision transformers to gigapixel images via hicrarchical self supervuee learning. In CVPR, pages 16144-16155, June 2022.
[10] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general- purpose foundation model for computational pathology. Nature Medicine, 30(3):850-863 2024.
[11] Richard J. Chen, Ming Y. Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y. Chen, Drew F. K. Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks, 2021.
[12] Richard J. Chen, Ming Y. Lu, Wei-Hung Weng, Tiffany Y. Chen, Drew F.K. Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015 4025, October 2021.
[13] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. /2009, 2023.
[14] Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky. Kerple: Kerne-- ized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386-8399, 2022.
[15] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 10055 2019.

---

[16] Krzysziof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for proteins via linearly scalable long-context transformers. Proceedings of ICLR, 2020.
[17] Yufei CUI, Ziquan Liu, Yixin CHEN, Yuchen Lu, Xinyue Yu, Xue Liu, Tei-Wei Kuo, Miguel R. D. Rodrigues, Chun Jason Xue, and Antoni B. Chan. Retrieval-augmenter multiple instance learning. In Teirty-seventh Conference on Neural Information Processing Systems, 2023.
[18) Yufei CUI, Ziquan Liu, Xiangyu Liu, Xue Liu, Cong Wang. Tei-Wei Kuo, Chun Jason Xue, and Antoni B. Chan. Bayes-MII.: A new probabilistic perspective on attention-base multiple instance learning for whole slide images. In The Eleventh International Conference Learning Representations, 2023.
[19] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL, 2019.
[20] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficiee exact attention with 10-awareness. In Advances in Neural Information Processing Systems, 2022.
[21] Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052, 2022.
[22] Yihe Dong, Jean- Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, 2023.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[24] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint MiXiv:2012.09699 2020.
[25] Olga Fourkioti, Matt De Vries, and Chris Bakal. CAMIL: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In The Twelfth International Conference on Learning Representations, 2024.
[26] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprini arXiv:23/2.00755 2023.
[27] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022.
[28] Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang, Pei Dong, Jinxi Xiang, Wei Yang, Junzhou Huang, Yuyao Zhang, and Xiao Han. Node-aligned graph convolutional network for whole- slide image representation and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11881318823,, 2022.
[29] Dongehen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention, 2023.
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CoRR, 116/2111100677, 2021.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[32] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-basel deep multiple instance learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2127-2136. PMLR, 10-15 Jul 2018.
12

---

[33] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Car- reira. Perceiver: General perception with iterative attention. arXiv preprint arXiv:2103.03206, 2021.
[34] Hanhwi Jang, Joonsung Kim, Jac-Eon Jo, Jaewon Lee, and Jangwoo Kim. Mnnfast: fast and scalable system architecture for nneorry-augmeet neural networks. In Proceedings of the 46th International Symposium on Computer Architecture, pages 250-263, 2019.
[35] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, and aa- ditya prakash. Additive MIL: Intrinsically interpretable multiple instance learning for pathology. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[36] Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira. Benchmarking self-supervised learning diverse pathology datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3344-3354, June 2023.
[37] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. arXiv preprint 1022266000.162.66 2020.
[38] Nikita Kitaev, tukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0444 2020.
[39] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pages 201-216, 2018
[40] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14318-14328. 2021.
[41] Honglin Li, Yusuan Sun, Chenglu Zhu, Yunlong Zhang, Shichuan Zhang, Zhongyi Shui, Pingyi Chen, Jingxiong Li. Sunyi Zheng, Can Cui, et al. Large-scale cervical precancerous screening via ai-assisted cytology whole slide image analysis. arXiv preprint arXiv:2407. 195.1.9......... 2024.
[42] Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottleneck for weakly-superriied pathology whole slide image classification. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1464-7463, June 2023.
[43] Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li, and Junzhou Huang. Graph enn for survival analysis on whole slide pathological images. International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 174- 182. Springer, 2018
[44] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J. Chen, Zi- hao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-PPilippe Moreney. Quantifying & modeling multimodal interactions: An information decomposition framework. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[45] Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209 2023.
[46] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXhv:2401.10166, 2024.
[47] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
13

---

[48] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guil- laume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, al. A languagg foundation model for computational pathology. Nature Medicine, 30(3):863-877 2024.
[49] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Andrew Zhang, Long Phi Le, et al. Towards a visual-language foundation model for computational pathology. arXiv preprint arXiv:2307. 12914, 2023.
[50] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology whole-slide images. Nature Biomedical Engineering, (66):555-770 2021.
[51] Oded Maron and Tomás Lozano-Pérez. A framework for multiple- instance learning, Advances in neural information processing systems, 10, 1997.
[52] OpenAL. Gpt-4 technical report. arXiv, 2023.
[53] Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.066349 2023.
[54] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305. 13048, 2023.
[55] Nicholas A Petrick, Shazia Akbar, Kenny HH Cha, Sharon Nofech-Mozes, Berkman Sahiner, Marios A Gavrielides, Jayashree Kalpathy-Cramer Karen Drukker, Anne LL Martel, et al. Spie- aapm-nci breastpathq challenge: an image analysis challenge for quantitative tumor cellularity assessment in breast cancer histology images following neoadjuvant treatment. Journal of Medical Imaging, 8())003501,, 2021.
[56] Etienne Pochet, Rami Maroun, and Roger Trullo. Roformer for position aware multiple instance learning in whole slide image classification, 2023.
[57] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302. 10866, 2023.
[58] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
[59] Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang. and Jie Tang. Blockwise self-attention for long document understanding. arXiv preprint 1rXiv: 1.0992, 2019.
[60] Linhao Qu, xiaoyuan Luo, Kexue Fu, Manning Wang, and Zhijian Song. The rise of AI language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. In Seirtt-seventh Conference on Neural Information Processing Systems, 2023.
[61] Linhao Qu, xiaoyuan Luo, Manning Wang, and Zhijian Song. Bi-directional weakly supervised knowledge distillation for whole slide image classification. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[62] Markus N. Rabe and Charles Staats. Self-attention does not need o(n²) memory, 2022.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[64] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Proceedings of TACL, 2020.
14

---

[65] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, and yongbing zhang. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1136-2147, Curran Associates, Inc., 2021.
[66] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[67] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
[68] Katarzyna Tomezak, Patrycja Czerwińska, and Maciej Wiznerowicz. Review the cancer genome atlas (tega): an immeasurable source of knowledge. Contemporary Oncology/sspólczeenn Onkologia, 2015(1):68-77) 2015.
[69] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[70] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558, NIH Public Access, 2019.
[71] CHAO TU, YU ZHANG, and Zhenyuan Ning. Dual-curriculum contrastive multi-instance learning for cancer prognosis analysis with whole slide images. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[73] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Proceedings of NeurlPS, 2020.
[74] Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chur Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. Cluster-former: Clustering- based sparse transformer for long-range dependency encoding. Proceedings of ACI-IJCNLP (Findings). 2020.
[75] Sinong Wang. Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 047688 2020.
[76] Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, and Furu Wei. When an image worth 1,024 x 1,024 words: case study in computational pathology. arXiv preprint arXiv:2312.03558 2023.
[77] Xiyue Wang, Jinxi Xiang, Jun Zhang, Sen Yang. Zhongyi Yang, Ming-Hui Wang. Jing Zhang, Yang Wei, Junzhou Huang, and Xiao Han. SCL-WC: Cross-slide contrastive learning for weakly-supervised whole-slide image classification. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[78] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEF/CVE international conference on computer vision, pages 22-31, 2021.
[79] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In 2021 IEFF/CVF International Conference Computer Vision (ICCV), pages 10013-10021, 2021.
15

---

[80] Jinxi Xiang and Jun Zhang. Exploring low-rank property in multiple instance learning for whole slide image classification. In The Eleventh International Conference on Learning Representations, 2023.
[81] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint vrXiv:2309.17453 2023.
[82] Ronald Xie, Kuan Pang, Sai W Chung, Catia Perciani, Sonya MacParland, BO WANG, and Gary Bader. Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[83] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[84] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pages 1 1-8, 2024.
[85] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:23/2.06635, 2023.
[86] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
[87] Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in cross-entropy-baae based training of deep survival networks. IEEE transactions on paltern analysis and machine intelligence, 13(9):3126- 3137, 2020.
[88] Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer, 2021.
[89] Hongrun Zhang, Yanda Meng. Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E. Coupland, and Yalin Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. ArXiv, abs/2203. 120081 2022.
[90] Jingwei Zhang, Saarthak Kapse, Ke Ma. Prateek Prasanna, Joel Saltz, Maria Vakalopoulou, and Dimitris Samaras. Prompt-mil: Boosting multi-instance learning schemes via task-specific prompt tuning, 2023.
[91] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106. 2022.
[92] Yunlong Zhang, Honglin Li, Yuxuan Sun, Sunyi Zheng, Chenglu Zhu, and Lin Yang. Attention- challenging multiple instance learning for whole slide image classification. arXiv preprint arXiv:2311.07/1 125,222522 2023.
[93] Yunlong Zhang, Zhongyi Shui, Yunxuan Sun, Honglin Li, Jingxiong Li, Chenglu Zhu, Sunyi Zheng, and Lin Yang. Adr: Attention diversification regularization for mitigating overfit- ting in multiple instance learning based whole slide image classification. arXiv preprint arXiv:2406. 15303, 2024.
[94] Yunlong Zhang, Yuxuan Sun, Honglin Li, Sunyi Zheng, Chenglu Zhu, and Lin Yang. Bench- marking the robustness of deep neural networks to common corruptions in digital pathology. In International Conference on Medical Image Computing and Computer-Assistee Intervention, pages 242-252. Springer, 2022.
16

---

A Appendix
A.1 Why linear complexity important
We find that FlashAttention 20 using memory-efficient trucking and hardware 10-aware opera- tions is good enough in both memory and speed to cope with 20x magnification (about twice slower) WSI model (as result we use Full Attention as the last global attention layer of our hybrid Long- MIL model for 20x magnification, we also provide replacement version using linear attention as last layer as shown Table 9 signed as MongMI++V-Mambil However it unacceptable in dealing with 40x magnification (about 30-times slower in BRACS) which takes us 2 days more to train 5-fold runs models on BRACS, and takes longer stacking more layers and on larger WSI with more rounds (e.g.average over 50000 instances of TCGA-BRCA dataset with double WSI num and traditionally using fold-cross validation). This hinders the improvement of 40x magnification which includes more useful details) for both development and deployment. Thus use LongMIL+ M-mmmba (bybrid transformer as local+local+Hinean global attention) in 40x and also get strong performance as shown in Table left-column, which is faster in speed and comparable in performance compared to FlashAttention.
A.2 Extrapolation: Train small but test large
We first split BRACS dataset into training (small images) and validation+testing (large image) by sorting them via instance number and then use train-val-test ratio as 6:2:2. The experimental results plot in lower-right area of Fig. 4
A.3 HIPT Region Slicing, Local-mask Matrix and 2-d ALiBi
In Fig. we show the difference and similarity between HIPT region slicing, local-mask matrix and 2-d ALiBi, where our local mask can be seen generalization of HIPT and 2-d ALiBi.
A.4 Details of Datasets
For the slide-level tumor subtyping performance, our method is evaluated on two datasets:
BReAst Carcinoma Subtyping (BRACS) 4 collect H&E stained Histology Images, containing 547 WSIs for three lesion types, i.e., benign, malignant and atypical, which are further subtyped into seven categories. Here, since the WSIs number is limited, we only perform three class subtyping. The WSIs are segmented in 20 magnification and non-overlapping patching with 224 x 224 size. The Cancer Genome Atlas Breast Cancer (TCGA (TCGA-BRCA) [68 55) is public dataset for breast invasive carcinoma cohort for Invasive Ductal Carcinoma versus Invasive Lobular Carcinoma subtyping. The WSIs segmented into non-overlapping cistaincn patches with a size of 256 x 256 (keep consistency to previous work [9] at 20 magnification patches were curated from 1038 WSIs. For the slide-level survival prediction, despite I'CGA-BRCA we further includes 2 TCGA histology datasets: 1) A combination dataset of the Colon adenocarcinoma and Rectum adenocarcinoma Esophageal carcinoma (TCGA-COADREAD) Stomach adenocarcinoma (TCGA-STAD dataset including 321 WSIs. For pre-processing, we using the implementation of CLAM [50] which mainly includes HSV, Blur, Thresholding, and Contours methods to localize the tissue regions in each WSI.
which includes 316 WSIs as used in HIPT 9 2)
A.5
Further Experiments and Ablations
A.5.1 TCGA-BRCA 2-categories tumor subtyping
The results are reported in Table and right column of Table 6 We could observe a significant improvement our method when using HIPT pre-trained patch embeddings. but only a slight improvement with Lunit. This could be because this task reaches an upper bound with the high- quality Lunit embeddings. Given that only predicts binary categories, even simple max -poling can outperform almost all previous MIL methods.
17

---

x10*
WSI 2d-shape histogram
8
4
2
0
x10*WSI 2d-foreground histogram
8
6
2
0
5
10
15
x10*
ANOVA p-value: 0. 0.1171
0.75
0.70
area needs extrapolation
F1-score
40
0.60
20
0-
100
150
200
LangMIL
Figure upper left: The WSI fore-ground shows irregularity (inner the green line). upper right and lower left: The 2-d position index of WSI foreground patches mainly scattered within index < 100, thus area enclosed by the dashedline suffers under-fitting with previous method. lower right: TransMIL and full self-attention (FSA) get relatively low performance during testing on unseen larger WSI. Assisted by our method, this case show significant performance improvement (p-value near 0.1).
18

---

121
121
121
12
6.20
5.01
230
3:.0
2.00
1.46
222
2..00
4.00
5.80
0.10
2.08
3.00
1:04
6.00
0.09
0.00
308
1.00
2.44
1.00
128
3.00
1.80
2.01
640
1000
Figure 5: Difference and similarity between various methods. upper left: HIPT slicing with extremely hard pattern, upper right: our proposed local mask, lower left: 2-d ALiBi, or 2-d Euclid distance, lower right: attention mask of Prov-GigaPath from their paper (their causal attention, only focus on lower triangular matrix, may be a drawing problem). Apparently the local mask of Prov-GigaPath mainly focus 1-d interactions (weigh x-axis of WSI more than y-axis). e.g. the interactions when distance less than 2.0 are almost missed, as depicted the red text areas of the lower-left 2-d Euclid distance subfigure. We have checked their code implementation, which directly apply 1-d LongNet to the serialized (via z-scan) patch sequence.
19

---

TCGA-BRCA tumor subtyping
VIT-S Lunit 36
VIT-S HIPT
9
Method
F1
AUC
F1
AUC
KNN (Mean)
0.669
0.821 +0.038
0.585
0.742:+0.06
KNN (Max)
0.657
0.799
0.516110.033
0.69110016
Mean-pooling
0.841 100050
0.934
0.731 1:0.09
1886710.077
Max Max-pooling
0.849 10.051
0.949 +0,022
0.688.10.074
0.826110058
AB-MIL 32
0.820=000037
0.922=00.02
0.757=+0.069
0.873+0.036
DS-MIL 40
1.841:0.047
0.925=0.02
0.723++0.068
0.834
CLAM-SB [50]
0.850=10.039
0.942=00020
0.733+t0.057
0.86 #0.04
DTFD-MIL MaxS 89
0.802::0.04
0.91 =0.031
0.678+t0.062
0.781 #00067
DTFD-MIL AFS 89
0.843
0.93 10.01
1..04
0.85 ::00066
TransMIL 66
0.024
0.933
0.715
0.840
HIPT 9
0.752:+0.042
0.874
Full Attention
0.843
0.044
0.758+t0.046
0.852+00.046
LongMIL (ours)
0.845+0.046
0.953
0.762:+0.04
0.880=+0.44
Table 5: Slide-Level Tumor Subtyping TCGA-BRCA.
A.5.2 ResNet-50 ImageNet pre-trained embedding results of tumor subtyping
The results experimented in Table
ResNet-50 ImageNet pre-trained embedding
BRACS
TCGA-BRCA
Method
Fl
AUC
F1
AUC
Mean-pooling
0,483+00018
0.710==0.04
#..099
00.066
Max-pooling
0,495++0018
0.763-20.005
0.780=000027
0,886=10.301
AB-MIL
0.553 10.003
0.752:10.005
0..60
0.85110.057
DS-MIL 40
0.564+10.37
0.779
0.79710.033
0.89411.0229
CLAM-SB
[50]
0.548 +0,026
0.769
0.709
0.87810.027
TransMIL
65
0.500
0.734
10.11
0,85411.055
Full Attention
0.544
0.775.10.018
0.800+0.0014
0.901+0.014
LongMIL. (ours)
0.591 100084
0.810.10003
0.781+00004
0.919
Table Slide-Level Tumor Subtyping on BRACS and TCGA-BRCA based on ResNet-50 embedding pre-trained via ImageNet supervised learning.
A.5.3 Hyper-parameters of Transformer training
Number of Transformer blocks and multi-head, bias slope coefficient and local window size, weight decay and dropout ratio: Here we include following hyper-parameters for our results on BRACS with ViT-S patch embedding pre-trained by [36]: Transformer blocks and multi-head number, dropout ratio, weight decay, and learning rate, finally the local window size. Due to time-consumption we fixed other hyper-parametto
when ablation on selected variant (The default setting Transformer local blocks number 2, multi-head number = I, dropout ratio 0,0, weight decay = 1c-2, and learning rate le-4, local-window size 10 (radius)), the details can be found Fig.
A.5.4 Multi-scale and magnification
There are large differences in speed and performance for 20x and 40x magnification, since FlashAt- tention (0) will quite slow if given over 20k instances compared linear attention our local attention. For performance and speed please check Table 7 and Fig. 7b, respectively.
20

---

ablation an Transformer blocks num
ablation Transformer multi head nun
AUC Fccco
AUC
Fl score
0,9
0.9
0.7
0.7
0.6
0.6
Tranformer brocks
head
ablation wei decay
dropout
1.0
AUC
AUC
0.8
0.8
0.7
0.5
0.5
0.005
0,0001
weight decay ratio
diopout
ablation on learning rate
ablation on window size (radius)
F1-score
F1-scare
0.9
0.9
0.8
0.6
0.01
0.005
0.003
0.0005
0.0001
Se 05
1e:05
10
leaming
local attention window size, radius
Figure 6: Ablations results on BRACS with ViT-S Lunit
36
patch embedding.
BRACS tumor subtyping
40x
20x
Method
F1
AUC
F1
AUC
AB-MIL 32
0.610
0.81110.013
0.668
0.866:100016
TransMIL 65
0.506
0.777
0.648 100054
0.835
Full Attention
0.618
0.831
0.689.10.036
0.870:10.010
LongMIL (full global)
0.624
0.842 0.022
0.706:10.025
0..08
LongMIL (linear global)
0.622
0.835 20.026
0.693
0.870.10.06
Table 7: Ablations on magnification (40x and 20x) BRACS tumor subtyping.
21

---

We also experiment on larger patch size (from 224 to 448, Table 8 to decrease the overall token number, but find that our method still shows stronger performance.
Simple attentions (ABMIL,, CLAM without pair wise interactions) gain improvement, and we speculate that the larger mage--izz can modelling the local context better.
DTFD try to split the whole bag into sub-bags, but smaller bag size may result in larger label noise of sub-bags which may answer its performance drop.
The gap between LongMILL and TransMIL decreases given closer and d. Full attention and LongMIL show small drops, since less interactions can be modelled with less patches.
4. LongMIL still out put-performs full attention. We speculate that local attention also works better when dealing with the shape-varying WSI even with less n.
BRACS tumor subtyping
224
448
Method
F1
AUC
F1
AUC
AB-MIL 32
0.692=0003
0.875
0.695
0,875
CLAM-SE 50
0.640=0.06
0.844
0.654
0.85 10.02
DTFD-MIL 89
0.655=0.03
0.878:0.00
0.625
0.839
TransMIl 665
0.592210044
0.859 10.02
0.646
0.855
Full Attention
0.715=0.0
0.884
0.700
0.874
LongMIL
0.728 0.05
0.887 1.000
0.722
0.883
Table 8: Ablations patch size (224 and 448) in BRACS tumor subtyping based UNI feature.
A.5.5
Memory efficcincy and speed
We show the memory efficiency and speed various transformer structures in Fig.
A.5.6 Linear Attention
We provide ablation on different linear attention e.g. RetNet, GLA (67 85 and linear RNN structure like Mamba [26] to uncover their advantages and limitations in WSI analysis. As shown in Table [ we first show the results of these vanilla Linear attention or RNN directly as MIL model (first row), but none of these methods can compete with Full Attention in performance. Then, we combine these modules into our LongMIL to replace its last global attention layer and we observe that this can provide us strong performance as well as linear complexity in total (2 layers of local attention + 1 layer of linear attention, better than two layers of Full Attention both speed and performance).
A.6 Detailed comparison to Prov-GigaPath
1. The motivation /contribution: our paper not only focus proposing an efficient self-attention mechanism for WSI, but also showing analysis on why some previous work like Roformer and TransMIL fail for WSI from the low-rank perspective, which we believe to be insightful to the digital pathology community. However, both the Prov-GigaPath [84 and Long ViT 76 focus on scaling up large-scale of data with pre-training, which is more empirical. We believe that our analysis may also work for Prov-GigaPath and could be one potential explain on why Prov-GigaPath success and how to improve further.
2. The method details: Prov-GigaPath does not treat interactions inside x-axis and y-axis equally, though the 2-d positional embedding is applied. By putting all patches into a 1-d sequence in 'z-scan' manner like VIT, their 1-d local attention focus more on x-axis but less on y-axis, as depicted in Fig. [S Although this be alleviated by their higher-level dilated attention term, the x-y inequality still exists. Whereas, our local-attent attentionnion is designed for 2-d (based on 2d Euclid distance), thus treat them equally.
The pretrained Pat-GGia WSI-head seems relying heavily on their own patch-pretrained encoder, which may be potential barrier to wide usage, e.g. there are still some cases when GigaPath
22

---

training GPU usage
40
Vanilla SelfAttention
TCGA, BRCCABRCAACC 20x
35
Flash Attention
TCGA-BRCA 40X
Nystomnaamee(Traeell
BRACS 20x
30
BRACS 40x
memory (GB)
25
16
10
10000
20000
30000
40000
50000
60000
70000
sequence length (instance number)
training speed
Vanilla Self Attention
Flash Attention
time consumption
300
Nystronfomerr(Tanskkl
Hybrid Attenlion(LongMML))
100
2000
4000
6000
8000
10000
sequence length (instance number)
Figure 7: Training memory usage and speed using different Attentions. Upper: (depicted as Flash Attention) for self-attention calculation convert memory complexity linear, even better than Nyströmformer. Lower: lash Attention with chunk method still suffers quadratic complexity in speed even with hardware-awarr accelerated operations. Our introduced LongMIL for WSI analysis can convert it as linearity with local-window mask. We also show markers about the max instance number of WSI used in this paper show potentials on higher (e.g. 40x) magnification learning. We omit more instances (e.g. over 50k) speed test since it takes long time, but based on its quadratic complexity of full self-attention, will be about 25 35 times slower than linear attention.
The chunk method
patch features weaker than UNI Conch, posted in the github repo of UNL The WSI pretraining is indeed useful as the key to their superior performance, which covers their problem of spatial inequality on and y. When dealing with the case 'BRACS', shown in the following table, method (even AB-MIL.) with better UNI feature can outperform their 'worse patch feature with stronger pretrained slide encoder'.
23

---

BRACS tumor subtyping
Method
F1
AUC
AB-MIL 32
0.668
0.866:10016
TransMIl. 66
0.648 10.054
0.835
RetNet 66
0.628
0.805=+0009
GLA [85]
0.589+0.032
0.794=#0013
Mamba 26 (random)
0.650=0.024
0.816=10.028
Mamba (single)
0.633 0.0094
0.834:10.037
V-Mamba [66 (cross)
0.642 00000
0,821 +0.008
Full Attention
0.689
0.870=10000
LongMIL (ours)
0.706
0.019:
RetNet
0.690
0.848
GLA
0.667=0.037
0,860=20014
Mamba (random)
0.678 10.044
0.8561100030
Mamba (single)
0.650.100052
0.838 10.027
V-Mamba
0.693
0.870.10.016
Table 9: Experiment Linear Attention and combine it into our LongMIL as hybrid local-local-
linear-atter attention Transformer model.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is clear that the low-rank with sparsity of attention matrix could be the key improve effectiveness and efficiency for Transformer based WSI analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the two main limitations including application scene and current employed limited embedding size considering speed and fair comparison, in the last section of main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: We provide full set of assumptions and complete proof for both low-rank bottleneck and high-rank clear and we also provide a full local band matrix (n = in main paper) supplemental material.
nk of local band matrix. For the latter one, the proof by sub-matrix is
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide comprehensive details of dataset selection and pre- pre-processing. together with training details. We will release full code implementation.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All our data is public and use 'CLAM' for pre-processing. We will release full code implementation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
haaibs..ccpublli//uuuueee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide model backbone selection and implementation details in core paper and further provide data split and pre-processing in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For each experiments we have reported the mean and std via multi-runs or cross-folds. We also show some p-value significance in train small test large experiment in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our memory efficiency method is RTX3090 GPU (24g), detailed in core paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripp: http://cc..pppplctcccccccceeeeee
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and understood the policies, and we believe that neither the manuscript nor the study violates any these.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is no negative societal impact our paper. Our work focus on computer aided diagnosis for potential medical use, which is discussed in conclusions of core paper.
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work mainly focus on recognition for real-world medical pathology image, without any generative problem.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the data and tools are public accessible and well referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
29

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There is no new assets right now.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "R2eXYuqv52", "reviewer": "Reviewer_UUFY", "summary": "This work examines the problem of extrapolating Transformer attention to long sequences in WSI representation learning. The main technical contribution is in examining the low-rank bottleneck problem of Transformer attention for WSIs, and proposing LongMIL which introduces modifications via local attention masking + 2D ALIBI in order to improve the rank of the attention matrix and enable extrapolation capabilities.", "questions": "Would the authors be able to address my concern in updating the results of this work to using a different encoder (to simplify the presentation of results)? Choice of pretrained encoder should not matter significantly (as the main focus is in fairly comparing LongMIL), but having 1-2 comparisons would be nice in a Supplemental Figure.\n\nRating at the time of reviewing this work is slightly negative, but am enthusiastic of this work and would raise my rating to borderline / weak accept if my concerns were addressed.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Core contribution of this work is novel and would have a lot of interest in the CPath community. Extrapolating to long contexts is an exciting problem that has had little investigation (outside of Prov-GigaPath). I believe this work can be re-organized in presenting a more systematic understanding of they key components needed to extrapolating to long contexts.\n- Supplement includes some interesting ideas and ablation experiments. A.4 discusses similarities and differences between HIPT, adding local mask attention, and 2D alibi. A.6.1 includes a comparison with the ViT-S in HIPT for equivalent comparisons. A.6.3 examines different hyper-parameters during Transformer training. A.6.4 looks at the difference between 20X and 40X magnification. A.6.6. ablates other straightforward extensions of Transformer attention with subquadratic complexity (including Mamba and V-Mamba). \n- Figures are illustrative (in both main text and Supplement).", "weaknesses": "- Main limitation of this work is that a comparison with Prov-GigaPath [1] (a concurrent work that appeared at the time of NeurIPS submission) is warranted. Prov-GigaPath also presents overlapping contributions in solving this problem, though I think there is room for more than 1 study examining this problem.\n- The writing feels a bit rushed and informal. I was often scrolling back and forth to understand the different comparisons being made. Ideally, there should be one table for results that compare MIL architectures and one table for ablating pretrained encoders. Other areas where the writing / presentation of figures and results could be significantly polished: \n- - \"We omit the ResNet-50 embedding for survival prediction since it get quite low and unacceptable results.\" Informal and non-scientific. \n- - Section 4.3 should be able to summarize the findings in the Supplement in a more clear and descriptive manner.\n- - One of the main issues I see in this work is that the authors are juggling many different encoders for different tasks. A.6.1 compares ViT-S Lunit and HIPT for TCGA-BRCA subtyping (evaluating multiple MIL models across multiple encoders on 1 task), but A.6.2. shows Resnet-50 features for BRACS and TCGA-BRCA subtyping (evaluating multiple MIL models across multiple tasks with the same encoder). Many of these issues can be drastically simplified if the authors were to use an encoder not pretrained on TCGA such as Prov-GigaPath [1], UNI [2], or PLIP [3] - with the main emphasis on comparing LongMIL with other competing works. Ref [4] provides an example on how the findings of this work can be better organized.\n\nReferences\n1. Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y. and Xu, Y., 2024. A whole-slide foundation model for digital pathology from real-world data. Nature, pp.1-8.\n2. Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M. and Williams, M., 2024. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3), pp.850-862.\n3. Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J. and Zou, J., 2023. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9), pp.2307-2316.\n4. Park, N. and Kim, S., How Do Vision Transformers Work?. In International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UUFY,\n\nWe appreciate your time and valuable feedback. We are glad that you found the formulation and analysis being novel and sound, and the figures and ablations being illustrative. Below, please find our point-to-point response to your comments:\n> **W1: Main limitation of this work is that a comparison with Prov-GigaPath (a concurrent work that appeared at the time of NeurIPS submission) is warranted. Prov-GigaPath also presents overlapping contributions in solving this problem, though I think there is room for more than 1 study examining this problem.**\n\nSince another reviewer Ghwr also points out this problem, we post the detailed comparisons in the general response, where we make systematical comparisons to Prov-GigaPath including:\n\n    d_1. Method: their receptive field weigh more on x-axis than y-axis, however our method as 2D locality treat x-y equally.\n    d_2. Contribution: we focus more on analyzing why previous transformers failed then deriving our method, while they empirically scale up to big data based on dilated attention. \n    d_3. We find that when their patch feature is not the best in some task cases, their heavily pretrained WSI head with problem in 'd_1' only shows sub-optimal performance. \n    We provide some quick experiments to compare their WSI-architecture and our method.\n\n>**W2: The writing feels a bit rushed and informal. I was often scrolling back and forth to understand the different comparisons being made. Ideally, there should be one table for results that compare MIL architectures and one table for ablating pretrained encoders. Other areas where the writing / presentation of figures and results could be significantly polished ...**\n\nWe thank a lot for your suggestions on the presentation of our paper, we will polish it for better reader-experience in next version.\n\n>**W2.3 + Q1: Would the authors be able to address my concern in updating the results of this work to using a different encoder (to simplify the presentation of results)? Choice of pretrained encoder should not matter significantly (as the main focus is in fairly comparing LongMIL) but having 1-2 comparisons would be nice in a Supplemental Figure.**\n\nWe indeed find that there are more and more stronger patch encoders pretrained without TCGA should be validated, which is also highlighted by almost all the other reviewers. So, we also add it into the general response, including both UNI and GigaPath pretrained patch encoders with main findings: \n\n    1. Our method still outperforms previous work for some complicated tasks like BRACS and survival prediction. \n    2. The results in TCGA-BRCA tumor-subtyping are similar for almost all the WSI methods. We speculate that this task as binary classification might have reached some sort of upper limit when equipped with strong patch encoder. We will be going to evaluate our method on more difficult tasks in future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lKorETjRvY", "reviewer": "Reviewer_GgQd", "summary": "The authors point out that that MIL often has insufficient ability to offer accurate slide level classifications.  There is a long (now) history of attempting to better consider sub-slide level context in the aggregation function.  TransMIL, GNN-based methods all have provided attempts to this end.  \n\nThe authors argue that transformer based aggregatioins functions have limited ability to consider both local and global context for tile-attention-rankings.  They provide theoretical arguments for why a \"low-rank bottleneck\" exists because the total number of patch embeddings is much greater than the embedding size.  \n\nThe method addresses this limitation but calculating local self attention followed by pooling function before doing a \"global\" self attention function. \n\nIn main paper experiments are done with BRACS dataset using two different encoders for feature extraction.  The proposed method is benchmarked against cotext aware and non-context aware aggregations functions.  Similar experiment is done with survival predictions.  Extensive additional supporting experiments are in supplement.", "questions": "In final version of paper, can you please improve orientation of figure 3.  It is hard to zoom in sufficiently to understand second panel.  It would also benefit from having sub labels (eg. a, b, c) to improve the legend description of the sub panels.", "limitations": "I concur with listed limitations.  I have pointed out other limitations in the weakness section.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well written and thorough.  The authors are very meticulous about addressing the problem proposed with traditional transformer based aggregators.  The claims are backed by the results.  The supplement provides extensive figures and ablation studies as well as additional details (including memory efficiency studies) on the methodologies.", "weaknesses": "A recent preprint (https://arxiv.org/abs/2407.07841) shows that context aware aggregations functions offer less performance boost over ABMIL when you have a high feature extraction encoder.  In this work the encoders used are less robust than the now publically available encoder (UNI, Gigapath and Virchow). All of these are very recently released so is understanable that they are not part of this submission. \n Going forward these encoders should be used for any aggregation function assessment.   The supplemental table showing that the boost of this method is much stronger with an image net pretrained cnn.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GgQd,\n\nWe would like to express our sincere gratitude for your thoughtful review and insightful feedback on our manuscript. We appreciate your recognition of the thoroughness and approach taken in addressing the limitations of traditional transformer-based aggregators. Your positive evaluation of our work's soundness, presentation, and contribution is greatly encouraging.\n\nBelow, we provide responses to your comments and questions:\n\n>**W1: A recent preprint (https://arxiv.org/abs/2407.07841) shows that context aware aggregations functions offer less performance boost over ABMIL when you have a high feature extraction encoder. In this work the encoders used are less robust than the now publicly available encoder (UNI, Gigapath and Virchow). All of these are very recently released so is understandable that they are not part of this submission. Going forward these encoders should be used for any aggregation function assessment. The supplemental table showing that the boost of this method is much stronger with an image net pretrained cnn.**\n\nWe find this question being very constructive and insightful to the completeness our paper. \nOn the one hand, we have performed some experiments on stronger patch encoder including UNI and GigaPath in this stage (check it in the general response), with main findings:\n\n    1. Our method continues to outperform previous work in complicated tasks such as BRACS and survival prediction, demonstrating a notable performance boost in consistency compared to other methods, with the exception of TransMIL.\n    2. The results in the TCGA-BRCA tumor subtyping task are similar across almost all the methods. We speculate that this task may have reached an upper limit when using a strong patch encoder. In future versions, we plan to validate our method with robust encoders on larger datasets to further explore the potential for improvement.\n\nOn the other hand, motivated by this suggestion, we realize that strong patch encoders with high-level semantics may lose important low-level spatial-context or fine-grained details. Thus, in future work we may be going to aggregate more features from different depth of layers in ViT or different pretrained ViT. This may not only include more detailed spatial contexts, but also helps improving the feature size $d$.\n\n>**Q1: In final version of paper, can you please improve orientation of figure 3. It is hard to zoom in sufficiently to understand second panel. It would also benefit from having sub labels (eg. a, b, c) to improve the legend description of the sub panels.**\n\nThank you for your detailed feedback on Figure 3. We understand that the current orientation and labeling make it difficult to interpret, particularly the second panel. To address this issue, we will take actions to improve the clarity and readability of the figure in the final version of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGxXtxZSjM", "reviewer": "Reviewer_Ghwr", "summary": "This paper focuses on the issue of attention computation for long sequences in WSI (Whole Slide Image) images. The authors first analyze how the low-rank nature of the long-sequence attention matrix constrains the representation ability of WSI modeling. They then propose a method using local attention masks to compute attention within local regions, followed by the computation of global attention. Experimental results demonstrate that the combination of local and global attention computations outperforms full attention.", "questions": "see weakness", "limitations": "no", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The computation of attention for ultra-long sequences is a significant challenge in WSI slide-level feature learning, and addressing this issue is highly valuable.\n2. The paper provides a detailed analysis of the low-rank and sparsity problems in the attention matrix of long sequences, based on which the use of local attention is proposed.\n3. The authors propose the longMIL method, which achieves better results than the baseline in both subtyping and prognostic tasks.", "weaknesses": "1. Although the paper identifies and analyzes the bottleneck issues in long-sequence attention and proposes the use of local attention based on this analysis, there is a lack of innovation in using local attention. Other methods, such as LongViT, LongNet, and Prov-GigaPath, have also used local attention more elegantly. Additionally, the paper does not discuss the potential for directly transferring numerous attention optimization methods from the fields of computer vision (CV) and natural language processing (NLP).\n2. The paper lacks a review of relevant literature, such as the aforementioned works.\n3. In the experimental results, for instance, the classification results in Table 1 and the prognostic results in Table 2 show only a 0.01 improvement over full attention, which is not significant.\n4. Although the paper claims to reduce computational costs, which is evident, it should provide corresponding comparisons to substantiate this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ghwr, \n\nWe appreciate your time and constructive feedback. We are glad that you found that our analysis and method are valuable. Below, please find our point-to-point response to your comments:\n\n> **W1 & W2: Although the paper identifies and analyzes the bottleneck issues in long-sequence attention and proposes the use of local attention based on this analysis, there is a lack of innovation in using local attention. Other methods, such as LongViT, LongNet, and Prov-GigaPath, have also used local attention more elegantly. Additionally, the paper does not discuss the potential for directly transferring numerous attention optimization methods from the fields of computer vision (CV) and natural language processing (NLP).\nThe paper lacks a review of relevant literature, such as the aforementioned works.**\n\nWe have great respect for the significant contributions made by Prov-GigaPath, LongViT, and LongNet, as well as the innovative dilated-attention method that underpins them. These advancements have brought substantial progress to the field of Computational Pathology. We regret that our initial submission did not adequately acknowledge these important works. The Prov-GigaPath paper, published in Nature on May 22, 2024, coincided with the NeurIPS submission deadline, while the LongViT was available as an arXiv preprint formatted as a technical report and titled as a 'Case Study' thus we missed it. We sincerely apologize for this oversight and have now included extensive discussions on these works in our rebuttal. We are committed to providing a more thorough analysis in the future version of our paper.\n\nFor the detailed comparisons of our method and Prov-GigaPath, please infer to the general response, where we make systematical analysis including:\n\n    d_1. Method: their receptive field weigh more on x-axis than y-axis, however our method as 2D locality treat x-y equally. Please also check the figure illustration in the rebuttal PDF.\n    d_2. Contribution: we focus more on analyzing why previous transformers failed then deriving our method, while they empirically scale up to big data based on dilated attention. \n    d_3. We find that when their patch feature is not the best in some task cases, their heavily pretrained WSI head with problem in 'd_1' only shows sub-optimal performance. \n    We provide some quick experiments to compare their WSI-architecture and our method.\n\n>**W3: In the experimental results, for instance, the classification results in Table 1 and the prognostic results in Table 2 show only a 0.01 improvement over full attention, which is not significant.**\n\nIt is not so easy to totally beat the full attention in performance since our main goal is to achieve comparable results as full attention but with less computational complexity. Even in the area of NLP currently, like LLM, the vanilla full attention is the most widely adopted choice [1][2][3] when equipped with the best hardware, although a lot of methods [4][5][6], and also the LongNet, are trying to replace it to solve long sequence problem. However, in the area of digital pathology given less computational resources, the inference speed is quite important. Moreover, we have tested the training and inference speed when dealing with larger resolution WSI or higher magnification like 40x with 0.25 mpp, the latency is unbearable even with flash attention. Whereas our method can highly alleviate the speed problem without performance drop (even improvement) if we need more detailed features in 40x.\n\n\n>**W4: Although the paper claims to reduce computational costs, which is evident, it should provide corresponding comparisons to substantiate this claim.**\n\nPlease check A.6.5 and Figure 8 of our main paper manuscript, where we have tested the speed (time consumption in deployed GPU), and shown the comparisons in paper submission stage.\n\nThe theoretical complexity is also included in the *line 232-236* of our main paper manuscript.\n\nIf you have additional questions, we’re happy to engage further.\n\n### Ref:\n\n*[1]. Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.*\n\n*[2]. Jiang A Q, Sablayrolles A, Roux A, et al. Mixtral of experts[J]. arXiv preprint arXiv:2401.04088, 2024.*\n\n*[3]. Bai J, Bai S, Chu Y, et al. Qwen technical report[J]. arXiv preprint arXiv:2309.16609, 2023.*\n\n*[4]. Gu A, Dao T. Mamba: Linear-time sequence modeling with selective state spaces[J]. arXiv preprint arXiv:2312.00752, 2023.*\n\n*[5]. Sun Y, Dong L, Huang S, et al. Retentive network: A successor to transformer for large language models[J]. arXiv preprint arXiv:2307.08621, 2023.*\n\n*[6]. Peng B, Alcaide E, Anthony Q, et al. Rwkv: Reinventing rnns for the transformer era[J]. arXiv preprint arXiv:2305.13048, 2023.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oXK8Hr8jri", "reviewer": "Reviewer_UesU", "summary": "The authors provide LongMIL, a hierarchical and hybrid of local and global attention mechanism, to address the inherent low-rank bottleneck of MIL problems in computational pathology. Through extensive evaluations across feature encoders and subtyping/survival tasks, the authors indeed demonstrate the superior performance as well as computational efficiency of LongMIL.", "questions": "- Since n >> d is MIL model-agnostic, would other approaches such as AB-MIL also suffer from low rank nature?\n- It seems LongMIL supersedes existing positional encodings (Or am I undsteranding this correctly?) - Can they be combined?\n- Equation 11, p -> p_{i,j}. Otherwise, readers might confuse it as being uniform probability.", "limitations": "Please see Weakness & Questions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Despite the recipes for LongMIL being simple (Masking of attention), I find the problem well-motivated and solution intuitive enough to be implemented in future MIL studies. It's been long well-known that TransMIL, although being the first self-attention mechanism in MIL, have not been demonstrating good performance and was in need of some alternative implementations.\n\nI also appreciate the fact that the authors performed extensive ablation studies over several different feature encoder choices as well as tasks of different nature (subtyping and survival), to truly show that LongMIL can be a meaningful contribution to the field.", "weaknesses": "There are several weaknesses of the studies that I think the authors need to address for this to be a meaningful contribution to the field.\n\n**Novelty**: Although the authors tried hard to distance from HIPT, I still consider LongMIL solution to be very similar to HIPT - For HIPT, if the first patch-level stage ViT is replaced with the pretrained feature extractors, wouldn't this be same solution as LongMIL, with the difference being how ROI regions are masked? Can authors expand on this point?\n\n**Motivation**: While I agree that the low-rank nature of MIL problem is problematic due to n >> d, I am not entirely convinced that \"the representation ability of self-attention is limited by the low-rank bottleneck, thus vanilla Transformer based model in WSI analysis suffers sub-optimal performance\" is always the case.\n\n1. There exists lots of morphological redundancy in WSI, so the effective number of distinct features might be much lower than actual n [1], [2]. Therefore low-rank might not always be the issue? Can authors expand on this point?\n\n2. To concretely show n>>d is indeed the issue, the authors should also evaluate their algorithm on tissue biopsies (not tissue resections), where the number of patches would be way lower (few thousand patches) or TMAs (few hundred patches). On these datasets, the gap between LongMIL and other frameworks should decrease, since this is not a low-rank setting. \n\n**Presentation**: Although the paper was not hard to follow, there are several items that needs improvement. There are lots of typos in the paper that need to be ironed out (e.g., line 157 \"are got\", line 172 \"is design\", line 298 \"which may because\"). I am not sure if Figure 2 contributes meaningfully to the paper (also it's impossible to read the axial labels). My suggestion would be to make it smaller and use extra space for the experiments section. Same goes for Equation 12 - I think this can be moved to supplemental section. \n\nThe \"pre-training backbones\" section (line 263~276) was hard to follow. Perhaps make it as a table?\n\n**Experimentation**: To follow up on the Motivation section, I think the authors could run few more ablation studies to demonstrate the severity of low rank issue, by trying to reduce the gap between n and d. This could include larger patch size (256->512) or random sampling patches, both of which have been used in literature and results in lower number of patches.\n\nI think the authors emphasize the survival experimentation over the subtyping (BRACS), since prognosis is known to depend on context [3], [4].\n\nThe authors might also consider using latest pathology foundation models (all of these have not been trained on TCGA) - PLIP, UNI, GigaPath - for future studies (It will be too much to do in the given time)\n\nReferences\n[1] Song, Andrew H., et al. \"Morphological prototyping for unsupervised slide representation learning in computational pathology.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Vu, Quoc Dang, et al. \"Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images.\" Medical image analysis 85 (2023): 102743.\n\n[3] Lee, Yongju, et al. \"Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning.\" Nature Biomedical Engineering (2022): 1-15.\n\n[4] Jaume, Guillaume, Andrew H. Song, and Faisal Mahmood. \"Integrating context for superior cancer prognosis.\" Nature Biomedical Engineering 6.12 (2022): 1323-1325.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UesU,\n\nWe appreciate your time and valuable feedback. We are glad that you found that our method is well-motivated and intuitive. Below, please find our point-to-point response to your comments:\n\n> **W1: Novelty: Although the authors tried hard to distance from HIPT, I still consider LongMIL solution to be very similar to HIPT ...**\n\nFor the comparison to HIPT, please find the A.4 and Figure 6 of paper manuscript for illustration. Here we provide some details:\n\n* HIPT first slice the whole image into regions (4096x4096, ~50 regions per WSI), then each region *r_4096* is sliced into patches (256x256, 256 patches per region). Each patch *p_256* are extracted by a ViT as feature, the patch-level operation is same as ours and other methods. \n* However, their self-attention on extracted patch features only focuses inside on each region *r_4096* with a Transformer layer. As pointed out in our paper (line 56-57, 180-183), **the adjacent patches may be separated into two regions, and the interactions between them are ignored in HIPT**. The features of each region *r_4096* is processed by further pooling, and a higher Transformer layer globally focus on slide-level.\n* Conversely, **our method do not need 4096 region slicing but use 2-d attention mask to treat all patches equally, thus all patches will interact with other adjacent patches**.\n\n> **W2.1: Motivation: .... There exists lots of morphological redundancy in WSI, so the effective number of distinct features might be much lower than actual n. Therefore low-rank might not always be the issue ...**\n\nWe find this question being very constructive to our paper. Though in patch-level the morphology and feature semantic is quite similar, in region-level, similar patches with different spatial combinations may construct different tumor type (carcinoma in situ V.S. invasive cancer). Some previous methods, e.g. DSMIL [1] [2] try multi-scale information to solve this problem, which indicates the importance. Our method can reach similar goal as multi-scale by adaptively local-global interactions in a uniformed transformer, which we believe to be more elegant.\n\n>**W2.2 + W4.1: To concretely show n>>d is indeed the issue... evaluate their algorithm on tissue biopsies (not tissue resections), where the number of patches would be way lower ... On these datasets, the gap between LongMIL and other frameworks should decrease, since this is not a low-rank setting. .... the authors could run few more ablation studies to demonstrate the severity of low rank issue, by trying to reduce the gap between n and d ...***\n\nWe run some experiments on larger patch size 448 on BRACS (224 originally), where the largest patch num n is less than 2k, near the feature size of UNI (given the limited time, we will further run biopsies and TMAs in next version). The results are shown in the following table, with main findings that:\n\n    1. Simple attentions (ABMIL, CLAM without pair wise interactions) gain improvement, and we speculate that the larger image-size can modelling the local context better.\n    2. DTFD try to split the whole bag into 3 sub-bags, but smaller bag size may result in larger label noise of sub-bags which may answer its performance drop.\n    3. The gap between LongMIL and TransMIL decreases given closer n and d. Full attention and LongMIL show small drops, since less interactions can be modelled with less patches.\n    4. LongMIL still out-performs full attention. We speculate that local attention also works better when dealing with the shape-varying WSI even with less n.\n\n* *BRACS, 224x224 VS 448x448*\n| Patch Encoder | UNI-224| | UNI-448| |\n|:---|:---|:---|:---|:---|\n| Method\\Metric | F1 | AUC | F1 | AUC |\n| AB-MIL | 0.692±0.03 | 0.875±0.02 | 0.695±0.01 | 0.875±0.01 |\n| CLAM-SB | 0.640±0.06 | 0.844±0.03 | 0.654±0.03 | 0.851±0.02 |\n| DTFD-MIL | 0.655±0.03 | 0.878±0.02 | 0.625±0.03 | 0.839±0.01 |\n| TransMIL | 0.592±0.04 | 0.859±0.02 | 0.646±0.07 | 0.855±0.02 |\n| Full Attention | 0.715±0.04 | 0.884±0.02 | 0.700±0.04 | 0.874±0.02 |\n| LongMIL (ours) | 0.728±0.05 | 0.887±0.01 | 0.722±0.03 | 0.883±0.01 |\n\n>**W3 + Q3: Presentation: ...**\n\nWe appreciate your advice and will definitely focus on refining the paper's presentation for the next version to enhance clarity.\n\n>**W4.2: ... consider using latest pathology foundation models ...**\n\nWe have finished part of this experiments in this stage, which can be inferred in the general response. \n\n>**Q1: Since n >> d is MIL model-agnostic, would other approaches such as AB-MIL also suffer from low rank nature?**\n\nFor AB-MIL, CLAM, etc., their attentions are in fact adaptive weighted average ($1 \\times n$) over all patches, and there is no pair-wise interaction like self-attention does ($n \\times n$ attention matrix).  It's hard to say the rank of a $1 \\times n$ array, but we find that they are sparse like what doctors do: determine the lesion level of a slide based on some specific lesion regions. However, this process needs quite high semantics align with doctors. Conversely, our method with self-attention can be understood as trying to convert the feature representation (learned by patch-level self-supervised learning) aligning better to doctors' diagnosis level by introducing important context. After layers of attention, cls-token and average pooling on these features will do the same thing as AB-MIL.\n\n>**Q2: It seems LongMIL supersedes existing positional encodings (Or am I understanding this correctly?) - Can they be combined?**\n\nWe have combined the rotary positional embedding, since we still need to determine the position information within the local areas (about a size of $20 \\times 20$). We feel sorry to ignore this implementation detail and will add it into the next version. \n\n\n### Ref:\n\n[1]. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning.\n \n[2]. Cross-scale multi-instance learning for pathological image diagnosis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to all the reviewers for your time and effort during the review process. We appreciate that you found our work insightful and solid.

We have responded to each reviewer individually, uploaded a rebuttal PDF, and collected the below response to general concerns. If you find our answers responsive to your concerns, we would be grateful if you considered increasing your score, and if you have additional questions, we’re happy to engage further.


> **All reviewers suggest evaluating on better pre-trained patch encoders**

Here, we list experiment results of tumor-subtyping on BRACS and TCGA-BRCA. 
We find that our method still outperforms previous method in BRACS with UNI and GigaPath. But for TCGA-BRCA tumor-subtyping, almost all the methods show similar results, which may because the binary classification is too simple, and this task meet its upper-bound given so strong patch feature encoders. We will experiment on more data with higher complexity in future work.

* *BRACS, tumor subtyping*
| Patch Encoder | UNI | UNI | GigaPath | GigaPath |
|:---|:---|:---|:---|:---|
| Slide Method\Metric | F1 | AUC | F1 | AUC |
| AB-MIL | 0.692±0.033 | 0.875±0.020 | 0.640±0.022 | 0.837±0.010 |
| CLAM-SB | 0.640±0.057 | 0.844±0.025 | 0.624±0.023 | 0.826±0.014 |
| DTFD-MIL | 0.655±0.031 | 0.878±0.022 | 0.610±0.032 | 0.843±0.017 |
| TransMIL | 0.592±0.036 | 0.859±0.023 | 0.599±0.058 | 0.838±0.048 |
| Full Attention | 0.715±0.043 | 0.884±0.017 | 0.663±0.023 | 0.850±0.018 |
| LongMIL (ours) | 0.728±0.045 | 0.887±0.008 | 0.673±0.023 | 0.856±0.015 |

* *TCGA-BRCA, tumor subtyping*
| Patch Encoder | UNI | UNI | GigaPath | GigaPath |
|:---|:---|:---|:---|:---|
| Slide Method\Metric | F1 | AUC | F1 | AUC |
| AB-MIL | 0.865±0.039 | 0.945±0.018 | 0.872±0.038 | 0.946±0.021 |
| CLAM-SB | 0.862±0.031 | 0.943±0.020 | 0.864±0.049 | 0.937±0.027 |
| DTFD-MIL | 0.867±0.034 | 0.941±0.024 | 0.870±0.035 | 0.937±0.034 |
| TransMIL | 0.853±0.049 | 0.949±0.019 | 0.830±0.048 | 0.934±0.020 |
| Full Attention | 0.849±0.043 | 0.942±0.017 | 0.860±0.041 | 0.946±0.023 |
| LongMIL (ours) | 0.863±0.033 | 0.945±0.008 | 0.871±0.030 | 0.947±0.022 |

Due to the limited time and large model architecture of UNI and GigaPatch, we only experiment survival prediction on TCGA-BRCA since its features are extracted in tumor-subtyping.

* *TCGA-BRCA, survival prediction*
| Patch Encoder | UNI | GigaPath |
|:---|:---|:---|
| Slide Method\Metric | c-index | c-index|
| AB-MIL | 0.630±0.054 | 0.635±0.033 | 
| AMISL | 0.627±0.080 | 0.620±0.040 | 
| DS-MIL | 0.616±0.034 | 0.612±0.086 | 
| TransMIL | 0.598±0.059 | 0.599±0.064 | 
| Full Attention | 0.638±0.056 | 0.617±0.069 | 
| LongMIL (ours) | 0.656±0.061 | 0.645±0.055 | 


> **Reviewers UUFY and Ghwr concern on the comparison to Prov-GigaPath / LongViT WSI-Architecture**

Although reviewers UUFY and Ghwr point out that both our paper and Prov-GigaPath use similar local attention mechanism for efficient transformer modelling in slide-level, we find that there are some important differences between them. 

    1. The motivation /contribution: our paper not only focus on proposing an efficient self-attention mechanism for WSI, but also showing analysis on why some previous work like Roformer and TransMIL fail for WSI from the low-rank perspective, which we believe to be insightful to the digital pathology community. However, both the Prov-GigaPath and LongViT focus on scaling up to a large-scale of data with pre-training, which is more empirical. We believe that our analysis may also work for Prov-GigaPath and could be one potential explain on why Prov-GigaPath success and how to improve further.
    2. The method details: Prov-GigaPath does not treat interactions inside x-axis and y-axis equally, though the 2-d positional embedding is applied. By putting all patches into a 1-d sequence in a 'z-scan' manner like ViT, their 1-d local attention focus more on x-axis but less on y-axis, as depicted in Fig. 1 of our rebuttal PDF. Although this can be alleviated by their higher-level dilated attention term, the x-y inequality still exists. Whereas, our local-attention is designed for 2-d (based on 2d Euclid distance), thus treat them equally. 
    3. The pretrained Prov-GigaPath WSI-head seems relying heavily on their own patch-pretrained encoder, which may be a potential barrier to wide usage, e.g. there are still some cases when GigaPath patch features weaker than UNI or Conch, as posted in the github repo of UNI. The WSI pretraining is indeed useful as the key to their superior performance, which covers their problem of spatial inequality on x and y. When dealing with the case 'BRACS', as shown in the following table, our method (even AB-MIL) with better UNI feature can outperform their 'worse patch feature with stronger pretrained slide encoder'.

Experimentally, we here perform evaluation on the two WSI-level architectures. Since the WSI params are pre-trained in Prov-GigaPath, we also experiment it using random initialization for fair comparison. For the mismatch of UNI patch encoder and GigaPath WSI head, we add a nn.Linear layer as a feature projector. We find that the pre-training plays a key role to the success of Prov-GigaPath WSI-head, since transformers are much more over-parameterized than previous simple attention-based MIL. Given the limited time, we are going to validate on more tasks in future version.

* *LongMIL V.S. Prov-GigaPath in slide-level on BRACS, tumor subtyping*
| Patch Encoder | UNI | UNI | GigaPath | GigaPath |
|:---|:---|:---|:---|:---|
| Slide Method\Metric | F1 | AUC | F1 | AUC |
| AB-MIL | 0.692±0.033 | 0.875±0.020 | 0.640±0.022 | 0.837±0.010 |
| TransMIL | 0.592±0.036 | 0.859±0.023 | 0.599±0.058 | 0.838±0.048 |
| GigaPath (random init) | 0.648±0.041 | 0.837±0.033 | 0.627±0.038 | 0.808±0.038 |
| GigaPath (pre-trained) | 0.668±0.026 | 0.861±0.030 | 0.677±0.033 | 0.862±0.034 |
| LongMIL (ours) | 0.728±0.045 | 0.887±0.008 | 0.673±0.023 | 0.856±0.015 |

Meta-review: This submission investigating transformers for histopathology image analysis received mixed review comments. The rebuttal added more comparison results. Reviewers raised scores or reinforced positive score after reading rebuttal and discussion with authors. The key discussing point of this submission is the contribution of its methodological innovations. The AC heard majority of the reviewers recognizing the paper's domain-oriented contribution. Despite the attention mechanism has been used in the literature, this paper has made solid method on designing the transformer architecture such that it fits the practical challenge for SWI data processing. Overall, the strengths of the work such as rigor & extensive experimentation outweigh its weaknesses. A final decision of acceptance is recommended.
Decision: Accept (poster)