Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis

Authors: honglin  li
yunlong  zhang
pingyi  chen
zhongyi  shui
chenglu  zhu
lin  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 03:38:58

Abstract: Histopathology Whole Slide Image (WSI) analysis serves as the gold standard for clinical cancer diagnosis in the daily routines of doctors. To develop computer-aided diagnosis model for histopathology WSIs, previous methods typically employ Multi-Instance Learning to enable slide-level prediction given only slide-level labels.
Among these models, vanilla attention mechanisms without pairwise interactions have traditionally been employed but are unable to model contextual information. More recently, self-attention models have been utilized to address this issue. To alleviate the computational complexity of long sequences in large WSIs, methods like HIPT use region-slicing, and TransMIL employs Nystr\"{o}mformer as an approximation of full self-attention. Both approaches suffer from suboptimal performance due to the loss of key information. Moreover, their use of absolute positional embedding struggles to effectively handle long contextual dependencies in shape-varying WSIs.
In this paper, we first analyze how the low-rank nature of the long-sequence attention matrix constrains the representation ability of WSI modelling. Then, we demonstrate that the rank of attention matrix can be improved by focusing on local interactions via a local attention mask. Our analysis shows that the local mask aligns with the attention patterns in the lower layers of the Transformer. Furthermore, the local attention mask can be implemented during chunked attention calculation, reducing the quadratic computational complexity to linear with a small local bandwidth. Additionally, this locality helps the model generalize to unseen or under-fitted positions more easily.
Building on this, we propose a local-global hybrid Transformer for both computational acceleration and local-global information interactions modelling. Our method, Long-contextual MIL (LongMIL), is evaluated through extensive experiments on various WSI tasks to validate its superiority in: 1) overall performance, 2) memory usage and speed, and 3) extrapolation ability compared to previous methods.

Main paper: 
 introduction: 
 Though digital pathology images have been widely used for Cancer diagnosis [50, 65, 89, 94, 17, 44] and prognosis [9, 12, 71] and gene expression [82] via automatic computer-assisted analysis, the Giga-pixels of resolution, as large as 150, 000 × 150, 000 pixels [50, 12, 60] of Whole Slide Image (WSI), still poses great challenges on both annotation labelling and efficient computation for model training [42] . Thus, previous methods [9, 12, 42, 5, 40, 92, 93, 41] focus on developing annotation-& computational-efficient learning to cope with those problems by employing Multiple Instance Learning (MIL) [51, 32] with only WSI-level supervision.Currently, there are mainly three steps (or mainstream genres) of WSI analysis framework: 1) access better instance-level patch embedding via Self-supervised Learning [30, 7, 9, 40, 77] . The computational complexity of transformers, denoted as O(n 2 ), becomes prohibitive in such cases, leading to computational explosion.2) design WSI head architectures [50, 65, 89] and train the head with frozen instance embedding. 3) fine-tune patch embedding with WSI level weak label for better task-specific results [90, 42] . Here in this paper, we focus on the step-2 and uncovering that there are still some room for improvement: Firstly, the vanilla attention used in AB-MIL, DS-MIL, CLAM, etc. [32, 40, 50, 89] , despite its computational efficiency (compared to self-attention), is unable to model contextual or interaction information across instances within a WSI. These interactions, which play a crucial role in prediction decisionmaking [11, 65] indeed, can be modelled via selfattention mechanism. However, the long sequence of WSI instances pose O(n 2 ) computation complexity with self-attention (Fig. 1 ). Although this complexity can be alleviated by self-attention approximation methods like Nyströmformer [83, 75] used in Trans-MIL [65] , this approximation only get sub-optimal performance compared to self-attention as pointed out in [20, 29] . Authors in HIPT [9] mitigates the complexity by non-overlap large region slicing, but the interactions of instances from different region slicing are highly ignored (e.g. adjacent patches may be separated into two regions).The above issues highlight a strong need for an effective and efficient Transformer for WSI modelling.To begin, we discuss the performance bottleneck of basic Transformer [72] for WSI. Different to Vision Transformer [23] for natural image modelling where the number of patched tokens are smaller than patch embedding size (e.g. ViT-Small-p16 with embedding size 384 attend on 196+1 tokens), the Transformer-based WSI model suffers severe low-rank bottleneck of attention matrix [3, 22] given the long-sequence (n >> 1024) of WSI but limited embedding size (d ≤ 1024). We reveal this problem in WSI theoretically, thus finding that one self-attention layer with limited embedding size can not model local contexts and global interactions at the same time. By stacking multiple self-attention layers, we notice that the low layer focus more on local context (Fig. 2a ) after training while high layer focus on global. However, the rank of the attention matrix is still limited (Fig. 2a+d ), resulting in constrained performance.We assert that the low-rank bottleneck causes the attention mechanism to become confused between local and global interactions, even after training. In other words, using Q and K ⊤ with only 2dn points, it's hard to model n × n interactions comprehensively in the context of WSI where n >> d.We believe that it would be better focusing on less interactions in one layer. Motivated by the low layers of Transformer showing highly sparse attention pattern (Fig. 2a+d ) with locality, we propose a local attention mask to learn local interaction more directly. This local mask, more importantly, can highly improve the rank of the attention matrix, showing better representation ability. Furthermore, the local attention mask can be implemented during chunked attention calculation, reducing the quadratic computational complexity to linear with a small local bandwidth. In addition, this locality helps the model generalize to unseen or under-fitted positions more easily (where absolute position embedding used in methods like TransMIL may fail, see Appendix A.2 for more illustration).Building on this, we propose a local-global hybrid Transformer for both computational acceleration and local-global information modelling.Our main contributions can be summarized as 3 folds:1) We firstly theoretically uncover why Transformer model for WSI-MIL analysis fails, based on the low-rank bottleneck of attention matrix for long sequence but limited embedding size.We then further analyze the sparsity and locality pattern of attention matrix empirically to hint our local attention design.2) We convert the full self-attention into local attention which shows three advantages: higher rank for better representation ability, lighter computational complexity and extrapolation ability for shape-vary WSIs. We further combine the full self-attention for global long-range dependency after stacking layers of local attention.3) Our WSI-analysis experiments are performed on both diagnosis and prognosis tasks on 4 WSI datasets including Breast, Stomach, Colon and Rectal Carcinoma, which show strong universality of the method and practical potential for real-world applications. 2 Related Work 
multiple instance learning for wsi analysis: 
 Whole Slide Images (WSIs) contain a rich set of visual information that can aid in pathological analysis [6, 50] . However, accurate annotation of cell-level information within WSIs is labor-intensive and time-consuming [6, 50, 9] . To address this issue, weakly-supervised methods have gained popularity in pathology WSI analysis. Attention-based Multi-Instance Learning (AB-MIL) [32] is adopted to learns instance adaptive weights, allowing the model to focus on informative regions within the WSIs. This approach significantly reduces the annotation burden of pathologists while still providing valuable insights for patient-level diagnosis. In the context of weakly-supervised pathology WSI analysis, several innovative approaches, DS-MIL, CLAM, DTFD, etc. [32, 40, 50, 89, 35, 61, 18, 2] have been proposed. However, their utilized vanilla attention with light computational cost can not model WSI contextual information, which is useful in pathologist diagnosis decision making [11, 65] . The fine-grained details and global contextual information can also be captured by multi-scale modelling [9, 40] . Graph Network [11, 43, 28, 8, 25] is also useful to make model be context-aware. Similar to this, HIPT [9] and TransMIL [65] have explored the advantages of Transformer with pairwise interactions to model this contextual information. Since Transformer can be generalized to Graph Network [24] , both modelling the pairwise interaction, in this paper we focus more on Transformer and try to adapt it better to fit the shape varying and long context properties of WSI. Unlike the authors in [80] who focus on the low-rank properties of pathology images, we investigate from the perspective of low-rank in the attention matrix of Transformer.The primary goal of this area is to alleviate the computation and memory complexity of self-attention mechanism on long sequence input. A lot of modifications sparsify the attention matrix [59, 15, 1] with some fixed patterns. Extend to this, some work [73, 74, 64] using learnable patterns in a datadriven fashion, e.g. Reformer [38] introduces a hash-based similarity measure to efficiently cluster tokens into chunks. Linformer [75] technique leverage low-rank approximations of the self-attention matrix, decomposing the N × N matrix to N × k. The kernels also serve as an approximation of the attention matrix, including Performers [37] , Linear Transformers [16] . Another popular method of reducing computation cost is to reduce the resolution of the sequence, hence reducing computation cost by a commensurate factor, e.g. Perceiver [33] , Swin Transformer [47] . The recent Nyströmformer [83] used in TransMIL [65] , can also be seem as kernel-based low-rank approach. Above work mainly focus on a light approximation of self-attention or using sparse attention, which is indeed worse than the full attention [20] . Recent work like FlashAttention [20] and others [62, 34] using chunked computation scheme and IO-aware mechanism to be memory-efficient and gain full ability like self-attention. Another lines of work try to merge RNN and Transformer, e.g. Transformer-XL [19] proposed a segment-level recurrence mechanism that connects multiple segments and blocks, and now is widely used in most successful LLMs [52, 91, 69] . Recently, linear RNNs [88, 54, 53, 27] and its variants [21, 57] are also proposed, but these recurrent ability is designed for 1-d sequence with causal or auto-regressive property, not fit well for image recognition. To fit longer sequence, better positional embeddings like RoPE, ALiBi, etc. [66, 58, 14] are also proposed. Different to these work focus on NLP task, here in this paper we try to build an efficient Transformer for WSI analysis, which is a unique challenge in vision task. 
preliminary: attention-based wsi analysis: 
 Given a WSI X as input, the goal is to make slide-level prediction Ŷ by learning a classifier f (X; θ). X is firstly patched into a long sequence of small instances X = {x 1 , ..., x n } because of its extremely large resolution, where n is the number of instance. The slide-level supervision Y is given by doctors who consider the latent label y i of all instance x i . Most previous work [6, 50, 9] try to model this process by a Max-pooling operation, so initially, this annotation process is treated as: Y = max{y 1 , ..., y n }.Since the end-to-end training from raw image input to WSI-level output is infeasible because of large memory cost, conventional approaches convert it into two separate stages: Firstly, convert all small patches into instance embeddings Z = {z 1 , ..., z n } by a pre-trained backbone such as ResNet [31] or ViT [79] , which refers to general features from public ImageNet, or learned on the related dataset to extract the domain-specific representations [9, 36] . Then, aggregate all patches' features within a slide and producing the slide-level prediction Ŷ = g(Z; θ). In this paper, we mainly focus on the latter one, where g is an vanilla attention function followed by a linear classifier head as:EQUATIONwhere a i is attention weights and σ(•) is a linear head.However, above vanilla attention method assigning adaptive weight to each instance to make simple summation or pooling can not model the interactions among different instances. Thus, to handle this problem, Transformer with self-attention is employed in TransMIL [65] and HIPT [9] , where the attention sublayer computes the attention scores for the i-th query q i ∈ R 1×d , (1 ≤ i ≤ n) in each head, where d is the head dimension. In other words, each instance will compute an attention score list as interactions with all instances. These attention scores are then multiplied by the values to return the output of the attention sub-layer as:EQUATIONwhere the {Q : q i , K : k i , V : v i } ∈ R n×d are obtained through linear transform from the input embedding Z, the softmax(q i K ⊤ ) is the attention score and O ∈ R n×d is the output. Given O, which encodes the interactions among instances, we can further use Equation ( 2) and input O to replace Z for final prediction, mean-pooling and class token in ViT [79] can also be adopted. Note that here we omit dropout, FFN, residual connection and some detailed blocks in Transformer for simplicity.Positional Embedding: Since the operation in Equation ( 3) is position-agnostic, Transformer [72, 23] try to model contextual interactions by incorporate position information. Absolute positional embedding assigns a positional vector p m to each position m and adds it to the embedding vector as:z i = z i + p m,i .In HIPT [9] , the absolute positional embedding [23] for 2-d is employed, while TransMIL [65] use convolutions as implicit positional embedding [78] but treat data as 1-d sequence.Relative positional embedding that model the positional difference m-n has become popular. Rotary positional embedding (RoPE) [66] encodes the position with rotations: f (q m , m) = R m q m , where R m is a rotation matrix with angles proportional to m. With the rotation's property, the query-key product exhibits a positional difference:EQUATIONThe core idea of RoPE is to insert position m, n signal on q, k and reflect the relative position on the newly attention matrix. Though the RoPE is designed for 1-d language sequence, it can also be extended to 2-d paradigm for application on WSI analysis [56] .Computational Complexity: Though above Transformer with self-attention can well model the interactions among instances, its computational cost O(n 2 d) is too heavy for long sequence of WSI due to the interactive attention score calculation (see Appendix A.5.5 for 40x magnification WSI modelling). Previous WSI Transformer SOTA like TransMIL [65] and HIPT [9] relieve this problem with different ways: 1) attention approximation: TransMIL [65] utilizes Nyströmformer [83] , a mechanism employs kernel-based low-rank approximation to approximate full self-attention for acceleration. 2) region slicing: HIPT [9] utilizes the locality of image by slicing WSI into 4096×4096 squares without overlapping. Given the fixed window size, in each square there are fixed 16 * 16 = 256 patches with shape of 256 × 256, thus the computational cost can also be seen as linear complexity.Low-rank bottleneck: Considering all queries in {Q : q i }, the Equation (3) can be seen as:EQUATIONwhere the rank of QK ⊤ can be derived as:EQUATIONIn the context of WSI analysis, the patched sequence length n of most WSIs is larger than 1024 (Fig. 1 ) As a result, the representation ability of self-attention is limited by the low-rank bottleneck, thus vanilla Transformer based model in WSI analysis suffers sub-optimal performance. Though the non-linear softmax operation can change the rank, but we still observe limited rank of softmax(QK ⊤ ) after training (Fig. 2a+d ). This is an extremely different problem compared to ViT modelling, e.g. ViT-Small with embedding size of 384 only need to focus on 196 patch tokens (with image size of 224 and patch size of 16), which can model both local contextual and global interactions simultaneously with full-rank attention matrix. We contend that, under this circumstance, Transformer for WSI modelling may become confused when handling local contextual and global interactions with a single layer.Under this assumption, it is also more easy to understand the limitation of previous SOTA transformer for WSI: TransMIL [65] with Nyströmformer [83] employs kernel-based low-rank approximation to approximate full self-attention. However, it is worth noting that the approximation may produce lower rank of attention matrix compared to basic self-attention, thus resulting lower performance in TransMIL. Similar problems also happens in other models like softmax-free linear attention [54, 67, 85] . We show a lot of experiment of these linear attention model in Appendix A.5.6.An intuitive modification to handle the low-rank problem is to set a larger embedding size d, but this makes computational complexity O(n 2 d) more severe, let alone most pathology patch pre-trained foundation models [36, 49, 10] carry fixed embedding size. Noting that it is infeasible to fully represent a matrix with a shape of n × n if n >> d given totally 2nd feature points from Q n×d and K n×d , why not focus the attention to more important interactions? Thus in the contrast, we alleviate the low-rank bottleneck together with the problem of computational cost by focusing on locality, motivated by the sparse and local pattern in low layers of attention (Fig. 2a ).Here we define the the selection index for retained sparse attention matrix when the softmax probability is greater than a threshold:EQUATIONwhere the threshold τ is normalized by sequence number n, e.g. τ = 0.0001/n. Then, the sparsity ratio can be denoted as:EQUATIONWe note that there is an obvious sparsity and local pattern in lower Transformer layers (Fig. 2a ) under this protocol. Given the learned locality and sparsity, we introduce to learn local contextual information with a addable local attention mask for A = softmax(QK ⊤ ) (without loss of generality, here we mainly derive on 1-d sequence for simplicity):EQUATION)where b is the local band width. Then the attention matrix is converted as:EQUATIONwhere 0 < p < 1 is the softmax probability.We claim that our proposed sparse local attention capture both higher rank and reduced computational complexity, which will work better for WSI modelling:1) Higher rank: It is easy to prove that the band matrix A in Equation 11is of a lower-bound of rank as n -b. Here we give a intuitive verification with a small band matrix: A <n=9,b=3> =             a 11 aEQUATION)Then, let's consider the lower-left sub-matrix ranging from (b + 1, 1) to (n, n -b):EQUATIONwhich is apparently a upper triangular matrix with a full rank of n -b. Thus, we have:EQUATIONSince n > 1024 >> b practically, our model with higher rank carry stronger representation ability which can focus more on local contextual information.2) Reduced computational complexity: Given a larger n but fixed small b in Equation 12, there will be a lot of zeros in upper-right and lower-left areas. We note that these zeros can be omitted during attention matrix calculation by a chunking method [20] . Since most operations in softmax(QK ⊤ ) can be skipped, the modified complexity O(bnd) will linearly related to sequence length and band-width, which is heavily reduced compared to O(n 2 d).3) Extrapolation ability: Despite above advantages, here we further show that our model can tackle WSIs with varying input shape better, in other words, extrapolation ability. RoPE need to be well trained or fine-tuned on unseen or seldom seen longer length [45, 13, 81] . Another strategy Attention with Linear Bias (ALiBi) adds pre-defined bias term after the query-key dot-product attention matrix before softmax. For the original 1-d ALiBi [58] , the bias is a static, non-learned matrix softmax(q i k ⊤ j -ρ |i -j| computed by the distance between tokens from different positions. We also introduce 2-d ALiBi by 2-d Euclidean distance among token positions, and we find that it shows similar pattern (Appendix A.3) compared to our 2-d local attention but it needs to focus on all instances. patches 2d positions ••• 𝑥 ! , 𝑦 ! 𝑥 " , 𝑦 " 𝑥 # , 𝑦 # ••• 𝑄𝐾 $ ••• patch embeddings ••• ••• ••• 𝑄 𝐾 𝑉••• global SA classi3ier head 𝑥 ! , 𝑦 ! 𝑥 " , 𝑦 " 𝑥 # , 𝑦 # ••• 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 { 𝑥 % , 𝑦 % , 𝑥 & , 𝑦 & }To realize long contextual MIL modelling and better WSI analysis performance, the overall framework (as depicted in Fig. 3 ) of our method includes 3 stages: 1) Segment and patch WSI into instances, then save its corresponding foreground patch feature embedding and 2-d positions for preparation.2) Calculate the local self-attention matrix by the local window mask given position distance. This process is finished by trunk method like FlashAttention [20] to omit non-masking areas.3) After multi layers (two as default) of local attention focusing on local contextual information interactions, a pooling function with window size 2 × 2 is employed to reduce token number by 4 times. Then a basic self-attention focus on global interactions is computed to get final feature and prediction.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We provide model backbone selection and implementation details in core paper and further provide data split and pre-processing in supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 In this section, we present the performance of the proposed method and compare it with various baselines. Ablation experiments are performed to further study the proposed method, for paper length, more experimental results are presented in the Appendix A.5.Datasets and Tasks. We use four datasets to evaluate our method for both tumor subtyping and survival prediction. For data details and pre-processing, please see Appendix A.4. Pre-training Patch Encoders. Our work mainly focus on the WSI-head results based on some good pre-trained encoders for histopathology including HIPT [9] , Lunit [36] and newly foundation models like UNI [10] and GigaPath [84] . We also include ResNet-50 pretrained in ImangeNet-1k and ViT-small pretrained in BRACS patch data by ourself with DINO [7] .Implementation Details. We train our model with PyTorch on a RTX-3090 GPU, with a WSI-level batchsize of 1, learning rate of 1e-4, and weight decay of 1e-2. We add positional encoding into the framework, please check our code for details.Evaluation Metrics. For all the experiments, we report the macro-AUC and macro-F1 scores since all these dataset suffering class imbalance. For TCGA-BRCA, we perform 10-fold cross-validation with the same data split adopted in HIPT [9] . Besides, the dataset BRACS is officially split into training, validation and testing, thus the experiment is conducted 5-times with different random seeds. The mean and standard variance values of performance metrics are reported for multi-runs or cross-validation runs.Baselines for Comparison. We first show the results of Mean-/Max-pooling and KNN for traditional evaluation. Then we directly evaluate several classical WSI-MIL methods, including AB-MIL [32] , DS-MIL [40] , CLAM [50] , DTFD-MIL [89] . Then we compare our method with Full Attention (RoFormer) and TransMIL [65] . We omit HIPT [9] for BRACS since it need WSI larger than a threshold and should based on their pre-trained backbone.Results Analysis: For BRACS 3-categories tumor subtyping, the results are reported in Table 1 . We can first observe that both Full Attention and our LongMIL show improvement respectively. For Full Attention, attributing to its full self-attention for pairwise interaction ability, it shows better performance compared to all vanilla attention modules [32, 40, 50] and especially TransMIL [65] which use attention approximation, but it is not quite stable to beat DTFD [89] .For TCGA-BRCA 2-categories tumor subtyping, we show the results in the Appendix A.5.1. Evaluation Metrics. For all the experiments, C-Index scores are reported for the 3 datasets. We follow the data splits and pre-trained patch embedding provided in HIPT [9] for fair comparison.The performance results are also reported via the mean and standard variance values of performance metrics by multiple folder cross-validation with the same running setting to HIPT [9] .Baselines for Comparison. For this task, we use the survival cross-entropy loss proposed by Zadeh et al. [87] . The results are summarized in Table 2 , where we directly evaluate several survival prediction WSI-MIL methods, including AB-MIL [32] , AMISL [86] , DS-MIL [40] , GCN-MIL [43] . Then we compare our method with some state-of-the-art combining position embedding on Transformer: TransMIL [65] and HIPT [9] . Though our method show some improvement, the C-index score is still too low to daily clinical usage depending on only WSI information. In the near future, we would like to investigate more on this task, e.g. combining multi-modality features as used in [12] , since Transformer also born with great ability on multi-modality fusion [39, 70, 12, 63] .Since recent Pathology Foundation Model (PathFMs) [10, 48, 84] have been emerging as strong patch encoders, we here further provide evaluations based on PathFMS including UNI [10] and GigaPath [84] . The pre-processing procedure is the same to previous sections. Since the WSI params are pre-trained in GigaPath, we also experiment it using random initialization for fair comparison.For the mismatch of UNI patch encoder and GigaPath WSI head, we add a nn.Linear layer as a feature projector. The results is shown in Table 3 , we find that our method also show consistency improvement with PathFMs. Furthermore, we find that the pre-training plays a key role to the success of Prov-GigaPath WSI-head, since transformers are much more over-parameterized than previous simple attention-based MIL. However, the WSI-level pretrained model relies on patch encoder (e.g. UNI patch encoder + GigaPath WSI do not show competing result). We also provide more difference analysis of efficient attention mechanism compared to GigaPath WSI head in Appendix A.6. In table 4, we also include survival prediction based on PathFMs. We also provide abundant ablations in Appendix A.5.3 to select best setting including: Transformer blocks and multi-head number, dropout ratio, weight decay, learning rate and the local window size.For linear attention result, please check Appendix A.5.6 with main findings that the linear attentions show low performance like TransMIL since it get low-rank problem more easily. For linear RNN method like Mamba, the result is also relatively lower than Transformer since 2-d WSIs do not hold causal property like 1-d data. For extrapolation validation, please check Appendix A.2 where our method show significant performance improvement (p-value ≈ 0.1). We also find that our method show consistency improvement when equipped on different magnification (e.g. 40x) and patch size (224 -> 448) as shown in Appendix A.5.4. 
conclusions and limitations: 
 In conclusion, our work introduces advancements in computer-aided diagnosis for histopathology WSI analysis. By analyze the low-rank bottleneck and sparsity property and proposing a local-global hybrid Transformer model, our method, Long-contextual MIL (LongMIL), demonstrates superior performance in handling large and shape-varying WSIs. The evaluations across various tasks highlight its accuracy, extrapolation ability, and efficiency compared to previous methods. Our contributions enhance WSI analysis and provide valuable insights for future research. The LongMIL has two limitations: First, its application is restricted to very large image via Transformer modelling. Second, limited embedding size is adopted for practical aim and fair comparison, which is the key to stronger performance based on the low-rank assumption. Future work will aim to address these limitations. TCGA-BRCA tumor subtyping ViT-S Lunit [36] ViT-S HIPT [  
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: For each experiments we have reported the mean and std via multi-runs or cross-folds. We also show some p-value significance in train small test large experiment in supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our memory efficiency method is run on RTX3090 GPU (24g), as detailed in core paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work mainly focus on recognition for real-world medical pathology image, without any generative problem.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the data and tools are public accessible and well referenced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Number of Transformer blocks and multi-head, bias slope coefficient and local window size, weight decay and dropout ratio: Here we include following hyper-parameters for our results on BRACS with ViT-S patch embedding pre-trained by [36] : Transformer blocks and multi-head number, dropout ratio, weight decay, and learning rate, finally the local window size. Due to time-consumption, we fixed other hyper-parameters when ablation on selected variant (The default setting is Transformer local blocks number = 2, multi-head number = 1, dropout ratio = 0.0, weight decay = 1e-2, and learning rate = 1e-4, local-window size = 10 (radius)), the details can be found in Fig. 6 .There are large differences in speed and performance for 20x and 40x magnification, since FlashAttention [20] will be quite slow if given over 20k instances compared to linear attention our local attention. For performance and speed please check Table 7 and Fig. 7b , respectively. [36] We also experiment on larger patch size (from 224 to 448, Table 8 ) to decrease the overall token number, but find that our method still shows stronger performance.1. Simple attentions (ABMIL, CLAM without pair wise interactions) gain improvement, and we speculate that the larger image-size can modelling the local context better.2. DTFD try to split the whole bag into 3 sub-bags, but smaller bag size may result in larger label noise of sub-bags which may answer its performance drop.3. The gap between LongMIL and TransMIL decreases given closer n and d. Full attention and LongMIL show small drops, since less interactions can be modelled with less patches.4. LongMIL still out-performs full attention. We speculate that local attention also works better when dealing with the shape-varying WSI even with less n. We show the memory efficiency and speed of various transformer structures in Fig. 7 .We provide ablation on different linear attention e.g. RetNet, GLA [67, 85] and linear RNN structure like Mamba [26] to uncover their advantages and limitations in WSI analysis. As shown in Table 9 , we first show the results of these vanilla Linear attention or RNN directly as MIL model (first row), but none of these methods can compete with Full Attention in performance. Then, we combine these modules into our LongMIL to replace last global attention layer and we observe that this can provide us strong performance as well as linear complexity in total (2 layers of local attention + 1 layer of linear attention, better than two layers of Full Attention in both speed and performance).A.6 Detailed comparison to Prov-GigaPath 1. The motivation /contribution: our paper not only focus on proposing an efficient self-attention mechanism for WSI, but also showing analysis on why some previous work like Roformer and TransMIL fail for WSI from the low-rank perspective, which we believe to be insightful to the digital pathology community. However, both the Prov-GigaPath [84] and LongViT [76] focus on scaling up to a large-scale of data with pre-training, which is more empirical. We believe that our analysis may also work for Prov-GigaPath and could be one potential explain on why Prov-GigaPath success and how to improve further.2. The method details: Prov-GigaPath does not treat interactions inside x-axis and y-axis equally, though the 2-d positional embedding is applied. By putting all patches into a 1-d sequence in a 'z-scan' manner like ViT, their 1-d local attention focus more on x-axis but less on y-axis, as depicted in Fig. 5 . Although this can be alleviated by their higher-level dilated attention term, the x-y inequality still exists. Whereas, our local-attention is designed for 2-d (based on 2d Euclid distance), thus treat them equally.3. The pretrained Prov-GigaPath WSI-head seems relying heavily on their own patch-pretrained encoder, which may be a potential barrier to wide usage, e.g. there are still some cases when GigaPathEQUATIONTraining memory usage and speed using different Attentions. Upper: The chunk method (depicted as Flash Attention) for self-attention calculation convert memory complexity to linear, even better than Nyströmformer. Lower: lash Attention with chunk method still suffers quadratic complexity in speed even with hardware-aware accelerated operations. Our introduced LongMIL for WSI analysis can convert it as linearity with local-window mask. We also show markers about the max instance number of WSI used in this paper to show potentials on higher (e.g. 40x) magnification learning. We omit more instances (e.g. over 50k) speed test since it takes a long time, but based on its quadratic complexity of full self-attention, it will be about 25 to 35 times slower than linear attention.patch features weaker than UNI or Conch, as posted in the github repo of UNI. The WSI pretraining is indeed useful as the key to their superior performance, which covers their problem of spatial inequality on x and y. When dealing with the case 'BRACS', as shown in the following table, our method (even AB-MIL) with better UNI feature can outperform their 'worse patch feature with stronger pretrained slide encoder'.F1 AUC AB-MIL [32] 0.668±0.032 0.866±0.016 TransMIL [65] 0.648±0.054 0.835±0.031 RetNet [67] 0.628±0.034 0.805±0.009 GLA [85] 0.589±0.032 0.794±0.013 Mamba [26] Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: We have read the NeurIPS Code of Ethics and understood the policies, and we believe that neither the manuscript nor the study violates any of these.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no negative societal impact of our paper. Our work focus on computer aided diagnosis for potential medical use, which is discussed in conclusions of core paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).q4, 1 k1, 1 q4, 1 k1, 2 q4, 1 k1, 3 q4, 1 k1, 4 q4, 1 k2, 1 q4, 1 k2, 2 q4, 1 k2, 3 q4, 1 k2, 4 q4, 1 k3, 1 q4, 1 k3, 2 q4, 1 k3, 3 q4, 1 k3, 4 q4, 1 k4, 1 q4, 1 k4, 2 q4, 1 k4, 3 q4, 1 k4, 4 q4, 2 k1, 1 q4, 2 k1, 2 q4, 2 k1, 3 q4, 2 k1, 4 q4, 2 k2, 1 q4, 2 k2, 2 q4, 2 k2, 3 q4, 2 k2, 4 q4, 2 k3, 1 q4, 2 k3, 2 q4, 2 k3, 3 q4, 2 k3, 4 q4, 2 k4, 1 q4, 2 k4, 2 q4, 2 k4, 3 q4, 2 k4, 4 q4, 3 k1, 1 q4, 3 k1, 2 q4, 3 k1, 3 q4, 3 k1, 4 q4, 3 k2, 1 q4, 3 k2, 2 q4, 3 k2, 3 q4, 3 k2, 4 q4, 3 k3, 1 q4, 3 k3, 2 q4, 3 k3, 3 q4, 3 k3, 4 q4, 3 k4, 1 q4, 3 k4, 2 q4, 3 k4, 3 q4, 3 k4, 4 q4, 4 k1, 1 q4, 4 k1, 2 q4, 4 k1, 3 q4, 4 k1, 4 q4, 4 k2, 1 q4, 4 k2, 2 q4, 4 k2, 3 q4, 4 k2, 4 q4, 4 k3, 1 q4, 4 k3, 2 q4, 4 k3, 3 q4, 4 k3, 4 q4, 4 k4, 1 q4, 4 k4, 2 q4, 4 k4, 3 q4, 4 k4, 4q1, 1 k1, 1 q1, 1 k1, 2 q1, 1 k1, 3 q1, 1 k1, 4 q1, 1 k2, 1 q1, 1 k2, 2 q1, 1 k2, 3 q1, 1 k2, 4 q1, 1 k3, 1 q1, 1 k3, 2 q1, 1 k3, 3 q1, 1 k3, 4 q1, 1 k4, 1 q1, 1 k4, 2 q1, 1 k4, 3 q1, 1 k4, 4 q1, 2 k1, 1 q1, 2 k1, 2 q1, 2 k1, 3 q1, 2 k1, 4 q1, 2 k2, 1 q1, 2 k2, 2 q1, 2 k2, 3 q1, 2 k2, 4 q1, 2 k3, 1 q1, 2 k3, 2 q1, 2 k3, 3 q1, 2 k3, 4 q1, 2 k4, 1 q1, 2 k4, 2 q1, 2 k4, 3 q1, 2 k4, 4 q1, 3 k1, 1 q1, 3 k1, 2 q1, 3 k1, 3 q1, 3 k1, 4 q1, 3 k2, 1 q1, 3 k2, 2 q1, 3 k2, 3 q1, 3 k2, 4 q1, 3 k3, 1 q1, 3 k3, 2 q1, 3 k3, 3 q1, 3 k3, 4 q1, 3 k4, 1 q1, 3k4, 2 q1, 3 k4, 3 q1, 3 k4, 4 q1, 4 k1, 1 q1, 4 k1, 2 q1, 4 k1, 3 q1, 4 k1, 4 q1, 4 k2, 1 q1, 4 k2, 2 q1, 4 k2, 3 q1, 4 k2, 4 q1, 4 k3, 1 q1, 4 k3, 2 q1, 4 k3, 3 q1, 4 k3, 4 q1, 4 k4, 1 q1, 4 k4, 2 q1, 4 k4, 3 q1, 4 k4, 4 q2, 1 k1, 1 q2, 1 k1, 2 q2, 1 k1, 3 q2, 1 k1, 4 q2, 1 k2, 1 q2, 1 k2, 2 q2, 1 k2, 3 q2, 1 k2, 4 q2, 1 k3, 1 q2, 1 k3, 2 q2, 1 k3, 3 q2, 1 k3, 4 q2, 1 k4, 1 q2, 1 k4, 2 q2, 1 k4, 3 q2, 1 k4, 4 q2, 2 k1, 1 q2, 2 k1, 2 q2, 2 k1, 3 q2, 2 k1, 4 q2, 2 k2, 1 q2, 2 k2, 2 q2, 2 k2, 3 q2, 2 k2, 4 q2, 2 k3, 1 q2, 2 k3, 2 q2, 2 k3, 3 q2, 2 k3, 4 q2, 2 k4, 1 q2, 2 k4, 2 q2, 2 k4, 3 q2, 2 k4, 4 q2, 3 k1, 1 q2, 3 k1, 2 q2, 3 k1, 3 q2, 3 k1, 4 q2, 3 k2, 1 q2, 3 k2, 2 q2, 3 k2, 3 q2, 3 k2, 4 q2, 3 k3, 1 q2, 3 k3, 2 q2, 3 k3, 3 q2, 3 k3, 4 q2, 3 k4, 1 q2, 3 k4, 2 q2, 3 k4, 3 q2, 3 k4, 4 q2, 4 k1, 1 q2, 4 k1, 2 q2, 4 k1, 3 q2, 4 k1, 4 q2, 4 k2, 1 q2, 4 k2, 2 q2, 4 k2, 3 q2, 4 k2, 4 q2, 4 k3, 1 q2, 4 k3, 2 q2, 4 k3, 3 q2, 4 k3, 4 q2, 4 k4, 1 q2, 4 k4, 2 q2, 4 k4, 3 q2, 4 k4, 4q3, 1 k1, 1 q3, 1 k1, 2 q3, 1 k1, 3 q3, 1 k1, 4 q3, 1 k2, 1 q3, 1 k2, 2 q3, 1 k2, 3 q3, 1 k2, 4 q3, 1 k3, 1 q3, 1 k3, 2 q3, 1 k3, 3 q3, 1 k3, 4 q3, 1 k4, 1 q3, 1 k4, 2 q3, 1 k4, 3 q3, 1 k4, 4 q3, 2 k1, 1 q3, 2 k1, 2 q3, 2 k1, 3 q3, 2 k1, 4 q3, 2 k2, 1 q3, 2 k2, 2 q3, 2 k2, 3 q3, 2 k2, 4 q3, 2 k3, 1 q3, 2 k3, 2 q3, 2 k3, 3 q3, 2 k3, 4 q3, 2 k4, 1 q3, 2 k4, 2 q3, 2 k4, 3 q3, 2 k4, 4 q3, 3 k1, 1 q3, 3 k1, 2 q3, 3 k1, 3 q3, 3 k1, 4 q3, 3 k2, 1 q3, 3k2, 2 q3, 3 k2, 3 q3, 3 k2, 4 q3, 3 k3, 1 q3, 3 k3, 2 q3, 3 k3, 3 q3, 3 k3, 4 q3, 3 k4, 1 q3, 3 k4, 2 q3, 3 k4, 3 q3, 3 k4, 4 q3, 4 k1, 1 q3, 4 k1, 2 q3, 4 k1, 3 q3, 4 k1, 4 q3, 4 k2, 1 q3, 4 k2, 2 q3, 4 k2, 3 q3, 4 k2, 4 q3, 4 k3, 1 q3, 4 k3, 2 q3, 4 k3, 3 q3, 4 k3, 4 q3, 4 k4, 1 q3, 4 k4, 2 q3, 4 k4, 3 q3, 4 k4, 4 q4, 1 k1, 1 q4, 1 k1, 2 q4, 1 k1, 3 q4, 1 k1, 4 q4, 1 k2, 1 q4, 1 k2, 2 q4, 1 k2, 3 q4, 1 k2, 4 q4, 1 k3, 1 q4, 1 k3, 2 q4, 1 k3, 3 q4, 1 k3, 4 q4, 1 k4, 1 q4, 1 k4, 2 q4, 1 k4, 3 q4, 1 k4, 4 q4, 2 k1, 1 q4, 2 k1, 2 q4, 2 k1, 3 q4, 2 k1, 4 q4, 2 k2, 1 q4, 2 k2, 2 q4, 2 k2, 3 q4, 2 k2, 4 q4, 2 k3, 1 q4, 2 k3, 2 q4, 2 k3, 3 q4, 2 k3, 4 q4, 2 k4, 1 q4, 2 k4, 2 q4, 2 k4, 3 q4, 2 k4, 4 q4, 3 k1, 1 q4, 3 k1, 2 q4, 3 k1, 3 q4, 3 k1, 4 q4, 3 k2, 1 q4, 3 k2, 2 q4, 3 k2, 3 q4, 3 k2, 4 q4, 3 k3, 1 q4, 3 k3, 2 q4, 3 k3, 3 q4, 3 k3, 4 q4, 3 k4, 1 q4, 3 k4, 2 q4, 3 k4, 3 q4, 3 k4, 4 q4, 4 k1, 1 


Decision: This submission investigating transformers for histopathology image analysis received mixed review comments. The rebuttal added more comparison results. Reviewers raised scores or reinforced positive score after reading rebuttal and discussion with authors. The key discussing point of this submission is the contribution of its methodological innovations. The AC heard majority of the reviewers recognizing the paper's domain-oriented contribution. Despite the attention mechanism has been used in the literature, this paper has made solid method on designing the transformer architecture such that it fits the practical challenge for SWI data processing. Overall, the strengths of the work such as rigor & extensive experimentation outweigh its weaknesses. A final decision of acceptance is recommended.

Reviews: review: paper_summary: paper_summary: This work examines the problem of extrapolating Transformer attention to long sequences in WSI representation learning. The main technical contribution is in examining the low-rank bottleneck problem of Transformer attention for WSIs, and proposing LongMIL which introduces modifications via local attention masking + 2D ALIBI in order to improve the rank of the attention matrix and enable extrapolation capabilities.
strength_weakness: strength_weakness: - Main limitation of this work is that a comparison with Prov-GigaPath [1] (a concurrent work that appeared at the time of NeurIPS submission) is warranted. Prov-GigaPath also presents overlapping contributions in solving this problem, though I think there is room for more than 1 study examining this problem.
- The writing feels a bit rushed and informal. I was often scrolling back and forth to understand the different comparisons being made. Ideally, there should be one table for results that compare MIL architectures and one table for ablating pretrained encoders. Other areas where the writing / presentation of figures and results could be significantly polished: 
- - "We omit the ResNet-50 embedding for survival prediction since it get quite low and unacceptable results." Informal and non-scientific. 
- - Section 4.3 should be able to summarize the findings in the Supplement in a more clear and descriptive manner.
- - One of the main issues I see in this work is that the authors are juggling many different encoders for different tasks. A.6.1 compares ViT-S Lunit and HIPT for TCGA-BRCA subtyping (evaluating multiple MIL models across multiple encoders on 1 task), but A.6.2. shows Resnet-50 features for BRACS and TCGA-BRCA subtyping (evaluating multiple MIL models across multiple tasks with the same encoder). Many of these issues can be drastically simplified if the authors were to use an encoder not pretrained on TCGA such as Prov-GigaPath [1], UNI [2], or PLIP [3] - with the main emphasis on comparing LongMIL with other competing works. Ref [4] provides an example on how the findings of this work can be better organized.

References
1. Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y. and Xu, Y., 2024. A whole-slide foundation model for digital pathology from real-world data. Nature, pp.1-8.
2. Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M. and Williams, M., 2024. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3), pp.850-862.
3. Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J. and Zou, J., 2023. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9), pp.2307-2316.
4. Park, N. and Kim, S., How Do Vision Transformers Work?. In International Conference on Learning Representations.
questions: questions: Would the authors be able to address my concern in updating the results of this work to using a different encoder (to simplify the presentation of results)? Choice of pretrained encoder should not matter significantly (as the main focus is in fairly comparing LongMIL), but having 1-2 comparisons would be nice in a Supplemental Figure.

Rating at the time of reviewing this work is slightly negative, but am enthusiastic of this work and would raise my rating to borderline / weak accept if my concerns were addressed.
limitations: limitations: N/A
paper_summary: paper_summary: The authors point out that that MIL often has insufficient ability to offer accurate slide level classifications.  There is a long (now) history of attempting to better consider sub-slide level context in the aggregation function.  TransMIL, GNN-based methods all have provided attempts to this end.  

The authors argue that transformer based aggregatioins functions have limited ability to consider both local and global context for tile-attention-rankings.  They provide theoretical arguments for why a "low-rank bottleneck" exists because the total number of patch embeddings is much greater than the embedding size.  

The method addresses this limitation but calculating local self attention followed by pooling function before doing a "global" self attention function. 

In main paper experiments are done with BRACS dataset using two different encoders for feature extraction.  The proposed method is benchmarked against cotext aware and non-context aware aggregations functions.  Similar experiment is done with survival predictions.  Extensive additional supporting experiments are in supplement.
strength_weakness: strength_weakness: A recent preprint (https://arxiv.org/abs/2407.07841) shows that context aware aggregations functions offer less performance boost over ABMIL when you have a high feature extraction encoder.  In this work the encoders used are less robust than the now publically available encoder (UNI, Gigapath and Virchow). All of these are very recently released so is understanable that they are not part of this submission. 
 Going forward these encoders should be used for any aggregation function assessment.   The supplemental table showing that the boost of this method is much stronger with an image net pretrained cnn.
questions: questions: In final version of paper, can you please improve orientation of figure 3.  It is hard to zoom in sufficiently to understand second panel.  It would also benefit from having sub labels (eg. a, b, c) to improve the legend description of the sub panels.
limitations: limitations: I concur with listed limitations.  I have pointed out other limitations in the weakness section.
paper_summary: paper_summary: This paper focuses on the issue of attention computation for long sequences in WSI (Whole Slide Image) images. The authors first analyze how the low-rank nature of the long-sequence attention matrix constrains the representation ability of WSI modeling. They then propose a method using local attention masks to compute attention within local regions, followed by the computation of global attention. Experimental results demonstrate that the combination of local and global attention computations outperforms full attention.
strength_weakness: strength_weakness: 1. Although the paper identifies and analyzes the bottleneck issues in long-sequence attention and proposes the use of local attention based on this analysis, there is a lack of innovation in using local attention. Other methods, such as LongViT, LongNet, and Prov-GigaPath, have also used local attention more elegantly. Additionally, the paper does not discuss the potential for directly transferring numerous attention optimization methods from the fields of computer vision (CV) and natural language processing (NLP).
2. The paper lacks a review of relevant literature, such as the aforementioned works.
3. In the experimental results, for instance, the classification results in Table 1 and the prognostic results in Table 2 show only a 0.01 improvement over full attention, which is not significant.
4. Although the paper claims to reduce computational costs, which is evident, it should provide corresponding comparisons to substantiate this claim.
questions: questions: see weakness
limitations: limitations: no
paper_summary: paper_summary: The authors provide LongMIL, a hierarchical and hybrid of local and global attention mechanism, to address the inherent low-rank bottleneck of MIL problems in computational pathology. Through extensive evaluations across feature encoders and subtyping/survival tasks, the authors indeed demonstrate the superior performance as well as computational efficiency of LongMIL.
strength_weakness: strength_weakness: There are several weaknesses of the studies that I think the authors need to address for this to be a meaningful contribution to the field.

**Novelty**: Although the authors tried hard to distance from HIPT, I still consider LongMIL solution to be very similar to HIPT - For HIPT, if the first patch-level stage ViT is replaced with the pretrained feature extractors, wouldn't this be same solution as LongMIL, with the difference being how ROI regions are masked? Can authors expand on this point?

**Motivation**: While I agree that the low-rank nature of MIL problem is problematic due to n >> d, I am not entirely convinced that "the representation ability of self-attention is limited by the low-rank bottleneck, thus vanilla Transformer based model in WSI analysis suffers sub-optimal performance" is always the case.

1. There exists lots of morphological redundancy in WSI, so the effective number of distinct features might be much lower than actual n [1], [2]. Therefore low-rank might not always be the issue? Can authors expand on this point?

2. To concretely show n>>d is indeed the issue, the authors should also evaluate their algorithm on tissue biopsies (not tissue resections), where the number of patches would be way lower (few thousand patches) or TMAs (few hundred patches). On these datasets, the gap between LongMIL and other frameworks should decrease, since this is not a low-rank setting. 

**Presentation**: Although the paper was not hard to follow, there are several items that needs improvement. There are lots of typos in the paper that need to be ironed out (e.g., line 157 "are got", line 172 "is design", line 298 "which may because"). I am not sure if Figure 2 contributes meaningfully to the paper (also it's impossible to read the axial labels). My suggestion would be to make it smaller and use extra space for the experiments section. Same goes for Equation 12 - I think this can be moved to supplemental section. 

The "pre-training backbones" section (line 263~276) was hard to follow. Perhaps make it as a table?

**Experimentation**: To follow up on the Motivation section, I think the authors could run few more ablation studies to demonstrate the severity of low rank issue, by trying to reduce the gap between n and d. This could include larger patch size (256->512) or random sampling patches, both of which have been used in literature and results in lower number of patches.

I think the authors emphasize the survival experimentation over the subtyping (BRACS), since prognosis is known to depend on context [3], [4].

The authors might also consider using latest pathology foundation models (all of these have not been trained on TCGA) - PLIP, UNI, GigaPath - for future studies (It will be too much to do in the given time)

References
[1] Song, Andrew H., et al. "Morphological prototyping for unsupervised slide representation learning in computational pathology." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Vu, Quoc Dang, et al. "Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images." Medical image analysis 85 (2023): 102743.

[3] Lee, Yongju, et al. "Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning." Nature Biomedical Engineering (2022): 1-15.

[4] Jaume, Guillaume, Andrew H. Song, and Faisal Mahmood. "Integrating context for superior cancer prognosis." Nature Biomedical Engineering 6.12 (2022): 1323-1325.
questions: questions: - Since n >> d is MIL model-agnostic, would other approaches such as AB-MIL also suffer from low rank nature?
- It seems LongMIL supersedes existing positional encodings (Or am I undsteranding this correctly?) - Can they be combined?
- Equation 11, p -> p_{i,j}. Otherwise, readers might confuse it as being uniform probability.
limitations: limitations: Please see Weakness & Questions.

Comments: title: Response
response
comment: Thank you very much for the acknowledgement. We deeply appreciate your time and effort in the review. We will further revise manuscript in next version based on your suggestions.
In response to Ghwr:

I agree that there are many works that focus on efficient / local attention methods, but disagree that these works diminish the contributions of this work. In my view, this work addresses a very nuanced-but-interesting/relevant problem in Transformer architectures for WSIs - namely, how to interpolate positional embeddings to gigapixel imagery + avoiding rank collapse. Comparisons with local attention (HIPT) and efficient attention (TransMIL, GigaPath) are performed, with LongMIL doing better. 

I believe the proposed tricks to Transformer Attention implementation for WSIs would be insightful for researchers working in pathology. I agree that similar works have addressed this issue (e.g., GigaPath), but think the experimentation is different enough and having >1 article investigating this topic is overall useful to have, which is why I have rated this as Borderline / Weak Accept (leaning closer to Weak Accept).
Dear authors,

Thank you for your response. I am happy with the author's rebuttal, and have thus raised my score.
We appreciate the constructive feedback aimed at enhancing the clarity of our paper. We agree with the points and suggestions raised and will address them in the revised manuscript.
Thank you again for your review.
Considering that it is the last day of the discussion period, we would like to confirm whether our rebuttal has adequately addressed the concerns you raised.

We continue to welcome any supplementary observations or clarification to bolster our work.
I revised my rating.
Since the author claims that other work is also addressing local attention, what is the essential difference between local attention in this paper? There are so many local attention innovations in CV and NLP that can be directly applied to pathological images. The number of tokens in the NLP space is much larger than the tens of thousands of tokens here. I don't think it's a neurips level of innovation enough. I respect the opinion of other reviewers, but my final rating is still negative.
Thank you for addressing my concerns - I trust the authors in the revised version will do their best to address the clarity issue I raised. I increase my score accordingly.
> the number of effective patches (256) for a WSI image is between a few thousand and ten thousand.

This is not always the case. There are various cases facing larger patch numbers need computational efficiency:

    1. 40x magnification, with about n=40k~70k patches (also shown in Fig. 8), we have test training speed of full attention, which is about 25x to 35x times than ours. We have talked about this in the caption of Fig. 8, and also show some experimental results of performance in Table 5 of A.6.4. Though currently 40x is not the mainstream, we speculate this is caused by 20x captures better context thus performs better using AB-MIL (without context modelling ability) paradigm. Moreover, there is study [1] show better performance via 40x.
    2. Overlapped patching, it will be resulting 2~4x patches if the overlap ratio is 0.25~0.5. This helps alleviating the edge effect of image modelling neither by CNN nor ViT.
    3. Some slides contain >=2 histology tissues; Survival prediction need multiple slides for one patience.

> if the local attention that cannot be proposed in this paper is not to solve the performance bottleneck and other effects of full attention caused by too many tokens... I have doubts about the actual value in terms of solving computational efficiency.

We find that you are doubting on the motivation of the whole paper, but our paper shares the same motivation to the structure of TransMIL, HIPT and even Prov-Gigapath (to the best of our knowledge, in Prov-Gigapath, they also use patch size 256 and extract patch features from stage-1, then perform stage-2 slide-level 'dilated attention encoder' fine-tuning ). An interesting point is that there are very limited papers using full attention, even in the famous works like UNI and Prov-Gigapath, which we believe, is incurred by its unacceptable complexity. 
The complexity has impeded it being widely use or scaling to larger data setting, including:

    1. 40x, overlapped patching ... as we mentioned above.
    2. pretraining on over 100k slides just like Prov-Gigapath do.
    3. slow speed in the clinical or deployed setting, where the GPU hardware is not so good as training.
    4. using larger feature embedding d (e.g. carrying both high-level and low-level feature) as we mentioned in the line 207~208 of our paper: 'An intuitive modification to handle the low-rank problem is to set a larger embedding size d, but this makes computational complexity O(n^2 d) more severe...'
    5. more transformer layers, as you said.

These prospective directions or future work which may improve the WSI diagnosis or prognosis, we think, can be better implemented via our method. Though is this paper we cannot include all above topics, we are actively working to enhance these factors so that the model becomes a more valuable resource for the community.

### Ref:
[1]. Yu, Jin-Gang, et al. "Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images." Medical Image Analysis 85 (2023): 102748.
Since the two-stage method is adopted in this paper, the first stage has used the pre-trained model to extract features offline. The second stage feature dimension has been smaller. Usually the number of effective patches (256) for a WSI image is between a few thousand and ten thousand, and the second stage network is smaller. In the experiment, the second stage of the network usually only needs a few simple transformer layers, which can be completed in a few minutes or ten minutes of training. In view of this, if the local attention that cannot be proposed in this paper is not to solve the performance bottleneck and other effects of full attention caused by too many tokens, it cannot exceed full attention (I think too many tokens will lead to inefficient learning, full attention is not necessarily better than local attention performance), and I have doubts about the actual value in terms of solving computational efficiency.