TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
Year: 2022

Abstract:
Multiple instance learning (MIL) is a powerful tool to solve the weakly supervised classification in whole slide image (WSI) based pathology diagnosis. However, the current MIL methods are usually based on independent and identical distribution hypothesis, thus neglect the correlation among different instances. To address this problem, we proposed a new framework, called correlated MIL, and provided a proof for convergence. Based on this framework, we devised a Transformer based MIL (TransMIL), which explored both morphological and spatial information. The proposed TransMIL can effectively deal with unbalanced/balanced and binary/multiple classification with great visualization and interpretability. We conducted various experiments for three different computational pathology problems and achieved better performance and faster convergence compared with state-of-the-art methods. The test AUC for the binary tumor classification can be up to 93.09% over CAMELYON16 dataset. And the AUC over the cancer subtypes classification can be up to 96.03% and 98.82% over TCGANSCLC dataset and TCGA-RCC dataset, respectively. Implementation is available at: https://github.com/szc19990412/TransMIL.

heading: 1 Introduction
text: The advent of whole slide image (WSI) scanners, which convert the tissue on the biopsy slide into a gigapixel image fully preserving the original tissue structure [1], provides a good opportunity for the application of deep learning in the field of digital pathology [2, 3, 4]. However, the deep learning based biopsy diagnosis in WSI has to face a great challenges due to the huge size and the lack of pixel-level annotations[5]. To address this problem, multiple instance learning (MIL) is usually adopted to take diagnosis analysis as a weakly supervised learning problem.
In deep learning based MIL, one straightforward idea is to perform pooling operation [6, 7] on instance feature embeddings extracted by CNN. Ilse et al. [8] proposed an attention based aggregation operator, giving each instance additional contribution information through trainable attention weights. In addition, Li et al. [9] introduced non-local attention into the MIL problem. By calculating the similarity between the highest-score instance and the others, each instance is given different
∗Contributed equally: {shaozc0412,h2495067728,sky374263410}@gmail.com. †Corresponding author: ybzhang08@hit.edu.cn.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
attention weight and the interpretable attention map can be obtained accordingly. There were also other pioneering works [10, 11, 12, 13, 14] in weakly supervised WSI diagnosis.
However, all these methods are based on the assumption that all the instances in each bag are independent and identically distributed (i.i.d.). While achieving some improvements in many tasks, this i.i.d. assumption was not entirely valid [15] in many cases. Actually, pathologists often consider both the contextual information around a single area and the correlation information between different areas when making a diagnostic decision. Therefore, it would be much desirable to consider the correlation between different instances in MIL diagnosis.
At present, Transformer is widely used in many vision tasks [16, 17, 18, 19, 20, 21] due to the strong ability of describing correlation between different segments in a sequence (tokens) as well as modelling long distance information. As shown in Figure 1, different from bypass attention network in the existing MIL, the Transformer adopts self-attention mechanism, which can pay attention to the pairwise correlation between each token within a sequence. However, traditional Transformer sequences are limited by their computational complexity and can only tackle shorter sequences (e.g., less than 1000) [22]. Therefore, it is not suitable for large size images such as WSIs.
To address these challenges mentioned above, we proposed a correlated MIL framework, including the convergence proof and a generic three-step algorithm. In addition, a Transformer based MIL (TransMIL) was devised to explore both morphological and spatial information between different instances. Great performance over various datasets demonstrate the validity of the proposed method.

heading: 2 Related Work

heading: 2.1 Application of MIL in WSI classification
text: The application of MIL in WSIs can be divided into two categories. The first one is instance-level algorithms [7, 23, 24, 25, 26], where a CNN is first trained by assigning each instance a pseudo-label based on the bag-level label, and then the top-k instances are selected for aggregation. However, this method requires a large number of WSIs, since only a small number of instances within each slide can actually participate in the training. The second category is embedding-level algorithms, where each patch in the entire slide is mapped to a fixed-length embedding, and then all feature embeddings are aggregated by an operator (e.g., max-pooling). To improve the performance, the MIL attention based method [8, 10, 11, 12, 13] assigns the contribution of each instance by introducing trainable parameters. In addition, the feature clustering methods [14, 27, 28] calculated the cluster centroids of all the feature embeddings and then the representative feature embeddings was employed to make the final prediction. Recently, non-local attention [9] was also adopted in MIL to pay more attention to the correlation between the highest-score instance and all the remaining instances.

heading: 2.2 Attention and Self-attention in Deep Learning
text: Attention was initially used to extract important information about sentences in machine translation [29]. Then the attention mechanism was gradually applied to computer vision tasks, including giving different weights to feature channels [30] or spatial distribution [31], or giving different weights to time series in video analysis [32]. Recently, attention was also applied in MIL analysis [10, 11, 12, 13]. However, all these methods did not consider the correlation between different instances.
The most typical self-attention application was the Transformer based NLP framework proposed by Google [33]. Recently, Transformer was also applied in many computer vision tasks, including object detection [16, 17], segmentation [18, 19], image enhancement [20, 21] and video processing [34]. In this paper, for the first time, we proposed a Transformer based WSI classification, where the correlations among different instances within the same bag are comprehensively considered.

heading: 3 Method

heading: 3.1 Correlated Multiple Instance Learning
text: Problem formulation Take binary MIL classification as an example, we want to predict a target value Yi ∈ {0, 1}, given a bag of instances {xi,1,xi,2, . . . ,xi,n} with Xi, for i = 1, . . . , b, that exhibit both dependency and ordering among each other. The instance-level labels {yi,1, yi,2, . . . , yi,n} are unknown, and the bag-level label is Yi, for i = 1, . . . , b. A binary MIL classification can be defined as:
Yi =
{ 0, iff ∑ yi,j = 0 yi,j ∈ {0, 1}, j = 1 . . . n
1, otherwise (1)
Ŷi = S(Xi), (2) where S is a scoring function, Ŷi represents the prediction. b is the total number of bags, n is the number of instances in ith bag, and the number of n can vary for different bags.
Compared to the MIL framework proposed by Ilse et al. [8], we further introduce the correlation between different instances. Theorem 1 and Inference give an arbitrary approximation form of the scoring function S(X), and Theorem 2 provides the advantage of correlated MIL.
Theorem 1. Suppose S : X → R is a continuous set function w.r.t Hausdorff distance dH(·, ·). ∀ε > 0, for any invertible map P : X → Rn,∃ function σ and g, such that for any X ∈ X :
|S(X)− g( P X∈X {σ(x) : x ∈ X})| < ε. (3)
That is: a Hausdorff continuous function S(X) can be arbitrarily approximated by a function in the form g( P
X∈X {σ(x) : x ∈ X}).
Proof. By the continuity of S, we take ∀ε > 0,∃ δε, so that |S(X)−S(X′)| < ε for any X,X′ ∈ X , if dH (X,X′) < δε.
Define K = ⌈ 1δε ⌉ and define an auxiliary function: σ(x) = ⌊Kx⌋ K . Let X̃ = {σ(x) : x ∈ X}, then:
|S(X)− S(X̃)| < ε, (4) because dH(X, X̃) < 1K ≤ δε.
Let P : X → Rn be any invertible map, its inverse mapping is expressed as P−1: Rn → X . Let g = S ( P−1 ) , then:
S ( P−1( P
X∈X ({σ(x) : x ∈ X}))
) = S ( P−1( P
X̃∈X (X̃))
) = S(X̃). (5)
Because |S(X)− S(X̃)| < ε and S(X̃) = S ( P−1(P (X̃)) ) = g(P (X̃)), we have:
|S(X)− g( P X∈X {σ(x) : x ∈ X})| < ε. (6)
This completes the proof.
Inference Suppose S : X → R is a continuous set function w.r.t Hausdorff distance dH(·, ·). ∀ε > 0, for any function f and any invertible map P : X → Rn,∃ function h and g, such that for any X ∈ X :
|S(X)− g( P X∈X {f(x) + h(x) : x ∈ X})| < ε. (7)
That is: a Hausdorff continuous function S(X) can be arbitrarily approximated by a function in the form g( P
X∈X {f(x) + h(x) : x ∈ X}).
Proof. The proof is in the Appendix A.
Theorem 2. The Instances in the bag are represented by random variables Θ1,Θ2, . . . ,Θn, the information entropy of the bag under the correlation assumption can be expressed as H (Θ1,Θ2, . . . ,Θn), and the information entropy of the bag under the i.i.d. (independent and identical distribution) assumption can be expressed as ∑n t=1 H (Θt), then we have:
H (Θ1,Θ2, . . . ,Θn) = n∑ t=2 H (Θt | Θ1, . . . ,Θt−1) +H (Θ1) ≤ n∑ t=1 H (Θt) . (8)
Proof. The proof is in the Appendix A.
Theorem 2 proved the correlation assumption has smaller information entropy, which may reduce the uncertainty and bring more useful information for the MIL problem. Motivated by the Inference and Theorem 2, a generic three-step method like Algorithm1 was developed. The main difference between the proposed algorithm and existing methods is shown in Figure 2.
Algorithm 1: A generic three-step approach under the correlated MIL Input: The bag of instances Xi = {xi,1,xi,2 . . . ,xi,n} Output: Bag-level predicted label Ŷi
1) Extracting morphological and spatial information of all the instances by f and h, respectively;
Xf ← f(Xi), Xh ← h(Xi), Xfh ← Xf +Xh, where Xf ,Xh,Xfh ∈ Rn×d; 2) Aggregating the extracted information for all instances by Pooling Matrix P; XP ← PXfh, where P ∈ Rn×n; 3) Transforming XP to obtain the predicted bag-level label by g; Ŷi ← g(XP ).

heading: 3.2 How to apply Transformer to correlated MIL
text: The Transformer uses a self-attention mechanism to model the interactions between all tokens in a sequence, and the adding of positional information further increases the use of sequential order information. Therefore it’s a good idea to introduce the Transformer into the correlated MIL problem where the function h encodes the spatial information among instances, and the Pooling Matrix P uses self-attention for information aggregation. To make this clear, we further give a formal denition.
Transformer based MIL. Given a set of bags {X1,X2, . . . ,Xb} , and each bag Xi contains multiple instances {xi,1,xi,2, . . . ,xi,n} and a corresponding label Yi. The goal is to learn the mappings: X → T → Y, where X is the bag space, T is the Transformer space and Y is the label space. The specific mapping form of X→ T and T→ Y are available in the Appendix B.

heading: 3.3 TransMIL for Weakly Supervised WSI Classication
text: To better describe the mapping of X → T, we design a TPT module with two Transformer layers and a position encoding layer, where Transformer layers are designed for aggregating morphological information and Pyramid Position Encoding Generator (PPEG) is designed for encoding spatial information. The overview of proposed Transformer based MIL (TransMIL) is shown in Figure 3.
Long Instances Sequence Modelling with TPT. The sequences are from the feature embeddings in each WSI. The processing steps of the TPT module are shown in Algorithm 2, where MSA denotes Multi-head Self-attention, MLP denotes Multilayer Perceptron, and LN denotes Layer Norm.
Algorithm 2: TPT module processing flow Input: A bag of feature embeddings Hi = {hi,1, . . . ,hi,n}, where hi,j ∈ R1×d is the embedding of the jth instance, Hi ∈ Rn×d Output: Bag-level predicted label Ŷi
1) Squaring of sequence;√ N ← ⌈ √ n⌉, M ← N − n, HS ← Concat (hi,class, Hi, (hi,1, . . . ,hi,M )), where
hi,class ∈ R1×d represents class token, HS ∈ R(N+1)×d; 2) Correlation modelling of the sequence; HℓS ← MSA(HS), where ℓ denotes the layer index of the Transformer, HℓS ∈ R(N+1)×d; 3) Conditional position encoding and local information fusion; HPS ← PPEG ( HℓS ) , where HPS ∈ R(N+1)×d;
4) Deep feature aggregation; Hℓ+1S ← MSA ( HPS ) , where Hℓ+1S ∈ R(N+1)×d; 5) Mapping of T→ Y; Ŷi ← MLP ( LN ( (Hℓ+1S ) (0) )) , where (Hℓ+1S ) (0) ∈ R1×d represents class token.
For most cases, the softmax used in Transformer for vision tasks such as [17, 18, 35] is a row-byrow softmax normalization function. The standard self-attention mechanism requires the calculation of similarity scores between each pair of tokens, resulting in both memory and time complexity of O(n2). To deal with the long instances sequence problem in WSIs, the softmax in TPT adopts the Nystrom Method proposed in [22]. The approximated self-attention form Ŝ can be defined as:
Ŝ = softmax ( QK̃T√
dq
)( softmax ( Q̃K̃T√
dq
))+ softmax ( Q̃KT√
dq
) , (9)
where Q̃ and K̃ are the m selected landmarks from the original n dimensional sequence of Q and K, and A+ is a Moore-Penrose pseudoinverse of A. The final computational complexity is reduced from O(n2) to O(n). By doing this, the TPT module with approximation processing can satisfy the case where a bag contains thousands of tokens as input.
Position encoding with PPEG. In WSIs, the number of tokens in the corresponding sequence often varies due to the inherently variable size of the slide and tissue area. In [36] it is shown that the adding of zero padding can provide an absolute position information to convolution. Inspired by this, we designed the PPEG module accordingly. The overview is shown in Figure 4, and the pseudo-code for the processing is shown in Algorithm 3.
Figure 4: Pyramid Position Encoding Generator. 1) The sequence is divided into patch tokens and class token; 2) Patch tokens are reshaped into 2-D image space; 3) Different sized convolution kernels are used to encode spatial information; 4) Different spatial information are fused together; 5) Patch tokens are flattened into sequence; 6) Connect class token and patch tokens.
Our PPEG module has more advantages over the method proposed in [37]: (1) PPEG module uses different sized convolution kernels in the same layer, which can encode the positional information with different granularity, enabling high adaptability of PPEG. (2) Taking advantage of CNN’s ability to aggregate context information, the tokens in the sequence is able to obtain both global information and context information, which enriches the features carried by each token.
Algorithm 3: PPEG processing flow Input: A bag of feature embeddings HℓS after correlation modelling, where HℓS ∈ R(N+1)×d. Output: The feature embeddings HPS after conditional position encoding and local information
fusion, where HPS ∈ R(N+1)×d. 1) Split: HℓS is divided into patch tokens Hf and class token Hc; Hf ,Hc ← Split ( HℓS ) , where Hf ∈ RN×d,Hc ∈ R1×d; 2) Spatial Restore: patch tokens Hf are reshaped to H f S in the 2-D image space; HfS ← Restore (Hf ), where H f S ∈ R √ N× √ N×d;
3) Group Convolution: using a set of group convolutions with kernel k and k−12 zero paddings(k = 3, 5, 7) to obtain Hft , t = 1, 2, 3;
Hft ← Conv ( HfS ) , where Hft ∈ R √ N× √ N×d, t = 1, 2, 3;
4) Fusion: HfS and the H f t , t = 1, 2, 3 obtained from the convolution block processing are
added together to obtain HFS ; HFS ← H f S +H f 1 +H f 2 +H f 3 , where H F S ∈ R √ N× √ N×d;
5) Flatten: HFS are flattened into sequence Hse; Hse ← Flatten ( HFS ) , where Hse ∈ RN×d; 6) Concat: connect Hse and class token Hc to obtain HPS ; HPS ← Concat (Hse,Hc), where HPS ∈ R(N+1)×d.

heading: 4 Experiments and Results
text: To demonstrate the superior performance of the proposed TransMIL, various experiments were conducted over three public datasets: CAMELYON16, The Caner Genome Atlas (TCGA) non-small cell lung cancer (NSCLC), as well as the TCGA renal cell carcinoma (RCC).
Dataset. CAMELYON16 is a public dataset for metastasis detection in breast cancer, including 270 training sets and 130 test sets. After pre-processing, a total of about 3.5 million patches at ×20 magnification, in average about 8,800 patches per bag were obtained.
TCGA-NSCLC includes two subtype projects, i.e., Lung Squamous Cell Carcinoma (TGCA-LUSC) and Lung Adenocarcinoma (TCGA-LUAD), for a total of 993 diagnostic WSIs, including 507 LUAD slides from 444 cases and 486 LUSC slides from 452 cases. After pre-processing, the mean number of patches extracted per slide at ×20 magnification is 15371. TCGA-RCC includes three subtype projects, i.e., Kidney Chromophobe Renal Cell Carcinoma (TGCA-KICH), Kidney Renal Clear Cell Carcinoma (TCGA-KIRC) and Kidney Renal Papillary Cell Carcinoma (TCGA-KIRP), for a total of 884 diagnostic WSIs, including 111 KICH slides from 99 cases, 489 KIRC slides from 483 cases, and 284 KIRP slides from 264 cases. After preprocessing, the mean number of patches extracted per slide at ×20 magnification is 14627.
Experiment Setup and Evaluation Metrics. Each WSI is cropped into a series of 256×256 nonoverlapping patches, where the background region (saturation<15) is discarded. In CAMELYON16 we trained on the official training set after splitting the 270 WSIs into approximately 90% training and 10% validation, and tested on the official test set. For TCGA datasets, we first ensured that different slides from one patient do not exist in both the training and test sets, and then randomly split the data in the ratio of training:validation:test = 60:15:25. For the evaluation metrics, we used accuracy and area under the curve (AUC) scores to evaluate the classification performance, where the accuracy was calculated with a threshold of 0.5 in all experiments. For the AUC, the official test set AUC was used on the CAMELYON16 dataset, the average AUC was used on the TCGANSCLC dataset, and the average one-versus-rest AUC (macro-averaged) was used on the TCGARCC dataset. All the results over TCGA datasets are obtained by 4-fold cross-validation.
Implementation Details. In the training step, cross-entropy loss was adopted, and the Lookahead optimizer [38] was employed with a learning rate of 2e-4 and weight decay of 1e-5. The size of mini-batch B is 1. As in [13], the feature of each patch is embedded in a 1024-dimensional vector by a ResNet50 [39] model pre-trained on ImageNet. During training, the dimension of each feature embedding is reduced from 1024 to 512 by a fully connected layer. Finally, the feature embedding of each bag can be represented as Hi ∈ Rn×512. In the inference step, the softmax is used to normalize the predicted scores for each class. All experiments are done with a RTX 3090.
Baseline. The baselines we chose include deep models with traditional pooling operators such as mean-pooling, max-pooling and the current state-of-the-art deep MIL models [8, 9, 13, 23, 40], the attention based pooling operator ABMIL [8] and PT-MTA [40], non-local attention based pooling operator DSMIL [9], single-attention-branch CLAM-SB[13], multi-attention-branch CLAM-MB[13], and recurrent neural network(RNN) based aggregation MIL-RNN [23].

heading: 4.1 Results on WSI classification
text: We will present the results of both binary and multiple classification. The binary classification tasks contain positive/negative classification over CAMELYON16 and LUSC/LUAD subtypes classification over TCGA-NSCLC. The multiple classification refers to TGCA-KICH/TCGA-KIRC/TCGAKIRP subtypes classification over TCGA-RCC. All the results are provided in Table 1.
In CAMELYON16, only a small portion of each positive slide contains tumours (averagely total cancer area per slide <10%), resulting in the presence of a large number of negative regions disturbing the prediction of positive slide. The bypass attention-based methods and proposed TransMIL all outperform the traditional pooling operators. However in AUC score, TransMIL was at least 5% higher than ABMIL, PT-MTA and CLAM which neglect the correlation between instances, and
do not consider the spatial information between patches. DSMIL only considers the relationship between the highest scoring instance and others, leading to limited performance.
In TCGA-NSCLC, positive slides contain relatively large areas of tumour region (averagely total cancer area per slide >80%), consequently the pooling operator can achieve better performance than in CAMELYON16. Again, TransMIL performed better than all the other competing methods, achieving 1.40% higher in AUC and 2.16% in accuracy, compared with the second best method.
In TCGA-RCC, as MIL-RNN did not consider the multi-classification problem, it was not included in this comparison result. The TCGA-RCC is unbalanced distributed in cancer subtypes and has large areas of tumour region in the positive slides (averagely total cancer area per slide >80%). However, TransMIL is equally applicable to multi-class problems with unbalanced data. It can be observed that TransMIL achieves best results in both accuracy and AUC score.

heading: 4.2 Ablation Study
text: To further determine the contribution of the PPEG module and the conditional position encoding for the performance, we have conducted a series of ablation studies. Since the high classification accuracy of most methods over TCGA-RCC is not obvious, all ablation study experiments are based on the CAMELYON16 and the TCGA-NSCLC dataset. All experiments were evaluated by AUC.

heading: 4.2.1 Effects of PPEG
text: The position encoding of the Transformer typically explores absolute position encoding (e.g., sinusoidal encoding, learnable absolute encoding) as well as conditional position encoding. However, learnable absolute encoding is commonly used in problems with fixed length sequences, and does not meet the requirement for variable length of input sequences in WSI analysis, so it is not taken into account in this paper. Here, we compared the effect of sinusoidal encoding and PPEG module which represents multi-level conditional position encoding. The same experiments are performed over CAMELYON16 and TCGA-NSCLC dataset, and the results are shown in Table 2. It should be noted that sinusoidal encoding is added to the original sequence with a multiplication of 0.001 as described in [33].
Table 2: Effects of PPEG.
Model Params Camelyon16 NSCLC
w/o 2.625M 0.8416 0.9287 sin-cos 2.625M 0.8941 0.9374
3×3 2.630M 0.8913 0.9355 7×7 2.651M 0.9015 0.9336 both 2.669M 0.9059 0.9402
PPEG 2.669M 0.9309 0.9603
Figure 5: Effects of Positional Encoding.
Compared with the model without position encoding, it can be seen that both sinusoidal encoding and conditional position encoding can improve the classification performance, and conditional position information encoded by PPEG can be more effective in diagnosis analysis. In contrast to the 3 × 3 and 7 × 7 convolutional block, adding different sized convolution kernels in the same layer allows for multi-level positional encoding and adds more context information to each token.

heading: 4.2.2 Effects of Conditional Position Encoding
text: Here, by disrupting the order of the input sequences, we explore actual improvements for conditional position encoding. The performance of the model under different configurations is shown in Figure 5, where order represents sequential data input and w/o represents random and disordered data input. It can be seen that conditional position information did enhance the model performance, e.g., the improvement can be up to 0.9% over CAMELYON16 and 0.61% over TCGA-NSCLC in AUC.
Compared with the model without position encoding or with sinusoidal encoding, conditional position information encoded by PPEG can be more effective in diagnosis analysis. Compared with the results trained over the sequential and disordered training sets, conditional position information did enhance the model performance, e.g., the improvement can be up to 0.9% over CAMELYON16 and 0.61% over TCGA-NSCLC in AUC.

heading: 4.3 Interpretability and Attention Visualization
text: Here, we will further show the interpretability of TransMIL. As shown in Figure 6(a), the area within the blue curve annotation is the cancer region, which was provided by Gao et al. [41] over the TCGA-RCC dataset. In Figure 6(b), attention scores from TransMIL were visualised as a heatmap to determine the ROI and interpret the important morphology used for diagnosis, and Figure 6(c) is a zoomed-in view of the black square in Figure 6(b). Obviously, there is a high consistency between fine annotation area and heatmap, illustrating great interpretability and attention visualization of the proposed TransMIL.

heading: 4.4 Fast Convergence
text: Traditional MIL methods as well as the latest MIL methods such as ABMIL, DSMIL and CLAM usually require a large number of epochs to converge. Different from these methods, TransMIL makes use of the morphological and spatial information among instances, leading to approximately two to three times fewer training epochs. As shown in Figure 7, TransMIL has better performance in terms of convergence and validation AUC than other MIL methods.

heading: 5 Conclusion
text: In this paper, we have developed a novel correlated MIL framework that is consistent with the behavior of pathologists considering both the contextual information around a single area and the correlation between different areas when making a diagnostic decision. Based on this framework, a
Transformer based MIL (TransMIL) was devised to explore both morphological and spatial information in weakly supervised WSI classification. We also design a PPEG for position encoding as well as a TPT module with two Transformer layers and a position encoding layer. The TransMIL network is easy to train, and can be applied to unbalanced/balanced and binary/multiple classification with great visualization and interpretability. Most importantly, TransMIL outperforms the state-of-the-art MIL algorithms in terms of both AUC and accuracy over three public datasets. Currently, all the experiments were conducted over the dataset with × 20 magnification, however the WSIs with higher magnification will result in longer sequence and inevitably pose great challenges in terms of both computational and memory requirements, and we will explore this issue in the follow-up work.
Broader Impact Our proposed approach shows greater potential for MIL application to real-world diagnosis analysis, particularly in problems that require more correlated information such as survival analysis and cancer cell spread detection. In the short term, the benefit of this work is to provide a model with better performance, faster convergence and clinical interpretability. In the long term, the proposed TransMIL network is more applicable to real situations, and it is hoped that it will provide more novel and effective ideas about further applications of deep MIL to diagnosis analysis.
Acknowledgment This work was supported in part by the National Natural Science Foundation of China (61922048&62031023), in part by the Shenzhen Science and Technology Project (JCYJ20200109142808034), and in part by Guangdong Special Support (2019TX05X187).

Review summaries:
	The authors propose a Transformer-based MIL method for WSI classification.   The method is designed to avoid the IID assumption inherent to standard MIL inference methods (e.g. mean, max pooling). Secondly, they propose a positional encoding module for patches in WSIs based on CNNs to incorporate spatial information into the prediction, along with semantic information learned in the MSA layers. In the ablation study, they demonstrated that PPEG model performed better than conventional sin-cos positional encoding. They used three WSI classification datasets from open sources with previously benchmarked results. The authors were able to achieve SOTA results on each task.  

The paper presents a transformer based approach for WSI classification, correlation among instances are considered to address the iid assumed by previous methods. The methods were tested on three public datasets, i.e. Camelyon16, TCGA-NSCLC and TCGA-RCC datasets, and the results suggest that the performance for binary and multiple category classification is better than state of the art. 

The authors propose a transformer based model for the classification of whole slide histopathology images. The method first extract ResNet features locally across WSIs, then apply a transformer based model on top of the features. The authors also propose a Pyramid Position Encoding Generator (PPEG) module for transforming local features and encode positional information. The proposed framework is general and can be applied to other non-medical image analysis tasks.  The proposed method achieves better result compared to recent baselines.