CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images
Authors: Olga Fourkioti
Matt De Vries
Chris Bakal
Venue: ICLR 2024 spotlight (2024)
Keywords: Multiple Instance Learning
Histopathology
Nearest Neighbors
Graph Representation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We develop a multiple instance learning model for cancer detection and subtyping in whole slide images that leverages dependencies and imposis contextual constraints between tiles. This lead to state-of-the-art classification on two public datasets.

Abstract: The visual examination of tissue biopsy sections is fundamental for cancer diagnosis, with pathologists analyzing sections at multiple magnifications to discern tumor cells and their subtypes. However, existing attention-based multiple instance learning (MIL) models used for analyzing Whole Slide Images (WSIs) in cancer diagnostics often overlook the contextual information of tumor and neighboring tiles, leading to misclassifications. To address this, we propose the Context-Aware Multiple Instance Learning (CAMIL) architecture. CAMIL incorporates neighbor-constrained attention to consider dependencies among tiles within a WSI and integrates contextual constraints as prior knowledge into the MIL model. We evaluated CAMIL on subtyping non-small cell lung cancer (TCGA-NSCLC) and detecting lymph node (CAMELYON16 and CAMELYON17) metastasis, achieving test AUCs of 97.5\%, 95.9\%, and 88.1\%, respectively, outperforming other state-of-the-art methods. Additionally, CAMIL enhances model interpretability by identifying regions of high diagnostic value. Our code is available at https://github.com/olgarithmics/ICLR_CAMIL.

Published as a conference paper at ICLR 2024
CAMIL: CONTEXT-AWARE MULTIPLE INSTANCE LEARNING FOR CANCER DETECTION AND SUBTYP- ING IN WHOLE SLIDE IMAGES
Olga Fourkioti, Matt De Vries & Chris Bakal
The Institute of Cancer Research, London, United Kingdom [ofourkioti; mvrries cbakal fiicr. wea.:eicc.uuuu....... ac.uk
ABSTRACT
The visual examination of tissue biopsy sections is fundamental for cancer diag- nosis, with pathologists analyzing sections at multiple magnifications discern tumor cells and their subtypes. However, existing uttention-based multiple in- stance learning (MIL) models used for analyzing Whole Slide Images (WSIs) cancer diagnostics often overlook the contextual information of tumor and neigh- boring tiles, leading to misclassifications To address this, we propose the Context- Aware Multiple Instance Learning (CAMIL) architecture. CAMIL incorporates neighbor-constrainee attention to consider dependencies among tiles within WSI and integrates contextual constraints as prior knowledge into the MIL model. We evaluated CAMIL on subtyping non-small cell lung cancer (TCGA-NSCLC) and detecting lymph node (CAMELYON16 and CAMELYONI7) metastasis, achiev- ing test AUCS of 97.5%, 95.9%, and 88.1%, respectively, outperforming other state-of-the-ard methods. Additionally, CAMIL enhances model interpretability by identifying regions of high diagnostic value)
INTRODUCTION
Deep learning (DL) methods have revolutionized the development of highly accurate diagnostic machines (Morales et al. 2021) that rival or even surpass the performance of expert pathologists
[Tong et al. Melendez et al., 2015) Quellec et al. [2166] Das al., [018 (Siindhi al.) 2019] Wang et al. 2021). These advancements have been facilitated by the emergence of weakly super- learning which eliminates the need for laborious pixel-level annotations. Models trained using weakly supervised learning. relying solely on slide-level labels, have demonstrated exceptional clas- sification accuracy on whole slide imaging (WSI) data, paving the way for scalable computational
decision support systems in clinical practice (Xu al. 2014 Courtiol et al. 2018 Xu et al. 2019 Zhou et al. 202T).
In the context of cancer histopathology. WSIs Instead, WSIs are frequently subdivided into smaller tiles, which serve as an input. The task is, then, to classify the WSI based on the features extracted from the individual tiles. Most current methods for weakly supervised WSI classification use the Multiple Instance Learning (MIL) framework, which considers each WSI as 'bag' of tiles and attempts learn the slide-level label without prior knowledge about the labels of the individual tiles.
processed single image by DL models.
A major bottleneck in the deployment of MIL models, and the weakly-supervised learning paradigm in general, is that the MIL model is either permutation invariant, meaning that the tiles within a WSI exhibit no ordering among each other (Sharma et al., [202 Xie al. 2020). or permutation aware without explicit information guidance. in other words, the spatial relationship of one tile to another is either ignored, or the dependencies between the tiles are implicitly modeled during training without
requiring direct instructions (Shao al. 202 Landini al. 2020 Campanella al. 2019).
However, explicit knowledge about a tile's spatial arrangement particularly relevant in cancer histopathology. where cancer and normal cells are not necessarily distributed randomly inside an
Our code is available at
gari thmi cs/ (CIR_CAM)

---

Published as a conference paper at ICLR 2024
image. Contextual insights into the cellular landscape, such as the spatial dispersion of cells, the arrangement of cell clusters, and the broader characteristics of the tissue microenvironment, provide more comprchensive view of the tile's local environment, enabling better assessment of subtle variations and abnormalities that may indicate the presence of cancer.
In this paper, we propose novel framework dubbed Context-Aware Multiple Instance Learning (CAMIL) to harness the dependencies among the individual tiles within a WSI and impose contex- tual constraints as prior knowledge on the multiple instance learning model. By explicitly accounting for contextual information, CAMIL aims to enhance the detection and classification of localized tu- mors and mitigate the potential misclassification of isolated or noisy instances, thereby contributing to an overall improvement in performance for both individual tiles and WSIs. Moreover, the atten- tion weights enhance the interpretability of the model by highlighting sub-regions of high diagnostic value within the WSI.
2
RELATED WORK
Under the MIL formulation, the prediction of a WSI label (i.e., cancerous or not) can come either directly from the tile predictions (instance-based) (Campanella et al. 2019 Landini et al. 2020 Hou
(t al. at 2016 Xu et al. 2019). or from a higher-level bag representation resulting from the aggregation of the tile features (bag embedding-
(Ilse et al. 2018; Lu et al. 2021: Sharma et al. 2021
Wang et al.] 2018). The bag bebbbeddinn--bssee approach has empirically demonstrated superior performance (Sharma et al. [202 Wang et al. 2018). Most recent bag embedding- approaches employ attention mechanisms /Vaswani et al. (017)). which assign an attention score to every tile reflecting its relative contribution to the collective WSI-level representation. Attention scores enable the automatic localization of sub-regions of high diagnostic value in addition to informing the WSI-
level label (Zhang et al. 2021 Ben Taieh & Hamarneh] 2018 Lu al. 202T
Attention tieett MIL models vary in how they explore tissue structure in WSIs. Many are permu- tation invariant, assuming the tiles are independent and identically distributed. Building upon this assumption, llse et al. (2018) proposed learnable attention-based MII pooling operator that com- putes the bag embedding as the average of all tile features in the WSI weighted by their respective attention score. This operator has been widely adopted and modified with the addition of clus- tering layer (Lu et al. 2021 Li et al. 2021b Yao et al., 2020 to further encourage the learning of semantically-rich, separable and class-specific Teatures. Another variation of the same model uses "pseudo bags' (Zhang et al. 2022). splitting the WSI into several smaller bags to alleviate the issue of the limited number of training data. Recently, data augmentation has been adopted to inflate the
number of bags (Gadermayr et al. 2023 Liu et al. 2023 Shao et al., 2023).
However, permutation invariant operators cannot inherently capture the structural dependencies among different tiles at the input. The lack of bio-topological information has partially been reme- died by the introduction of feature similarity scores instead of positional encodings to model the mutual tile dependencies within a WSI (Xie et al. 2020 Tellez et al. 2022 Adnan et al. 2020). For instance, DSMIL (Li et al., 2021a) utilizes non-local operator to compute an attention score for each tile by measuring the similarity of its feature representation against that of a critical tile. To consider the correlations between the different tiles of a WSI, transformer-bait based architectures have been introduced, which usually make use of a learnable position-dependent signal to incorporate the spatial information of the image (Zhao et al.) TTu et al. (019). For instance, TransMII. (Shao al., 2021) is transformer-like architecture trained end-to-end to optimize for the classification task and produce attention scores while simultaneously learning the positional embeddings.
In CAMIL, we provide explicit guidance regarding the context of every tile we argue that it can provide a valuable, rich source of information. Unlike most existing MIL approaches where the relationships developed between neighboring tiles are omitted, in approach, we propose neural network architecture that explicitly leverages the dependencies between neighboring tiles of a WSI by enforcing bio-topological constraints to enhance performance effectively.
2

---

Published as a conference paper at ICLR 2024
preprocessing
Neighhbor-constraineed attention module
2. Feature extraction
3. Nystrom- former
Value
Feature aggregation classification
250
F(x)
T(h)
Key
n.
Fixl Backbone feature extractor
Query
ufltaax
product
matrix
neighborhsod
pooling
product
sumn pooling
Figure 1: An overview of the CAMIL model architecture. First, WSIs are preprocessed to separate tissue from the background. Then, the WSIs are split into fixed-size tiles of size 256 x 256 and fed through pre- trained feature extractor obtain feature representations size 1024 for each tile. A Nystromformer module then transforms these feature embeddings. These transformed feature embeddings are then used as input to our neighbor-constraincc attention module. This module allows attending over each patch and its neighboring patches, generating neighborhood descriptor of cach tilc's closest neighbors, and calculating their attention coefficients. The output layer then aggregates the tile-level attention scores produced in the previous layer emit a final slide classification score.
MATERIAL AND METHODS
CAMIL operates on the principle that the context and characteristies of a tile's surroundings hold substantial potential for enhancing the accuracy of whole slide classification. To illustrate this con- cept, we can draw a parallel between our framework and the examination process of pathologist analyzing a biopsy slide. Similar to how pathologist inspects sub-regions to comprehensively understand its broader surroundings, CAMIL expands the tile's view to examine the broader neigh- borhood of each tile thoroughly. This extension allows CAMIL. to gather additional information and facilitates a better understanding and assessment of the surrounding microenvironment and tissue context.
CAMIL, we recalibrate each tile's individual attention score by aggregating the attention scores of its surroundings. For example, tiles with high attention scores surrounded by other high-scoring tiles should be considered important. Conversely, the presence of a tile classified by the model as important in low-scoring neighborhood could be, in some cases, attributed to noise, and this should be reflected in its final attention score.
The overview of CAMIL can be seen in Figure It can be decomposed into five elements:
1. A WSI-preprocessing phase automatically segments the tissue region of each WSI and divides it into many smaller tiles (e.g., 256 256 pixels).
2. A tile and feature extraction module, consisting and linear layers transform the original tile input to low dimensional feature representa- tions: H {hi..... hi...., hy }. Raxd where is the embedding dimensions of a tile, the number of tiles within a WSI (n differs among different WSIs), and N the number of WSIs.
stack of convolutional, max pooling,
3. A Nystromformer module (Xiong al. 2021 transforms the tile embeddings to cise, descriptive hidden feature representation. It is crucial in aggregating global contexts, capturing the overall information and patterns across multiple tiles.
4. A neighbor-constrrinee, attention mechanism in CAMIL, coupled with contrastive learn- ing block, encapsulates the neighborhood prior and focuses on aggregating local concepts.
5. The feature aggregator and classification layer combine the local concepts derived from the previous layer with the features that describe the global contexts obtained from the transformer module. These features are merged to generate prediction at the slide level.
We elaborate each step in the following subsections.
3

---

Published as a conference paper at ICLR 2024
3.1 WSI PREPROCESSING
We followed methods in Lu et al. (2021) to segment tissue regions and split the whole slide image into individual non-overlapping tiles (1. in Figure (details of the hyperparameters used here shown in the appendix).
3.2 FEATURE EXTRACTOR
Effective feature representations significantly impact predictive accuracy, as demonstrated by the success of self- contrastive learning (Chen et al. 2020). Therefore, to extract rich, mean- ingful feature representations from individual tiles, we first train a feature extractor following the SimCLR (Chen al. 2020) approach. SimCLR is one of the most popular self-supervised learning frameworks that enable semantically rich feature representations to be learned by minimizing the distance between different augmented versions of the same image data.
Similar to the training approach followed by Li et al. (2021a), the data sets utilized SimCLR composed of patches derived from WSIs. These patches densely cropped with overlap and treated as separate images for the purpose of SimCLR training. During training, two different augmentations are done on the same tile. These augmentations (color distortion, zoom, rotation, and reflection) using a stochastic data augmentation module. These two augmentations of the same tile are fed through a ResNet-18 (He al. 2015) pre- trained on ImageNet (Deng et al. 2009) with an additional projection head, which is multi-layer perceptron (MLP) with two hidden layers that map the feature representations to space where contrastive loss is applied. The final convolutional block of ResNet- 18 and the projection head are then fine-tuned by minimizing the contrastive loss (temperature-scale cross entropy) between Ei, 2j. corresponding to two 'correlated' (differentially augmented) views of the same tile. Here, we minimized the normalized temperature-scalee cross entropy (NT-Xent) defined as
augmentations
chosen from four possible
log exp(sim(z,, 2,)/T)
li
k=1 [[ySs)] exp(sii((z; /T)
The trained network is then used as the base feature extractor (F(()) in Figure 1) to produce the fea- ture representations
[h1,..... hu............... .u...... hv h, 12d of each WSI, where is the number of tiles and d is the embedding dimension to represent each tile, and N the number of WSIs. This trained feature extractor is frozen when training CAMIL and is only used to extract features and calculate distances between neighboring patches. These distances between neighboring patches are calculated using the sum of squared differences between the features and are used in the following neighbor-constrained attention module.
3.3 TRANSFORMER MODULE TO CAPTURE GLOBAL CONTEXTS
To encode the feature embeddings H. our approach focuses on capturing the inter-tile relationships and dependencies, enhancing the global context understanding, and facilitating comprehensive fea- ture aggregation. This is achieved using a transformer layer, represented as T(h) in Figure which particularly effective for managing the complex structure of WSIs. To address the challenge of memory overload due to the long-range dependencies in large WSIs. we adopt the Nystrom- former architecture (Xiong et al., 2027], enabling CAMIL to model intricate feature interactions through an efficient, approximate self-attention mechanism. This produces "transformed" feature set T (t,....,t,,......... (N with each t; E Rnndd where,
(T Vah
(A)
@i(h)) (hi)
t, softmax
softmax
V1 (ha),
vdz
where (h4) and (h4) are the selected landmarks (see Xiong et al.
2021 and Appendix) +
(1(h)) vdl
from the original n-dimensional sequence of Q and K,, A+ softmax
the approximate inverse of A. Softmax is applied along the rows of the matrix. K, (hz), Q (hi).
4

---

Published as a conference paper at ICLR 2024
and V1 Vi(h;) are the first key, query, and value representations of h; shown T(h) in Figure | and defined in Vaswani et al. (2017).
3.4 NEIGHBOR-CONSTRAINE ATTENTION MODULE TO CAPTURE LOCAL CONTEXTS
The neighbor-constrane patterns within localized areas of the slide. It focuses on the immediate neighborhood of each tile and aims to capture the local relationships and dependencies within that specific region. By doing so, the module can emphasize the relevance and importance of nearby tiles, effectively incorporating fine-grained details and local nuances into the model.
attention module in CAMIL is designed to capture specifie features and
To model the tile and its surroundings, undirected graph G = (V, E), where V represents the set of nodes representing image tiles, and E represents the set of edges between nodes indicating adjacency. The graph can be represented by an adjacency matrix A with elements As,, where A = Sij if there exists an edge (e,, UU) E E and A.,j otherwise. Each image tile must be connected to other tiles and can be surrounded by eight adjacent patches. Each element of the matrix Sij represents the degree of similarity or resemblance between two connected tiles and is calculated as follows:
construct weighted adjacency matrix. Consider an
exp -h,, 0,
(o,, 25) E otherwise.
(3)
This design ensures injecting a -hootopollgical prior such that the weight of a tile is dependent on adjacent tiles with similar pattern.
The transformed tile representations T= (lq...., .,..,.,, bN} are again transformed by the weight matrices Wg E R=xd, W₂ Raxds and W. E Roxd, into three distinct representations: the query representation Qtt,) = WTt, the key representation ((t,) = W t. and the value repre- sentation (t,)) = WTt, where dy = dN = d, = The dot product of every query with all the key vectors produces attention matrix whose elements determine the correlation between the different tiles of a WSI (4. in Fi
The similarity mask is element wise multiplied with the dot product of the query and key embed- dings, generating a masked attention matrix whose non-zero elements reflect the contribution of a tile's neighbors the tile score.
After obtaining the attention coefficients that correspond to the neighbors of every tile, the last step to aggregate this contextual information to generate single attention weight. For each tile, sum the coefficients of their neighbors. The resultant tile score vector is passed through a softmax function to ensure all weights sum to one.
Therefore, the attention coefficient of the ith tile of a WSI is given by the following equation, where (.) denotes the inner product between two vectors:
Σn (Q(t))
exp
W/
(4)
Σ
Σ)=((0Q0 ((;;))s13
k= exp
The feature embeddings E Radd are then computed and weighted by their respective attention score to give neighbor-consstranee feature vector, 4, for each tile:
= wV
(5)
3.5 FEATURE AGGREGATION AND SLIDE-LEVEL PREDICTION
The mechanism utilized to fuse the local and global value vectors allows for the adaptive blending of local and global information described in Equation 6 The sigmoid function applied to the local values serves as a weighting factor, enabling the model to emphasize local characteristics when they deemed more relevant while still retaining the contribution from the global contexts.
5

---

Published as a conference paper at ICLR 2024
m m= 0l) Ot,
(6)
where o) denotes the sigmoid non-linearity.
The collective, WSI-level representation m E R is adaptively computed as the weighted average of the fused vector:
a (m.);
such that:
0; = exp w" (tanh( v o (U(t)))
(8)
k Σラ= exp un (tanh o (UI)))
where U. V, and w are learnable parameters, C is an element wise multiplication, and tanh() is the hyperbolic tangent function.
CAMIL achieves a synergistic effect by combining the value vector of the neighbor ceabbor-conntraieed at- tention module with that of the transformer layer. The transformer layer captures global interactions and dependencies across the entire slide, while the neighbor-constrainee attention module comple- ments it by capturing local details and context. Together, they enable CAMIL to integrate both local and global perspectives effectively.
Finally, the slide-level prediction is given via the classification layer W. Rexd
T
Yslide
W.
Σ
Zi
(9)
where e corresponds to the number of classes and Σ the sum pooling operation applied on z. The representation obtained from the high-attended patches used to minimize cross-entropy loss, and a final classification score is produced.
4 EXPERIMENTS AND RESULTS
To demonstrate the performance of CAMIL in capturing informative contextual relationships and improving classification and localization, various experiments were performed on three histopathol- ogy datasets: CAMELYON16 (Ehteshami Bejnordi et al., 2017). CAMELYONI7 (Bándi et al.) 2019), and TCGA-NSCLC. Additional information about the datasets, including details about the training and test sets and our baseline models, can be found in the appendix.
4.1 CLASSIFICATION PERFORMANCE
We evaluated the performance of our context-aware pooling operator by comparing its performance other iitention-based MIL models, including CLAM-SB. CLAM-MB (Lu et al. 2021). TransMIL. (Sao et al. (02T). DTFD-MI (Zhang et al. 2022). DSMIL al. (2021a) and GTP (Zeng et al., 2022). CLAM-SB and CLAM- MEMM-MBBBBE utilize an attention based pooling operator within the Attention- Based MIL. (AB-MIL) framework (lsse et al.) 2018). They focus on the features of individual tiles and incorporate a clustering layer to enhance performance further. TransMIL. is transformer-based aggregator operator, DTFD-MIL leverages class activation maps estimate the probability of an instance being positive under the AB-MIL framework, and DSMIL uses dual instance and bag clas- sifiers refine predictions. Lastly, GTP combines graph-based representation of WSI and vision transformer.
The results of using CAMIL to classify WSI in the CAMELYONI6, TCGA-NSCLC, and CAME- LYON17 datasets are presented in Table The evaluation of the model's 's performance in all exper- iments includes the area under the receiver operating characteristic curve (AUC) and the slide-level

---

Published as a conference paper at ICLR 2024
Table 1: Classification results on CAMELYONI6, TCGA-NSCLC, and CAMELYON17
CAMELYONI6
TCGA-NSCLC
CAMELYON17
Method
ACC(T)
AUC(T)
ACC(T)
AUC(T)
ACC(T)
AUC(T)
CLAM-SB
0.8770.029
0,933o.002
0.903o.011
0.972o. 004
0.802a.or 039
0.849v.041
CLAM-MB
0.894o.010
0.938p.00s
0.9046.010
0.973a.004
0.803a.036
0.858o.046
TransMIL
0.905e.u05
0,950p.a0s
0.905o.011
0.974a.00
0.804a.021
0.873v.031
DTFD-MIL
0.889..007
0.941o.00s
0.899o.010
0.964o.om 003
0.7976.029
0.884..033
GTP
0.883o.o8 o.026
0.9160.015
0.973o 006
0.800o.o0 .037
0.762o.
DSMIL
0.874o.666
0.949o.006
0.853o.031
0.954o.015
0.815o.03 15.031
0.863o.043
CAMIL-L
0.910a.010
0.953o.auz
0.975u.004 0.914u.011
0.828a.u2r
0,88 0.003
CAMIL-G
0.89 10.001
0.950p.009
0.907u.012
0.973a.ue
0.818a.u 18.1980099
0,875p.036
CAMIL
0.9176.006
0.959..001
0.9160.007
0.975g.003
0.8430.024
0.881 1
accuracy (ACC), which is determined by the threshold of 0.5. When implementing our baselines, we fine-tuned the hyperparameters used in the previously published original work to achieve the best performance. The GTP model has the same configurations as the one described in the original paper (Zheng et al. 2022). However, for the CAMELYON16 dataset, the batch size (k) was set to 2 due to memory limitations.
CAMIL outperforms other MIL models in ACC and AUC across CAMELYON16, TCGA-NSCLC, and CAMELYON17 datasets, narrowly trailing DTFD-MIL on CAMELYONI7 by 0.003 in AUC. Its effectiveness notably identifies sparse cancerous regions in WSI, where tumor cells are often minimal, such as in the CAMELYON datasets, where tumor cells may account for as little as 5% of any WSI, which is particularly common in metastatic sites (Cheng et al. 202I). Specifically, CAMIL outperforms the other models on the CAMELYON 16 dataset by significant margins, achieving at least 0.9% better in the AUC and 1.2% in ACC than the existing models on CAMELYON16 and 3.8% in ACC on CAMELYON17.
GTP, with its MinCUT pooling layer, which aims to reduce the complexity of self-attention, per- forms well on TCGA-NSCL.C CC but less so on CAMELYON datasets. Reducing the complexity may make computation more manageable. However, on large and complex datasets such CAME- LYON16 and CAMELYON17, this pooling operation may result loss of information, particularly in that of fine-grained details. TCGA-NSCLC is a smaller dataset, allowing the MinCUT pooling in GTP to retain sufficient information and remain competitive.
4.2 LOCALIZATION
To evaluate the localization capability of CAMIL compared our baselines, we examine both qual- itative and quantitative evidence. Similar to the ex- perimental design of Tourniaire et al. (023b)) we compute the Dice score to quantify the ability of the different models identify cancerous evidence in cancerous slides. For normal slides, we compute the tile-level specificity. The reference ground-truth masks are computed at the 5th magnification level using expert tumor delineations. Additionally, a tile is considered cancerous if it contains at least 20% annotated tumor. To produce the predicted masks, use the scaled attention scores for CAMIL, both CLAM models, TransMIL and DSMIL, the tile level logits for DIFD-MIL, and the GraphCAM for GTP. We apply a threshold of 0.5 to the model's 's output probabilities to generate the masks from the tile-level predictions. The results for the Dice score and Specificity shown in Table
Table 2: Localization on CAMELYON16 Specificity(t))
Method
Dice(t)
CLAM-SB
0.459h.0s7
0.987o.008
CLAM-MB
0.4066.o0
0.5730.045
TransMIL
O. 1030.000 004
0.9996.001
DTFD-MIL
0.525o.o5 033
0.9990.00
GTP
0.418o.ue 80.88
0.85 10.11G
DSMIL
0.259o.0s3
0.863u.043
CAMIL
0.5150.058
0.980v.040
CAMIL performs well, albeit slightly behind the DTFD-MIL model. We believe the decreased lo- calization performance might be attributed to integrating the Nystromformer module in model
7

---

Published as a conference paper at ICLR 2024
Annotated
CAMIL
TransMIL
Figure 2: visual example of the tumor regions attention maps produced by different models on the CAMELYONI6 cancer dataset. The left column shows the original whole slide image, including the pathologist ground truth annotations with tumor regions delineated by red lincs. The other columns show attention maps from left to right for CAMIL, GTP, and TransMIl., pinpointing diagnostically significant locations the CAMELYON16 cancer dataset.
design, akin to its role in TransMIL. TransMIL. indicated by the attention maps in our qualitative assessment and its slide-level performance, demonstrates the ability to grasp the general patterns within WSI and distinguish between normal and cancerous slides. However, despite this, firmed by its low Dice score, it falls short in effectively pinpointing specific cancerous evidence within slides. Integrating the Nystromformer into our model design might introduce trade-off' be- tween slide-level accuracy and localization performance, resulting in improved slide-level accuracy with an expense of slightly decreased localization performance.
Figure 2 provides a qualitative comparison of the attention maps generated by CAMIL, GTP, and TransMIl. on the CAMELYON16 cancer dataset. We visually compare these methods as they lever- age spatial information to enhance prediction. These attention maps underscore CAMIL's high localization performance, as it can discern the boundaries separating normal tissue from tumor tis- sue. Although effectively pinpointing the regions of interest, GTP attention maps appear fragmented and less dense in cancer-associated regions. This fragmentation could be attributed to the MinCUT- pooling operation, which may reduce the representation's granularity and affect the heatmap's herence.
TransMIL appears to sufficiently capture long-term dependencies within the WSIs, evidenced by the attention maps of Figure [ Specifically, TransMIl. can identify the presence of cancer and precisely pinpoint the cancer-associated regions. However, these maps also reveal TransMIL's in- ability to capture intricate details and local nuances. The attention scores are not only confined to the cancer regions but expand beyond those to the surrounding normal tissue, impeding the precise localization of tumor boundaries, indicating the model's 's limitations in representing close proximity relationships within the WSIs.
5 ABLATION STUDIES
Additionally, we performed ablation studies to evaluate the effectiveness of the Nystromformer mod- ule and that of the neighboring-come condtraind attention module our model. Specifically, we exam- ined the effect of the Nystromformer block by retaining it while excluding the neighbor-constrained attention module denoted as CAMIL-G. Table [ demonstrates that using only the Nystromformer block leads to satisfactory performance comparable to that of the TransMIL model, which also in- corporates the Nystromformer. In a distinct ablation study, we omitted the Nystromformer block and retained the neighbor-constraing marginal improvement over the CAMIL-G model. thereby underlining its crucial role in augmenting the model's performance. Optimal results were achieved through the amalgamation of both models,
attention module, referred to CAMIL-L. CAMIL-L exhibits
8

---

Published as a conference paper at ICLR 2024
Annotated
Attention Distribution
Annotated
CAMIL-G
Frequency
Tumo
Large cancerous region
Tumo
Frequency Frequancy
Small cancerous region
Normal
value
Figure 4: A visual example CAMIL-L fine- grained localization abilities. From left to right are the ground truth annotations with tu- mor regions delineated by red lines, generated heatmaps for CAMIL-LCAMIILL and the at- tention distribution of the neighbor-constrained attention module's weights (hefore applying softmax function).
Figure 3: A visual example of the tumor regions and tention maps produced by different models on the CAME- LYONI6 cancer dataset. From left to right are the ground truth annotations with tumor regions delineated by red lines, the heatmaps for CAMIL-G, and the heatmaps for CAMIL-
L.
harnessing both potential of the Nystromformer block to model long-term dependencies and prowess of the niighbor-attentiin module to comprehend local visual concepts.
5.1 VISUALISING GLOBAL AND LOCAL CONCEPTS
We also visualized the attention maps generated by the two versions of model. Notably, in Figure 3 CAMIL-G demonstrates commendable performance in understanding global concepts and overall patterns, it effectively detects tumor regions. However, it struggles with highly localized tumors, encountering difficulties capturing intricate, short-term dependencies within the image.
On the other hand, the attention maps produced by CAMIL-L excel in capturing the fine details within a WSI. This proficiency can be attributed to the context-aware module, which utilizes a sim- ilarity mask. This mask aggregates attention weights of similar neighboring feature representations, resulting in robust activations. contrast, less favorable weights, particularly those associated with negative regions, do contribute as strongly. This observation substantiated by the histograms of the unnormalized attention coefficients, which are not constrained within the range of O to (before applying the softmax function) provided in Figure These histograms underscore a notable patterm: while a significant portion of the attention coefficients falls below the 0.5 threshold, cancerous cases display outliers within the range of 0.5 2. These outliers represent stronger activations, which dis- tinctly characterize regions affected by cancer. Conversely, all other activations consistently remain below the 0.5 threshold, signifying a reduced emphasis on non-cancerous areas.
CONCLUSION
We have introduced CAMIL, a novel MIL vision transformer-basee method that considers the tumor microenvironment context while determining tile-level labels WSIs, mirroring the approach of skilled pathologist. This is achieved by employing unique neighbor-constrained attention mech- anism, which assesses the dependencies between tiles within a WSI and incorporates contextual constraints as prior knowledge into the MIL model. We have demonstrated that using the trans- former and the wethhborooo--tttetniion mechanism together is imperative in successful performance across datasets through our ablation studies. Importantly, CAMIL achieves state-of state-of-the-ar across multiple datasets regarding tile-level ACC, AUC, and Fl scores and patch-level localization and interpretability.

---

Published as a conference paper at ICLR 2024
REFERENCES
Mohammed Adnan, Shivam Kalra, and Hamid R. Tizhoosh. Representation learning of histopathol- ogy images using graph neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 4254 4261. Computer Vision Foundation IEEE, 2020. doi: 10. 109/CVPRW50499, w0w.555000.0200000000 2020.00502.
Aïcha BenTaaeb and Ghassan Hamarneh. Predicting cancer with a recurrent visual attention model for histopathology images. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, and Gabor Fichtinger (eds.), Medical Image Computing and Computer Intervention MICCAI 2018- 21 International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part 11, volume 11071 of Lecture Notes in Computer Science, pp. 129-137. Springer, 2018, doi:
15.
Péter Bándi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, Quanzheng Li, Farhad Ghazvinian Zanjani, Svitlana Zinger, Keisuke Fukuta, Daisuke Komura, Vlado Ovtcharov, Shenghua Cheng, Shaoqun Zeng, Jeppe Thagaard, Anders B. Dahl, Huangjing Lin, Hao Chen, Ludwig Jacobsson, Martin Hedlund, Melih Çetin, Eren Halicı, Hunter Jackson, Richard Chen, Fabian Both, Jörg Franke, Heidi Küsters- Vandevelde, Willem Vreuls, Peter Bult, Bram van Ginneken, Jeroen van der Laak, and Geert Litjens. From detection of individual metas- tases to classification of lymph node status the patient level: The camelyon17 challenge. IEEE Transactions on Medical Imaging, 38(2):550-585 2019. doi: 10.1 20.1109012.2111.88777777
Gabriele Campanella, Matthew G. Hanna, Luke Geneslaw, Allen P. Miraflor, Vitor Werneck Krauss Silva, Klaus J. Busam, Edi Brogi, Victor E. Reuter, David S. Klimstra, and Thomas J. Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, pp. 1-9, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Jun Cheng, Yuting Liu, Wei Huang. Wenhui Hong, Lingling Wang, Xiaohui Zhan, Zhi Han, Dong Ni, Kun Huang, and Jie Zhang. Computational Image Analysis Identifies Histopathological Image Features Associated With Somatic Mutations and Patient Survival in Gastric Adenocarcinoma. Frontiers in Oncology, 11, 2021. ISSN 2234-943X.
Pierre Courtiol, Eric W. Tramel, Mare Sanselme, and Gilles Wainrib. Classification and disease localization in histopathology using only global labels: A weakly-supervised approach. CoRR, 0oss/802.02221 2018
Kausik Das, Sailesh Conjeti, Abhijit Guha Roy, Jyotirmoy Chatterjee, and Debdoot Sheet. Multi- ple instance learning of deep convolutional neural networks for breast histopathology whole slide classification. In 15th IEEE International Symposium Biomedical Imaging, ISBI 2018, Wash- ington, DC, USA, April 4-7, 2018, pp. 578 7778-81. IEEE, 2018. doi: (0000//SBL2018.83633333
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 1091199/.PP222 20W.9 200.5526884..
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes van Diest, Bram van Ginneken, Nico Karsse- meijer. Geert Litjens, Jeroen A. W. M. van der Laak, and and the CAMELYONI6 Consortium. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA, 3 318(22)2119-2222 doi: 10. 00.1001/jama.2017.44888 URL /ttp: wttp://///...../.aoo..o
December 2017. ISSN 0098-7484.
1001/ 2017.14582.1 14585
Michael Gadermayr, Lukas Koller, Maximilian Tschuchnig, Lea Maria Stangassinger, Christina Kreutzer, Sebastien Suuillard-Despres Gertie Janneke Oostingh, and Anton Hittmair. Mixup-mil: Novel data augmentation for multiple instance learning and a study on thyroid cancer diagnosis. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Saleudean, James Duncan,
10

---

Published as a conference paper at ICLR 2024
Tanveer Ayeda-Mahmood and Russell Taylor (eds.), Medical Image Computing and Computer Assisted Intervention - MICCAI 2023, pp. 477-486. Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-43987-2.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Le Hou, Dimitris Samaras, Tahsin M. Kurç, Yi Gao, James E. Davis, and Joel H. Saltz. Patch- based convolutional neural network for whole slide tissue image classification. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2423-2433. IEEE Computer Society, 2016. doi: 10. 2.110//CVR.2006..222
Maximilian Ilse, Jakub M. Tomezak, and Max Welling, Attention-based deep multiple instance learning. Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockhoimsmüsssun, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2132-2141, 2018.
Gabriel Landini, Giovanni Martinelli, and Filippo Piccinini. Colour deconvolution: stain unmixing in histological imaging. Bioinformatics, 09 2020. ISSN 1367-4803. doi: 10. 10.19///oiinformaticc btaa847. btaa847.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14318-14328, 2021a.
Jiayun Li, Wenyuan Li, Anthony E. Sisk. Huihui Ye, W. Dean Wallace. William Speier, and Corey W. Arool.. A multi-resolution model for histopathology image classification and lo- calization with multiple instance learning. Comput. Biol. Medicine, 131:104253, 2021b. doi: 111016/..compionee 2021.
Pei Liu, Luping Ji, Xinyu Zhang, and Feng Ye. Pseudo-bag mixup augmentation for multiple in- stance learning-based whole slide image classification, 2023.
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6):555-570, 2021.
Jaime Melendez, Bram Ginneken, Pragnya Maduskar, Rick H. H. M. Philipsen, Klaus Re- ither, Marianne Breuninger, Ifedayo M. O. Adetifa, Rahmatulai Maane, Helen Ayles, and Clara I. Sánchez. A novel multiple-instance learning-based approach to computer-aided detec- tion of tuberculosis on chest x-rays. IEEE Trans. Medical Imaging, 34(1):179-192, 2015. doi: 10.110997M11221122555339
Sandra Morales, Kjersti Engan, and Valery Naranjo. Artificial intelligence in computational pathology challenges and future directions. Digit. Signal Process., 119:103196, 2021. doi: 10.1016/j.
Gwénolé Quellec, Mathieu Lamard, Michel Cozic, Gouenou Coatrieux, and Guy Cazuguel. Multiple-instance learning for anomaly detection in digital mammography. IEEE Trans. Med- ical Imaging, 35(7):1604-166) 2016. doi: 10. 10.1109/TM..2211.5524444
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, and Yongbing Zhang. Transmil: Transformer based correlated multiple instance learning for whole slide im- age classification. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 2136-2147, 2021.
Zhuchen Shao, Liuxi Dai, Yifeng Wang. Haoqian Wang, and Yongbing Zhang. Augdiff: Diffusion based feature augmentation for multiple instance learning in whole slide image, 2023.

---

Published as a conference paper at ICLR 2024
Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A. Moskaluk, Sana Syed, and Don- ald E. Brown. Cluster-to-conquee A framework for end-to-end multi-instance learning for whole slide image classification. In Mattias P. Heinrich, Qi Dou, Marleen de Bruijne, Jan Lellmann, Alexander Schlaefer, and Floris Ernst (eds.), Medical Imaging with Deep Learning, 7-9 July 2021, Lüibeck, Germany, volume 143 of Proceedings of Machine Learning Research, pp. 682- 698. PMLR, 2021.
Chetan L. Srinidhi, Ozan Ciga, and Anne L. Martel. Deep neural network models for computational histopathology: A survey. CoRR, abs/191 2911. 123788 2019.
David Tellez, Geert Litjens, Jeroen van der Laak, and Francesco Ciompi. Neural image compression for gigapixel histopathology image analysis. IEEE Trans. Pattern Anal. Mach. Intell., 43(2):567- 578, 2021. doi: 10.1 (0.1109/PAAM..201..9999
Tong Tong, Robin Wolz, Qinquan Gao, Ricardo Guerrero, Joseph V. Hajnal, and Daniel Rueckert. Multiple instance learning for classification of dementia in brain MRI. Medical Image Anal., 18 (5)8008-818, 2014. doi: 10.101 mmw.mmmia..011111. 2014.04.006
Paul Tourniaire, Marius Ilie, Paul Hofman, Nicholas Ayache, and Hervé Delingette. MS-CLAM: mixed supervision for the classification and localization of tumors in whole slide images. Medical Image Anal. 85: 102763,, 2023a. doi: 10.1016/j.media22022 202.1....0.311226.... URL https://doii org/10. 0ee5/10.1116//1 media. 2023. 2023.102763333
Paul Tourniiire Marius Ilie, Paul Hofman, Nicholas Ayache, and Hervé Delingette. Ms-clam: Mixed supervision for the classification and localization of tumors in whole slide images. Med- ical Image Analysis, 85:102763, 2023b. ISSN 1361-8415. doi: http:://ooiorg//10000//// 1016/ www.:scceeeeeieeet.cocceceeeeeee
2023. 102763. URL https: $1361841523000249
Ming Tu, Jing Huang. Xiaodong He, and Bowen Zhou. Multiple instance learning with graph neural networks. CoRR, 1488811900488 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 0198-6008,200 2017.
Xiaodong Wang. Ying Chen, Yunshu Gao, Huiqing Zhang, Zehui Guan, Zhou Dong, Yuxuan Zheng, Jiarui Jiang. Haoqing Yang, Liming Wang. Xianming Huang, Lirong Ai, Wenlong Yu, Hongwei Li, Changsheng Dong, Zhou Zhou, Xiyang Liu, and Guanzhen Yu. Predicting gastric cancer outcome from resected lymph node histopathology images using deep learning. Nature Com- munications, 12(1):1637, March 2021. ISSN 2041-1723. doi: 10. 10.1038841467.02111167747 Number: Publisher: Nature Publishing Group.
Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognit., 74: 15-24, 2018. doi: 10. http::/...aco..011111 20w.oo...1........ 02.6
Chensu Xie, Hassan Muhammad, Chad M. Vanderbilt, Raul Caso, Dig Vijay Kumar Yarla- gadda, Gabriele Campanella, and Thomas J. Fuchs. Beyond classification: Whole slide tissue histopathology analysis by end-to-end part learning. In Tal Arbel, Ismail Ben Ayed, Marleen de Bruijne, Maxime Descoteaux, Hervé Lombaert, and Christopher Pal (eds.), International Con- ference on Medical Imaging with Deep Learning, MIDL 2020, 6-8 July 2020, Montréal, QC. Canada, volume 121 of Proceedings of Machine Learning Research, pp. 843 433-856.22 2020.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 14138-144 14148. AAAI Press, 2021. URL https: //op::://oss/ jsss aaai, org/ orgginnde iinexx..xxx. php/ php/AAAI/ AAAI/
artica levvvevvii ew/ 17664
12

---

Published as a conference paper at ICLR 2024
Gang Xu, Zhigang Song, Zhuo Sun, Calvin Ku, Zhe Yang. Cancheng Liu, Shuhao Wang. Jianpeng Ma, and Wei Xu. CAMEL: A weakly supervised learning framework for histopathology image segmentation. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South) October 27 November 2, 2019, pp. 10681-10690, 2019. doi: 10.1109/
ICCV.2019.01078.
Yan Xu, Yan Zhu, Eric I-Chao Chang, Maode Lai, and Zhuowen Tu. Weakly supervised histopathology cancer image segmentation and classification. Medical Image Anal., 18(3):591- 604. 2014. doi: 1..1016/..eddia20144.....
Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas J. Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learn- ing networks. Medical Image Anal., 65:101789, 2020.
doi: 10. 1111//..meei....... 2020.10000000111
Hongrun Zhang. Yanda Meng, Yitian Zhao. Yihong Qiao, Xiaoyun Yang, Sarah E. Coupland, and Yalin Zheng. DTFD-MIL: doublc-tier feature distillation multiple instance learning for histopathology whole slide image classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 18780- 18790, 2022. doi: 10.1109/CVPR52680 2002000222288222201222222 2022.01824.
Jingwei Zhang, Ke Ma, John Van Arnam, Rajarsi Gupta, Joel H. Saltz, Maria Vakalopoulou, and Dimitris Samaras. A joint spatial and magnification based attention framework for large scale histopathology classification. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pp. 3776-3784. Computer Vision Foundation IEEE, 2021. doi: 10. 10.1100 (L////CVPRR.33388.2222.2...
Yu Zhao, Fan Yang. Yuqi Fang. Hailing Liu, Niyun Zhou, Jun Zhang, Jiarui Sun, Sen Yang. Bjoern H. Menze, Xinjuan Fan, and Jianhua Yao. Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020. Seat- tle, WA, USA, June 13-19, 2020, pp. 4836-4845. Computer Vision Foundation / IEEE. doi: 10.
Yi Zheng, Rushin H. Gindra, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, and Vijaya B. Kolachalama. A graph-transformer for whole slide image classification. IEEE Trans. Medical Imaging, +1(11):3003-33 2022. doi: 10. 10.110997M1222223166565. URL https:
//doi. 10. / TMI 2022. 3176598
Changjiang Zhou, Yi Jin, Yuzong Chen, Shan Huang, Rengpeng Huang, Yuhong Wang, Youcai Zhao, Yao Chen, Lingchuan Guo, and Jun Liao. Histopathology classification and localization of colorectal cancer using global labels by weakly supervised deep learning. Comput. Medical Imaging Graph., 88: 88:101861, 2021. doi: 10. 1..1016/..coppeeiing 20tpeeddmaaa.22211100011.1 2021.101861.
A APPENDIX
A.1 REPRODUCIBILITY STATEMENT
All seekv--sueervised deep learning models are trained using NVIDIA Tesla P100 GPUs. One GPU is used for training in each experiment. We intend to make the source code of our algorithm publicly available soon.
A.2 PERFORMANCE METRICS
Here, we provide ACC, FI, and AUC for both datasets. Fl scores were omitted from the main text save space.
13

---

Published as a conference paper at ICLR 2024
Table 3: Classification results on CAMELYON16 AUC(1)
METHOD
ACC())
F1(1)
CLAM-SB
0.877v.029
0.840v.ozz
0.933v.ouz
CLAM-MB
0.894o.010
0.845 0.019
0.938o.005
TransMIL
0.905o.005
0.888o.018
0.950p.005
DTFD-MIIL
0.8890.007
0,850b.004
0,9410.005
GTP
0.883o.oe 0.8630.026
0,92 0.006
DSMIL
0.8740.086
0.848o.056
0,9490.006
CAMIL-L
0.9100.010
0.872n.016
0.953p.002
CAMIL-G
0.891 00001
0.8666.012
0,950p.009
CAMIL
0.9170.006
0.881o.010
0.959..001
Table 4: Classification results on TCGA-NSCLC
METHOD ACC())
F1(1)
AUC(1)
CLAM-SB
0.903o.011
0.867 0.012
0.9720.004
CLAM-MB
0.904o.010
0.872n.037
0.9730.004
TransMIL
0.905o.011
0.00s
0.9749.003
DTFD-MIIL
0.899o.010
0.8990.014
0.9640.003
GTP 0.9160.015
0.9170.016
0,9730.006
DSMIL
0.8530.0a1
0.8640.024
0.954p.016
CAMIL-L
0.914o.b11
0.921 0.009
0.975p.004
CAMIL-G
0.907o.012
0.91 80.007
0.9730.001
CAMIL 0.9160.007
0.91 80.005
0.9750.003
A.3 WSI PRO-ESSCGS
Using the publicly available WSI-prepossessing toolbox developed by (Lu et al. 2021). for both datasets, we first automatically segmented the tissue region from each slide and exhaustively divided it into 256x256 256 non-overlapping patches using x20 magnification (Figure 1). Otsu's method was used to perform automatic WSI thresholding.
To avoid the computational overhead and capitalize the rich feature representations already learned during its previous training on CAMELYONI6, CAMELYONI7, and TCGA-NSCLO datasets, opted to the pre-trained ResNet- 18 feature extractor provided by (Li et al. 2021a
This model was extensively trained on large set of tiles from the CAMELYON16 dataset, densely cropped without overlap, making it powerful feature extractor. To make fair comparison, used the same contrastive bearning-base model as the feature extractor for all our baselines.
A.4 DATASETS
CAMELYON16 a significant publicly available Whole Slide Image (WSI) dataset for lymph node classification and metastasis detection. It includes 270 training and 129 test slides from two medical centers, all meticulously annotated by pathologists. Some slides have partial annotations, making challenging benchmark due to varying metastasis sizes.
The CAMELYON17 dataset consists of 1000 WSIs of similar type to the CAMELYON16 dataset. However, only half (500 WSI) of these images are labeled and accessible publicly. These images are expertly annotated by pathologic lymph node classification into pt-stage:
pNO: No micro-metastases or macro-metastases, isolated tumor cells (ITCs) found.
PNO(i+): Only ITCs found.
mi: Micro-metastases found, but no macro-metastases found.
PN1: Metastases found in 1-3 lymph nodes, of which at least one a macro-metastasis.
pN2: Metastases found in 4-9 lymph nodes, of which at least one is macro-metastasis.
14

---

Published as a conference paper at ICLR 2024
Adopting the approach discussed in Tourniaire et al. (2023b), created binary classification problem by treating pNO as normal and unifying all classes that were not pNO into a single class, cancerous.
The TCGA-NSCLC dataset comprises two non-small cell lung cancer subtypes, LUAD and LUSC, with 541 slides. Unlike CAMELYONI6, it lacks annotations.
A.5 DATA SPLITS
In the case of CAMELYONI6, the WSIs are partitioned into training and test set. The 270 WSIs of the training set are split five times into a training (80%) and validation (20%) set in 5-fold cross-validation fashion, and the average performance of the model on the competition test set is reported. The official test set comprising 129 WSIs is used for evaluation. For CAMELYON17, we used a 4-fold validation 65%, 15%, and 25% train, validation, and test splits. Regarding the TCGA-NSCLC dataset, 5-fold cross-validation across the available images performed. For each fold, the training set is split into 80% for training purposes and 20% for validation.
A.6 TRANSFORMER MODULE TO CAPTURE GLOBAL CONTEXTS
Working with large WSIs can lead to memory overload, as the self-attention mechanism used in the transformer layer requires computing pairwise interactions between all of the tiles in each WSI. To circumvent the memory overload associated with the long-range dependencies of large WSIs, model feature interactions that
adopt the Nystromformer architecture (Xiong et al. 2021 otherwise would be intractable.
The Nystromformer approach is based on the Nystrom method, which technique for approxi- mating a kernel matrix by selecting a small subset of its rows and columns. In the context of the transformer layer, this means selecting a subset of "landmark" tiles from each WSI represent the full set of tiles. The landmark tiles are chosen randomly, and their embeddings are used to compute a low-rank approximation the self-attention matrix. This approximation is then used instead of the full self-attentio matrix to compute the final output of the transformer layer.
The Nystromformer architecture is highly scalable regarding sequence length, making well-suited for processing large WSIs (Xiong et al. 202I). Additionally, by reducing the time complexity of the self-attention mechanism from (n to (n), the Nystromformer approach can significantly reduce the computational cost of processing each WSI.
A.7 LOCALISATION
Annotated WSI
Model
Model
Model
Model
Model
Figure 5: Displayed attention maps produced Whole Slide Images (WSIs) throughout the 5-fold cross- validation runs. The leftmost column shows the original WSIs, including ground truth annotations of tumor regions delineated by lines, while the succeeding columns depict the predicted probabilities from the cross- validated model runs. The colormap signifies the likelihood with which a specific region within a WSI corre- lates to the target output label.
In addition to the guantitative outcomes, we assessed our approach qualitatively by visually present- ing attention maps generated by different validation of our model overlaid expert-annotated
15

---

Published as a conference paper at ICLR 2024
Table 5: CAMIL. using ResNet18 feature extractor pre-trained ImageNet without SimCLR fine- tuning.
Dataset CAMELYONI6 TCGA-NSCLC
ACC
AUC
0.723o.093
0.743p.077
0.692o.083 082
0.798o.059
tumor regions (Figure 5). These attention maps pinpoint diagnostically significant locations in the image that are crucial for accurate tumor identification.
We notice substantial agreement between the regions of interest identified by experts and those generated by our attention maps. These steps are harmonized with an attention map, forming transformer relevancy map. Notably, our method consistently highlights the same regions within Whole Slide Images (WSIs) across different cross-validation folds, underscoring the reliability and robustness of our model.
A.8 SIMCLR ABLATION STUDY
SimCLR feature extraction backbone integral part of our model, and it plays pivotal role in generating concise and descriptive feature representations. Leveraging the pre-trained SimCLR weights lays the foundation for generating meaningful similarity scores and, therefore, crucial for the optimal performance of the neighbor-constraine attention mask similar to many similarity- based histopathology approaches. Incorporating ImageNet weights directly into our model notably decreases its performance (as it does with most models Tourniaire et al. (2023a): [Li et al. (2021a)), emphasizing the necessity of SimCLR within our model architecture. Table shows these results when using ResNet-18 18 pre-trained on ImageNet on CAMELYONI6 and TCGA-NSCLC.
16


Reviews:
[{"review_id": "3mn6O5LzCq", "reviewer": "Reviewer_5Vaj", "summary": "The authors highlight one of the central limitations of multiple instance learning that the relative position of tiles are not typically considered and have addressed this by implementing the neighbor-constrained attention module.  This augments the weighting of each individual tile based on the relative attention weighting of the neighbors of the tile from the whole slide image.  These local features are then integrated back with the global features for the slide level prediction. CAMELYON16 and TCGA-NSCL were used as benchmarking datasets and the performance of CAMIL model exceeded all other MIL frameworks except GTP on TCGA-NSCL dataset. It is noted that the local and global features independently perform well compared to other models but the combination show good outperformance.", "questions": "You use SimClr to train the feature extraction encoder.  Have you test performance from other self-supervised pretrained encoders? Recent pre-prints suggest that larger ViT SSL trained encoders can benefit performance on less sophisticated aggregation functions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Transmil had suggested that context is important on the whole slide level, perhaps as much as the context within individual tiles.  What the authors have done appears to have better defined this context awareness which better constrains the concept of neighborhood awareness for important information in a whole slide image.", "weaknesses": "I would like to see performance on more challenging datasets for which the slide level labels are not visual features, such as mutation prediction or something similar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXLoWaaWSN", "reviewer": "Reviewer_SSWm", "summary": "This work proposes CAMIL - a MIL framework that applies neighborhood constraints to the transformer based MIL method, thus adding spatial context information. The method is tested on two MIL datasets - Camelyon16 and TCGA-NSCLC subtyping, where it compares favorably to existing methods. Ablations explore the importance of adding the neighborhood constraint and Nystromformer module.", "questions": "- There is typo in table 1 heading: it should be 'TCGA-NSCLC'\n- There are some spelling mistakes in the work. Example - 'are impirative in successfull performance` in Conclusion section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The central idea behind the paper is simple yet effective; current methods do not account for the spatial dependencies among the patches in a WSI. This information is something that human experts use, and this work proposes an intuitive similarity mask to modify the attention values to reflect the local context.\n- The paper is well-written and easy to read.\n- The ablation studies serve to motivate the need for adding the neighborhood constraint module.", "weaknesses": "- In the appendix, the F1 score for TransMIL is better than CAMIL for Camelyon16 dataset. Can the authors explain about the level of imbalance in this dataset, and whether accuracy is the correct metric for this?\n- Regarding the effectiveness of localization with this technique, the authors have produced some qualitative evidence in figure 3 and 4. However, since one of the main merits of this work is around better localization esp for problems with small foci of interest, can the authors a quantitative evaluation like in Table 1 in [1]?\n- Currently, a comparison with DS-MIL [1] is missing. This would help to evaluate this work against a model that uses multi-scale features and motivate the need for local context even with a multi-scale featurizer. \n\n\n\n[1] - Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning - Li et al", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2KQC2eEXBq", "reviewer": "Reviewer_2WUS", "summary": "The paper propose a new method for multiple-instance learning. The main novelty is the idea of adjusting the attention score using surrounding context (neighbor tiles). The proposed method achieved improved performance on two datasets.", "questions": "1. In the compared baselines, are self-supervised learning also employed to enhance the feature extractors?\n2. In page4, what is Q1 and K1, did you define it before? What are the m selected landmarks?\n3. Right below eq2, you talked about tile representations T, did you ever define it before? How is it different from H?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I appreciate the concept of incorporating contextual information to enhance the performance of Multiple Instance Learning (MIL). In particular, the intuition that 'tiles with high attention scores, surrounded by other high-scoring tiles, should be recognized as significant' sounds reasonable with the dataset the paper primarily focuses on.", "weaknesses": "1. The intuition that \"tiles with high attention scores that are surrounded by other high scoring tiles should be recognized as important. Conversely, the presence of a tile classified by the model as important in a low-scoring neighborhood ....could attributed to noise..\" make sense for the particular dataset and task (\" tumor cells are interspersed within large regions of normal cells\") ,and also the crop size, that the paper looks at. However, it remains unclear how well this method generalizes to datasets beyond the particular choices of dataset and task focuses on in this paper.\n\n2. Currently the neighbor-contrained idea is operationalized by masking the attention matrix (from self-attention) using the neighborhood mask. However, the attention scores from self-attention typically represent how much a tile attends to other tiles, not necessarily how much the slide-level prediction attends to that specific tile. It is not clear how directly masking on this attention matrix relates to the intuition above.\n\n3. Feature extractor backbone is an important factor for final performance. Comparing with prior method TransMIL that uses similar transformer backbone, the performance gain from the proposed method is minimum.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JHlqnc7fL2", "reviewer": "Reviewer_15GF", "summary": "This paper proposes a multi-instance framework for WSI classification that incorporates the spatial context of patch instances into the architecture of the model via a combination of transformer and neighbor-constrained attention with constrastive learning. The input images are preprocessed into features using a model trained with self-supervised contrastive learning (SimCLR) on pathology images with an ImageNet-pre-trained ResNet18. The transformer module follows the NystroFormer architecture. The attention module is the main contribution of this paper and trains an feature-similarity-weighted adjacency matrix. Experiments are performed on two well-known pathology datasets (CAMELYon-16 and TCGA-NSCLC) for a binary classification task (cancer/non-cancer). SOTA accuracy and AUC are reported for both datasets. An ablation study examines the separate effects of the Nystroformer and the attention module, showing the attention module to provide the largest improvement.", "questions": "* The published TransMIL accuracy numbers for CAMELYON16 and TCGA-NSCLC are: (ACC=0.8837, ACC=0.8835). The paper cites: 0.905 and 0.905. I did not verify the other numbers, but please double-check. [update: this question has been answered - the papers uses different feature extractors SimCLR]\n\nOverall, this is a solid paper implementing a grounded intuition using a MIL framework and demonstrating SOTA results.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The main idea behind this paper (leveraging neighbor correlations among patches in a transformer framework) is substantiated by domain-grounded intuition.\n* The chosen architecture appears to provide SOTA results on two well-known pathology benchmarks.\n* The ablation study shows the strong effect of the neighbor attention module.", "weaknesses": "* The number of datasets used for benchmarking is too small (2). [update: the authors have added the C17 dataset]\n\n* The paper does not discuss data augmentation within MIL training. Several recent papers have explorer ways to inflate the number of bags in MIL with various augmentation scheme.\n\n* The ablation study lacks an experiment showing the contribution of the pathology-specific self-supervised feature-extraction module. It would be easy to compare to a straight imagenet-pretrained ResNet18. [update: the authors have added an ablation study as requested][", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)