Exploring Low-Rank Property in Multiple Instance Learning for Whole Slide Image Classification
Authors: Jinxi Xiang
Jun Zhang
Venue: ICLR 2023 poster (2023)
Keywords: computational pathology
multiple instance learning
low-rank constraint
self-attention

Abstract: The classification of gigapixel-sized whole slide images (WSIs) with slide-level labels can be formulated as a multiple-instance-learning (MIL) problem. State-of-the-art models often consist of two decoupled parts: local feature embedding with a pre-trained model followed by a global feature aggregation network for classification. We leverage the properties of the apparent similarity in high-resolution WSIs, which essentially exhibit \textit{low-rank} structures in the data manifold, to develop a novel MIL with a boost in both feature embedding and feature aggregation. We extend the contrastive learning with a pathology-specific Low-Rank Constraint  (LRC) for feature embedding to pull together samples (i.e., patches) belonging to the same pathological tissue in the low-rank subspace and simultaneously push apart those from different latent subspaces. At the feature aggregation stage, we introduce an iterative low-rank attention MIL (ILRA-MIL) model to aggregate features with low-rank learnable latent vectors to model global interactions among all instances. We highlight the importance of instance correlation modeling but refrain from directly using the transformer encoder considering the $O(n^2)$ complexity. ILRA-MIL with LRC pre-trained features achieves strong empirical results across various benchmarks, including (i) 96.49\% AUC on the CAMELYON16 for binary metastasis classification, (ii) 97.63\% AUC on the TCGA-NSCLC for lung cancer subtyping, and (iii) 0.6562 kappa on the large-scale PANDA dataset for prostate cancer classification. The code is available at https://github.com/jinxixiang/low_rank_wsi.

Published as a conference paper at ICLR 2023
EXPLORING LOW-RANK PROPERTY IN MULTIPLE IN- STANCE LEARNING FOR WHOLE SLIDE IMAGE CLAS- SIFICATION
Jinxi Xiang, Xiyue Wang, Jun Zhang', Sen Yang, Xiao Han, Wei Yang
Tencent Lab
[jinxixang, june june haroldhar willyang] etencent. com (xiyue. wang. scu, sen. yang. scu) (1ggmalll... com
ABSTRACT
The classification of gigapixel-sized whole slide images (WSIs) with slide-level labels can be formulated as a tuaiipl-innttane-leaaniii (MIL) problem. State-of- the-art models often consist of two decoupled parts: local feature embedding with a pre-trained model followed by a global feature aggregation network for classi- fication. We leverage the properties of the apparent similarity in high-resolution WSIs, which essentially exhibit low-rank structures in the data manifold, to de- velop novel MII. with boost in both feature embedding and feature aggregation. We extend the contrastive learning with a pathology -specific Low-Rank Constraint (LRC) for feature embedding to pull together samples (i.e., patches) belonging the same pathological tissue in the low-rank subspace and simultaneously push apart those from different latent subspaces. At the feature aggregation stage, we introduce an iterative low-rank attention MIL (ILRA-MIL) model to aggregate features with low-rank learnable latent vectors. We highlight the importance of cross-instance correlation modeling but refrain from directly using the transformer encoder considering the (n?²) complexity. IIRA-MII. with LRC pre-trained features achieves strong empirical results across various benchmarks, including (i) 96. 96.49% AUC on the CAMELYON16 for binary metastasis classification, (ii) 97.63% AUC the TCGA-NSCLC for lung cancer subtyping, and (iii) 0.6562 kappa on the large-scale PANDA dataset for prostate cancer classification. Code is available
wwm wwwm//inxxiaanga/.mo.mccma low rank wsi
INTRODUCTION
Recent artificial intelligence in digital pathology has presented the potential to analyze gigapixel whole-slide images (WSIs). However, some challenges remain unsolved, including limited samples for training deep learning models and the extremely high resolution of WSI images (Lu et al. 2021e
Campanella et al. 2019 Shao et al. 2021 Sharma et al. 2021 Lu et al. (021b).
Since the relationship between input images and target labels is highly ill-posed, e.g., on CAME- LYONI6, 1.5 million x x24x224 224 input image tiles against 270 WSI-level labels, one has to decompose the model into two separate stages, local feature embedding and global feature aggregation. Bio- logical tissues in WSIs exhibit a wide variation, and there are still high semantic and background similarities among different image patches from the same type of tissue. Therefore, one fundamen- tal challenge is performing feature embedding that only captures relevant biological information and allows for quantitative comparison, categorization, and interpretation. After embedding, the taak/mmman-pooliinn to perform slide-level classification. Such
standard MIL uses non-pon paramertttei simplified schemes might lead to sub-optimal feature aggregation for WSI classification, and the models cannot learn cross-instance correlation due to the weak supervision signal.
As consistent with the findings in natural images (Cong et al. 2013 Zhou et al. 2014 Zhang et al., 2013 Liu et al. 2012). empirically find that gigapixel WSIs exhibit essentially low-rank prop- erties in the data manifold (see evidence in Appendix A). We aim to harness the low-rank property
"corresponding author

---

Published as a conference paper at ICLR 2023
whole slide image
patches
feature extractor with LRC
iterative low-rank atlention (IL((IRR)
Figure 1: The proposed pipeline. WSI is cropped into patches and then embedded into vectors for classification. We design LRC for feature embedding and ILRA-MIL for feature aggregation.
for WSI classification. The first intention is to learn low-dimensional feature embedding in discriminative way by extending contrastive loss with low-rank constraint. For global feature ag- gregation, it would be beneficial for MIL. to learn potential cross-instance correlation, which may help the model become more context-aware (Lu et al., 2021c). To this end, the second intention is to introduce self-attention with low-rank matrix that forms an attention bottleneck with which all in- stances must interact, allowing it to handle large-scale bag sizes with a small computation overhead. resolves the quadratic complexity O(n²) caused by global self-attention.
Our main contributions: (1) We extend contrastive learning with a low-rank constraint (LRC) to learn feature embedding using unlabeled WSI data; (2) We use iterative low-rank attention MIL (ILRA-MIL) to process large bag of instances, allowing it to encode cross-instance correlation naturally: (3) Extensive experiments on public benchmarks are conducted. Remarkably, I--A-MMI. improves over baselines, including attention-pooling and consformer-basee MII., by a large margin.
2
RELATED WORK
2.1 LOCAL FEATURE EMBEDDING IN MIL
Most methods conduct feature embedding with the ResNet50 pre-trained on ImageNet (Lu et al., 2021c Campanella al. 2019). However, there is a significant domain deviation between patho- logical and natural images, which might lead to sub-optimal patch features for WSI classification. Contrastive learning paves way for pathology-specific image pre-training (Lu
2019 Li al., 2021 Chen et al., 2020a Ciga et al. 2022, Stacke et al., 2021 The fundamental idea is to pull to- gether an anchor and a "positive" sample in embedding space and push apart the anchor from many "negative" samples. Nevertheless. it is infeasible for pathology images since they usually consist of multiple positive instances (Li et al. 2022). SupCon extends the self-supervised contrastive approach to the fully-supervised setting. allowing us to leverage label information (Khosla et al.) 2020) effectively. Nevertheless, fine-grained local annotations for WSIs are hardly available; thus, cannot adapt SupCon directly. We exploit the low-rank properties to generalize the supervised contrastive loss for WSIs without patch-level label information.
2.2 GLOBAL FEATURE AGGREGATION IN MIL
Traditional poolings robust noisy data and unbalanced distribution. MIL-RNN (Campanella eet al. 2019) built on recurrent network achieved clinical grade using more than 10,000 slides, but is data- hungry and constrained for binary classification. The local attention method, i.c. ABMIL (llse et al. 2018) uses the attention weights to allow us to find key instances, bringing significant improvements and robustness. CLAM (Lu et al. 2021c) further improves ABMIL with a clustering constraint by pulling the most and least attended instances apart. As concluded by (Lu et al., 2021c), one limitation of CLAM and MIL-based approaches that they typically treat different patches in the slide independent and do not learn the potential cross-interactions, which may help the model become context-aware To this end, global uttention-bart networks (Li et al.) 2021 Shao et al. 2021 Lu et al. 2021a), are introduced with non-local pooling transformer encoder to compensate for the shortness of local attention MIL that considers no cross-ine instance correlations. We aim to improve the global attention model with ILRA-MII. further.
2

---

Published as a conference paper at ICLR 2023
3
METHOD
The proposed pipeline boosted with the low-rank property of WSI, consisting local feature embedding module and global feature aggregation module, as illustrated in Fig. [
3.1
LOCAL FEATURE EMBEDDING
3.1.1 PRELIMINARY
Contrastive learning implements the heuristic to discern positive samples from negative samples (Chnn et al. 2020a b Grill et al., 2020 Gao et (021]. Given a randomly sampled minibatch of N images, get pars of projected feature vectors from augmented examples {z, (zi,, {1.... 2N). The self-supervised contrastive loss is (Chen et al. 2020a):
exp (sim(zi, /T(
Lcon
Σ log VEl
(1)
E LaEN(s) exp (sim( 2;, z /T)/T)
where sim(u4 v) u "ru' w/ |//|||||||||||| is the dot product between l normalized u and e; M(i) = M(o);; j(t) the index of the other augmented sample from the same image; is temperature negative samples.
parameter. For each anchor z;, there is one positive sample Z)(8) and 2(N
3.1.2 EXTENSION OF CONTRASTIVE LOss
Most pathology cases have high semantic and background similarity, thus resulting in multiple pos- itives in batch, introducing estimation errors (I) One straightforward approach is the general- ization of supervised contrastive learning (i.c., SupCon (Khosla et al. 20200) to an arbitrary number of positives by extending:
exp (sim Zi, xp)/T)
LSupCon
Σ PEP())
log
(2)
P())
ELer exp (sim Zi, za)/T)
SEI
where P(t) is the set of indices of all positive samples in the minibatch given anchor 2;; IP(t) its cardinality. For images with labels, it is intuitive to constitute positive samples with the same labels.
3.1.3 PATHOLOGY SPECIFIC LOW-RANK LOSS
SupCon in (2) extents vanilla contrastive loss by leveraging label information. But we refrain from adopting SupCon for WSIs because putch-level labels are available. We thus propose new self-supervised learning loss named LRC tailored for pathology images, which is shown to be generalization of SupCon unlabeled scenarios.
Given set of feature samples, each of which can be represented as a linear combination of the bases in dictionary, aim finding the representations that have low-rank similarity matrix between two sets of augmented representations:
R(T { ( O'TERNA raank T = =r, < N)
where T is similarity matrix of T [t].... ty). T [t,.... {N]: & and t; are two augmented representations of the same image. A low-rank matrix can be decomposed as the product of dictionary and a diagonal B such that (Liu et al. 2012 Wright & Ma 2022):
B
0
0
0
B2
0
0
TTT= D D D.]
+
(4)
0
0
0
0
B,
where E is an emor matrix which should be minimized; D3 E Nxab, B,
E Raxx with
=
1,... .T; Sa, 9b represent the shape of subspace B,
Intuitively, pairs belonging to the same subspace are more semantically similar than randomly sam-
pled ones. This has also been recognized as latent classes (Chuang al.) 2020 Saunshi et al.
3

---

Published as a conference paper at ICLR 2023
2019). For self-supervised contrastive loss Lcon with only one positive pair for each anchor, T T is considered to be a full-rank diagonal matrix, i.c., all entries are zeros except for the diagonal ones. SupCon loss SupCon further leverages label information to access more positive samples, enforcing T to explore the semantic similarity in the embedding space. In this way, SupCon loss could make low-rank constraints implicitly on T T, where r is the rank of the matrix corresponding to the total number of classes in SupCon. This observation recognizes the connections of contrastive loss with the low-rank property.
Since the low-rank decomposition of (3) is not tractable for online learning, as an alternative, we use SupCon loss in (2) as a surrogate by accessing more positives belonging to the same subspace B,. Suppose we get set of descendingly sorted indices based on their similarity to the anchor:
C(a) {A(1). .. if <
then sim(ta: Ea(s) 2 sim(ta; EA()))
(5)
Given an anchor fus we get subspace Cs(a),b b 1, , as stated in the low-rank repre- sentation Eq. ( We can intuitively consider that each subspace corresponds to a latent class, where (a)
+ 92)). A( Note that qı, 42, ... g.... is the column dimension of B,, B2.... ,,. Instead of partitioning all samples to get all subspace, which is computationally infeasible without solving (4). we only optimize the objective over the least- and most-distant subspace Ci(a), (a(a) with respect to the anchor. For any positive sample p E (a) and negative sample E Ca(a), would like to achieve the following:
{AANN - gr + 1), (A(1). A(qı), C2(a) = {A(g) + 1).... .Agi
C+(a)
sim(ta: ip) 2 sim (fa; (fm, fm))+ E.
where ę 0.5 is constant margin for all pairs of negative. We should add a threshold & rather than just ensure sim ta tp) 2 sim tx) to avoid trivial solution where features collapse together, i.e. sim(ta, Ep) = sim ta t.
We can incorporate low-rank constraint loss with margin into the supervised contrastive loss function in (2) by adding it after the cosine similarity term, giving us:
exp (sim(ta:
CLRC = - Σ
Σ log
(7)
Ci(a)
Eje (C(())CC-())))a exp (sim(tas {3) Ex)
a=1..N
(EC)(()
where Ej = 0f E otherwise E, = (a) qi is the number of elements in (a).
The loss 77) is minimized when all positive pains are correctly identified with condition (6) satisfied, thus enforcing our low-rank constraints. We set the top 5% of instances in a training batch C (a)
and the bottom 5% as (a). The derivation and analysis of 7 is provided in the Appendix A
The total loss for sule-supervised learning for feature embedding is:
L = XLcon + (1 ALCL
(8)
Without the self-supervised contrastive loss TT. there is a chicken-and-egg issue that good features will not be learned and low-rank loss in (28 is not sufficiently good. Incorporating contrastive loss Leon with LLCC is an incremental self-updating learning process. In our default setting where 0.5, no unstable training is observed.
3.2 GLOBAL FEATURE AGGREGATION
3.2.1
PRELIMINARY
Without loss of generality, we take the binary MIL classification example. The learning task is to learn a nonlinear function from feature space X' to label space y = {1,0} using the training data set ((Xi, i, y1) -.),.... (Xm: m }. where X; {,.,,,.... Di,m; is WSI; m. is the bag size of X;; Ci is an instance. The corresponding instance labels {VG,1 . Vi.m; } are unknown, i.e.
{ 1, otherwise
0, iff Σ, Vi, = 0: yc E {0, 1).j 1, 1.-
V/
MIL processes a bag of instances with permutation invariance property, stating that the label of the bag remains unchanged regardless of the order of input instances on the bag (Ilse et al. 2018 Li
4

---

Published as a conference paper at ICLR 2023
al. 2021). A simple example of permutation invariant model is network that performs pooling over embedding extracted from the patches of a bag. Mathematically:
logits (X.) (pool (10 (æt,1).
(es.m,))))
(10)
where 'pool' is pooling operation; & and p denote instance-level network and bag-level classifier,
respectively. Attention based pooling is commonly used (Ilse et al. 2018 Lu et al. 2021c
Tomita
al. 2019 Hashimoto al. 2020):
exp WWT (tanh (Væ,,) 3 sigm (Uz.,)))
( 2j=1 exp {WT (tanh (Væig) O sigm (Uz,)))}
(11)
with learnable parameters W, V, and U.
3.2.2 TRANSFORMER- BASED MIL.
Eq. 11) a local attention network where the score )(xr))) of instance æa only depends on the instance itself. We aim to explore the dependence and interaction among all instances. One straightforward approach is the application of transformer.
The transformer encoder consists of alternating layers of multi-headed attention and MLP blocks.
Here, we denote the feature matrix the feature bag X; as X = [0i,,,--. m. T An attention head maps queries Q E R xd to outputs using m, key-value pairs K RR,, xd E Rmm xd and d is the query/key dimension:
head, (X.) Attention Qn, Ki, Vn. softmax (Q.KT Vn {
(12)
where Q X/W NEET K, = X W. b V, = W h 1, ,,... H,
where W W h Wk. are learnable; é = 1, 1.... A is the index of the transformer layer; k is the h total number of layers. Transformer uses multi-head attention to project Q: K; V onto H different vectors and then concatenate all attention outputs:
( = concat (head), headh) X MHA (LN (X()) + X
(13)
where LN is the layer norm. The output layer is MLP with skip connection:
x(+1 = MLP
IN { + X
(14)
Considering a large number of instances in each bag (hundreds of thousands), obstacle with transformer for MII. is the quadratic time and memory complexity O (m?) Despite the linear the- oretical complexity with some approximations like Nystromformer (Xiong et al., 2021), Linformer
(Wang et al., (2200)), or Performer (Choromanski et al. 2020), it overlooks the innate characteristic of input instances.
3.2.3 ITERATIVE LOW-RANE RANK ATTENTION MIL
Medical image including WSI is extensively nigh-dimensional in its raw form. As such, it is effective to explore the hidden structures in the forms of low-rank matrices of high-dimensional data (Wang al. 2020b Li et al. 2018 2020). We thus introduce learnable low-rank latent matrix L E R" One basic module
to interact with all input instances as the proposed RA-A-- shown Fig. 2. of the network is the cross-attention (CAtt), defined as:
CAULL x²)- = Attention Q K, V) softmax (QK (QKT/VL WVTV V
(15)
where Q = LW? K = X = X(WV.
Note that L is unified matrix for all layers to keep the low-rank consistency for different layers. As shown in the right-hand side of Fig. we also use unified layer with cross-attention and Gated Linear United (GLU), named Gated Attention Block (GAB):
GAB X = (U O VIW?
(16)
U- du (LWL), V XA)
5

---

Published as a conference paper at ICLR 2023
ILRA
layers
Iatent matrix
K
GAB
GABp
Non-tocal
Pooling
Logits
s (QK
Emput instance X
CAl
GAB
Figure 2: ILRA-MIL iterates over k IRLA layers. Each layer consists of two GAB blocks. The first GAB block projects input instance X [mm;xa to low-rank space by attending to the latent vectors E R'xd r < m, and the second GAB recovers the input dimension. w represents softmax. The output layer uses non-local pooling to make predictions. Layer normalization is omitted for brevity.
where : stands for element-wise multiplication: óu is Sigmoid Linear Units SiLU (Elfwing et al,, 2018 Hua et al. 2022;; W W are linear transforms. The inputs of GAB are not permutation invariant as the first is the query and the second is the key-value. An ILRA block consists of:
P GLB X GAB. P).
(17)
Eq. 17 is analogous to low-rank projection auto-encoder models. GAB, first projects the high- dimensional X to the low-rank space L. Then the projection result P E RR"x is reconstructed to
high-dimensional space X with GAB, where the query is X and key-value is P.
There some desirable properties of IIRA-MMIL. (i) The latent vectors L encode global features that help to explain input instances. For example, in the cancer subtyping problem for computational pathology, the latent vectors could be approximately some mutual and universal information of key cancerous regions so that the ILRA module can compare instances in the query indirectly through to all inputs. (ii) The Q-K-V pair not longer symmetric as in MHA because for the shapes L Brxd K E Rmaxdd V E Rm < mi, Thus, the complexity of gross-attention operation significantly is reduced from quadratic (mm)) to linear O(rms). We set r = 64 by default.
Constraining the latent vectors to be low-rank may restrict the network's s ability to capture all of the necessary details from the imput instances. To improve expressivity, the model stacks &o (k by default) ILRA layers extract information from the input instances:
X ILRA(ILRA(X))
(18)
layers
where LN should be applied before the input of each layer. X;
1 02. -mm) encodes
cross-instance correlations in the bag. A bag feature E 1 is obtained through max pooling over X, Then, trainable linear classifier is used to conduct non-local pooling at the output layer:
172g
exp (#b 2r
logits X2)
pl
Wy Zz),
Wj
mi 9=1
exp (xb x.g)
(19)
1=1
EXPERIMENTS
4.1 DATASET
CAMELYONI6 is public dataset for metastasis detection in breast cancer (binary classification), including 270 training slides and 130 test slides. total of about 1.5 million patches at x10 mag- nification are obtained. TCGA-NSCLC includes two subtype projects (binary classification), i.e., Lung Squamous Cell Carcinoma (TGCA-LUSC) and Lung Adenocarcinoma (TCGA-LUAD), for total of 993 diagnostic WSIs, including 507 LUAD slides from 444 cases and 486 LUSC slides from 452 cases. We obtain 3.4 million patches total at x10 magnification. PANDA is the largest prostate biopsy public dataset to date (Bulten al. 2022). We use 4369 slides from Karolinska Institute for training. The independent test set from Radboud University has 2591 slides. A total of 1.1 million patches at x 10 magnification obtained. More details introduced in the Appendix.

---

Published as a conference paper at ICLR 2023
Table 1: Classification Results on Benchmarks.
CAMELYON16
TCGA-NSCLC
PANDA
Acc
AUC
Acc
AUC
Acc
kappa
Mean-poolle heaa-pooling
0.6511
0.6755
0.7282
0.8401
0.5691
0.4422
Max-pooling
0.7674
0.8169
0.8593
0.9263
0,6100
0,5830
ABMIL
0.8527
0.8503
0.8384
0.9205
0.6834
0.5998
MIL-RRNN
0.8449
0.8580
0.8619
0.9107
NA
NA
CLAM-SB
0.8682
0.8709
0.8632
0.9307
0.6648
0.5782
CLAM-MB
0.8604
0.8779
0.8492
0.9377
0.6760
0.6067
DSMIL
0.8759
0.8944
0.8690
0.9439
0.6737
0,5562
SSMIL+
0.8867
0.9175
0.9048
0.9551
0.7017
0.5837
TransMIL
0.8449
0.8769
0.8565
0.9303
0.6720
0.5638
DTFD-MIL (MaxS)
0.8543
0.9103
0.8701
0.9097
0.6334
0.5462
DTFD-MIL (AFS)
0,9010
0,9401
0.8941
0,9612
0,6573
0,5437
ILRA-MIA (ours)
0.8992
0.9278
0.9004
0.9592
0. 7.90
0.6236
ILRA-MIL + LRC (ours)
0.9218
0.9649
0.9213
0.9763
0.7287
0.6562
4.2 IMPLEMENTATION DETAILS
Training. In CAMELYON16, the 270 training WSIs are split into approximately 90% training and 10% validation and tested on the official test set. In PANDA. we split the 4219 slides from Karolinska into 80% training and 20% validation and tested on the 2591 slides from Radboud. For TCGA datasets, first ensured that different slides from patient do not exist in both the training and test sets, and split the data in the ratio of sainiin:vllidation.te. = 60:15:25. For self-supervised ResNet50 encode 224 224 images into 1024-dimensional vectors. The same training data used to develop feature embedding with LRC and feature aggregator II.A-MMI..
learning,
Evaluation. For the evaluation metrics, used accuracy and under the curve (AUC) scores to evaluate the classification performance, where the accuracy was calculated with a threshold of 0.5 in all experiments. The multi-class PANDA scored based Cohen's kappa.
Baseline methods include mrean/max-pooling and deep MIL models, i.e., ABMIL (Ilse et al. 2018 DSMIL. (Li et al. 202T), CLAM-SB CLAM-MBB (Lu al. 2021c), MIL.-RNN (Campanella al. Shao et al. 2021), and DTFD-MIL (Zhang et al. 2022).
2019.
transMiL
5 RESULTS
5.1 RESULTS ON CLASSIFICATION
All results are provided in Table 6 'DSMIL denotes DSMIL with SimCLR features as reported in (Li et al. (021). Other baselines ImageNet pre-trained features without notice. In all cases, ILRA with LRC feature embedding consistently improves over ImageNet pre-trained feature embedding, the statistic in the last row shows.
In CAMELYONI6, tumors are minor regions in positive slides (averagely < 10% per slide), result- ing highly imbalanced distribution of positive and negative instances in bag. Attention-based methods all outperform the traditional mean or max pooling operators. Nonlocal poolings, including DSMIL and TransMML outperform attention pooling with nonlocal operator that models the cross- instance correlation. DTFD-MIL is the best-performed competing method which is particularly designed to address the small sample cohorts. The proposed II-RA-MIL processes cross-instance correlation and the AUC score at least 4.99% higher than CLAM-MB, which only local instance for aggregation.
In TCGA-NSCLC, positive slides contain relatively large areas of the tumour region (average total cancer area per slide > 80%). As result, both the max pooling and attention pooling operators work pretty well in this scenario. Non-local pooling methods are consistently stable, and ILRA- MIL performed better than all the other competing methods, achieving 1.53% improvement in AUC and 1.51% % in accuracy, compared with the best competing results.
7

---

Published as a conference paper at ICLR 2023
Table 2: Ablation on Different Pretrained Models.
ILRA-MIL
CLAM-SB
Acc
AUC
Acc
AUC
ImageNet
0.8992
0.9278
0.8682
0.8709
SimCLR
Chen et al.
2020a
0.9082
0.9392
0.8895
0,9106
BYOL
Gnill a..
2020)
0.9002
0.9330
0.8701
0.8902
SimSiam
Chen & Hc
2021
0.9032
0.9354
0.8837
0.8962
MoCov3
(Chen ct al.
2020c)
0.9158
0.9490
0.9021
0.9123
LRC (ours)
0.9218
0.9649
0.9088
0.9377
nogattve
nogattiee
Uree
Live
LRC
M3Co-v3
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
Prob
Prob
Figure 3: Probability distribution with MoCo- V3 and LRC.
For PANDA, MIL-RNN does work for multi-clasiifcationn problems, we exclude from the comparison result. PANDA is unbalanced distributed in cancer subtypes, and is challenging to differentiate glandular patterns with intermediate morphological structures (Nagpal al. 2020).
ILRA-MIL can also be applied to multi-class problems with unbalanced data, and it can be observed that the best results are achieved in both accuracy and kappa. For comparison, existing clinical- grade AI system trained pixel-wise annotations from highly urological pathologists scores from 0.62 (Bulten al., 2020 0.66 (Tolkach al. 2020
5.2 ABLATIONS ON LRC
To demonstrate the effectiveness the proposed clustering-constraie contrastive loss, we com- pare its performance with alternative contrastive learning: SimCLR (Chen et al. (020a), BYOL (Grill et al. 2020), SimSiam (Chen & He 2021 and MoCoV3 (Chen et al., 2020c). and an Ima- geNet pre- praiteeieed model, shown in Table 2 The same ILRA-MIL model is used to evaluate the ACC and AUC performance. Unsurprisingly, all self-supervised features significantly bootstrap the performance against the ImageNet pre-trained features.
MoCo-V3 and SimCLR JR outperform BYOL and SimSiam without negative samples. The proposed LRC achieves 1.59% AUC improvement over MoCo-v3. Similar results also apply to CLAM-SB, as shown in the table. Fig. 3 shows the predicted probability on the CAMELYONI6 test set using ILRA-MIL trained with MoCo-v3 and LRC features. As we set 0.5 the classification threshold, we can observe that with LRC features, there are fewer false positives and false negatives samples compared with the probability distribution of MoCo-V V3.
5.3 PARAMETER ANALYSIS AND ABLATIONS ON ILRA-MIN
We conduct some ablations on ILRA-MIL terms of some key modules: (i) low-rank latent vectors attention in 15 (ii) local pooling in [9 : (iii) iterative attention mechanism in (8). Ablation
studies are performed on the CAMELYONTO dataset with ImageNet features; see Table 3
(1) The default rank of L in Eq. 15 is 64. We adjust the rank from 32 to 128, and the result demonstrates that with large- vector rank, can attend all input instances with negligible loss of information. Then, compare it with the self-attention module.
8

---

Published as a conference paper at ICLR 2023
Table Parameter Analysis and Ablations on ILRA-MIL
Settings
# params
AUC
Settings
# params
ALC
rank 16
2.97
0.9205
iteration
1.39
0.9102
rank r 32
2.99
0.9231
iteration 2
1.94 M
0,9221
(iv)
rank r 64
3.02 M
0,9278
iteration &o == 4
3.02 M
0,9278
rank 128
3.09 M
0.9279
iteration & = 6
4.11 M
0.8947
full self atlf-attentinn
2.64
0.8127
iteration 8
5.19 M
0.8418
(ii)
low-rank attention
3.02 M
0.9278
max pooling
2.76 M
0.8061
iiii)
local Pooling
5.01 M
0.8612
nonlocal att. pooling
3.02
0.9278
naw image
CLAM
CLAM-LRC
LO
Figure 4: Heatmap visualization of "test__75" and 026" for CLAM with and without LRC.
(2) We cannot directly apply the full self-attention considering the large bag size, and instead, use the Nystrom transformer as an approximation. The same number of heads and layers are used for evaluation. The results indicated that low-rank attention achieves 0.9278 AUC, outperform- ing 0.8127 AUC of full self-attention by a large margin. Although with linear approximation, full self-attention involves excessively redundant and task-irrelevant interactions among instances and is challenging to optimize where only tiny amount of slide-level labels are available.
(3) After TLRA iteration in 17), non-local pooling is used with (9)) aggregate global feature. We ablate it with the commonly used max pooling and local attention pooling in (II). Remarkably, nonlocal pooling can improve max pooling and local attention pooling by 12.17% and 6.66% AUC.
(4) ILRA-MIL can make deeper network through iterative attention. As the number of iterations increases, the model performance growth tends level off, which indicates that it is sufficient to characterize cross-instance correlations in the dataset. The iteration number greater than k = 4 leads significant decrease performance caused by the over-fitting dataset.
5.4 INTERPRETABILITY
Our feature embedding LRC boosts the performance of CLAM-SB (see Table and it can also en- hance interpretability. We use the trained CLAM-SB model with LRC features to draw the predicted heatmap shown in Fig. 4 The heatmaps show remarkable consistency with expert annotation, especially for "test 075" where the ROIs only occupied a small area; the most significant regions located and identified. We show visual comparisons in the Appendix.

---

Published as a conference paper at ICLR 2023
6 CONCLUSION
In this paper, we address the problem of WSI classification by optimizing the feature embedding and feature aggregation with low-rank properties. We improve the vanilla contrastive loss with additional low-rank constraints to collect more positive samples for contrast. We also devise an iterative low- rank attention feature aggregator to make efficient cross-instance correlations. All these designs boost the performance across various benchmarks, as the results show. One limitation of our model that has been validated on multi-center larger-scale clinical datasets, In addition, ILRA-MIL cannot directly provide local attention score for each instance, which might hinder an intuitive clinical analysis of each patch image.
REFERENCES
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122, 2011.
Wouter Bulten, Hans Pinckaers, Hester van Boven, Robert Vink, Thomas de Bel, Bram Gin- neken, Jeroen van der Laak, Christina Hulsbergen-van de Kaa, and Geert Litjens. Automated deep-learning system for gleason grading of prostate cancer using biopsies: diagnostic study. The Lancet Oncology, 21(2):233-241, 2020.
Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester Boven, Robert Vink, et al. Artificial intelli- gence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine, pp. 1-10, 2022.
Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning whole slide images. Nature medicine, 25(8):3301-13) 2019.
Emmanuel Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi cemi-supervised learners. Advances in neural information pro- cessing systems, 02222243-22252 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint vrXiv:2003.04299, 2020c.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Ching-Yar Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775. 2020.
Ozan Ciga, Tony Xu, and Anne Louise Martel. Self supervised contrastive learning for digital histopathology. Machine Learning with Applications, 7:100198, 2022.
Yang Cong, Ji Liu, Junsong Yuan, and Jiebo Luo. Self-supervised online metric learning with low rank constraint for scene categorization. IEEE Transactions on Image Processing, 22(8):3179- 3191, 2013.
10

---

Published as a conference paper at ICLR 2023
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Semooddweiihte linear units for neural network function approximation in reinforcement learning. Neural Newworks, 107:3-11, 2018.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.088),, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-212882 2020.
Noriaki Hashimoto, Daisuke Fukushima, Ryoichi Koga, Yusuke Takagi, Kaho Ko, Kei Kohno, Masato Nakaguro, Shigeo Nakamura, Hidekata Hontani, and Ichiro Takeuchi. Multi-scale domain-adversarial multiple-instanee enn for cancer subtype classification with unannotated histopathological images. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3852-3861, 2020.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoe V Le. Transformer quality in linear time. arXiv preprint arXiv:2202.1044 2022.
Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-basee deep multiple instance learn- ing. In International conference machine learning, pp. 2127-2136. PMLR, 2018.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 23:18661-18677 2020.
Bin Li, Yin Li, and Kevin W Elicciri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14318 14328, 2021.
Huafeng Li, Xiaoge He, Dapeng Tao, Yuanyan Tang, and Ruxin Wang. Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning. Pattern Recognition, 79:130-145 146, 2018.
Huafeng Li, Xiaoge He, Zhengtao Yu, and Jiebo Luo. Noise-robust image fusion with low-rank sparse decomposition guided by external patch prior. Information Sciences, 523:14-37, 2020.
Zhaowen Li, Yousong Zhu, Fan Yang, Wei Li, Chaoyang Zhao, Yingying Chen, Zhiyang Chen, Jiahao Xie, Liwei Wu, Rui Zhao, et al. Univip: unified framework for self-supervised visual pre-training. arXiv preprint arXiv:2203.06960 2022.
Guangean Liu, Zhouchen Lin, Shuicheng Yan, Sun, Yong Yu, and Ma. Robust recovery of subspace structures by low-rank representation. IEEE transactions pattern analysis and machine intelligence, 35(1):
2012.
Mengkang Lu, Yongsheng Pan, Dong Nie, Feihong Liu, Feng Shi, Yong Xia, and Dinggang Shen. Smile: Sparse-attention based multiple instance contrastive learning for glioma sub-type clas- sification using pathological images. In MICCAI Workshop on Computational Pathology, pp- 159-169. PMLR, 2021a.
Ming Y Lu, Richard J Chen, Jingwen Wang. Debora Dillon, and Faisal Mahmood. Semi-supervised histology classification using deep multiple instance learning and contrastive predictive coding. arXiv preprini arXiv:
2019.
Ming Lu, Tiffany Y Chen, Drew FK Williamson, Melissa Zhao, Maha Shady, Jana Lipkova, and Faisal Mahmood. Ai-based pathology predicts origins for cancers of unknown primary. Nature, 594(7861):106-111 10, 2021b.
Ming Y Lu. Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021c

---

Published as a conference paper at ICLR 2023
Kunal Nagpal, Davis Foote, Fraser Tan, Yun Liu, Po-Hsuan Cameron Chen, David F Steiner, Naren Manoj, Niels Olson, Jenny L Smith, Arash Mohtashamian, et al. Development and validation of deep learning algorithm for gleason grading of prostate cancer from biopsy specimens. JAMA oncology, 6(9):1372-1380, 2020.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Con- ference Machine Learning, pp. 5628-56233 5637. PMLR, 2019.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Ad- vances Neural Information Processing Systems, 34, 2021.
Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A Moskaluk, Sana Syed, and Donald Brown. Cluster-to-conquer: A framework for end-to-end multi-instance learning for whole slide image classification. In Medical Imaging with Deep Learning, pp. 682-698, PMLR, 2021.
Karin Stacke, Jonas Unger, Claes Lundström, and Gabriel Eilertsen. Learning representa- tions with contrastive self-supervised learning for histopathology applications. arXiv preprint arXiv:2112.05766. 2021.
Peter Ström, Kimmo Kartasalo, Henrik Olsson, Leslie Solorzano, Brett Delahunt, Daniel M Berney, David G Bostwick, Andrew J Evans, David Grignon, Peter A Humphrey, al. Artificial intel- ligence for diagnosis and grading of prostate cancer in biopsies: population-based based, diagnostic study. The Lancer Oncology, 21(2):222-232 2020.
Yuri Tolkach, Tilmann Dohmgörgen, Marieta Toma, and Glen Kristiansen. High-aceuracy prostate cancer pathology using deep learning, Nature Machine Intelligence, 2(7):41 2020.
Naofumi Tomita, Behnaz Abdollahi, Jason Wei, Bing Ren, Arief Suriawinata, and Saeed Hassan- pour. Attention-based deep neural networks for detection of cancerous and precancerous esopha- gus tissue on histopathological slides. JAMA network open, 2(1 (11))e9114445-1111
2019.
Madeleine Udell, Corinne Horn, Reza Zadeh, Stephen Boyd, et al. Generalized low rank models.
Foundations and Trends@ in Machine Learning. (1):1-1118 2016.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 20t1::20000047766 2020a.
Yongjun Wang, Baiying Lei, Ahmed Elazab, Be-Leng Tan, Wei Wang, Fanglin Huang, Xuehao Gong, and Tianfu Wang. Breast cancer image classification via multi-network features and dual- network orthogonal low-rank learning. IEEE Access, 1-27779-27792. 2020b.
John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vīkas Singh. Nyströmformer: A nystöm-based algorithm for approximating self-attention. In Proceedings of the... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, volume 35, pp. 14138. NIH Public Access, 2021.
Hongrum Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathol- ogy whole slide image classification. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 18802-18812. 2022.
Yangmuzi Zhang, Zhuolin Jiang, and Larry Davis. Learning structured low-rank representations for image classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 676-683, 2013.
Xiaowei Zhou, Can Yang, Hongyu Zhao, and Weichuan Yu. Low-rank modeling and its applications in image analysis. ACM Computing Surveys (CSUR), 47(2):1-33, 2014.
12

---

Published as a conference paper at ICLR 2023
A LOW-RANK PROPERTY OF WSI
High-dimensional WSI data bring great challenges to data analysis. But fortunately, the high- dimensional WSI data often lie in low-dimensional subspace, consistent with findings including natural images in computer vision, documents in natural language processing et al, 2016, Zhang al. 2013 Zhou et al. 2014 Wright & Ma 2022). Pursuing the low-rank property of high- dimensional data is to identify the intrinsic manifold or physical mechanisms from which the data
generated.
Given a bag of feature embedding from WSI, it can be formulated as a data matrix X [X,,... X,] where X; corresponds latent class i, is the total number of latent classes. Ide- ally. X can be decomposed into a low-rank component DB and a sparse error component E, i.c., X DB E with respect dictionary, the optimal representation matrix B for X should be block-diagonal:
B
0
0
0
0
B2
0
0
(20)
0
0
0
0
0
B,
The space matrix D = [DI
D2.... contains r sub-space. An example of optimal decomposi-
tion for feature embedding is illustrated in Fig. (5) For example, data X [X,,Xx X3) con-
tains features from classes, where X, contains 3 samples 21, 22 23. X₂ contains 4 samples
24 25, L6 27, and X3 contains samples æs, 22 2 10- D has 3 sub-space, and each has 2 support items.
0
0
1
0
0
0
0
0
2
X
0
1
1
1
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
26
Zig
Z10
X 22 Xg
di (ddgdddddd
class
class
class 3
class class 2 class
Figure 5: Example of an optimal low-rank decomposition.
Mathematically, the decomposition is achieved by optimizing:
minB.E E BB|| + -||
(21)
s.t X DB
We construct 270 data matrices with CAMLEYONI6 training WSIs for low-rank property analysis. The size of data matrix is IR where is the bag size of WSI and d is the fix-dimension of emhedding depending on the encoder, e.g., d = 1024 for ResNet50 backbone. The low-rank de- be optimized by ADMM algorithm (Alternating Direction Method of Multipliers) (Candès et al. 2011 Boyd et al. 2011).
composition problem in (21)
We plot the histogram of the rank of all matrices in Fig. 6 The average rank of ImageNet fea- ture embedding is 349, much smaller than the full-rank 1024. Remarkably, the average rank of self-supervised learning feature BYOL (Grill et al.) (020), MoCo (Chen & He 2021). and the pro- posed LRC can further reduce to 218, 195, and 181, respectively. As the Table in the main paper shows, the classification performance AUC with the same ILRA-MIL model using ImageNet, B BYOL,, MoCo, and LRC features is 0.9278, 0.9330, 0.9490, 0.9649,, respectively, i.e.:
avg. rank: ImageNet BYOL MoCo > LRC
(22)
classification AUC: ImageNet BYOL MoCo < LRC
Even without low-rank constraints, BYOL and MoCo tend to produce features with lower ranks than ImageNet. Also, Fig. (6) (d) indicates that the distribution of all WSIs feature embedding is more
13

---

Published as a conference paper at ICLR 2023
compact than ImageNet. This empirical evidence implicitly shows that low-rank features very likely to be beneficial to WSI representation.
0.00
demsity
Wobb density
0
0.005
000005
0.000
3.000
rank
rank
(a) ImageNet features, avg. rank 349.
(b) BYOL features, avg. rank 218.
deusity
prbb. density
0.15
0.30
0.04
0.00
rank
rank
(c) MoCo features, avg. rank =195.
(d) LRC features (proposed), avg. rank=181.
Figure 6: The histogram of the rank of feature embedding. Each sample in the histogram data matrix by embedding each patch in the WSI feature vector.
B DERIVATION OF LLRC
The derivation of LLRC incorporates the contrastive margin into the standard SupCon loss LappCon for positive pair classification.
Starting from triplet example with an anchor sample ta. positive sample εp, and negative sample in. The sigmoid function to identify the positive pairs is:
exp (simm(ta Ep))
(23)
exp (sim bp)) + exp (sim((aa in)) The positive pair is correctly classified if:
sim(ta, (,) 2 sim(ta; (ta:t (n)
(24)
We incorporate the margin constraint into the classification boundary so that the positive pair is correctly classified only if:
sim(ta, (p) 2 sim(ta, In) E
(25)
The sigmoid in
23 is modified accordingly:
exp (sim Ep)
(26)
exp (sim(ta, tp)) + exp (sim(ta, tw) + E)
14

---

Published as a conference paper at ICLR 2023
Therefore, the cross-entropy loss to identify the positive is:
exp (sim(ta: kp))
log exp (sim(ta, tp)) + exp (sim tn) +
(27)
Given an anchor la we get subspace Ca(a), 1, 1,,.. as stated in the low-rank representation Eq. (4). We intuitively consider that each subspace corresponds to latent class. We thus would like to discriminate between different subspaces, c.g. Ci()), (a(a), which are the least and most-distant subspaces to anchor to.
We extends (27 to Oj (a) positive and (a) negative pairs in sample batch with SupCon
Khosla
al. 2020), giving us:
exp (sim(tu: tp))
LLCC= = Σ [(((al a= N.NN
Σ log (aCi(a)
(28)
Eje (Ci(a)UC+(a)))e exp [sim(ta;& {2} +E,
where E j E (i)), otherwise Ej E.
C How SENSITIVE Is THE HYPER-PARAMIN PARAMETER IN LLRC?
Eq. 28 aims to push apart two subspaces. We set the top 5% of instances in a training batch as (a) and the bottom 5% as (a) by default. The percentage of Ci(a) controls the estimated positive samples in a minibatch given anchor tas and helps strike balance between the benefits brings with more true positive samples and the inverse effects of using false positive samples.
Table 4 shows the WSI classification evaluations on CAMELYON16 and TCGA-NSCLC datasets for ILRA-MIL models trained with different choices of C1 (a), ranging from 1% to 10%. The best result highlighted bold and the second best underlined. The results show that 5% achieves relatively optimal performance on both datasets. A larger percentage of 10% or smaller percentage of 1% generally leads to worse performance. We find the sensitivity of the hyperparameter reduced in the range of 3% to 7%.
Table 4: WSI classification evaluations for models trained with different choices of (1(a)
CAMELYON16
TCGA-NSCLC
Accuracy
AUC
Accuracy
AUC
1%
0.8899 0.0365
0.9260 0.0136
0.8897 = 0.0207
0.9551 0.0191
3%
0.9287 = 0.0090
0.9556
0.0098
0.9205 + 0.0224
0.9710 - 0.0185
5%
0.9218 = 0.0113
0.9649 - 0.0844
0.9213 + 0.0173
0.9763 + 0.0149
7%
0.9084 + 0.0215
0.9521 + 0.0095
0.9200 1 0.0204
0.9780 0.0234
10%
0.8884 0.0105
0.9137 0.0139
0.9113 1 0.0197
0.9663 1 0.0188
D TRAINING DETAILS
The training data splits are described in Section For each dataset, we use the same training data to first develop self-supervised learning model to conduct local feature embedding and then train MIL models implement global feature aggregation for classification.
One should pay attention not to exposing test datasets for the development of feature embedding model. although no labels are used. For example, if the test set of CAMELYON16 is used by MoCo- v3 for feature embedding pretraining, we can achieve 0.9885 AUC classification performance on the test set with CLAM-MB. This exceptionally high performance caused by data leakage.
D. SELF-SUPERVISED TRAINING DETAILS
We train self-supervised learning models conduct local feature embedding. We closely follow MoCo-V3 V3 (Chen et al. 2020c) and use the same training hyper-parameters. The data augmentation setting: (ixx224-iixll crop is taken from randomly resized image, and then undergoes random
15

---

Published as a conference paper at ICLR 2023
color jittering, random horizontal flip, and random grayscale conversion. For all methods, initial learning rate of 1.5e- We use AdamW as the optimizer and adopt a learning rate warmup for 20 epochs to alleviate instability. Each model is optimized on 16 Nvidia V100 GPUs with cosine learning rate decay schedule and a mini-batch size of 4096. We train for 200 epochs for CAMELYONI6, TCGA-NSCLC, and PANDA. The training takes about 4 days.
an
D.2 MIL TRAINING DETAILS
MIL models, including baseline and ours, are trained on a single Nvidia V100 GPU. The training of II-RA-MII. take less than hour for all datasets with an Nvidia V100 GPU. They are optimized end-to-end with Adam optimizer with a batch size of 1 and learning rate of le-1 for 200 epochs. The Adam optimizer has parameters 81 0.9, 82 = 0.95, and 1e-8.
E DATASET
Each WSI is cropped into a series of 224 224 non-overlapping patches using a binary mask for the tissue regions which is computed based on thresholding the saturation channel of the image.
CAMELYONI is a public dataset for metastasis detection in breast cancer (2-level classifica- tion), including 270 training sets and 130 test sets. A total of about 1.5 million patches at x 100 magnification are obtained after prep-process.
TCGA-NSCLO includes two subtype projects (2-level classification), i.c., Lung Squamous Cell Carcinoma (TGCA-LUSC) and Lung Adenocareinoma (TCGA-LUAD), for total of 993 diagnostic WSIs, including 507 LUAD slides from 444 cases and 486 LUSC slides from 452 cases. We obtain 3.4 million patches in total at x x10 magnification.
PANDA is the largest prostate biopsy public dataset to date (Bulten et al. 2022). We only use slides with pure and unequivocal patterns (from 0+0, 3+3, 4+4, or 5+5 slides) where the inter- observer variability making it 4-level classification problem. We use 4369 slides (1924 slides of 0+0, 1813 slides of 3+3, 466 slides of 4+4, 166 slides of 5+5) from Karolinska Institute for training, and 2591 slides (962 slides 0+0, 852 slides of 3+3, 660 slides of 4+4, 111 slides of 5+5) from Radboud University for testing. total of 1.1 million patches at x 10 magnification are obtained.
normally low (Tolkach al., 2020 Bulten et al. 2022 Ström et al. 2020),
F INFERENCE EFFICIENCY
We evaluate the inference runtime and MACs (nultiply-accumulate operations) of the proposed model. We use the CAMELYONI6 test set that contains 130 WSIs and the average bag size is 1600 at x10 magnification. The cvaluation involves data preprocessing including segmentation and patching, feature embedding using the LRC pre-trained ResNet50, and slide-level prediction with ILRA-MIL. The average inference run time for each WSI is represented in Table 5 The data preprocessing consume most of the time cost and this module is not accelerated by GPU. Feature embedding module converts 224 x 224 images into 1024-dimensional vectors and its MACs are relatively high. The feature aggregation module operating on embedding vectors is efficient and only takes about 4.4 ms.
Table 5: Average Runtime Per Slide on CAMELYON16 Using a P40 GPU.
Modules
runtime
MAC's
Data Preprocessing (non-parametric)
183.3 s
Feature Embedding (RestNet50)
3.6 s
13.22 T
Feature Aggregation (ILRA-MIL)
4.4 mas
2.89 G
'nttps//cammelyon16 brandaada-lllallleeeeor/rr/aaaa
"https:/pportal. http://oooaal.g......... wwt.cccccc..g.o///////
16

---

Published as a conference paper at ICLR 2023
G CLASSIFICATION RESULTS ON BENCHMARKS
The following Table 6 and Table 7 are extensions of Table the main paper with the standard deviation. Each experiment conducted for runs with respect different random startup seeds on the same data splits.
Table 6: Results on Benchmarks CAMELYON16 and TCGA-NSCLC.
CAMELYON16
TCGA-NSCLC
Accuracy
AUC
Accuracy
AUC
Mcan-pooling
0.6511 + 0.015
0.6755 + 0.0415
0.7282 0.0181
0.8551 = 0.0429
Max-pooling
0.7674 0.004
0.8169 + 0.0254
0.8593 0.0451
0.9263 0.0585
ABMIL
0.8527 + 0.0252
0.8503 + 0.0293
0.8384 = 0.0260
0.9205 = 0.0259
MIL-RNN
0.8449 + 0.0257
0.8580 + 0.0314
0.8619 0.040
0.9107 = 0.0430
CLAM-SB
0.8682 + 0.0133
0.8709 + 0.0135
0.8632 + 0.0267
0.9307 = 0.0162
CLAM-MB
0.8601 + 0.0215
0.8779 + 0.0193
0.8492 0.0294
0.9377 0.0139
DSMIL
0.8759 + 0.0231
0.8944 0.0184
0.8690 0.0277
0.9439 0.0215
DSMIL SimCLR
0.8867 + 0.0201
0.9175 + 0.0139
0.9048 0.0225
0.9551 = 0.0187
TransMIL
0,8449 + 0.0381
0.8669 + 0.0273
0.8565 + 0.0178
0.9303 = 0.0154
DTFD-MI MITD-MMLL (MaxS)
0.8543 + 0.0236
0.9103 + 0.0312
0.8701 + 0.0294
0.9097 = 0.0185
DTFD-MIL (AFS)
0.9010 + 0.0341
0.9401 + 0.0272
0.8941 0.0331
0.9612 0.0223
ILRA-MIL
0.8992 1 0.0184
0.9278 1 0.0121
0.9004 0.0218
0.9592 - 0.0176
ILRA-MIL LRC
0.9218 1 0.0113
0.9649 1 0.0096
0.9213 0.0173
0.9763 1 0.0149
Table 7: Results on PANDA test set.
PANDA
Accuracy
Kappa
Mean-pooling
0.5691 0.0493
0.4422 1 0.0248
Max-pooling
0.6100 1 0.0255
0.5830 1 0.0460
ABMIL
0.6834 1 0.0177
0.5998 0.0155 NA
MIL-RNN
NA 0.6648 0.0368
CLAM-SB
0.5782 + 0.0182
CLAM-MB
0,6760 + 0.0441
0.6067 + 0.0161
DSMIL
0.6737 + 0.0468
0.5562 + 0.0127
DSMIL+ SimCLR
0.7017 0.0530
0.5837 + 0.0231
TransMII.
0.6720 0.0434
0.5638 + 0.0135
DTFD-MIL (MaxS)
0.6334 0.0329
0.5462 + 0.0237
DTFD-MIL (AFS)
0.6573 + 0.0218
0.5437 + 0.0193
ILRA-MIL
0.7094 0.0309
0.6236 0.0143
ILRA-MIL LRC
0.7287 0.0210
0.6562 1 0.0244
H HEATMAPS
Figure 7 shows diverse examples on the CAMLEYON16 test set. In the "raw image" column, the tumor area is delineated by the blue line. In the "CLAM" and "CLAM+LRC" columns, brighter red indicates that the higher attention score is the tumor at the corresponding location. "CLAM" is the original CLAM-SB method (Lu al. 2021c) whereas "CLAM+LRC" incorporates CLAM-SB with our proposed LRC feature embedding method.
17

---

Published as a conference paper at ICLR 2023
raw image
CLAM
CLAM LRC
Figure Four examples of high-resolution heatmaps of the CAMELYON16 test set, namely test_016, test_073, te,t_ and test_092 from the top row to the bottom row. We compare the heatmap CLAM in the second column with our proposed LRC method in the third column. Our method is more consistent with the ground truth annotations, indicating superior performance.
18


Reviews:
[{"review_id": "qkFEdGDZCZ", "reviewer": "Reviewer_j84C", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a new model architecture and training method to classify digital pathology whole-slide images (WSI). These images are in the giga-pixel range and cannot be classified directly by a model. The paper proposes a multi-instance learning approach with contrastive self-supervised learning for patch level feature extraction and an iterative low-rank attention model for the multi-instance feature aggregation. The method is tested on 3 public digital pathology datasets and shown to significantly outperform SOTA. In an ablation study, other feature extraction and MIL techniques are compared to the proposed one, showing superior results of individual components. In a parameter analysis, low-rank size, pooling method and number of iterations are studied.\n\n", "summary_of_the_review": "Overall, a good paper with a clear and convincing approach that beats SOTA.\n", "strength_and_weaknesses": "Pros:\n\n* A sophisticated MIL model that significantly beats SOTA on several public datasets\n\n* Both feature-extraction and MIL steps are improved, as shown by the ablation study.\n\n* An exploration of several hyper-parameters show a sweet spot for the number of iterations in MIL, an asymptotic behavior for the rank of the low-rank decomposition and the superiority of nonlocal pooling against regular max-pooling or local-attention-pooling.\n\n\nCons:\n\n* There are no speed information given for training and inference. How long does it take to classify a slide on average ? How long does it take to train the feature extractor, the MIL model ?\n\n* no details are given on which type of augmentation is used in the contrastive loss.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nfX_q0zJxZX", "reviewer": "Reviewer_NvEE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel framework for Whole Slide Image (WSI)  classification based on two\nlow-rank methods:\n1. A Low-Rank Constraint (LRC) feature embedder to extract features from pathological slides in\na self-supervised fashion.\n2. An Iterative Low-Rank Attention MIL (ILRA-MIL) model for bag-level classification , derived\nfrom the vision transformers architecture.\n\nBoth methods rely on low rank assumptions, i.e. the existence of a data representation in a smaller dimension space that exhibit more discriminatory characteristics for feature embedding on the one hand, and lower computational complexity for bag-level prediction on the other.", "summary_of_the_review": "The paper brings an interesting idea worth being introduced in the community. But it lacks clarity and is impaired by many typos, missing words and unprecise formulas.", "strength_and_weaknesses": "Strength : \nThe approach presented here is quite new in the field of digital pathology, and the authors show\nthat the enforcement of low-rank constraints on top of existing self-supervision and attention-based\nMIL strategies improves the results on several whole slide image classification datasets.\n\nWeaknesses : some explanations are not clear enough or suffer from ambiguous notations, and overall there are several errors along the manuscript.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P1HGDihVvNC", "reviewer": "Reviewer_oayv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of multiple instance learning in pathology imaging where annotations are available only at slide level and the goal is to predict it at instance level. The author exploits the low-rank structure in the data to design a new contrastive learning approach. LRC is proposed as an extension SupCon approach for low-rank data by defining the loss function on the most- and least-distant subspaces. At the feature aggregation level, an iterative approach is proposed using transformers. A learnable low-rank latent matrix L is used across the layers to encode global features. The transformer model named ILRA is based on two transformer modules GABf which reduces the space dimension and GABb restores it. The ILRA transformer layer is applied k times. The proposed approaches are benchmarked on three datasets, showing the improved performance ", "summary_of_the_review": "I think the main drive for improvements in the proposed model is the contrastive learning loss LRC rather than the proposed global features. ILRA-MIL alone does not do better than the baseline methods. This would require further investigation and limits the novelty of the paper. ", "strength_and_weaknesses": "Strengths\nLow-rank property provides good inductive bias in applications such as in computational pathology\n\nWeakness\nNo validation with multi-center datasets\nThe computation time has not been discussed and could be prohibitive for real-world usage where inference time needs to be reduced.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "NpkI-tN0-Jk", "reviewer": "Reviewer_LxYk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of WSI classification by optimizing the feature embedding and feature aggregation with low-rank properties. Firstly, the paper improves the vanilla contrastive loss with additional low-rank constraints to collect more positive samples for contrast. Then, the authors devise an iterative low-rank attention feature aggregator to make efficient non-local interactions among instances. Experiments show the effectiveness of the proposed model. ", "summary_of_the_review": "Overall, the structure is clear and the proposed model is well-validated. ", "strength_and_weaknesses": "Strength:\n(1) The authors extend contrastive learning with a low-rank constraint (LRC) to learn feature embedding using unlabeled WSI data.\n(2) The authors design an iterative low-rank attention MIL (ILRA-MIL) to process a large bag of instances, allowing it to encode cross-instance interactions.\n(3) Experiments are conducted on multiple datasets and the effectiveness is well studied. \n\nWeakness:\n(1) In the Abstract, the authors state that “we highlight the importance of instance correlation modeling but refrain from directly using the transformer encoder considering the O(n^2) complexity”. However, in the main body part, there are not any details to discuss the importance of instance correlation. Besides, in the experimental section, the importance should be investigated.\n(2) The authors should also provide more details to discuss cross-instance interaction, and whether is it helpful to boost the classification performance?\n(3) Some details of experimental settings are missing. For example, k is the total number of layers. Thus, in the experiments, how to set k? Besides, for the used contrastive loss, which type of augmentation is adopted in this study? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster