Dual-Curriculum Contrastive Multi-Instance Learning for Cancer Prognosis Analysis with Whole Slide Images
Authors: CHAO TU
YU ZHANG
Zhenyuan Ning
Venue: NeurIPS 2022 Accept (2022)
Keywords: Prognosis Analysis
Whole Slide Images
Multi-Instance Learning
Curriculum Learning
Contrastive learning

Abstract: The multi-instance learning (MIL) has advanced cancer prognosis analysis with whole slide images (WSIs). However, current MIL methods for WSI analysis still confront unique challenges. Previous methods typically generate instance representations via a pre-trained model or a model trained by the instances with bag-level annotations, which, however, may not generalize well to the downstream task due to the introduction of excessive label noises and the lack of fine-grained information across multi-magnification WSIs. Additionally, existing methods generally aggregate instance representations as bag ones for prognosis prediction and have no consideration of intra-bag redundancy and inter-bag discrimination. To address these issues, we propose a dual-curriculum contrastive MIL method for cancer prognosis analysis with WSIs. The proposed method consists of two curriculums, i.e., saliency-guided weakly-supervised instance encoding with cross-scale tiles and contrastive-enhanced soft-bag prognosis inference. Extensive experiments on three public datasets demonstrate that our method outperforms state-of-the-art methods in this field. The code is available at https://github.com/YuZhang-SMU/Cancer-Prognosis-Analysis/tree/main/DC_MIL%20Code.

Dual-Curriculum Contrastive Multi-Instance Learning for Cancer Prognosis Analysis with Whole Slide Images
Chao Tu 12.33 Yu Zhang 1.2,3,+ Zhangl.233, Zhenyuan Ning 1.2.3,
SSchool Biomedical Engineering, *Guangdong Provincial Key Laboratory of Medical Image Processing, 3 GGanggdng Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China (tchaoc170smu, edu. cn. yuzhang@smu, edu. edu cn, jonnyning@f www.annnn@oommmll.mmmmmmmmm oxmail. com) "Corresponding authors.
Abstract
The multi-instance learning (MIL) has advanced cancer prognosis analysis with whole slide images (WSIs). However, current MIL methods for WSI analysis still confront unique challenges. Previous methods typically generate instance represen- tations via a pre-trained model model trained by the instances with bag-level annotations, which, however, may not generalize well to the downstream task due the introduction of excessive label noises and the lack of fine-grained informa- WSIs. Additionally, existing methods generally
tion across multi-magnificatiin aggregate instance representations as bag ones for prognosis prediction and have no consideration of intra-bag redundancy and inter-bag discrimination. To address these issues, we propose dual-curriculum contrastive MIL. method for cancer prognosis analysis with WSIs. The proposed method consists of two curriculums, i.e., saliency-guidee weakly-supervised instance encoding with cross-scale tiles and contrastive-enhance soft-bag prognosis inference. Extensive experiments on three public datasets demonstrate that our method outperforms state-of-the-an methods in this field. The code is available at https://github.co http://ithuub.coo/yu/ualae www..//ooo/yyogSSSS. SuMZZhaannSUU/Cnneerr Prognosis- Aataa//saaaaaiiiteeeaaaaaCCCCCCC Mtt.aaaaiiiiDDMMMMM00000
Introduction
Cancer prognosis analysis is of great significance for risk-benefit assessment and clinical decision [1]. Computational whole slide image (WSI), which entails the quantitative profiling of spatial patterns and tumor microenvironments in tissue slides, has advanced the application of deep learning in cancer prognosis analysis [2. 3]. However, deep learning in WSIs often hindered by the gigapixel size and the lack of pixel-level annotations [4]. Recently, multi-instance learning (MIL) methods have attracted increasing attention performing inference weakly-supervised manner and have been successfully applied to WSI analysis [5, 6, 7].
In general, the MII. consists of two stages, i.e., instance encoding stage and instance aggregation stage [8, 9]. In the encoding stage, several methods have utilized the pre-trained model (e.g.. ResNet on ImageNet) extract instance representations an unsupervised way (i.c., without the reference of outcome labels) [10, 11, 12]. However, recent studies have demonstrated that the pre-trained model may not generalize well to the downstream tasks it task-agnostic and inclined to overfit
'The "task-specific" and "task-agnostic refer training manner that is oriented by the task target This work aims conduct cancer prognosis. The "task-specific" means using prognosis labels (e.g., survival time or death risk) as supervision to train the model, while the "task-agnostic" means unsupervised or using mognosis-irrelevar annotations as supervision.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the pretraining objective [13, 14, 15]. Alternatively, some work directly assigned each instance the bag-level unnotation (i.e., the survival time of patient) for network training, which introduces task-specific' supervision and improves model's generalisability [16, 17, 18). Actually, the prognosis evaluation primarily and comprehensively determined by certain representative regions, yet those regions might only occupy small portion of WSI [17]. As result, the strong supervision on all instances, especially on grognosis-irrelevar tiles, will introduce excessive label noises and limit model's performance [19, 20]. For alleviating this issue, a feasible way is to supervise the instance encoding weak and easy-to-hard manner. On the other hand, existing methods often generate mono-scale instance representations [11, 21] typically concatenate multi-scale information from multi-maanifiiationn WSIs [22, 23), which may cause feature redundancy and ignore fine-gratd -ra-grained details across multi- multi-mmaniiiiation images.
In the aggregation stage, most previous studies aggregated instance representations within a bag for prognosis estimation by certain fusion strategies (e-g., pooling operation and attention mechanism) (24, 25]. However, it may overwhelm bognosis-relevan information, cause intra-bag redundancy, and reduce inter-bag discrimination, if many instances from irrelevant regions are enrolled [12, 26]. To address the aforementioned issues, we arm MIL with a dual-curriculum strategy and propose a dual-curriculum contrastive MIL method for cancer prognosis analysís with WSIs. The proposed method consists two easy-to-hard curriculums, i.e., saliency-guided supely--epereised instance encoding with cross-scale tiles and contrastive-enhanet soft-bag prognosis inference. The main contributions are summarized as follows:
1) We present dual-curricllum contrastive MIL. method which includes two easy-to-hard curriculums. We first conduct a preliminary task to learn instance representations by considering risk stratification status (degraded from survival time) as annotation, followed by the prognosis inference with survival time as supervision.
2) We design the first curriculum of saliency-guided reakly-supervised instance encoding with cross-scale tiles. It is supervised by relatively weak annotations so to reduce label noises and maintain prognosis reelated-related guidance. Additionally, to imitate the reviewing procedure of pathologists, we leverage the wwwgainifiiationn saliency map guide the encoding of magginfaniiitttion instances for exploring fine-gu mme-grained information across multi muulti-manniiicationn WSIs
3) We develop the second curriculum of contrastive-enhance soft-bag prognosis inference. Instead of enrolling all instances, we adaptively identify and integrate representative instances within a bag (as the soft-bag) for prognosis inference and leverage the constrained self-attention strategy obtain extra sparseness for soft-bag representations, which can help reduce intra-bag redundancy in both instance and feature levels. Meanwhile, improve the Cox loss with two-tier contrastive learning for enhancing intra-bag and inter-bag discrimination.
4) We evaluate the proposed method on three public cancer datasets, and extensive experiments demonstrate its superiority in cancer prognosis analysis with WSIs.
2 Related Work
2.1 WSI-based Cancer Prognosis Model
In recent years, many methods, especially convolutional neural networks (CNNs), have been devel- oped for cancer prognosis evaluation with WSIs [27, 28, 29]. Due to the gigapixel size of WSI, some methods took small-size ROIs selectively sampled from WSIs as network's 's input for training and inference, in which all ROIs were assigned with patient-level outcome labels [30, 31]. However, these methods generally require prior knowledge from pathologists, which is may suffer from high inter-observer variation. Therefore, current researches have proposed crop the entire WSI into many tiles for model's 's input so as to make full use of available information contained by WSI [10, 32, 33]. However, these methods often confront the lack of tile-level annotations and may result in sub-optimal solution. To this end, MIL methods have been introduced to perform cancer prognosis analysis in a weakly-supervised manner and have shown promising performance in this field [5, 28).
and

---

2.2 Multi-instance Learning in WSI Analysis
The MIL methods in WSI analysis can be roughly divided into two categories according to the way instances are leveraged in the model [12, 17]. The one infers bag-level prediction from instance-level probabilities via average pooling, maximum pooling, or other ensemble strategies [34, 35]. By com- parison, the other one, referred to as the bag embedding method, aggregates instance representations as bag ones, upon which a model is constructed for bag-level prediction [36, 37J. The former is simple and straightforward but empirically proven to be inferior to the latter in terms of generalization performance. Therefore, the bag embedding method has attracted extensive attention in WSI analysis [17]. For example, Courtiol et al. [6] developed a MIL method based on CNN, called MesoNet, to accurately predict the overall survival of mesothelioma patients with WSIs. Yao et al. [5] proposed a deep attention multi-instance survival model to efficiently learn features from WSIs and fuse them for prognosis prediction. Our method essentially belongs to the bag embedding method. Different from previous work, the proposed method consists of two easy- easy-to- to--ard curriculums, comprehensively considering fine-gred one-grained information across multi- multi-magnification WSIs, intra intra-bag redundancy, and inter-bag discrimination.
3 Method
3.1 Problem Formulation
Denote the triplet {Xa, Tn, 6 as the dataset consisting of N patients, where X,, is the n-th patient with one more WSIs, and T,, and ö, are respectively its observed survival time and event indicator (viz. equals to and 0 for uncensored and censored samples, respectively). Prognosis analysis is an ordinal regression task that models time-to-event distribution, which is formulated
T s(X,,;
where S and Tn denote the prognosis inference function and the estimated survival time, respectively. In practice, is difficult to directly feed the WSI into the model due to its gigapixel size. Accordingly, many studies [10, 32, 33] have cropped the WSI into a large amount of tiles and generated a bag Xn { J.m that contains N,, instances (tiles) for the n-th patient. Then, MIL is leveraged to construct the prognosis model as follows:
T, = S(AA Xn,i E x,,)),
(2)
where EE is an instance encoding function that extracts feature representation for each instance, and A is crmmutation-invariaan instance aggregation function that pools the instance representations into the bag ones. However, previous studies [10, 11, 12, 16. 17. 18] typically generate instance representations via pre-trained model or model trained by the instances with bag-level annotations, which may not generalize well to the downstream task. As result, we decompose the model into two easy-to-hard curriculums, including (1) Curriculum (C-I): instance encoding (via E) during preliminary risk stratification (via C, a risk stratification function), and (2) Curriculum II (C-II): prognosis inference (via S) after instance aggregation (via A). Formally, the optimization process can be formulated as
N
N
C.I: arg min
Σ
(Y, == 1|
Σ [C [C Xn, E Xn), Ynla,
E,C
1=1
(3)
N
C-I1: S. arg min
[(((((( Σ S.A
xn,s E X, Tn, 8,1cn
where t(-) outputs (0) if true (false), and Li and Lu denote the loss functions for C-1 and C-II, respectively. The risk stratification status Y, is determined by (Tn, 5n) with three-year time threshold (denoted as T,) as follows:
T, S T, & òn = 1
Y, n
0 T, T
others
3

---

F
Z
E,
B
Faran Ensser
Furas
H
R
Lag Patee
P
8
B
Figure 1: The pipeline of the proposed dual curl-curriculum contrastive MIL method for cancer prognosis analysis with WSIs. The left is sereendy-guiedd supervisd instance encoding with cross- scale tiles (Curriculum D), while the right one is contrastive-enhanced soft-bag prognosis inference (Curriculum II).
3.2 Curriculum I: Saliency- gugdded Tiles
Instance Encoding with Cross-scale
In this section, we introduce the first curriculum of guiided weakly-supervise instance encoding with cross-scale tiles, as shown in Figure 1. Given the input instance with S magnifications 3 in this work), the proposed method contains $ branches and each branch takes
[x= (S 3 iifferent-maanifiictii
tiles input, which aims to explore multi-scale information from multi- magnification images. For the s-th branch, mainly consists of a feature extractor F", feature aggregator G and classifier C And {F and (G*)s= form E, while {C+}s |s= constitute C in E3)( The feature extractor I contains s residual modules with the parameters {0) [0...... ... 0..
and the first 1 modules share the same architecture with F if 1. Figure 2 illustrates the architecture details?. Different from previous studies [22, 23] that separately feed {x into the network and ignore fine-grained details across multi-magnification images, encourage the network to focus on salient regions by utilizing the prior knowledge of the (s - 1)-th branch to highlight the input x which can be formulated as
1
(5)
>
where @ denotes the element-wise multiplication operator, and m is the salient mask indicating high-risk regions. For convenience, we introduce how to acquire m which is easily generalized to Specifically, m is computed by Class Activation Map [38] follows:
m n.i
m = w @ F* (8%,))
(6)
where O is the shannel-wise multiplication operator, and w" indicates the importance of channel of feature maps for risk stratification prediction in the s-th branch. To obtain w", we first generate instance embedding vector z,,; by
" G (F* (&,,).
where G is feature aggregator that performs channel-directin attention (as shown in Figure 2).
Incorporating comprehensive information from current and previous branches, subsequently feed instance embeddings {z/,,)= into the classifier C* for predicting the high-risk probability P which is computed by
P ((,,)))))) ó(w"
[Z
2,, 2--11"" w (z)
"The feature extractor F shares the same architecture with the top-14 14 layers of ResNet-18, 18, upon which g² additionally deepens the network by introducing two identify blocks with the modification of kernel number (from [128, 128, 512] [128, 128, 256J). And Fd deepens the architecture using the same way as F

---

Big
2,1
1-
Figure The detailed architectures of some modules in the model. The first shows the architectures of feature extractor IF, feature aggregator G and classifier C in Curriculum I, while the second row illustrates the architectures of soft-bag learning model B, the constrained self-attention module D and contrastive-enhanee prognosis inference function & in Curriculum II.
where C is implemented by linear layer with the parameters ( ,w" (as shown in Figure 2), and and T denote softmax activation function and transposition operation, respectively.
We introduce hierarchical transfer strategy to leverage the prior from different- iiterrenttmmaiiiiattioo images, accelerate network convergence, and maintain the specificity of current branch. Specifically, as shown in Figure 1, the parameter 0ş. (for any s) of the s-th module in E completely learnable and has not any constraints, while 0f. (for > 1) is also learnable but constrained to approximate 0p.-1 by the structural loss Lo follows:
Ln 00 0p-
For the remaining parameters {0).................. 0g.²) (for s 2) F they are first initialized with (fp--,,.... ... 0 }, and then are frozen. Furthermore, we propose hybrid loss function 4 that contains parts, i.e., empirical loss Le and structural loss La, train the network:
L = Le + BaCa.
(10)
where Bg the weight coefficient, and Es is defined as
N N Li NN, ΣΣ y log(p"..) + (1 - log(1 - Ph,)),
(11)
where yn,; E Y, denotes the risk stratification status of the i-th instance in the n-th bag.
Finally, we can obtain the multi-scale instance representation Zn,s for subsequent prognosis inference task by Zn,i ",,.."... i
3.3 Curriculum II: Contrastive-enhancee Soft-bag Prognosis Inference
In this section, develop the second curriculum of inntrastiveeenhance soft-bag prognosis in- ference, as shown in Figure 1. Given a set of instance representations from the n-th bag, i.e., Z, = |2,,1: 2n,2. :2,,N_] 1T N., where C denotes the feature dimension, the proposed method mainly consists of a soft-bag learning module IB, constrained self- atff-attentinn module D, contrastive-enhance prognosis inference function 8, where B and D form A in (3).
For the soft-bag learning module (as shown in Figure 2), it first learns new bag representation E, E JRN.x (where D denotes the feature dimension) from Zn via linear layer. Instead of enrolling all instances, we adaptively identify and integrate representative instances within a bag by introducing the function H:
É, HE,,, h,,) (hdiog(h..) w.." (diag(h,) E.).
(12)
where E,, N (NB is the number of selected instances in each bag, and NB N,, the feature representation of those selected instances, and W. NB denotes the weight matrix
5

---

that is generated by feeding E,, into two linear layers (as shown in Figure 2). And h, E R x. is an indicator vector that adaptively selects NB representative instances and is computed by
Nu s.f.
h = arg max S (ID E, h)))),
h = NB.
(13)
b
The constrained self-attention module D and contrastive-enhancee prognosis inference function $ are introduced below.
We leverage to obtain extra sparseness for soft-bag representations. Given the input E,, first linearly projects E, into three feature spaces (i.e.. Qn E RN Kn E RNxxD and Vn e RNxx and Î is the feature dimension) via three mapping matrices (i.e., We DRD Wx
DDD and Wy E DxD)) in which the mapping matrices are constrained by
L. = (Ill + |[ + [Wvll1)
(14)
Then, the sparse soft-bag representation B,, E g1xDD (where DB is the feature dimension) can be obtained by the correlation-baser activation mechanism [39], as illustrated in Figure 2. It worth mentioning that the collaboration of B and DD can significantly help reduce intra- bag redundancy in both instance and feature levels so as to improve model's 's generalization.
For S, takes B, as input to infer the relative risk T,, via linear projection matrix W3 Da as shown in Figure 2. In order to boost the ability of soft-bag inference, we perform two-tier contrastive learning (TCL) to enhance intra-bag and inter-bag discrimination. We assume that a bag B, is sampled from the distribution O {0+, O-}. where the positive source O+ denotes the distribution of high-hazard samples (i.e., those with survival time S Ty), while the negative source O presents the distribution of low-hazard samples (i.e., those with survival time T,). We first focus inter-bag discrimination. Let B, = (B1, ....... BN.., }/B,, present the collection of bags (expect B,,) from the same source O+ (or O-) as B,, where NB., denotes the number of such bags. We can obtain the representation of B, (denoted as B,) by merging the representations of all bags in B, via an attention function IK (as illustrated Figure 2). Inspired by [40], inter-bag discrimination can be enhanced by maximally preserving the mutual information between B, and B,, follows:
Σ p(B,, B,) log p(B, (B.) RE(N]
Σ B... ME[N]
(15)
((B..)
According to (12( partial instances within a bag are adaptively discarded, and the rest remained. However, all instances within the bag belong to homologous tissues sampled from the same patient, which means that it should show potential correlation among these instances. Therefore, similar to (15), we maximize the following mutual information to improve intra-bag discrimination:
B.). Σ I(B,, (a) = Σ p(B., B.) log p(B..) NE[N]
(BB.
(16)
nE[N]
where B,, is obtained via K to merge the representations of those discarded instances E,, that has the definition of E,, IH( (E,, h,), illustrated in Figure 2. To this end, we propose to jointly maximize (15) and (16), i.c., mar (nerNE I(B,, B,.) + I(B,, B,)),, which is formulated by
((B BB) p(B,.)
+ log ((B,, = max( Σ p(B..)
((B,, ( BR)). log
max
Σ log N
(17)
PB.)) (B..)
nE[N]
Instead of constructing a generative model pBB.. (B, B,), we leverage a log-bilinear function L. to model the density ratio which preserves the mutual information:
L(B L(B,, B,, B.) (B W,,,B,,BBB Debp)) V,
(18)
eeppBB...i B x PBB B. (B, B.)
(19)
p (B,, (B.))

---

where WL and VL are trainable parameters. Motivated by [41], we design the TCL loss based on the InfoNCE function, which has the following definition:
L(B,, B,.
Leel E log
(20)
Σ LiE[N] L(B,, B,, B.)
where B denotes the collection of all bags. Additionally, utilize the negative log- likelihood of Cox proportional hazard regression model [42] train the network, which is computed by
(B, Ws)
II Leer log( 1=66,1]
E E)ER(T,) Cr Ws) -- Σ
(Bn Ws llog( > SER(TA) eap(B, Ws))),
where Wg is the linear projection matrix in &, and R(T,) = (i T, > n, is the collection of all samples (including censored and uncensored ones) with survival time being longer than TT, Finally, the loss function Lu for the second curriculum can be defined as
Lι = Looz + Leet + ByCo,
(21)
where B, is the weight coefficient.
Experiments and Results
In this section, we briefly describe datasets, preprocessing, and implementation details, followed by experimental results.
4.1 Datasets and Preprocessing
In this work, we evaluated the proposed method on three datasets from The Cancer Genome Atlas (TCGA) database, including colon adenocarcinoma (COAD)(N = 365), hepatocellular carcinoma (LIHC)(N 334), and bladder urothelial carcinoma (BLCA)(N = 396). Each dataset contained WSIs stained with hematoxylin and cosin (H&E) and clinical information (i.e., survival time and event status). After segmenting tissue areas via the Otsus method. set of tiles under different magnifications (20x, 10x, 5x) were non-overlappingly sampled from each segmented tissue area, and the window sizes were set 512 512, 256 256, and 128 x 128, respectively.
4.2 Implementation Details
In our experiments, we adopted the 5-fold cross-validation strategy to comprehensively evaluate our proposed method and the ratio training and validation sets was set to 4:1. Specifically, we divided the entire dataset into five folds, among which four folds for model training while the remaining one for model evaluation. Notably, the cross-validation strategy was conducted on patient level to prevent data leakage which means that the WSI of each patient only appeared in one of these subsets. During the training stage, we selected the best model in terms of two specific evaluation metrics for different curriculums. That is, the model with the highest accuracy was selected for Curriculum while the model with the highest C-index for Curriculum II. The concordance index (CT), the receiver operating characteristic (ROC) curve, the area under ROC curve (AUC), and Kaplan-Mest Meier (KM) curve were used to assess model's performance. We chose three-year time as threshold for risk stratification status and AUC calculation, which lies two reasons as follows: 1) Three-year survival time is an important indicator for clinically evaluating the prognosis of patients, especially for three cancer types studied in our work [43, 44, 45). 2) For the used datasets, using three-year survival time as threshold can make the distribution of positive and negative samples relatively balanced, which is beneficial to network training. We implemented all competing methods using the Pytorchl.9 .9 library on Python3.6. 6. All intensive calculations were offloaded to a 12 GB NVIDIA Pascal Titan X GPU.
Curriculum I. We adopted Adam optimizer with the weight decay of 0.2 for network training. For different magnifications s E {1, 2, 3), the learning rate, batch size and epoch number were set to {10-5,3 5, 3 10-6 3 10 00-6), (32, 16, 16} and (50, 50, 50}, respectively. The weight coefficient Bn was set to 10
Curriculum I.. We adopted SGD optimizer with the momentum of 0.9 for network training. The learning rate, batch size and epoch number were set to 10-4, 16 and 1000, respectively. The soft-bag size N B set to 3 and the weight coefficient B, was set to 10
7

---

Table 1:
The results achieved by all competing methods on three datasets (i.e., COAD, LIHC, and
BLCA). The boldface denotes the best result.
COAD
LIHC
BLCA
Overall
AUC
AUC'
AUC
WSISA [46]
0.641
0.673.10.037
0.601200002
0.700:10.071
0.639.10.018
0,651 1:00043
0.647
0.675
DeepGraphSury [47]
0.629=0.054
0.639:::00388
0.113:0.027
0.633:10:047
0.620=+0.015
0.687=0.046
0.620
0.653
MesoNet [6]
0.643=0..39
0.684=0.0022
0.629=00022
0.667::0.042
0.630=t0.021
0.648=0.052
0.634
0.666
DeepAtMMiSL
0.683=00025
0.654:0.045
0.672+00.027
0.676:1:00119
0.642:+0.020
0.661=00007
0.666
0.663
Patch GCN [10]
0.650.10.013
0.562.10.00
0.565110.024
0.630.10.061
0.501
0.542.10.027
0.595
0.578
DC MC_r)
0.717=00012
0.754
0.705
0.745::0.012
0.672:10.029
0.720
0.698
0.740
WSISA
DeepCrahhSary
MesoNet
DeepAtrMMSS
Patch-GCN
DC MIL(
1.8-
Tow nit
knn ri
Iow nak
low
low nic
Jun rik
COAD
survival rate
high nk
0.8
highrik
[U5-
ligh nik
0.--
rol
0.5
nigh
0.8-
bight rik
0.6.
0.6
0.5
0.2
0.
387
0.2
0,003
0.2
0,753
0.2-
0.078
0.2
0.865
0.2
0.002
0.0
0.0
10
0.0
0.5
0.0
Iow ric
Inor ri
lox nik
survival rafe
high
0.8
bigh
high mic
0.8
high tik
0.8
high m1d:
bigh mi
LIHC
0.6-
0.6-
0.6-
0.6-
0.6-
0.5
0.4-
0.4
p
0=0.001
0.2
p=0.55
p 009994
0000009
p
p=0.415
440077-
0.
75
10.0
0.9
255
5/0
15
50
75
10.0
9.
0.0
255
5.0
10.0
0.0
2.5
50
7/5
255
75
10.0
low
1ow
Jon
survival rate
ligh
bigh chik
ligh rid
high: r5k
0.5
ligh ridk
high c5a
BLCA
6.-
0.6
0.6-
0.6
0.5
0.6
0.4
n4
8.6
04
p 00=00 0. 152
p=
-4.9655
p 0.466
p=333
p=9.919
p 3,991
0.0
10
0.0
10 time (year)
5
0.0
5
10
0.0
0
10
0.0
5
0.0
15 time (year)
5
(year)
time (year)
time (year)
time (year)
Figure 3: The KM curves with the p-values of all competing methods on three datasets. The farther apart the two curves are, the better the model works.
4.3 Comparison with State-of-the-Art Methods
We compared our method with other weakly-supervisee methods in cancer prognosis analysis, including clustering approaches [46], graph networks [10, 47], and multi-instance learning methods [5, 6]. Table 1 presents the 5-fold cross-validation results (in terms of CI and AUC) of all competing methods. The KM curves as well as p-values, are provided in Figure 3. We can observe that the proposed method outperforms others in head-to-head comparisons, achieving an overall CI of 0.698 and AUC of 0.740. It may benefit from several potential advantages in DC MILL 1) Unlike some methods that leverage the dimension reduction technique [46] or pre-trained model [5, 6, 10, 47], encodes instances in a weakly-supervised manner so to reduce label noises and maintain prognosis-related guidance. It utilizes the low-magnification saliency map to guide the encoding of high- maggifiaiiiii instances for exploring fine-graines information across multi- muulti-maniificttion WSIs. 3) It introduces a soft-bag learning method and a constrained self-attention strategy to help reduce intra-bag redundancy in both instance and feature levels, respectively. 4) It equips the Cox loss with two-tier contrastive learning for enhancing intra-bag and inter-bag discrimination. Additionally, we also find that WSISA [46]. MesoNet [6], and MISL [5] outperform DeepGraphSury [47] and Patch-GCN [10], which may owe to the selection of discriminative tiles. More results can refer to Appendix D.
4.4 Ablation Study
Ablation on Curriculum I. We validated the efficacy of each crucial component in Curriculum From Table 2, we can observe several key points: 1) Compared with pre-trained ResNet-34 (regardless with MM or not), the proposed model shows significant performance improvement, which demonstrates the effectiveness of Curriculum 2) The model with MM outperforms the one without MM and gets CI gain of 0.039 and AUC gain of 0.061, which benefits from the utilization
8

---

Table The results of ablation experiments on three datasets (i.e., COAD, LIHC, and BLCA). The boldface denotes the best result.
DC. C.I+C.II
COAD
LIHIC
BLCAA
MM
SG
HT
C-11
CI
CI
AUC
AUC
X
X
X
0.676:10.031
0.696:10.055
0.662:10.021
0.659-10.012
0.640:20.038
0.680:10.046
x
J
0.69240.0049
0.705-+0.091
0.671
0.720=00.000
0.647=0.019
0.678::00007
X
0.707
0.714 001
0.699#0.019
0,721=00018
0.654=0.016
0.700=:0.043
PT
w/o
MM
0.642=0.000
0.643=000022
0.644
0,640=0.018
0.618=0.013
0.635=00.028
PT
w/
MM
0.660
0.683+00.025
0.662#0.021
0.691=0.036
0.633=0.015
0.639:00.020
C.I
SB
CSA
TCL
CI
AUC
CI
AUC
CI
AUC
X
0.601=00.000
0.697=10.040
0.653=t0.028
0.680=000022
0.621=00.023
0.630:1:.002
0.670::00044
0.715:20.066
0.673=t0.031
0.717=00.06
0.655:20.024
0.668
0.709:10.025
0.729:10.063
0.693:10.033
0.736:10.029
0.669:100017
0.696
0.678100.031
0.721
0.680.10.046
1.0044
0.659
0.680 0.680:10.066
0.717
0.754.10.00
0.705 1.0015
0.745 1.0012
0.672 0.029
0.720 1 0.039
C-LL Curriculum MM. multi- magnilication stralegy, SG, saliency-guided guided method; HT, hierarchical transfer stralegy: PT, pre-trained stratogy; C-I1, Curriculum II; SB, soft-bag learning, CSA, constrained sclf-attcation module; TCL. two-tirr contrastive learning.
of
information. 3) The model without SG or FT suffers from performance degradation (average CI drops of 0.028 and 0.011 and AUC drops of 0.039 and 0.028). It indicates that fine-grained information across multi-scale tiles and hierarchical transfer strategy can help significantly improve the performance of model. More results can refer to Appendix D.
Ablation on Curriculum II. We also investigated the efficacy of each key component in Curriculum II. From Table 2, we have the following observations: 1) The baseline method performed prognosis prediction with the Cox model, which achieves average CI of 0.642 and AUC of 0.669. Compared to the baseline method, the model with soft-bag strategy achieves average CI gain of 0.024 and AUC gain of 0.031, which mainly profits by adaptively identifying and integrating the representative instances within a bag. 2) The models with CSA. TCL, and both of them show better performance with average CI gains of 0.006, 0.024, and 0.032 and AUC gains of 0.011. 0.02, and 0.04, respectively. which benefits from the reduction of intra-bag redundancy and the enhancement of intra-bag and inter-bag discrimination. More results can refer to Appendix D.
4.5 Parameter Analysis
Influence of the Soft-bag Size Nn. To explore the influence of the soft-bag size NB, we conducted of experiments by varying NB within the set of 2,, 4, 5]. Note that NB corresponds to the standard MIL method. From Table 3, we can observe that the model shows the best performance when NB equals (for COAD and LIHC) or (for BLAC). indicates that the top 3~4 instances are adequate to represent the bag. Besides, the model with NB > 1 works better than the standard MIL method, indicating the effectiveness of the soft-bag strategy for prognosis inference.
Influence of the Magnification Number S. We conducted experiments to investigate the influence of the magnification number S by varying S in the range of {1, 2,,2,33, 4).. Note that 5 = corresponds to the mono-magnification input and S = 4 corresponds to the multi-magnification input with (20x, 10x, 22, 2.5x). }. As shown in Table 3, the model's performance improved as S increases. And the model can achieve the best performance on all datasets and reach a stable state after S = 3.
Influence of the Weight Coefficients 3n and 3.. We turned to the influence of the weight coefficients 3n and Bg by varying them within the sets of (2e - 4, 5e - 5, le - 5, 2e - 6} and (2e - 2, 5e observe from Table 4, the performance of the model fluctuates slightly when both 3n and 83 vary, which verifies the robustness of the proposed method to the weight coefficients.
3, 1e 3, 2e - 4). respectively. As
4.6 Visualization
This section exhibits the salient maps output by the proposed method for some samples. As illustrated in Figure& 4, the model mainly focuses on the fine-grained regions (under 20x WSIs) that are wrapped by the salient regions under 10x WSIs, indicating the proposed method can effectively guide the

---

Table 3: Results (CI) achieved by the proposed model with different Ng and S.
Ng=1
Ng=2
Ng=3
Ng-4
Nga5
S=1
Sw2
5a3
S=4
COAD
0.660
0.679
0.717
0.671
0.705
0.663
0.700
0.708
0,703
LIHC
0.631
0.691
0.705
0.704
0.705
0.667
0.686
0.703
0.697
BLCA
0.648
0.657
0.672
0.686
0.667
0.628
0.645
0.658
0.650
Table 4: Results (CT) achieved by the proposed model with different weight coefficients 8n and 3...
Bn=2e-4
Pa=5e-5
Ba=le-5
0n=2e-6
9,=2o-2
0,=50-3
8,=1o-3
8,=2e-4
COAD
0.703
0.711
0.709
0.706
0.700
0.708
LITCC
0.692
0.696
0.695
0.691
0.689
0.892
0.695
0.686
0.669
0.672
0.667
RISK
Figure 4: Visualization of salient maps output by the proposed method. The first column shows the original WSIs, and the second and third columns exhibit the salient maps under 10x and 20 magnifications, respectively.
encoding of niigh-mannifiaation instances so to encourage the network to focus on more fine-grained information. We have provided more intuitive results and analyses in Appendix D.
Conclusion
In this paper, we present a dual-curriclum contrastive MIL method for cancer prognosis analysis with WSIs, which contains two easy-to-hard curriculums, i.e., saliency-guider guided instance encoding with cross-scale tiles and contrastvve-enaanee soft-bag progmosis inference. The proposed network is easy to train in each curriculum and can be applied to gigapixel WSI analysis with good visualization. Besides, the proposed method outperforms state-oftthe of-the--r methods in cancer prognosis analysis over three public datasets.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 61971213 and Grant 6167 1230, in part by the Basic and Applied Basic Research Foundation of Guangdong Province under Grant 201941515010417, and in part by the Guangdong Provincial Key Laboratory of Medical Image Processing under Grant No. 20.2020B 2222000B2220000 2060039.. We would like to express our gratitude to The Cancer Genome Atlas (TCGA) database for the open-source data and to anonymous reviewers for their insightful comments.
10

---

References
[1] Ole-Johan Skrede et al. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. The Lancer, (95(22))) 360, 2020.
[2] Richard Colling et al. Artificial intelligence digital pathology: roadmap to routine use in clinical practice. The Journal of Pathology, 249(2):143-150, 2019.
[3] Chetan Srinidhi, Ozan Ciga, and Anne I. Martel. Deep neural network models for computa- tional histopathology: survey. Medical Image Analysis, 67:10181 2021.
[4] Jana Lipkova al. Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies. Nature Medicine, 28(3):575-582, 2022.
[5] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
[6] Pierre Courtiol et al. Deep learning-bused classification of mesothelioma improves prediction of patient outcome. Nature Medicine, 25(10):1519-1525 2019.
[7] Hassan Muhammad al. Epic-survival: end-to-end part inferred clustering for survival analysis, with prognostic stratification boosting. In Medical Imaging with Deep Learning, 2021.
[8] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14318-14328, 2021.
[9] Gabriele Campanella et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, 25(8):1301-1309, 2019.
[10] Richard J Chen et al. Whole slide images are 2d point clouds: context-aware survival prediction using patch-based graph convolutional networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 339-349, 2021.
[11] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology whole-slide images. Nature Biomedical Engineering, 5(6):555-577 570, 2021.
[12) Zhuchen Shao et al. Transmil: transformer based correlated multiple instance learning for whole slide image classification. In Advances Neurul Information Processing Systems, volume 34, 2021.
[13] Chetan L Srinidhi and Anne L Martel. Improving self-supervised learning with hardness- aware dynamic curriculum learning: an application to digital pathology. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 562-571, 2021.
[14] Chetan L Srinidhi, Seung Wook Kim, Fu-Der Chen, and Anne L Martel. Self-supervised driven consistency training for annotation efficient histopathology image analysis. Medical Image Analysis, 75:102256, 2022.
[15] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: improving generalization of visual representations. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6509-6518. 2020.
[16] Noriaki Hashimoto et al. Multi-scale domain-adversse adversarial multiple-instame instance enn for cancer sub- type classification with unannotated histopathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3852-3861, 2020.
[17] Hongrun Zhang et al. Difd-mil: double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18802-18812, 2022.

---

[18] Trung Vu, Phung Lai, Raviv Raich, Anh Pham, Xiaoli Z Fern, and UK Arvind Rao. A novel attribute-base based symmetric multiple instance learning for histopathological image analysis. IEEE Transactions on Medical Imaging. 39(10))3125-3133 2020.
[19] Pierre Courtiol, Eric W Tramel, Marc Sanselme, and Gilles Wainrib. Classification and disease localization in histopathology using only global labels: a weakly-supervised approach. arXiv preprint arXiv: 1102.820222222 2018.
[20] Weixin Li and Nuno Vasconcelos. Multiple instance learning for soft bags via top instances. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4277-4285, 2015.
[21] Wei Shao, Tongxin Wang, Zhi Huang, Zhi Han, Jie Zhang, and Kun Huang. Weakly supervised deep ordinal cox model for survival prediction from whole-slide pathological images. IEEE Transactions on Medical Imaging, (11(1)))333--37477)) 2021.
[22] Nima Hatami, Mohsin Bilal, and Nasir Rajpoot. Deep histopathology image analysis. arXiv preprini
dictionary learning for 2021.
[23] Hiroki Tokunaga, Yuki Teramoto, Akihiko Yoshizawa, and Ryoma Bise. Adaptive weight- ing multi-fieldoof-viee enn for semantic segmentation in pathology. In Proceedings of the IEEL/CVF Conference Computer Vision and Pattern Recognition, pages 12597-12606, 2019,
[24] Naofumi Tomita, Behnaz Abdollahi, Jason Wei, Bing Ren, Arief Suriawinata, and Saeed Hassanpour. Attention- based deep neural networks for detection of cancerous and precancerous esophagus tissue on histopathological slides. JAMA Network Open, 2(1 (11 11119914445-999940 2019.
[25] Nikhil Naik al. Deep Geaminn-enablle breast cancer hormonal receptor status determination from base-level h&e stains. Nature Communications, 11(1):1-8 1,, 2020.
[26] Shujun Wang et al. Rmdl: recalibrated multi-instance deep learning for whole slide gastric image classification. Medical Image Analysis, 58:101549, 2019.
[27] Dmitrii Bychkov et al. Deep learning based tissue analysis predicts outcome in colorectal cancer. Scientific Reports, 8(1):1111, 2018
[28] Richard J Chen et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4025, 2021.
[29] Charlie Saillard et al. Predicting survival after hepatocellular carcinoma resection using deep learning on histological slides. Hepatology, 72(6):2000-20 2013,, 2020.
[30] Pooya Mobadersany et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences, 115(13):F29900 E2979, 2018.
[31] Richard J Chen et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging. 2020.
[32] Shuai Jiang, George J Zanazzi, and Saeed Hassanpour. Predicting prognosis and idh mutation status for patients with lower-grade gliomas using whole slide images. Scientific Reports, 11(1):1-9, 2021.
[33] David Tellez, Geert Litjens, Jeroen van der Laak, and Francesco Ciompi. Neural image compression for gigapixel histopathology image analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2))567-578, 2019.
[34] Gang Xu et al. Camel: weakly supervised learning framework for histopathology image segmentation. In Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 10682-10691, 2019.
12

---

[35] Fahdi Kanavati et al. Weakly-supervised learning for lung carcinoma classification using deep learning. Scientific Reports, 10(1):1-11, 2020.
[36] Chensu Xie et al. Beyond classification: whole slide tissue histopathology analysis by end-to- end part learning. In Medical Imaging with Deep Learning, pages 843-856, 2020.
[37] Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A Moskaluk, Sana Syed, and Donald Brown. Clustertooconquer:: a framework for end-to-end multi-instance learning for whole slide image classification. In Medical Imaging with Deep Learning, pages 682-698, 2021.
[38] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921-29222 2016.
[39] Ashish Vaswani et al. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[40] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXhv:1807.03748, 2018.
[41] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607, 2020.
[42] John Fox and Sanford Weisberg. Cox proportional-hazzate regression for survival data. An and S-PLUS Companion to Applied Regression, 2002, 2002.
[43] Daniel Sargent et al. Disease-free survival versus overall survival as primary end point for adjuvant colon cancer studies: individual patient data from 20,898 patients on 18 randomized trials. Journal of Clinical Oncology. 23(34):864-88670, 2005.
[44] Josep M Llovet, Andrew Burroughs, and Jordi Bruix. Hepatocellular carcinoma. The Lancet, 16279199) 2003.
[45] Ashish M Kamat et al. Bladder cancer. The Lancet, 388( 388(10061):276--288) 2016.
[46] Xinliang Zhu, Jiawen Yao, Feiyun Zhu, and Junzhou Huang. Wsisa: making survival prediction from whole slide histopathological images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2017.
[47] Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li, and Junzhou Huang. Graph cnn for survival analysis on whole slide pathological images. International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 174-182, 2018
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See the claimed contributions in Introduction and Abstract.
(b) Did you describe the limitations of your work? [Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This work has no negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read it and ensured that our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section 4.1, and the code is avaliable at GitHub.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.2 and Section 4.5.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We have used 5-fold cros-validatiin strategy to evaluate the proposed method and the results with mean and standard deviation are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.
(b) Did you mention the license of the assets? [Yes] See Section 4.1. The used datasets are publicly available.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] See Section 4. The used datasets are publicly available. The patient/participant consent has been obtained by the organizer.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See Section 4. The used datasets publicly available, but the personally identifiable information is shielded.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "enaIGDtSht", "reviewer": "Reviewer_LZCQ", "summary": "The paper proposed an effective framework for Cancer Prognosis Analysis, which outperforms existing methods on three datasets. First, this framework can leverage knowledge from multi-scale information from multi-magnification WSIs. This design is referred to by Saliency-guided Weakly-supervised Instance Encoding with Cross-scale Tiles in the paper. Second, it implements two-tier contrastive learning to enhance intra-bag and inter-bag discrimination.", "questions": "1. What does \"task-specific\" mean in the work? It appears in lines 30, 245, and 262 and is stated as a unique feature of the paper. However, it has never been introduced officially in the paper. Is it relative to using imagenet Pre-trained CNNs as feature extractors? \n2. Where is the architecture of the feature extractor F different from what commonly used ResNet variants? Why do you decide to introduce such architectural-wise changes? How will it perform if we adopt, e.g. ResNet 50, pre-trained on ImageNet, and fine-tune the network? \n3. How to understand the choice of $N_B \\ll N_n$? Why does it perform the best to use only 3 instances per bag?   ", "limitations": "I did not find a discussion about the limitation of the proposed methods.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method implemented several novel ideas that are designed to accommodate particular characteristics of WSI. \n2. The authors conducted comprehensive experiments and validated the effectiveness of the proposed framework. \n3. Ablation studies are also provided to show the impact of each introduced component. \n\nWeakness:\n\nThe paper provides a complicated system but the writing makes it very hard to understand. \n\nSpecific problems include:\n* Figure 1 is hard to render by many PDF readers and makes it slow to load. The figure itself is not as informative as the authors may expect due to the overloaded details. Personally, I found Figure 1 in the supplementary materials very helpful in addition to the mathematical explanations. \n* There are many denotations introduced first without any explanation or being explained very late in the paper. It gives the reader hard time understanding the system.  \n* The authors may consider using less mathematical denotations. It is not necessary to write down math formulas for many concepts/implementations, such as MLP, attention, et al, which are standard in a deep learning context. The current way in fact makes it too complicated to focus on the novel components proposed in the paper.\n* In line 137, it sounds to explain how to obtain w, however, w has never appeared in the part below. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XD2xxCFl8sz", "reviewer": "Reviewer_cBZX", "summary": "The authors propose a framework for cancer diagnosis of WSI by combining a saliency-guided weakly supervised multiscale instance encoding, and soft bag prognosis, using contrastive multi instance learning. The proposed method is tested on three datasets of different cancer tissue and outperforms five other relevant approaches in survival rate prognosis. An extensive ablation study is presented, targeting the contributions of the individual components of the framework.\n", "questions": "1.) How were training and test sets chosen? Can WSI from one patient be in the training as well as the test set?\n2.) Are the areas highlighted by the network as visualized in Fig. 3 also clinically relevant or would a pathologist identify other areas of cancerous tissue when making a diagnosis?\n3.) How stable is the training with respect to the chosen hyperparameters?\n", "limitations": "The authors do not address potential negative impacts of their study. Could this framework be applied to remote sensing/satellite data?", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The introduction is very well formulated to motivate the issue addressed by this paper.\n- The paper is well structured and written.\n- The evaluation is extensive with a thorough ablation study.\n- The proposed method targets a very relevant problem in biomedical imaging, in which labels are very expensive obtain\n- The authors succeed combine a big number of state of the art learning techniques to significantly improve the classification performance \n- The results are verified on three different types of cancer tissue.\n- The results can be visualized in an interpretable way.\n- The mathematical background and the notation are thorough.\n\nWeaknesses:\n\n- The method targets one particular task of application in particular and is not generally transferable to other tasks, though I believe a number of concepts are.\n- The framework consists of a large number of steps and components, each of which have to be tuned for other applications and it is not clear how sensitive the method is to various hyperparameters.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TgeDjxWIuEp", "reviewer": "Reviewer_h4Rb", "summary": "This paper presents a dual-curriculum contrastive Multiple Instance Learning (MIL) method for predicting cancer prognosis in whole-slide histopathology images (WSI’s). The proposed method consists of two key steps to reduce intra-WSI redundancy and increase inter-WSI discrimination: 1) a saliency-guided weakly-supervised multi-scale instance representation learning approach supervised by risk stratification; 2) an instance-level feature aggregation approach via a contrastive-enhanced soft-bag prognosis technique with survival time as supervision. The proposed method has been validated on three publicly available large-scale TCGA datasets: colon adenocarcinoma (COAD), hepatocellular carcinoma (LIHC), and bladder urothelial carcinoma (BLCA). Extensive experiments and comparisons with previous state-of-the-art methods demonstrate the effectiveness of the proposed idea. ", "questions": "There are very few minor concerns with this paper:\n\n— What is the ratio of training and validation sets used across 5-fold cross-validation in experiments.\n\n— How is the best model selected for the validation set to measure the predictive performance on the testing set. \n\n— Is there any specific reason why three year survival time threshold is adopted? Is there any specific reason behind the 3-year survival time with the three cancer types that have been used in this paper?\n\n— Please clearly state the difference between Figure 4 and Figure 5 (Captions) in the Supplementary Material.\n\n— In Table 2, Please abbreviate the short forms that are used in the paper for better readability\n\n— In Line 241, Section 4.3, there is a typo: Table 3 --> Table 1.", "limitations": "The authors have adequately addressed their work's limitations and potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "— Overall, in my opinion, the dual-step curriculum learning strategy based on two easy-to-hard curriculums is novel and generic, which is applicable to multi-cancer prognosis prediction tasks in computational pathology. \n\n— A robust set of baseline comparisons with existing state-of-the-art methods, including clustering, graph network, and multi-instance learning approaches, demonstrates the strength of the proposed idea. Further, an in-depth ablation experiment depicts the importance of encoding multi-magnification instance representations, along with constrained self-attention and the two-tier contrastive learning modules for prognosis prediction across three cancer types.\n\n—  Overall, the paper is well written and easy to follow the methodological contributions of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept