SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MIL Pre-training. For the backbone MIL model we use the the same parameter setup as CLAM. The model parameters are updated via the Adam optimizer with an L2 weight decay of 1e-5 and a learning rate of 2e-4. Each result is obtained with 10-fold splits of training/validation/testing sets.
Table 8: Ablation study for the percentage of instances used on CAMELYON16 and CAMELYON17.
In-Domain (CAM16)	Out-of-Domain (CAM17)
AUC	Accuracy	AUC	Accuracy
10% attention Retri	0.9440±0.037	0.8975±0.052	-	-
10% attention RetriIO	0.9365±0.052	0.9200±0.050	0.7974±0.054	0.7433±0.073
10% attention Retro	0.9414±0.046	0.8975±0.056	0.7775±0.050	0.7392±0.063
20% attention Retri	0.9451±0.036	0.8925±0.050	-	-
20% attention RetriIO	0.9341±0.051	0.8925±0.053	0.7651±0.056	0.7714±0.030
20% attention Retro	0.9419±0.048	0.9175±0.051	0.7681±0.058	0.7795±0.021
Neighbor Selection. After pre-training the MIL model, we obtain the slide-level feature and the attention scores predicted by the network. As computing the optimal transport distance based on all instances is time-consuming, we approximate the distance with a part of samples in a bag.
$$
d _ {\mathrm {O T}} (\mu , \nu) = \min  _ {\boldsymbol {T} \in \mathcal {T} (\boldsymbol {\alpha}, \tilde {\boldsymbol {\alpha}})} \sum_ {i = 1} ^ {| \boldsymbol {\alpha} |} \sum_ {j = 1} ^ {| \tilde {\boldsymbol {\alpha}} |} c \left(\boldsymbol {h} _ {i}, \tilde {\boldsymbol {h}} _ {j}\right) T _ {i j} + \beta \cdot \sum_ {i j} T _ {i j} \log T _ {i j} \tag {11}
$$
$$
s. t., \boldsymbol {T} ^ {\top} \mathbf {1} _ {K} = \boldsymbol {\alpha}, \boldsymbol {T} \mathbf {1} _ {\tilde {K}} = \tilde {\boldsymbol {\alpha}}, \boldsymbol {T} \geq 0
$$
where  $\alpha$  and  $\tilde{\alpha}$  are the new attention vector obtained by selecting top  $\eta \%$  from  $a$  and  $\tilde{a}$ . In other words, we approximate a bag with  $\eta \%$  of instances with the highest attention values generated by pretrained MIL model. As shown in Table 8, we set  $\eta = 10$  and  $\eta = 20$  for the ablation study. In this experiment, we use Regularization term of 0.5 and Max number of iterations 1000.
It is observed from Table 8, improving the percentage of data improves the performance on  $\mathrm{Retr}_{\mathrm{I}}$  and  $\mathrm{Retr}_{\mathrm{O}}$ . On  $\mathrm{Retr}_{\mathrm{IO}}$ , the performance is saturated when using only  $10\%$  of all patches. Differentiating in-domain and out-of-domain data for retrieval could be easily accomplished by representing a bag by a few amount of instances.
Table 9: Ablation study for different merge functions on CAMELYON16 and CAMELYON17.
In-Domain (CAM16)	Out-of-Domain (CAM17)
AUC	Accuracy	AUC	Accuracy
Mergeadd(2-feats) RetRI	0.9409±0.038	0.9000±0.049	-	-
Mergeadd(2-feats) RetRIO	0.9341±0.051	0.8925±0.053	0.7651±0.056	0.7714±0.030
Mergeadd(2-feats) RetRO	0.9414±0.046	0.8975±0.056	0.7775±0.050	0.7392±0.063
Mergeadd(3-feats) RetRI	0.9383±0.050	0.9175±0.051	-	-
Mergeadd(3-feats) RetRIO	0.9313±0.044	0.9000±0.045	0.7641±0.059	0.7553±0.043
Mergeadd(3-feats) RetRO	0.9391±0.051	0.9175±0.045	0.7644±0.059	0.7754±0.022
Mergeconvex(2-feats) RetRI	0.9451±0.036	0.8925±0.050	-	-
Mergeconvex(2-feats) RetRIO	0.9365±0.052	0.9200±0.050	0.7974±0.054	0.7433±0.073
Mergeconvex(2-feats) RetRO	0.9419±0.048	0.9175±0.051	0.7681±0.058	0.7795±0.021
Mergeconvex(3-feats) RetRI	0.9398±0.043	0.8975±0.052	-	-
Mergeconvex(3-feats) RetRIO	0.9435±0.038	0.8975±0.052	0.7652±0.052	0.7714±0.030
Mergeconvex(3-feats) RetRO	0.9417±0.048	0.9050±0.050	0.7690±0.056	0.7755±0.021
Merge Function. Table 9 presents the results using different merge functions. To generate the bag representations, we employ two merge functions: 1) simple addition, referred to as Merge_add; 2) convex combination, referred to as Merge_convex. Additionally, '2-feats' and '3-feats' refer to bag representation that are merged with 1 nearest neighbor or 2 nearest neighbors, respectively. For convex combination, '2-feats' uses coefficients of 0.6 and 0.4, while '3-feats' uses coefficients of 0.6, 0.2 and 0.2, where the greatest coefficient corresponds to the original representation. This experiment is done with  $\eta = 10$ .
It is derived from Table 9 that using 1 nearest neighbor and convex combination presents the best performance. Using 2 nearest neighbors and addition presents the similar results.
Classification Training. Finally, we train a single logistic regression classifier using the merged representation. The Adam optimizer is used to update the model parameters, with a L2 weight decay of 1e-4 and a learning rate of 2e-4. The models are trained for a minimum of 40 epochs and up to a maximum of 200 epochs if the early stopping criterion is not met. This criterion involves monitoring the validation loss each epoch and if it has not decreased from the previous low for over 15 consecutive epochs, early stopping is used.
D Patch-level results on tumor slides of CAMELYON16.
Table 10: Patch-level results on tumor slides of CAMELYON16.
P-Prec.(↑)	FROC(↑)
DSMIL	0.1030	0.4443
CLAM	0.6068	0.4792
TransMIL	0.1726	0.4797
Bayes-MIL	0.8107	0.4919
RAM-MIL	0.6114	0.5281
The Tumor-Precision is calculated by the precision of classifying the tumor patches. The Patch-Precision is calculated by averaging the precision of classifying both normal and tumor patches. The Patch-FROC is defined as the average sensitivity (recall) at 6 predefined false positive rates:  $1/4$ ,  $1/2$ ,  $1$ ,  $2$ ,  $4$  and  $8$  FPs per WSI.
In Table 17, RAM-MIL presents the second best precision and the best FROC on the patch-level segmentation. This indicates that using transport matrix for interpreting the patch-level classification achieves the best overall performance in the trade-off of false positive rate and recall. By contrast, Bayes-MIL could only obtain a high precision, which reduces the number of false alarm. However, for the application of medical WSI, reducing false negative (better recall and FROC) is supposed to be more important as classifying a positive instance to be negative is unacceptable in the application of medical prognosis or diagnosis.
E Results on CAMELYON17.
A general setting is to treat ITC as the negative class in the CAMELYON17 dataset.
Table 11: Results on CAMELYON16 and CAMELYON17 for in-domain classification and unsupervised domain adaptation, for comparisons with related methods.
In-Domain (CAM16)	Out-of-Domain (CAM17)
AUC	Accuracy	AUC	Accuracy
CLAM	0.9177±0.044	0.8650±0.060	0.8399±0.059	0.7796±0.099
Mixup RetrI	0.9260±0.051	0.8825±0.051	-	-
Mixup RetrIO	0.9271±0.048	0.8850±0.046	0.8418±0.054	0.8016±0.059
Mixup Retro	0.9271±0.045	0.8825±0.048	0.8397±0.057	0.8095±0.063
l2 RetrI	0.9281±0.047	0.8800±0.055	-	-
l2 RetrIO	0.9241±0.048	0.8950±0.048	0.8403±0.049	0.7914±0.052
l2 Retro	0.9398±0.045	0.8950±0.055	0.8412±0.055	0.7998±0.056
RAM-MIL RetrI	0.9451±0.036	0.8925±0.050	-	-
RAM-MIL RetrIO	0.9365±0.052	0.9200±0.050	0.8475±0.055	0.8236±0.051
RAM-MIL Retro	0.9419±0.048	0.9175±0.051	0.8413±0.053	0.8457±0.027
Table 12: Ablation study evaluated on CAMELYON16 and CAMELYON17.
In-Domain (CAM16)	Out-of-Domain (CAM17)
AUC	Accuracy	AUC	Accuracy
Approx-OT-min Retri1	0.9301±0.033	0.8800±0.043	-	-
Approx-OT-min RetriIO	0.8601±0.080	0.8000±0.062	0.8380±0.056	0.7935±0.065
Approx-OT-min Retro	0.9196±0.063	0.8750±0.056	0.8397±0.056	0.7915±0.048
Approx-OT-avg Retri1	0.9227±0.038	0.8600±0.036	-	-
Approx-OT-avg RetriIO	0.8865±0.063	0.8425±0.061	0.8401±0.056	0.7902±0.051
Approx-OT-avg Retro	0.9195±0.063	0.8750±0.056	0.8412±0.053	0.7823±0.063
Approx-OT-max Retri1	0.9174±0.049	0.8625±0.044	-	-
Approx-OT-max RetriIO	0.9182±0.047	0.8650±0.048	0.8406±0.056	0.7854±0.052
Approx-OT-max Retro	0.9195±0.063	0.8750±0.056	0.8412±0.063	0.7925±0.058
l2 Retri1	0.9281±0.047	0.8800±0.055	-	-
l2 RetriIO	0.9241±0.048	0.8950±0.048	0.8403±0.049	0.7914±0.052
l2 Retro	0.9398±0.045	0.8950±0.055	0.8412±0.055	0.7998±0.056
Hausdorff Retri1	0.9273±0.057	0.8850±0.049	-	-
Hausdorff RetriIO	0.9226±0.055	0.8725±0.060	0.8403±0.056	0.7932±0.053
Hausdorff Retro	0.9322±0.046	0.8950±0.048	0.8395±0.063	0.8000±0.061
RAM-MIL Retri1	0.9451±0.036	0.8925±0.050	-	-
RAM-MIL RetriIO	0.9365±0.052	0.9200±0.050	0.8475±0.055	0.8236±0.051
RAM-MIL RetriO	0.9419±0.048	0.9175±0.051	0.8413±0.053	0.8457±0.027
Table 13: Ablation study for the percentage of instances used on CAMELYON16 and CAMELYON17.
In-Domain (CAM16)	Out-of-Domain (CAM17)
AUC	Accuracy	AUC	Accuracy
10% attention Retri	0.9440±0.037	0.8975±0.052	-	-
10% attention RetriIO	0.9365±0.052	0.9200±0.050	0.8475±0.055	0.8236±0.051
10% attention Retro	0.9414±0.046	0.8975±0.056	0.8466±0.047	0.8037±0.062
20% attention Retri	0.9451±0.036	0.8925±0.050	-	-
20% attention RetriIO	0.9341±0.051	0.8925±0.053	0.8473±0.052	0.8216±0.030
20% attention Retro	0.9419±0.048	0.9175±0.051	0.8413±0.053	0.8457±0.027
Table 14: Ablation study for different merge functions on CAMELYON16 and CAMELYON17.
In-Domain (CAM16)	Out-of-Domain (CAM17)
AUC	Accuracy	AUC	Accuracy
Mergeadd(2-feats) RetRI	0.9409±0.038	0.9000±0.049	-	-
Mergeadd(2-feats) RetRIIO	0.9341±0.051	0.8925±0.053	0.8473±0.052	0.8216±0.030
Mergeadd(2-feats) RetRO	0.9414±0.046	0.8975±0.056	0.8390±0.056	0.7695±0.056
Mergeadd(3-feats) RetRI	0.9383±0.050	0.9175±0.051	-	-
Mergeadd(3-feats) RetRIIO	0.9313±0.044	0.9000±0.045	0.8432±0.055	0.8176±0.040
Mergeadd(3-feats) RetRO	0.9391±0.051	0.9175±0.045	0.8402±0.056	0.8457±0.020
Mergeconvex(2-feats) RetRI	0.9451±0.036	0.8925±0.050	-	-
Mergeconvex(2-feats) RetRIIO	0.9365±0.052	0.9200±0.050	0.8475±0.055	0.8236±0.051
Mergeconvex(2-feats) RetRO	0.9419±0.048	0.9175±0.051	0.8413±0.053	0.8457±0.027
Mergeconvex(3-feats) RetRI	0.9398±0.043	0.8975±0.052	-	-
Mergeconvex(3-feats) RetRIIO	0.9435±0.038	0.8975±0.052	0.8473±0.049	0.7755±0.067
Mergeconvex(3-feats) RetRO	0.9417±0.048	0.9050±0.050	0.8474±0.047	0.8097±0.057
F Comparisons with domain adaptation methods
There have been a large volume of research studying the domain adaptation in patch-level models, categorized as staining transfer and domain adversarial learning.
Patch-level staining transfer: Cho et al. [31], Shaban et al. [32] and Zanjani et al. [33] use generative adversarial networks (GAN) to learn the staining difference implicitly. Images are transferred to the target domain, by the trained generators in GAN. A simple neural network is then fit on the generated images for discriminative tasks.
Patch-level domain adversarial learning: Lafarge et al. [34] and Ciga et al. [35] apply gradient reversal method for adapting the model for discriminative tasks in medical images. Ren et al. [36] uses Siamese network in domain adversarial learning. Brieu et al. [37], Kapil et al. [38] and Gadermayr et al. [39] for transfer images between domains for segmentation tasks. These methods are not directly comparable with our methods, as they require patch-level labels during training, which is not realistic in real-world scenario. In experiment, we adapt Shaban et al. [32] with least modification, to compare with the retrieval based multiple instance learning. We run Shaban et al. [32] in patch-level in the Camelyon17 domain adaptation setting. For obtaining the WSI features, we use averaged pooling. For fairness, the same classifier as RAM-MIL is used for Shaban et al. [32]. Here are the results:
Table 15: Comparisons with patch-level domain adaptation methods
AUC(↑)	Accuracy(↑)
Shaban et al. [32]	0.6458	0.6889
RAM-MIL	0.8209	0.7667
Slide-level methods: Yang et al. [48] propose a slide-level domain adaptation method that involves local and global adversarial loss to train a weakly supervised learning model. RAM-MIL outperforms this method in the domain adaptation evaluation on Camelyon17.
Table 16: Comparisons with slide-level domain adaptation methods
AUC(↑)	Accuracy(↑)
Yang et al. [48]	0.5867	0.6786
RAM-MIL	0.7974	0.7795
Other methods: Previous methods in slide-level classification [24] and methods in other fields [25] also consider the out-of-domain testing performance. However, they did not provide a method for domain adaptation, while RAM-MIL has. We compare RAM-MIL with several most advanced baselines in slide-level classification in our experiments.
G Comparisons with other retrieval methods
For the information retrieval works (Yottixel [40] and SISH [41]) of whole slide image, we will add them into the related works section. However, these two methods are not directly comparable, as they focus on the retrieval problem while our goal is to use retrieval to enhance the weakly supervised classification (MIL). An executable way is to replace the OT-retrieval module in RAM-MIL with these retrieval methods. As there is no available implementation of Yottixel, and SISH is a more advanced retrieval method, we integrate SISH retrieval into the RAM-MIL process. The experiment runs on Camelyon17 (including 5 hospitals), with 2 hospitals as the in-domain dataset and the rest being the out-of-domain dataset. Here are the experimental results:
Table 17: Comparisons with other retrieval methods
AUC(↑)	Accuracy(↑)
CLAM-SISH	0.7647	0.7444
RAM-MIL	0.8209	0.7667
H More results on Dimensionality Reduction.
Directly reducing the normal dimension to the estimated intrinsic dimension is a brute-force method to improve the performance. To validate if this argument holds, we use a masked auto-encoder (MAE) with output dimension to be 5 (the estimated intrinsic dimension) to extract the features from patches. To ensure a good quality of extracted features, we fine-tune the MAE with all patches from Camelyon16 for 60 epochs. The performance of CLAM fitted on the extracted features is at the second row of the Table 18:
Table 18: Additional results evaluated on Camelyon16 (in-domain) and Camelyon17 (out-of-domain)
CAMELYON16	CAMELYON17
AUC	Accuracy	AUC	Accuracy
CLAM-SVD-5D	0.9216	0.6625	0.8183	0.4729
CLAM-MAE-5D	0.8285	0.7368	0.7982	0.6987
RAM-MIL-SVD-5D	0.9273	0.8475	0.8199	0.7355