Title: Are nuclear masks all you need for improved out-of-domain generalisation? A closer look at cancer classification in histopathology
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep learning

domain generalization

histopathology

computational pathology

digital pathology

computer vision

single domain generalization
Abstract: Domain generalisation in computational histopathology is challenging because the images are substantially affected by differences among hospitals due to factors like fixation and staining of tissue and imaging equipment. We hypothesise that focusing on nuclei can improve the out-of-domain (OOD) generalisation in cancer detection. We propose a simple approach to improve OOD generalisation for cancer detection by focusing on nuclear morphology and organisation, as these are domain-invariant features critical in cancer detection. Our approach integrates original images with nuclear segmentation masks during training, encouraging the model to prioritise nuclei and their spatial arrangement. Going beyond mere data augmentation, we introduce a regularisation technique that aligns the representations of masks and original images. We show, using multiple datasets, that our method improves OOD generalisation and also leads to increased robustness to image corruptions and adversarial attacks. The source code is available at https://github.com/undercutspiky/SFL/

Are nuclear masks all you need for improved out-of-domain generalisation? A closer look at cancer classification in histopathology

Abstract

Domain generalisation in computational histopathology is challenging because the images are substantially affected by differences among hospitals due to factors like fixation and staining of tissue and imaging equipment. We hypothesise that focusing on nuclei can improve the out-of-domain (OOD) generalisation in cancer detection. We propose a simple approach to improve OOD generalisation for cancer detection by focusing on nuclear morphology and organisation, as these are domain-invariant features critical in cancer detection. Our approach integrates original images with nuclear segmentation masks during training, encouraging the model to prioritise nuclei and their spatial arrangement. Going beyond mere data augmentation, we introduce a regularisation technique that aligns the representations of masks and original images. We show, using multiple datasets, that our method improves OOD generalisation and also leads to increased robustness to image corruptions and adversarial attacks. The source code is available at https://github.com/ undercutspiky/SFL/

1 Introduction

Domain generalisation in histopathology is a crucial challenge because domain shifts naturally occur among hospitals and even within a single hospital or laboratory, e.g., temporally or among human operators and observers such as pathologists. Non-biological factors that substantially alter the images include differences in scanners, staining protocols, fixation of tissue, and even minor aspects like the manufacturer and storage conditions of stains [1].

Collecting data from numerous hospitals to address these domain shifts is often impractical and may not adequately reflect the full variability present in routine clinical practice, thus making it difficult to build computational histopathology models that generalise well. This leads us to focus on single-domain generalisation (S-DG) in this paper, specifically on how to train a model using data from only one hospital (considered a domain here) that generalises well to data from other hospitals. Popular S-GD methods in histopathology apply data augmentation and stain normalisation [2, 3]. The effectiveness of S-GD methods developed for natural images remains underexplored in histopathology [3]. Here, we compare these methods to a new, simple approach that we propose.

Research has shown that Convolutional Neural Networks (CNNs) tend to focus on texture over shape [4, 5]. However, in histopathology, the texture and colour of cell nuclei vary much more across domains than the shape and organisation of cell nuclei. As a result, focusing on shape features could improve a computational histopathology model's ability to generalise to unseen data because it may rely less on domain-specific features that vary across hospitals and more.

Nuclei in cancerous tissue exhibit distinct changes in shape, size, and overall organisation compared to nuclei in normal tissue [6-8]. Pathologists rely on these and other visual cues [9] for cancer diagnosis and grading, underscoring the biological importance and the consistency of nuclear morphology and organisation across domains. We hypothesise that focusing on nuclear morphology and organisation may be sufficient for cancer detection and that exploiting this during training could result in models with good generalisation.

We propose a method that encourages CNNs to focus more on nuclear morphology and organisation by using additional loss terms that prioritise shape-based features. Specifically, our method leverages nuclear segmentation masks during training to steer the learning towards nuclei. Through extensive experimentation, we demonstrate that this method improves performance on out-of-domain data without requiring nuclear segmentation masks at inference time, thus offering a promising and attractive solution for addressing domain generalisation in histopathology. Our contributions include:

We propose a novel training method that incentivises the model to focus on nuclei.

We evaluate our method on three datasets comprising hundreds of WSIs in total from various hospitals and organs. Our results show accuracy improvements over all other approaches.

We evaluate the sensitivity of our method to image corruptions and adversarial attacks. Our results show performance improvements over the baseline.

We conduct extensive ablation studies to show that models trained with our method focus on nuclei.

2 Related work

The prediction of various properties such as malignancy, grading, and HER2 expression using segmented nuclei has been a well-studied topic for many years [10-16]. Researchers have employed techniques such as watershed segmentation [17], thresholding, level sets [18], and snakes [19], often followed by extracting explicit morphometric features from the segmentations. For example, early work by Hasegawa et al. [20] focused on counting segmented regions, while Lee and Street [21] applied neural networks to the segmentation outputs. In contrast to these approaches, our method does not rely on segmentation during inference. Instead, we adjust the training process to encourage the extraction of nuclear features.

Stain normalisation methods convert the colours of a source image to match those of a target image. These methods were typically designed specifically for the most common type of histopathology images, which are images of tissue stained with haematoxylin and eosin (H&E). One of the earlier methods, Macenko normalisation [1], estimates stain vectors for source and target images and uses them to normalise the source image. Vahadane et al. [22] proposed a method that decouples stain "density maps" from "colour appearances", allowing the combination of the source image's density maps with the target image's colour appearances. Reinhard et al. [23] pioneered colour transfer by adjusting the global statistics of images in a different colour space, effectively transferring the colour characteristics of the target to the source image. Random Stain Normalization and Augmentation (RandStainNA) [24] combines stain normalisation and augmentation. Unlike traditional approaches that normalise using a fixed template, RandStainNA generates random virtual templates in the LAB [23] colour space and uses them to normalise the images during training. The templates are drawn from Gaussian distributions whose means and variances are derived from the training data. For a more comprehensive review, we refer the readers to [25]. In summary, the stain normalisation methods primarily focus on manipulating colour information to remove stain variability. On the other hand, our approach shifts the focus from colour manipulation to nuclear features.

Data augmentation is a common way to facilitate domain generalisation. Tellez et al. [2] evaluated several stain colour augmentation and stain normalisation methods and found that colour augmentation was crucial for good performance on external test sets in histopathology. Faryna et al. [26] extended RandAugment [27] by including certain histopathology-specific augmentations and excluding the ones that produce unrealistic-looking images. Tellez et al. [28] developed a data augmentation method

specific to H&E-stained images and used it for domain generalisation in mitosis detection. Pohjonen et al. [29] developed StrongAugment, where varying numbers of transformations are applied to an image to improve domain generalisation. Marini et al. [30] proposed Data-driven colour augmentation (DDCA), which evaluates an augmented image as acceptable or not for training based on its distance from other images in a database. Faryna et al. [31] evaluated different data augmentation strategies in histopathology, including manually selected augmentations, and found them all to be competitive.

Single-domain generalisation (S-DG) methods do not require data from multiple domains during training. Representation Self-Challenging (RSC) [32] works by discarding the features with relatively high gradients, making the model predict with the remaining features during training. Adversarial Domain Augmentation (ADA) [33] generates adversarial examples iteratively to augment the source domain and creates an ensemble of models. Meta-Learning-based ADA (M-ADA) [34] uses Wasserstein Auto-Encoder [35] to generate new samples and uses adversarial training on top along with a meta-learning scheme. Progressive domain expansion network (PDEN) [36] uses multiple autoencoders to generate new samples to expand the training set. Learning to Diversify (L2D) [37] introduces a learnable style-complement module that generates augmented images. The style-complement module is trained to diversify the images as much as possible but still keep the semantic information intact.

Domain adaptation , unlike S-DG, requires having access to some samples from the target domain. In histopathology, domain adaptation methods commonly make use of GAN [38] and CycleGAN [39]. StainGAN [40] uses CycleGAN to make images in the source domain look like the target domain. Residual CycleGAN [41] modifies the CycleGAN objective to have the generator produce the residual between domains instead of recreating the input image. In [42], authors augment a generator in CycleGAN with a stain colour matrix as an auxiliary input to stabilise the training. NST_AD_HRNet [43] uses Neural Style Transfer [44, 45] and GAN to preserve the content of the source image while combining it with the style of the target image. In some earlier works [46, 47], the input image is converted to greyscale and then coloured using a generator network which is based on a target image. While domain adaptation is not S-DG and thus a bit tangential to the focus of this paper, it is worth noting that domain adaptation is impractical in many clinical settings and may result in worse generalisation than stain normalisation and colour augmentation [48].

3 Proposed Method

Our approach aims to enhance S-DG by incentivising the model to focus on shaped-based features of nuclei in histopathological images and thereby reduce overfitting to irrelevant features that may carry higher label noise.

The first step involves generating segmentation masks that highlight specific areas of interest in the image. This step is applied only during training, while test-time evaluation relies solely on H&Estained images. As we hypothesised that nuclear morphology and organisation contain sufficient information for cancer detection, our segmentation masks are binary images with nuclear pixels as foreground and other pixels as background.

One possible approach to using the segmentation mask is to include it as a fourth channel in the input image. Alternatively, the mask can be used as the sole input to the model. However, both methods necessitate running the segmentation model during inference, which increases computational demands and slows down processing.

Our method circumvents the need for a nuclear segmentation network at inference time by incorporating additional loss terms during training. For a given input image x and its corresponding segmentation mask x ′ , our method involves the following steps:

Execute a forward propagation through the neural network model on both the H&E-stained image x and its nuclear mask x ′ , saving the embeddings generated by the network as z and z ′ for x and x ′ , respectively.

Compute the Binary Cross-Entropy (BCE) loss for both x and x ′ .

Compute the ℓ 2 -distance between the embeddings z and z ′ .

Minimise the sum of the two CE losses and the ℓ 2 -distance.

ResNet-50

(without GAP and

classification layers)

BCE Loss

BCE Loss

Euclidean

distance

GAP and

classification

+

Probability:

50%

Probability:

50%

Our approach is illustrated in Figure 1. We employ a ResNet-50 [49] from Torchvision [50] as the base model. We next discuss some details of our approach.

ℓ 2 -regularisation: To encourage the network to focus on nuclei, we minimise the distance between the feature map of the original image and that of its nuclear segmentation mask. We use the flattened feature map from ResNet-50's penultimate layer, just before Global Average Pooling, to obtain the embeddings. The regularisation term consists of the ℓ 2 -distance between the embeddings of the original image z and its mask z ′ , which is added to the BCE losses for both the image and the mask. Let ˆ y be the model's prediction for x , ˆ y ′ for x ′ , and y be the ground truth. Then, the total loss L is:

where BCE is the Binary Cross-Entropy loss function for labels in { 0 , 1 } :

Original image times mask: Since the embeddings of the original image and its binary segmentation mask may differ significantly, minimizing their ℓ 2 -distance can be challenging for the model. To address this, with a probability of 0.5, we multiply the original image by its segmentation mask, i.e., the network receives x ∗ x ′ as input half the time instead of x . By multiplying with the segmentation mask, everything in the original image except the nuclei is set to 0. Figure 1 shows what the output looks like. By simplifying the task, the network can more easily reduce the distance between the embeddings of the nuclei-only image and the mask and gradually improve alignment between the embeddings of the original image and the mask. We found this augmentation to help stabilise training.

4 Experiments

4.1 Datasets

CAMELYON17 [51] dataset consists of 1000 H&E-stained Whole Slide Images (WSIs) of breast cancer metastases in lymph node sections from five medical centres in the Netherlands. It contains pixel-level annotations of tumours for 10 WSIs from each medical centre, giving us 50 WSIs to work with. WSIs from centres 0, 3 and 4 were scanned using the same scanner, while the other two centres used a different scanner each. All slides were scanned at 40 × resolution. We treat each centre as a different domain.

BCSS [52] dataset consists of 151 H&E-stained WSIs of histologically-confirmed primary breast cancer cases from The Cancer Genome Atlas (TCGA) with triple-negative status determined from

clinical data files. All WSIs have a resolution of 40 × . The WSIs were annotated at the pixel level using crowdsourcing. Each pixel can have one of the many labels. We consider the label "tumor" to define pixels with a tumour and all other labels except "outside_roi", "exclude", or "undetermined" to define pixels without a tumour.

Ocelot [53] dataset consists of pixel-level annotations of tumour vs non-tumour pixels for 303 WSIs from TCGA. It consists of WSIs of primary tumour from six different organs: Bladder, Endometrium, Head-and-neck, Kidney, Prostate, and Stomach. The annotations in the dataset are at a low resolution, so we upscale the annotations to 40 × . We exclude two WSIs that have only 20 × resolution; all other WSIs have 40 × resolution.

4.2 Dataset preparation

We use code from WILDS [54, 55] to prepare the CAMELYON17 dataset with modifications. Tiles are sized (270 × 270) at 40 × resolution. For each domain (medical centre), data is split by patient, ensuring all tiles from a patient are in a single subset. Since the number of tiles varies drastically across patients, we shuffle patients so that the validation subset contains 20%-25% of all tiles per domain. Our processed version of CAMELYON17 is available at [56].

4.3 Experiment setup

We train models using the CAMELYON17 dataset, treating each medical centre as a distinct domain. We use the BCSS and Ocelot datasets as external test datasets. To avoid multiple comparisons and overly optimistic performance estimates, we use the external test datasets only once during the entire project, solely to evaluate the final models [57].

For each combination of medical centre and method, we train ten models using the train subset of that centre. Thus, we train 50 models in total for each method. We use the loss on the validation dataset (of the training domain) to select the best model for each training. All models are trained for 50 epochs using the Adam optimiser with a learning rate 4e -5 and a weight decay of 1e -4 . We use exponential learning rate decay with a decay rate of 0 . 955 . For our method, we set the parameter λ in equation (1) to λ = 0 for the first five epochs, effectively training without ℓ 2 -distance loss in these epochs, and then use λ = 1 for the rest of the training. We start saving models for selection of the one with lowest validation loss after ten epochs for our method to allow the network to stabilise while from the first epoch for other methods. For all experiments, unless stated otherwise, we use a ResNet-50 [49] model pre-trained on ImageNet [58]. We use HoVer-Net [59] trained on the CoNSeP [59] dataset to generate nuclear segmentation masks.

While domain generalisation encompasses a wide variety of methods, we have selected several exemplary baselines for comparison: Macenko normalisation [1], RSC [32], L2D [37], RandStainNA (RandSNA in result tables) [24], and DDCA [30]. We also include a baseline where we initialise ResNet-50 with pre-trained weights from HoVer-Net [59]. These methods represent different approaches, including stain normalisation (Macenko, RandStainNA) and generating augmented images (L2D). By selecting these diverse techniques, we ensure a comprehensive evaluation of our method's performance across various S-DG strategies.

It is important to note that our method can be integrated with many existing S-DG approaches, making it a flexible plug-in solution rather than a direct competitor. We evaluate most methods with and without the photometric augmentations selected for ERM. After testing various augmentation strategies available in Torchvision [50], we identified the most effective combination to be: ColorJitter(brightness=[0.5, 1.5], contrast=[0.5, 1.5], saturation=[0.5, 1.5], hue=[-0.3, 0.3]) and GaussianBlur(kernel_size=3). Results using these augmentations are marked as '-Aug' in the results tables. In all experiments, including those without photometric augmentations, we apply the basic geometric augmentations: random horizontal and vertical flips. For all ViT-Tiny [60] experiments, we also add affine augmentations: random rotation (up to 90 ◦ ) and translation (up to 45 pixels).

We ran the experiments on two clusters with GPUs with 64 GB (AMD MI250X) and 24 GB (Nvidia RTX 3090) GPU RAM each. Each job consumed about 21 to 31 GB of GPU RAM. The proposed method took 5 to 20 hours to train, depending on the train data size while ERM took 2.5 to 11 hours.

We report tile-level accuracy for tumour vs non-tumour tile classification for all datasets. Additionally, we measure robustness to image noise by measuring the accuracy drop on CAMELYON17 for image

corruptions introduced in [61]. This includes Gaussian-, shot-, impulse- and snow-noise, and two blur types, elastic transform and JPEG compression.

Results on CAMELYON17 (lymph node sections) We test models on their respective out-ofdomain data. E.g., a model trained on Centre-3 is tested on all the data from Centre-0,1,2,4. Our method attains 10% higher accuracy than the next best method (L2D) when none used photometric augmentations and was also superior when photometric augmentations were used (Table 1).

Results on BCSS (primary breast cancer) The accuracy of the models trained on a centre in CAMELYON17 drops substantially (12% to 14%) for all the methods when tested on BCSS (Table 2) compared to when tested on other centres in CAMELYON17 (Table 1). This could be due to a mismatch between pathologists' annotations on CAMELYON17 and BCSS but also due to the biological differences between these tissue types. In particular, epithelial cells in lymph nodes would almost certainly be tumour cells, while they could be benign cells in ordinary breast tissue. These results show that the relative performance on CAMELYON17 for different methods is indicative of relative performance on an external test set, as the performance drop is similar for all methods.

Results on Ocelot (primary non-breast cancer) We test our model on the Ocelot dataset to evaluate if our method helps to train models that generalise to other organs as well. Ocelot does not have any data from breast tissue nor does it include lymph node sections (which all the models have been trained on). We report the results of these experiments in Table 3. While our method achieves the highest accuracy also in this case, the difference between our method and L2D is not as big as it is for CAMELYON17 and BCSS. Taking a closer look into the performance for separate organs (Tables 4, 5, 6, 7, 8, and 9 in the Supplement), we can see that our method performs worse than L2D in Endometrium and Kidney, where accuracies are generally lower, and better in the four other organs. This indicates that models trained with our method generalise worse to organs where the transferability from breast tissue is generally low. This is at least the case for Kidney which has by far the lowest accuracies across all methods. Generalising to different cancer types is an emerging experimental topic; see, for example, [62].

In summary, our method yields better accuracy than the baselines, including other S-DG approaches.

5 Ablation Study and Discussion

Impact of data augmentation Tables 1, 2, and 3 shows that data augmentation benefits all methods substantially, which is consistent with well-established knowledge.

Impact of ℓ 2 -regularisation The result labelled Ours-noℓ 2 -A in Table 1 shows that using data augmentation with nuclear masks alone is insufficient to achieve high accuracy. Without ℓ 2 -regularisation (i.e., setting λ = 0 in Equation (1)), our method only slightly outperforms most baselines that also uses data augmentation. The key factor for effective cross-domain generalisation is the ability to align the feature representation of input images with corresponding mask images, which lack colour and texture. Further evidence supporting this alignment effect is presented in the next paragraph.

Impact of mask-times-input-augmentation The result Ours-MO-Aug in Table 1 demonstrates an ablation with two changes: the absence of ℓ 2 -regulation and the removal of the 50% probability

Original

Mask

Original * Mask

Black Nuclei

White Nuclei

Inverted Mask

Inpainted Nuclei

(a)

(b)

(c)

(d)

(e)

(f)

(g)

mask-times-input augmentation of H&E images during training (Figure 1), but still having two CE loss terms, one of them over nuclear masks. We see a further decline of accuracies below most baselines with photometric augmentations.

Impact of learned features without nuclear-mask-like features Here, we bring further evidence for the effect of ℓ 2 -regularisation about pulling the features towards the representation of nuclear masks. Note that nuclear masks are not used during inference in standard evaluations such as all those in Tables 1, 2, and 3. In Table 10 in the Supplement, we can see results for predictions in which the embeddings (features before the GAP layer) of the H&E images are modified by subtracting the embeddings of the corresponding nuclear masks. By comparing to Table 1, we see a drop in performance for all methods. However, the drop is largest for our method, with an accuracy below random guessing. This shows that the features computed from H&E-stained images at test time are indeed more similar to features from nuclear masks for our method than for other methods.

Impact of removal of intranuclear texture and colour In Tables 11 and 14 in the Supplement, we consider the performance on modified H&E images, in which intranuclear texture is removed by masking it out with a constant colour (see examples in Figure 2d and 2e). This is of interest due to the observation that intranuclear texture is often different in cancerous nuclei, which can be informative to humans. We can see that if it is replaced by a colour similar to the colour of nuclei, we for our method obtain a performance (Table 11) very similar to the performance with original H&E images (Table 1). On the other hand, changing the colour to white seems to reduce the performance notably (Table 14). This is possibly due to the creation of images with outlier statistics. A more likely explanation is that it is common for H&E stains to have small holes or gaps of white background colour in the stroma, which usually are not discriminative information but rather shear stress artefacts from the tissue cutting process. Therefore, masking nuclei with white masks may effectively remove discriminative information about nuclei. This domain-specific observation may explain the asymmetry in behaviour when masking nuclei with black versus white.

Impact of removal of extranuclear information Table 23 in the Supplement shows results on data where all the non-nuclear background is set to white (see example in Figure 2c). These images can be viewed as an inside-out inverted case of the images evaluated to give the results in Table 11. The common information in both sets of images is the morphology and organisation of nuclei. The performance for our proposed method remains high on these images (Table 23), being close to the best result on original H&E data (Table 1). The experiments in Tables 23 and 11 demonstrate the strong generalisability of focusing on nuclear morphology and organisation in out-of-domain settings.

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

shot_noise

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

gaussian_noise

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

elastic_transform

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

impulse_noise

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

defocus_blur

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

jpeg_compression

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

motion_blur

Method

ERM-Aug

Ours-Aug

0

1

2

Corruption level

0

10

20

30

40

50

60

70

80

90

100

snow

Method

ERM-Aug

Ours-Aug

Impact of dilution of nuclear shapes We expand nuclei masks by a classic morphological dilation and then blacken the dilated regions in the H&E images. The nuclear shape information in these images is thus progressively reduced compared to the images where only the nuclei are filled with black. Across all methods, we observe a drop in accuracy with an increase in dilation (Tables 11, 12, and 13 in the Supplement), highlighting the critical role of shape in this domain. The proposed method is more robust to moderate shape dilution with a mask size of 5 than the baseline methods. A similar but stronger trend appears in Tables 14, 15, and 16 in the Supplement for whitened nuclei.

Impact of removing nuclei We dilate the nuclear mask image with a kernel size of 5 to encapsulate remnants of the boundary of nuclei and then use the dilated mask to remove nuclei by inpainting [63]. The accuracy with the resulting images (see example in Figure 2g) drops to random guessing for our method (Tables 17 and 18 in the Supplement). Essentially no tiles are classified as tumour, giving a nearly zero recall and low precision (Tables 19, 20, 21, and 22 in the Supplement). Also, this supports that models trained using our method focus on nuclear morphology and organisation, and shows that the models reasonably associate the absence of nuclei with no tumour.

Saliency maps via Integrated Gradients To further demonstrate that our method steers models to focus on nuclei, we generate saliency (pixel attribution) maps using Integrated Gradients [64] and show some randomly selected examples in Figures 6,7 in the Supplement. The saliency maps also indicate that a model trained using our method focuses on nuclei.

Evaluation of L2D and RSC combined with the proposed method Tables 26, 27, and 28 in the Supplement show the results of combining L2D and RSC with the proposed method. Combining the proposed method, which regularises, with L2D, which diversifies, yields mixed results, likely due to the opposing effects of these two interventions. Combining it with RSC results in a small gain over using our method alone. Overall, this demonstrates the effectiveness of the method proposed.

Evaluation on segmentation mask data For the sake of completeness, we show in Tables 24 and 25 in the Supplement that our method also performs well when tested on nuclear masks (as exemplified in Figure 2b) and their inversions (see example in Figure 2f).

Evaluation of robustness to image corruptions Figure 3 shows that the proposed method has notably higher robustness to image corruptions for most experiments in eight types of corruptions described in [61]. Samples of corrupted images are shown in Figure 5 in the Supplement.

Evaluation of robustness against adversarial attacks We evaluated the robustness of models against adversarial attacks [65] using the Projected Gradient Descent (PGD) attack [66]. Figure 4a demonstrates that models trained using our method have significantly higher robustness than ERM and L2D, the latter being the second-best performing method in Tables 1, 2, and 3. Additionally, we

0.0/255

0.5/255

1.0/255

1.5/255

2.0/255

eps

0

10

20

30

40

50

60

70

80

90

100

steps=6, alpha=3/255

Method

ERM

L2D

Ours

(a)

0.0/255

1.0/255

2.0/255

3.0/255

4.0/255

5.0/255

6.0/255

7.0/255

8.0/255

eps

0

10

20

30

40

50

60

70

80

90

100

steps=6, alpha=3/255

Tested on <-- Generated via

ERM<--L2D

ERM<--Ours

L2D<--ERM

L2D<--Ours

Ours<--ERM

Ours<--L2D

(b)

conduct cross-model attacks by generating adversarial images using models trained with one method and evaluating them on models trained with other methods. The results indicate that our models exhibit minimal performance degradation when exposed to adversarial images generated by models from other methods (Figure 4b). In contrast, the accuracy of models trained with ERM and L2D drops substantially. This further demonstrates the superior robustness of models from our method.

A preliminary evaluation on a transformer architecture We perform a comparison using a finetuned ViT-Tiny [60] model. The results are shown in Tables 29, 30 and 31 in the Supplement. These results show that our approach obtains superior out-of-domain performance for the CAMELYON17 dataset. The results are more mixed for the other datasets. In particular, it seems that models trained on one of the five centres (Center-4) in CAMELYON17 do not generalise well to other cancer types and are actually also performing sub-optimally in CAMELYON17. For models trained on each of the other four centres in CAMELYON17, the performance with our approach is, on average, better than with other approaches, but the performance increase is lower than for ResNet-50. However, in the same tissue type (CAMELYON17 data), the performance gain is similar for both ViT-Tiny and ResNet-50. Our interpretation of all these results is that our approach can improve out-of-domain performance also for ViT-Tiny, in particular across centres and scanners for the same tissue type, but that it might also fail for a minority of the training datasets. This experiment is preliminary because we took the same hyperparameters as used for ResNet-50, including the same learning rate and λ = 1 , both of which might not be optimal. Also, we note that ViT-Tiny has much fewer parameters than ResNet-50. Experiments with larger transformers might obtain bigger differences, as seen in [67].

Limitations of this study As a limitation, we identify that we have performed these experiments for only one classification task. For medical practitioners, it would be of interest to measure the impact for other tasks, such as tumour grading and survival prediction, when evaluated in an out-of-domain generalisation setup. However, this would require access to multi-centre datasets with relevant labels available. Secondly, we ran the full set of experiments only on one base network, ResNet-50, because we preferred to run a larger set of ablation experiments to understand what actually has been learned when using our method. While we expect results to be qualitatively similar for other CNNs, transformer networks might have different learning dynamics, and results for those with a larger capacity than the ViT-Tiny are of interest in future work. Finally, an extension to other cancer types, such as prostate or colon cancer, would also be of interest.

6 Conclusion

We have shown a simple method to enforce the learning of shape features at training time, which uses unmodified input images at inference time. It shows very good out-of-domain performance and can be combined as a plugin with other methods to enhance out-of-domain generalisation. Aside from out-of-domain accuracy, the proposed method gives improved robustness to image alterations.

Acknowledgments and Disclosure of Funding

The computations were performed on resources provided by Sigma2-the National Infrastructure for High-Performance Computing and Data Storage in Norway-through Project NN8104K. Additionally, we acknowledge Sigma2 for access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through Sigma2, Norway, Project 465000262. This work was supported by the authors' institutions, the Research Council of Norway (project number 309439), the National Research Foundation, Singapore, and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings, and conclusions or recommendations expressed in this work are those of the authors and do not reflect the views of their institutions, the Research Council of Norway, the National Research Foundation, Singapore, and Infocomm Media Development Authority.

References

Marc Macenko, Marc Niethammer, James S Marron, David Borland, John T Woosley, Xiaojun Guan, Charles Schmitt, and Nancy E Thomas. A method for normalizing histology slides for quantitative analysis. In 2009 IEEE international symposium on biomedical imaging: from nano to macro , pages 1107-1110. IEEE, 2009.

David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen Van Der Laak. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical image analysis , 58:101544, 2019.

Mostafa Jahanifar, Manahil Raza, Kesi Xu, Trinh Vuong, Rob Jewsbury, Adam Shephard, Neda Zamanitajeddin, Jin Tae Kwak, Shan E Ahmed Raza, Fayyaz Minhas, et al. Domain generalization in computational pathology: survey and guidelines. arXiv preprint arXiv:2310.19656 , 2023.

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations , 2018.

Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip J Kellman. Deep convolutional networks do not classify based on global object shape. PLoS computational biology , 14(12): e1006613, 2018.

Christopher W Elston and Ian O Ellis. Pathological prognostic factors in breast cancer. i. the value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology , 19(5):403-410, 1991.

Daniele Zink, Andrew H Fischer, and Jeffrey A Nickerson. Nuclear structure in cancer cells. Nature reviews cancer , 4(9):677-687, 2004.

Edgar G Fischer. Nuclear morphology and the biology of cancer cells. Acta cytologica , 64(6): 511-519, 2020.

William H Wolberg, W Nick Street, and Olvi L Mangasarian. Importance of nuclear morphology in breast cancer prognosis. Clinical Cancer Research , 5(11):3542-3548, 1999.

Shivang Naik, Scott Doyle, Shannon Agner, Anant Madabhushi, Michael Feldman, and John Tomaszewski. Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology. In 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro , pages 284-287, 2008. doi: 10.1109/ISBI.2008.4540988.

Jean-Romain Dalle, Wee Kheng Leow, Daniel Racoceanu, Adina Eunice Tutac, and Thomas C Putti. Automatic breast cancer grading of histopathological images. In 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society , pages 3052-3055. IEEE, 2008.

Laura E. Boucheron, B. S. Manjunath, and Neal R. Harvey. Use of imperfectly segmented nuclei in the classification of histopathology images of breast cancer. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 666-669, 2010. doi: 10.1109/ ICASSP.2010.5495124.

Mitko Veta, Robert Kornegoor, André Huisman, Anoek H J Verschuur-Maes, Max A Viergever, Josien P W Pluim, and Paul J van Diest. Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer. Modern Pathology , 25(12): 1559-1565, 2012. ISSN 0893-3952. doi: https://doi.org/10.1038/modpathol.2012.126. URL https://www.sciencedirect.com/science/article/pii/S089339522202943X .

Paweł Filipczuk, Marek Kowal, and Andrzej Obuchowicz. Automatic breast cancer diagnosis based on k-means clustering and adaptive thresholding hybrid segmentation. In Image processing and communications challenges 3 , pages 295-302. Springer, 2011.

Hela Masmoudi, Stephen M. Hewitt, Nicholas Petrick, Kyle J. Myers, and Marios A. Gavrielides. Automated quantitative assessment of her-2/neu immunohistochemical expression in breast cancer. IEEE Transactions on Medical Imaging , 28(6):916-925, 2009. doi: 10.1109/TMI.2009. 2012901.

L Sing Cheong, Angela Jean, Tsu Soo Tan, Waiming Kong, and Soo Yong Tan. Automated segmentation and measurement for cancer classification of her2/neu status in breast carcinomas. In Biotechno 2011: The Third International Conference on Bioinformatics . Citeseer, 2011.

Serge Beucher and Christian Lantuéjoul. Use of watersheds in contour detection. workshop published, September 1979. URL http://cmm.ensmp.fr/~beucher/publi/watershed. pdf .

Stanley J. Osher and James A. Sethian. Fronts propagating with curvature-dependent speed: algorithms based on hamilton-jacobi formulations. Journal of Computational Physics , 79: 12-49, 1988. URL https://api.semanticscholar.org/CorpusID:205007680 .

Michael Kass, Andrew Witkin, and Demetri Terzopoulos. Snakes: Active contour models. International Journal of Computer Vision , 1(4):321-331, Jan 1988. ISSN 1573-1405. doi: 10.1007/BF00133570. URL https://doi.org/10.1007/BF00133570 .

Akira Hasegawa, Kevin J. Cullen M.D., and Seong Ki Mun. Segmentation and analysis of breast cancer pathological images by an adaptive-sized hybrid neural network. In Murray H. Loew and Kenneth M. Hanson, editors, Medical Imaging 1996: Image Processing , volume 2710, pages 752 - 762. International Society for Optics and Photonics, SPIE, 1996. doi: 10.1117/12.237980. URL https://doi.org/10.1117/12.237980 .

Kyoung-Mi Lee and W.N. Street. An adaptive resource-allocating network for automated detection, segmentation, and classification of breast cancer nuclei topic area: image processing and recognition. IEEE Transactions on Neural Networks , 14(3):680-687, 2003. doi: 10.1109/ TNN.2003.810615.

Abhishek Vahadane, Tingying Peng, Amit Sethi, Shadi Albarqouni, Lichao Wang, Maximilian Baust, Katja Steiger, Anna Melissa Schlitter, Irene Esposito, and Nassir Navab. Structurepreserving color normalization and sparse stain separation for histological images. IEEE transactions on medical imaging , 35(8):1962-1971, 2016.

Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer graphics and applications , 21(5):34-41, 2001.

Yiqing Shen, Yulin Luo, Dinggang Shen, and Jing Ke. Randstainna: Learning stain-agnostic features from histology slides by bridging stain augmentation and normalization. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 212-221. Springer, 2022.

Md Ziaul Hoque, Anja Keskinarkaus, Pia Nyberg, and Tapio Seppänen. Stain normalization methods for histopathology image analysis: A comprehensive review and experimental comparison. Information Fusion , page 101997, 2023.

Khrystyna Faryna, Jeroen Van der Laak, and Geert Litjens. Tailoring automated data augmentation to h&e-stained histopathology. In Medical imaging with deep learning , 2021.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

David Tellez, Maschenka Balkenhol, Nico Karssemeijer, Geert Litjens, Jeroen van der Laak, and Francesco Ciompi. H and e stain augmentation improves generalization of convolutional networks for histopathological mitosis detection. In Medical Imaging 2018: Digital Pathology , volume 10581, pages 264-270. SPIE, 2018.

Joona Pohjonen, Carolin Stürenberg, Atte Föhr, Reija Randen-Brady, Lassi Luomala, Jouni Lohi, Esa Pitkänen, Antti Rannikko, and Tuomas Mirtti. Augment like there's no tomorrow: Consistently performing neural networks for medical imaging. arXiv preprint arXiv:2206.15274 , 2022.

Niccolò Marini, Sebastian Otalora, Marek Wodzinski, Selene Tomassini, Aldo Franco Dragoni, Stephane Marchand-Maillet, Juan Pedro Dominguez Morales, Lourdes Duran-Lopez, Simona Vatrano, Henning Müller, et al. Data-driven color augmentation for h&e stained images in computational pathology. Journal of Pathology Informatics , 14:100183, 2023.

Khrystyna Faryna, Jeroen van der Laak, and Geert Litjens. Automatic data augmentation to improve generalization of deep learning in h&e stained histopathology. Computers in Biology and Medicine , 170:108018, 2024.

Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves crossdomain generalization. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part II 16 , pages 124-140. Springer, 2020.

Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems , 31, 2018.

Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12556-12565, 2020.

Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein autoencoders. In International Conference on Learning Representations , 2018.

Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 224-233, 2021.

Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 834-843, 2021.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 2223-2232, 2017.

M Tarek Shaban, Christoph Baur, Nassir Navab, and Shadi Albarqouni. Staingan: Stain style transfer for digital histological images. In 2019 Ieee 16th international symposium on biomedical imaging (Isbi 2019) , pages 953-956. IEEE, 2019.

Thomas de Bel, John-Melle Bokhorst, Jeroen van der Laak, and Geert Litjens. Residual cyclegan for robust domain transformation of histopathological tissue slides. Medical Image Analysis , 70:102004, 2021.

Niyun Zhou, De Cai, Xiao Han, and Jianhua Yao. Enhanced cycle-consistent generative adversarial network for color normalization of h&e stained images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part I 22 , pages 694-702. Springer, 2019.

Harshal Nishar, Nikhil Chavanke, and Nitin Singhal. Histopathological stain transfer using style transfer network with adversarial loss. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part V 23 , pages 330-340. Springer, 2020.

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2414-2423, 2016.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 694-711. Springer, 2016.

Edwin Yuan and Junkyo Suh. Neural stain normalization and unsupervised classification of cell nuclei in histopathological breast cancer images. arXiv preprint arXiv:1811.03815 , 2018.

Hyungjoo Cho, Sungbin Lim, Gunho Choi, and Hyunseok Min. Neural stain-style transfer learning using gan for histopathological images. arXiv preprint arXiv:1710.08543 , 2017.

Karin Stacke, Gabriel Eilertsen, Jonas Unger, and Claes Lundström. Measuring domain shift for deep learning in histopathology. IEEE journal of biomedical and health informatics , 25(2): 325-336, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10.1109/CVPR.2016.90.

TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://github.com/pytorch/vision , 2016.

Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob Van de Loo, Rob Vogels, et al. 1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset. GigaScience , 7(6):giy065, 2018.

Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai AT Elsebaie, Lamia S Abo Elnasr, Rokia A Sakr, Hazem SE Salem, Ahmed F Ismail, Anas M Saad, et al. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics , 35(18): 3461-3467, 2019.

Jeongun Ryu, Aaron Valero Puche, JaeWoong Shin, Seonwook Park, Biagio Brattoli, Jinhee Lee, Wonkyung Jung, Soo Ick Cho, Kyunghyun Paeng, Chan-Young Ock, et al. Ocelot: Overlapped cell on tissue dataset for histopathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23902-23912, 2023.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML) , 2021.

Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised adaptation. In International Conference on Learning Representations (ICLR) , 2022.

Dhananjay Tomar. Replication Data for: Are nuclear masks all you need for improved out-ofdomain generalization? A closer look at cancer classification in histopathology, 2024. URL https://doi.org/10.18710/NXPLFL .

Andreas Kleppe, Ole-Johan Skrede, Sepp De Raedt, Knut Liestøl, David J Kerr, and Håvard E Danielsen. Designing deep learning studies in cancer diagnostics. Nature Reviews Cancer , 21 (3):199-211, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. HoVer-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image analysis , 58:101563, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations (ICLR) , 2019.

Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, Eric Robert, Yi Kan Wang, Jeremy D. Kunz, Matthew C. H. Lee, Jan Bernhard, Ran A. Godrich, Gerard Oakley, Ewan Millar, Matthew Hanna, Juan Retamero, William A. Moye, Razik Yousfi, Christopher Kanan, David Klimstra, Brandon Rothrock, and Thomas J. Fuchs. Virchow: A million-slide digital pathology foundation model, 2024.

Alexandru Telea. An image inpainting technique based on the fast marching method. Journal of graphics tools , 9(1):23-34, 2004.

Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International conference on machine learning , pages 9269-9278. PMLR, 2020.

C Szegedy. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 1204-1213. IEEE, 2022. doi: 10.1109/CVPR52688. 2022.01179. URL https://doi.org/10.1109/CVPR52688.2022.01179 .

Technical Appendix / Supplement

Defocus Blur

JPEG Compression

Elastic Transform

Snow

Impulse Noise

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claim made in the abstract and introduction are supported by the results in the main Tables 1, 2, and 3, and for robustness in Figures 3 and 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalise to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see the end of the Section 5 Ablation Study and Discussion for limitations Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The method is described in Section Proposed method 3. The experiments are described in Section Experiments 4, and, there in particular in Subsection 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets are created and available via their owners. Code is available, shareable and presentable to external parties. It is available at https://github.com/ undercutspiky/SFL/

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiments are described in Section Experiments 4, and, in particular, in Subsection 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We reported variances of the performance measures over the runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We disclosed the GPU types, the GPU RAM and training times, see Section 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked against the Neurips ethics code. No privacy, security, safety impact expected. "Disclosure of essential elements for reproducibility" has been done in the paper, and we consider it to be sufficient for reproduciblity

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: No societal impact of the work expected

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No direct malicious use is expected from a method to improve out-of-domain generalisation.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets, the model and the deep learning toolbox are all cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We use established datasets [51-53] containing patient data, which were collected by third parties mentioned in the cited publications and which are already published. Applied crowdsourcing is described in those publications.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We use established datasets [51-53] containing patient data, which were collected by third parties mentioned in the cited publications and which are already published. As a consequence, IRB approval was not required for our study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the issue of out-of-domain (OOD) generalization in histopathology, which is complicated by domain shifts due to different scanners, staining procedures, and inter-patient variability. With a focus on single-domain generalization, this paper proposes to prioritize shape features over texture, as shape remains consistent across domains. This is achieved by using images and segmentation masks with data augmentation and regularization, emphasizing nuclear morphology critical for cancer detection. Experiments confirm this approach improves OOD generalization, validating an old hypothesis with deep learning and demonstrating the method's flexibility as an enhancement to other techniques.\n\nSTRENGTHS:\n- Vulnerability to cross-domain variances and shifts is essentially a critical challenge in the context of AI-driven digital pathology. Diving deeper into this topic could greatly enhance the reliability and usability of current learning-based approaches.\n\n- The experimental evaluations and ablation studies are quite comprehensive.\n\nWEAKNESSES:\n- The paper is overall poorly structured with convoluted paragraph organizations and intricate sentences. It is hard for readers to get the core idea of this paper.\n\n- The description of proposed methodology is quite informal, with ambiguous elucidation and unrigorous mathematical formulations. For example, what do you mean by stating “run a forward pass”?\n\n- The idea to leverage the shape of nuclei as domain-invariant markers is not new [Ref-A]. The morphology and spatial distribution of nucleus clusters are also common biomarkers for cancer diagnosis and grading [Ref-B]. It is unclear what the contribution of this work is. \n\n- Why not directly employing a segmentation to extract the morphometrics for all nuclei and then input the secured measurements to a simple classifier (e.g., SVM) to get the diagnosis prediction? If the color and texture are argued to be not robust against domain shifts, why not simply discard the original pathology slides?\n\n[Ref-A]:\nSharma, Yash, Sana Syed, and Donald E. Brown. \"Mani: Maximizing mutual information for nuclei cross-domain unsupervised segmentation.\" in MICCAI, 2022.\n\n[Ref-B]:\nBansal, Cherry, et al. \"Grading systems in the cytological diagnosis of breast cancer: a review.\" Journal of cancer research and therapeutics 10.4 (2014): 839-845.\n\nQUESTIONS:\n- Could you carefully revise the manuscript to make its content self-organized and easy to follow?\n\n- Compared to the suggested alternative solutions, what is the advantage of adopting the proposed approach? In-depth investigation and analysis are required in this respect.\n\nLIMITATIONS:\nThe authors have clearly identified the limitations of their work and suggested sound solutions. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method focusing on nuclei masks, along with an augmentation method, to train a feature extractor to increase\nthe out-of-domain generalization by directing the model toward learning nuclear features.\nThe method has been evaluated against 4 different approaches on three different cancer datasets comprised of different centers. It has been empirically shown that the nuclei-based approach is superior to other methods, and also the augmentation method can improve the performance of different approaches when used as a plugin.\n\nSTRENGTHS:\nThe methodology proposed is simple and straightforward and appears to be superior to other methods using a resnet50 backbone. There are different ablation studies supporting the authors' claims for the components of their design. Also, generally, the idea is biologically interesting to lead the model to focus on nuclei.\n\nWEAKNESSES:\nmajor:\nThe methods used to benchmark against are not the recent methods (with the newest method (L2D) being from 2021). This needs to be addressed by comparing it against some new pathology-specific methods like (1), (2), or (3).\n\nThe authors only used resnet50 for most of their experiments and used vit_tiny for limited experiments against ERM. In order to show the generality of the method, vit_tiny needs to be added to the full set of experiments, along with other methods for benchmarking.\n\n(1) Marini, Niccolò, et al. \"Data-driven color augmentation for H&E stained images in computational pathology.\" Journal of Pathology Informatics 14 (2023): 100183.\n\n(2) Shen, Y., Luo, Y., Shen, D., Ke, J. (2022). RandStainNA: Learning Stain-Agnostic Features from Histology Slides by Bridging Stain Augmentation and Normalization. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13432. Springer, Cham. https://doi.org/10.1007/978-3-031-16434-7_21\n\n(3) Bouteldja, Nassim, et al. \"Stain-independent deep learning–based analysis of digital kidney histopathology.\" The American Journal of Pathology 193.1 (2023): 73-83.\n\nminor:\nthe contributions of the work in the introduction can be written more clearly: it is a bit vague to understand the key points through the first read :)\n\nQUESTIONS:\nThe work has been built upon the idea that the network should focus on nuclei to generalize better. Yet, I would suggest the authors add a qualitative experiment visualizing the saliency map of resnt50 (or attention map of vit) to support indeed that the focus on nuclei is the key to the improvement. This can be compared with some sort of human annotation and also against other methods.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Are nuclear masks all you need for improved out-of-domain generalization? A closer look at cancer classification in histopathology.\" approaches the task of center classification for out-of-domain histopathology imagery. Motivated by classical shape-based segmentation, the authors suggest utilizing segmentation maps as an input to force the classification network to focus more on the morphology of cells. This enables the classification network to achieve a more robust classification accuracy on out-of-domain histopathology images.\n\nSTRENGTHS:\nThe presented approach is certainly interesting, using segmentation maps to shift the attribution/focus of the classification network to cells. While I'm not too familiar with the recent SOTA approach in this domain, this approach seems to be novel. Using the segmentation mask leads to an increased classification accuracy on OOD data.\n\nWEAKNESSES:\nThe paper lacks clarity; I often find myself rereading sentences and paragraphs to follow the ideas of the authors. For instance, the abstract does not clearly state which specific tasks the authors are approaching. \n\nMany statements in the introduction motivating the presented approach are not supported by citations. An example include the last sentence of the second paragraph.\n\nAdditionally, the related work section mainly provides an enumeration of existing approaches and does not clearly explain how the presented approach fits/extends related work.\n\nMy main concern is the use of more extensive annotations, namely segmentation maps. The presented approach uses the predictions of HoVer-Net that rely on dense segmentation labels. Subsequentially, the authors require substantially more supervision than the approaches they compare against (e.g., L2D). Providing a discussion on this is required. Additionally, a more competitive baseline also using mask supervision would be required. For instance, training a network on segmentation and then fine-tuning it on the downstream task.\n\nFurther, the presented method does not consistently outperform L2D when data augmentation is utilized while using extensively more supervision.\n\nAs the main aim of using segmentation maps is to shift the attribution of the classification network, the authors should also discuss the connection to attribution methods or consider analyzing the attribution of their approach and other approaches. \n\nFinally, many tables presented both in the supplement and the main paper do not adhere to the style guidelines and use too much horizontal space.\n\nMinor comments:\n- Line 17: Labeling noise is not really changing the data distribution.\n- Line 29: What does general vision refer to? I think the domain of natural images is way more concise here.\n- Line 51: “directing the attention of CNNs” sounds somewhat misleading. I guess the attribution is relevant in this case.\n- Line 130: PyTorch does not provide a ResNet-50 implementation. The authors probably refer to TorchVision.\n- Line: 31 is -> are.\n\nQUESTIONS:\nDoes the slight increase in classification performance justify the use of significantly more extensive supervision?\nHow does the attribution change when utilizing the presented approach?\nHow complimentary is the presented approach to other existing approaches? Can I combine L2D and your approach?\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations of their approach. As mentioned in my main concern, the presented method requires significantly more supervision than other approaches. This is not discussed in the limitations sections.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-20"}