Multi-modal Denoising Diffusion Pretraining for Whole-Slide Image Classification

Authors: wei  lou
guanbin  li
xiang  wan
haofeng  li
  multi
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 09:03:21

Abstract: Whole-slide image (WSI) classification methods play a crucial role in tumor diagnosis. Most of them use hematoxylin and eosin (H&E) stained images, while Immunohistochemistry (IHC) staining provides molecular markers and protein expression information that highlights cancer regions. However, obtaining IHC-stained images requires higher costs in practice. In this work, we propose a multi-modal denoising diffusion pre-training framework that harnesses the advantages of IHC staining to learn visual representations. The framework is trained with the H&E-to-IHC re-staining task and IHC-stained image reconstruction task, which helps capture the structural similarity and staining difference between two image modalities. The trained model can then provide IHC-guided features, by taking only H&E-stained images as inputs. Besides, we build a new class-constraint constrastive loss to achieve the semantic consistency between dual-modal features from our pre-training framework. To integrate with WSI classifiers based on multi-instance learning, we further propose a bag feature augmentation strategy to extend bags with the features extracted by our pre-trained model. Experimental results on three datasets show that our pre-training framework effectively improves WSI classification and surpasses the state-of-the-art pre-training approaches. Code and model are released via https://github.com/lhaof/MDDP

Main paper: 
 related work: 
 Most whole-slide image (WSI) classification methods [13, 28, 33, 37, 49, 63] are based on multiple-instance learning, which can be further divided into two groups: 1. using instance-level predictions directly, 2. using aggregation of instance-level features for baglevel classification. In the former group, Top-K MIL [8] employs the top-K instances for bag prediction. For the latter group, AB-MIL [23] calculates attention scores for all instances and computes the weighted average of instance features to obtain bag-level representations. HAG-MIL [58] utilizes multi-magnification images of WSIs, extracting features from higher magnification levels and distilling them to lower magnification levels to obtain bag-level representations. However, all these methods primarily focus on visual representations from H&E-stained images, which may be improved when adding more specialized information from other staining techniques. Image-to-image translation is the process of establishing a mapping between two image domains [22, 36, 46, 60, 61, 67] . Isola et al. [24] proposed Pix2Pix, a conditional generative adversarial network (GAN) for image-to-image translation using paired samples. Zhu et al. [66] proposed CycleGAN, which uses two GANs and allows the models to learn from unpaired images. However, due to their adversarial mechanism, GAN-based methods often encounter training difficulties and mode collapse issues. Recently, denoising diffusion models have been applied to image generation and translation due to their exceptional generative capabilities [9, 20, 38, 64] . BBDM [29] utilizes the Brownian bridge process to generate noisy images that resemble the target domain, rather than generating pure noise. In the area of histopathology, re-staining is an image-toimage task that involves translating the staining on histopathology images while preserving tissue structures and cell shapes. Current re-staining methods [11, 30, 35, 62] mainly employ GAN-based approaches trained on paired or unpaired images with different staining. However, these methods primarily focus on generating samples and do not thoroughly study the features learnt from the re-staining task for downstream image understanding tasks.Vision representation learning involves pre-training a neural network to generate features or initialize model weights for downstream tasks. One type of visual pre-training method learns features based on the image itself using pretext tasks such as predicting transformations [15] , inpainting [31, 41] or recovering masked regions [17, 21, 57] . Another type of visual pre-training method is based on contrastive learning [5, 40, 52, 65] . These methods utilize different views of the same sample or different samples as contrastive components for feature discrimination. Recently, several studies [6, 39, 54, 56, 59 ] have investigated the features learnt from diffusion denoising models (DDMs). However, current methods mainly rely on uni-modal image reconstruction tasks for denoising diffusion model pre-training, and it remains unclear whether crossmodal generative tasks can further enhance the representation ability of DDMs. 
methodology 3.1 multi-modal image registration and label: 
 Assignment.As shown in Figure 2 (a), in our data acquisition process, an unstained tissue section is initially stained with H&E, and then stained with IHC after removing the H&E stains. This sequential staining approach ensures the maximum preservation of the positional and structural correspondence between the two images with different stains. However, the staining process may still undergo unavoidable transformations such as translation or rotation in digitization, resulting in misalignment. To address this, we first employ a rigid registration method [42] to align the H&E and IHC stained WSIs from the same tissue slide. Subsequently, we use a technique called Yottixel [25] to select the valid image patches from the registered H&E-stained WSIs based on the tissue segmentation results and RGB histograms. In Yottixel, the square regions of size 1000 × 1000 are cropped from the segmented tissue areas and resized to 256×256 image patches. The corresponding IHC-stained regions are cropped and resized at the same positions as their paired H&E-stained image patches. After the above patch selection procedure, a large number of paired H&E-IHC stained samples (𝑥 ℎ𝑒 , 𝑥 𝑖ℎ𝑐 ) are generated.In IHC staining, specific protein expression is visualized as brown regions, while normal tissues tend to appear more white. For example, as the row of 𝑥 𝑖ℎ𝑐 in Figure 2 (a) shows, the left patch is a 'negative' sample and the right one with larger brown regions is a 'positive' sample. Therefore, the presence of specific protein expression can be evaluated by analyzing the proportion of brown areas in an IHC sample [62] . In our method, we define samples with brown regions larger than 1% as positive and those with less than 0.1% as negative, discarding the remaining samples. Each H&E-IHC pair is assigned a binary pseudo label 𝜌 (positive/negative) indicating the level of protein expression.The proposed multi-modal denoising diffusion pre-training framework learns to extract IHC-staining guided representation with only H&E-stained images as inputs. The framework is trained with two tasks. The first one is a multi-modal image-to-image translation task, where the denoising diffusion model learns to map H&E-stained images to their IHC-stained counterparts. The second one is a uni-modal image reconstruction task, requiring the denoising diffusion model to recover IHC-stained images from their noisy version. As Figure 2 (b) shows, given a pair of H&E-stained and IHC-stained patch 𝑥 ℎ𝑒 and 𝑥 𝑖ℎ𝑐 , we first employ a pre-trained encoder 𝜏 to extract their latent features 𝑧 ℎ𝑒 0 and 𝑧 𝑖ℎ𝑐 0 . Then, we add noise to these latent features separately, resulting in the noisy features 𝑧 ℎ𝑒 𝑡 and 𝑧 𝑖ℎ𝑐 𝑡 . Subsequently, a U-Net takes 𝑧 ℎ𝑒 𝑡 and 𝑧 𝑖ℎ𝑐 𝑡 as inputs to perform the re-staining and reconstruction tasks, respectively. Depending on the input features, the denoising U-Net model not only converts the latent features of H&E-stained patches into those of IHC-stained patches but also recovers the original features of IHC-stained patches from the noisy features. Let 𝑓 𝑡 denotes the generated features of the image-to-image translation task, we could use a pre-trained decoder D to output synthetic IHC-stained images by using 𝑓 𝑡 as input.3.2.1 Latent Feature Extraction. We describe how to obtain the above-mentioned pre-trained encoder and decoder in the following. Latent diffusion models (LDMs) [2, 43, 47] move the diffusion process into latent space and perform denoising diffusion process on latent features, which effectively reduces the time and memory cost. In particular, LDMs utilize an autoencoder architecture comprising an encoder and a decoder. The encoder 𝜏 maps an input image 𝑥 to a latent feature map 𝑧 = 𝜏 (𝑥), while the decoder D converts the latent feature map back into an image. Prior to training the denoising U-net, we first train an autoencoder named VQGAN [12] on the H&E-IHC stained dataset. Then, during the training of our proposed framework, the weights of encoder 𝜏 and decoder D are frozen. The encoder takes paired H&E and IHC stained patches (𝑥 ℎ𝑒 and 𝑥 𝑖ℎ𝑐 ) with the shape of ℎ × 𝑤 × 3 as inputs and outputs two latent features 𝑍 ℎ𝑒 0 and 𝑍 𝑖ℎ𝑐 0 of shape ℎ ′ × 𝑤 ′ × 𝑐 ′ . The two latent features will be used in the diffusion process. Process. The diffusion process in our framework introduces noise to input features in each time step 𝑡, resulting in noisy features after a long time. Our framework follows an image-toimage translation network called BBDM [29] to perform diffusion processes. In the diffusion process, noise is added to two latent features for different tasks. The forward diffusion process of the two tasks can be defined as:EQUATIONEQUATIONwhere 𝑞 ℎ𝑒 ->𝑖ℎ𝑐 represents the forward transition probability for the H&E feature to IHC feature translation task, and 𝑞 𝑖ℎ𝑐 ->𝑖ℎ𝑐 denotes the forward transition probability for the IHC feature reconstruction task. N is the Gaussian distribution. 𝑡 is a time step between 0 and 𝑇 . 𝑚 𝑡 = 𝑡/𝑇 denotes the proportion of the added noise. 𝛿 𝑡 is designed as 2(𝑚 𝑡 -𝑚 2 𝑡 ) following the best setting of BBDM [29] . In the 𝑡 𝑡ℎ time step, the diffused features for two tasks are computed as:EQUATIONEQUATIONwhere 𝑧 ℎ𝑒 𝑡 and 𝑧 𝑖ℎ𝑐 𝑡 denote the features of H&E-stained and IHCstained images after adding noises at time step 𝑡. 𝜖 𝑡 ∼ N (0, 𝑰 ) is Gaussian noise from a standard normal distribution.In the denoising process, given 𝑧 ℎ𝑒 𝑡 , 𝑧 𝑖ℎ𝑐 𝑡 and 𝑡, a U-net model 𝜇 𝜃 is deployed to predict the noise that is added on these two input noisy features respectively. It is important to notice that 𝜇 𝜃 learns to predict noise for different tasks (imageto-image translation/image reconstruction) purely relying on the input features. The denoised features can be calculated as 𝑓 𝑡 = 𝑧 ℎ𝑒 𝑡 -𝜇 𝜃 (𝑧 ℎ𝑒 𝑡 , 𝑡) and 𝑔 𝑡 = 𝑧 𝑖ℎ𝑐 𝑡 -𝜇 𝜃 (𝑧 𝑖ℎ𝑐 𝑡 , 𝑡). 𝑓 𝑡 ∈ R ℎ ′ ×𝑤 ′ ×𝑐 ′ represents a synthetic IHC virtual-stained feature map from the noisy feature map of its paired H&E-stained image patch. 𝑔 𝑡 ∈ R ℎ ′ ×𝑤 ′ ×𝑐 ′ denotes the reconstructed feature map of the input noisy feature map of the original IHC-stained patch.3.3.1 Reconstruction and Re-staining losses. In our proposed multimodal pre-training network, there are two generation losses that measure the synthesis quality for reconstruction and re-staining tasks, respectively. One is to evaluate whether the denoised IHC features generated through a reconstruction task are consistent with their original features extracted by the encoder 𝜏. This reconstruction loss helps the denoising U-net better learn the distribution of the original features of IHC-stained slides. The other generation loss is to assess whether the features converted by the denoising diffusion model using H&E-stained image patches through a re-staining task are similar to features of their corresponding IHCstained patches. This loss enables the denoising U-Net to predict cross-modal IHC-staining guided representations with uni-modal H&E-stained inputs. We use L1 loss for generation losses, and they are defined as L ℎ𝑒→𝑖ℎ𝑐 where L 𝑖ℎ𝑐→𝑖ℎ𝑐 rec and L ℎ𝑒→𝑖ℎ𝑐 rec denote the losses for reconstruction task and re-staining task, respectively.As mentioned in Section 3.1, each IHC-stained image patch can be categorized into 'positive' or 'negative' based on the percentage of brown regions. The H&E-stained image patches share the same categories with their paired IHC-stained image patches. Therefore, we propose a classconstraint contrastive loss to maintain the semantic consistency between re-stained features 𝑓 𝑡 and reconstructed features 𝑔 𝑡 . For a batch of re-stained features 𝐹 = 𝑓 0 𝑡 , 𝑓 1 𝑡 , ..., 𝑓 𝑏 𝑡 and a batch of corresponding reconstruction features 𝐺 = 𝑔 0 𝑡 , 𝑔 1 𝑡 , ..., 𝑔 𝑏 𝑡 , they share the same pseudo labels 𝜌 = 𝜌 0 , 𝜌 1 , ..., 𝜌 𝑏 . The features of the same label should be similar in the latent space. Therefore, we define a CLIP [44] -style contrastive loss to perform class-constraint feature alignment:EQUATIONwhere 𝑏 is the batch size. 𝑓 𝑖 𝑡 and 𝑔 𝑖 𝑡 denote the re-stained feature and reconstructed feature of the 𝑖th pair in an input batch, respectively. 𝑓 𝑗 𝑡 and 𝑔 𝑗 𝑡 represent the other features in the batch. 𝜙 is a global average pooling operation that converts all the features in 𝐹 and 𝐺 to feature vectors of shape 1 × 𝑐 ′ . 𝜎 is the temperature factor. The overall loss function has three terms:L (𝑋 ℎ𝑒 , 𝑋 𝑖ℎ𝑐 , 𝜌) = 𝜆 1 L ℎ𝑒→𝑖ℎ𝑐 rec + 𝜆 2 L 𝑖ℎ𝑐→𝑖ℎ𝑐 rec + 𝜆 3 L 𝑐𝑜𝑛 , (6)where 𝑋 ℎ𝑒 , 𝑋 𝑖ℎ𝑐 and 𝜌 denote a batch of H&E-stained image patches, IHC-stained image patches and their correspoding pseudo labels, respectively. 𝜆 1 , 𝜆 2 and 𝜆 3 are set 10, 1 and 0.1, respectively. The study of hyper-parameters is in Supplementary Materials.For the pre-trained denoising diffusion network, we employ it as a feature extractor for H&E-stained images, rather than an image generator. Considering that the denoising U-Net has learned to transform the features of H&E-stained images into the representations of IHC-stained images, we assume that it can provide not only the morphological and textural features of H&E-stained images but also the IHC-staining guided features indicating the level of specific protein expression. As depicted in Figure 3 , for the task of classifying whole-slide images (WSI), an H&E-stained WSI of shape 𝐻 × 𝑊 × 3 is cropped into 𝑛 image patches, which form a bag B for multi-instance learning. This bag is then sent into an ImageNet pre-trained encoder E to extract universal instance-level features 𝑓 𝑢𝑛𝑖 ∈ R 𝑛×𝐶 . Simultaneously, the bag is also input into the other feature extractor that is pre-trained using our proposed multi-modal framework, aiming to extract IHC-guided representations. The feature extractor consists of the pre-trained encoder 𝜏, a single-step diffusion process Ψ, and the denoising U-net 𝜇 𝜃 . For each image patch 𝑥 ℎ𝑒 in B, a feature map 𝑧 ℎ𝑒 0 is initially computed using 𝜏. Then, given a time step 𝑡, a noised feature 𝑧 ℎ𝑒 𝑡 can be obtained through a single-step diffusion process: 𝑧 ℎ𝑒 𝑡 = 𝑧 ℎ𝑒 0 + √ 𝛿 𝑡 𝜖 𝑡 , where 𝛿 𝑡 and 𝜖 𝑡 are the same as those in Eq. ( 3). The difference between Eq. ( 3) and the single-step diffusion process is that 𝑧 𝑖ℎ𝑐 0 in Eq. ( 3) is replaced by 𝑧 ℎ𝑒 0 since we aim to solve the downstream tasks with only uni-modal H&E-stained images. Subsequently, the denoising U-net model takes the noised features and time step 𝑡 as input, and outputs feature representations 𝑓 𝑑𝑒 ∈ R 𝑛×ℎ ′′ ×𝑤 ′′ ×𝑐 ′′ . To align features of different latent spaces from two pre-trained models, we apply a global average pooling operator and a linear layer (see 'GAP+Linear' in Figure 3 ) to the features of our pretrained model. Then, we concatenate 𝑓 𝑑𝑒 and 𝑓 𝑢𝑛𝑖 to build the augmented bag-level features 𝑓 𝑏𝑎𝑔 ∈ R 2𝑛×𝐶 . The extraction of the bag-level features can be formulated as follows:EQUATIONwhere 𝜙 denotes the global average pooling operation and 𝐹𝐶 is the linear layer (Fully-Connected layer). 𝜏, Ψ and 𝜇 𝜃 represent the pretrained encoder, the single-step diffusion process and the denoising U-Net in our proposed multi-modal framework, respectively. E represents the ImageNet pre-trained encoder. || means the concatenating operation. Then, the resulting bag-level features can be [37, 58, 63] to obtain the final predictions.The denoising U-Net model offers multi-scale feature representations. Previous studies [6, 56] have shown that utilizing features from different decoder layers can affect the performance of downstream tasks. Hence, we conducted experiments to study how to select the decoder layer of the denoising U-Net to extract features. We select the optimal feature from these layers to be the final output representations of our pre-trained model. Besides, the different choices of 𝑡 can also affect the strength of extracted features. Therefore, like former studies, we search for the best 𝑡 that achieves the superior performance for downstream tasks. The details are in the experimental section. 
experiments 4.1 implementation details: 
 For pre-training, we built an In-house paired H&E-IHC stained dataset, which has 184 pairs of H&E-stained and IHC-stained WSIs of Colon cancer from a local hospital. After registration, we extracted 13,248 pairs of image patches at a magnification of 20x. Each image patch was cropped and resized to 256 × 256. Subsequently, we divided the dataset into training and validation sets in an 8:2 ratio for pre-training. For downstream tasks, we use the following dataset. Camelyon16 [34] is a public WSI classification dataset of breast cancer, including 270 training WSIs and 129 testing WSIs. The training set contains 159 normal WSIs and 111 tumor WSIs. For each method, we perform 10-fold cross-validation on the training set of Camelyon16 to obtain 10 models of different weights, and average their results on the official testing set. TCGA-COAD is a WSI subtyping dataset of Colon cancer from the TCGA database [51] , collected by us. The dataset consists of 392 WSIs categorized as adenomas and adenocarcinomas (adenomas) and 65 WSIs categorized as cystic, mucinous, and neoplasms (non-adenomas). We further divided the dataset into training, validation, and test sets in a ratio of 6:1.5:2.5. 5-fold cross-validation is adopted, and the mean value of performance metrics of the 5 testing folds are reported. TCGA-NSCLC is a public WSI subtyping dataset from the TCGA database [51] , and has two kinds of Lung tumor: 541 Lung Squamous Cell Carcinoma WSIs (TGCA-LUSC) and 512 Lung Adenocarcinoma WSIs (TCGA-LUAD). We split the dataset into training, validation, and test sets by a ratio of 6:1.5:2.5. For evaluation, we employ 5-fold cross-validation and report the average result across the 5 testing folds.We use AUC, F1 score, and accuracy (Acc) as evaluation metrics. The thresholds for F1 score and accuracy are 0.5 [37, 58, 63] . To train our framework, the number of time steps 𝑇 , the batch size, the number of epochs are set to 1000, 96, and 100, respectively. Adam optimizer is used with a learning rate of 1 × 10 -4 . For WSI classification, the Encoder E in the bag feature augmentation is ResNet-50 [19] . We extract 9 feature maps (𝑢 1 -𝑢 9 ) from the denoising U-net 𝜇 𝜃 and the synthetic IHC virtual-stained feature map 𝑓 𝑡 to search for the optimal feature. The sizes of these extracted feature maps are 1024 × 4 × 4 (𝑢 1 -𝑢 3 ), 512 × 8 × 8 (𝑢 4 -𝑢 6 ), 128 × 16 × 16 (𝑢 7 -𝑢 9 ), and 256 × 16 × 16 (𝑓 𝑡 ). For the Camelyon16 / TCGA-COAD / TCGA-NSCLC dataset, we adopt the feature map 𝑢 4 / 𝑢 3 / 𝑢 5 and a time step of 40 / 20 / 20.We study if the features from our pre-training framework can enhance existing WSI classifiers with the proposed bag feature augmentation strategy. In Table 1 , we evaluate four WSI classifiers (CLAM-SB [37] , CLAM-MB [37] , DTFD-MIL [63] , and HAG-MIL [58] ), and compare their results without using and with our pre-trained features (+Ours) on three datasets. The approaches without '+Ours' only use ImageNet pre-trained features from a ResNet to achieve multi-instance learning as their original setting. As Table 1 presents, the best classification results on the Camelyon16, TCGA-COAD, and TCGA-NSCLC datasets are obtained by our pre-trained features and bag augmentation strategy with HAG-MIL, CLAM-SB, and HAG-MIL, respectively. Our learned features and augmentation strategy improve HAG-MIL by 1.8% F1 score on the Camelyon16 dataset, CLAM-SB by 9% F1 score on the TCGA-COAD dataset, and HAG-MIL by 1.6% accuracy on the TCGA-NSCLC dataset, respectively. Note that our proposed method consistently improves the four baseline models across three datasets and in all three metrics. The observed improvements range from 1.1% to 4% in AUC, 1.2% to 14.4% in F1 score, and 1.0% to 4.4% in accuracy. That shows the generalization ability of our method. In Table 2 , our pre-training framework is compared with the stateof-the-art methods on the Camelyon16 and TCGA-COAD datasets. Among these methods, Simmim [57] is based on masked image modeling, while DINO [3] , DINOv2 [40] and HIPT [4] are built on contrastive learning. HIPT is proposed for histopathology images.For fair comparison, we integrate their pre-trained encoder directly into our bag feature augmentation strategy, replacing our feature extractor. HAG-MIL is chosen as the WSI classifier. 'Baseline' is the HAG-MIL classifier that only uses ImageNet pre-trained features without our proposed bag augmentation strategy. For HIPT, we directly use its released weights trained on a large amount of histopathology images. For Simmim, DINO, and DINOv2, we compare their results using their ImageNet pre-trained weights and those using the weights re-trained on our in-house dataset and the training set of CAMELYON16 / TCGA-COAD, reporting the best results. Specifically, the reported results for Simmim and DINO are based on the re-trained models, while for DINOv2, the best result is based on its ImageNet pre-trained model. In Table 2 , our framework surpasses the state-of-the-art pre-training methods in WSI classification. Specifically, on the Camelyon16 dataset, our framework achieves 1.7% higher F1 score compared to the second best DINOv2. For the TCGA-COAD dataset, our framework outperforms the second best DINO by 1.4% AUC and 1.9% accuracy.We study the effectiveness of the H&E to IHC re-staining task, the IHC-stained image reconstruction task, the class-constraint contrastive loss in our pre-training framework and the bag feature augmentation strategy. The results are obtained on the Camelyon16 dataset with CLAM-SB as the WSI classifier. 𝑧 𝑖ℎ𝑐 0 that is extracted by the encoder 𝜏 using IHC-stained images. By comparing 'M3' and 'M5', we show that the IHC-stained image reconstruction task enhances the pre-trained features and increases the WSI classification results by 0.7% AUC and 1.1% F1 score. We investigate how different extracted feature maps and various time steps (𝑡) in our pre-trained model affect the WSI classification. To select the optimal feature maps, we evaluate the classification performance using each of the multi-scale feature maps (𝑢 1 -𝑢 9 ) from the decoder of the denoising U-net and the synthetic IHC virtual-stained feature map 𝑓 𝑡 . In Figure 4 , the height of each point on the 3D mesh represents the AUC value of a WSI classifier trained based on a specific feature map and time step 𝑡. We observe that the feature maps from intermediate decoder layers (𝑢 4 𝑢 8 ) are more likely to achieve better performance, exhibiting improved stability and reduced variance. For time steps, we selected six different values (0, 10, 20, 40, 80, 160) as the inputs to the single-step diffusion process and the denoising U-net. The results in Figure 4 suggest that the different choices of time step have limited influence on the representation quality for WSI classification task. There are three peaks in the 3D-mesh: P1 (u4,40,0.868), P2 (u6,10,0.857) and P3 (u7,10,0.857). For the three WSI classification datasets, we conduct experiments similar to Figure 4 to determine the feature map and time step on the validation sets.  
conclusion: 
 In this paper, we present a novel multi-modal denoising diffusion pre-training framework for solving the task of histopathology image analysis. Firstly, we train a denoising diffusion model on the H&Eto-IHC image translation task and IHC-stained image reconstruction task, allowing the model to provide multi-modal information from even an image modality. Next, we introduce a class-constraint contrastive loss that ensures semantic consistency between the restained features and the reconstructed ones, utilizing prior pseudo labels estimated from IHC-stained images. Moreover, to integrate our synthetic multi-modal features with existing MIL-based WSI classifiers, we propose a new bag feature augmentation strategy to expand bag features with the generated features from our pretraining framework. Experimental results demonstrate that our proposed framework effectively improves the performance of existing WSI classification methods and outperforms the state-of-the-art pre-training approaches. 
: 
 Histopathology images are widely used for medical diagnosis and research, facilitating objective assessments of diseases. Whole-slide images (WSIs) are high-resolution digital histopathology images that visualize an entire tissue section. The creation of WSIs involves tissue fixation, cleaning, infiltration, sectioning, staining, and digital scanning. WSI classification including cancer classification [14, 50] , grading [45, 55] and subtyping [7, 16] , are important in tumor diagnosis and treatment planning. Hematoxylin and eosin (H&E) staining is widely used to visualize WSIs for its simplicity and affordability. However, H&E staining mainly emphasizes tissue structures and cell morphology, and its capability to detect specific cells and protein markers is limited [1, 62] . Hence, the assessment accuracy of disease through H&E-stained images is limited [30, 35, 62] . Immunohistochemistry (IHC) staining [10, 26, 27] labels specific antigens or proteins, offering molecular markers and protein expression levels for disease diagnosis. However, IHC-stained slides require extra labor, time, and specialized laboratory equipment for preparation [30, 62] . Therefore, it is an interesting challenge to leverage multi-modal training images to infer the labels of WSIs with some missing modality.Some studies [11, 30, 35, 62] generate IHC-stained images from H&E-stained images using image-to-image translation. These methods typically employ generative adversarial networks (GANs) on paired H&E-IHC stained images to create virtual IHC-stained samples. However, their focus is limited to image generation and does not incorporate re-stained images or generated features to improve the performance of WSI analysis. Most WSI classification methods [37, 48, 58, 63] employ multi-instance learning (MIL) by dividing WSIs into patches and extracting patch features, which are then combined into a bag. Feature selection and classification are performed on the bag to obtain the category of the bag as the predicted WSI label. These methods are based solely on a single H&E staining modality, and few approaches have exploited the knowledge from multiple image modalities (such as IHC-stained images).Self-supervised pre-training is widely used in learning visual representation, providing high-quality image features for downstream tasks. Masked image modeling [17, 21, 53, 57] recovers missing parts of input images for feature learning. Some methods based on contrastive learning [3, 5, 18, 32] learn to draw the features of positive samples (such as different views of an image) closer to each other (Figure 1 (a)). Recent studies discovered that denoising diffusion models exhibit visual representation capabilities by estimating the added noise on input noisy images (Figure 1 (b)) [6, 54, 56] . However, most diffusion-based pre-training methods mainly focus on the reconstruction task, overlooking the potential visual representation from multi-modal generation tasks, such as cross-modal image-to-image translation.Inspired by the aforementioned observations, we propose a novel multi-modal denoising diffusion pre-training framework for wholeslide image classification (Figure 1(c) ). First, to address the high acquisition costs of IHC-stained images, we introduce a re-staining task that generates IHC-modality images from H&E-modality inputs using a denoising diffusion model. Through the learning of shared structural features between H&E and IHC stained images and capturing the differences in staining appearance, the re-staining task enables the trained model to extract cross-modal visual representations. These representations can serve as supplementary multi-modal information for WSI analysis models in the absence of the IHC modality. Second, to enhance the generalization ability of the model and enable it to capture visual patterns across different modalities, we take IHC-stained images as input during the training to tune the model on an extra IHC-stained image reconstruction task. The reconstruction task helps the framework to harvest specific antigen and protein expression features associated with IHC staining. To ensure semantic consistency between reconstructed and re-stained features, we propose to apply a contrastive loss between the features of the above two generative tasks. Furthermore, we observe that the presence of brown regions in IHC-stained samples correlates with the expression level of specific antibodies as well as the type of WSIs. Therefore, we can estimate binary pseudo-labels for each IHC-stained sample by quantifying the proportion of brown regions. To better guide the contrastive learning, we propose a class-constraint contrastive loss to encourage similarity between re-stained images and reconstructed images with the same pseudo labels, while ensuring lower feature similarity across different pseudo labels. The pseudo label based constraint is promising to preserve the similar clinical information between the reconstructed and re-stained representations.For applying the multi-modal pre-trained model to WSI analysis, we develop a MIL-based WSI classification framework that is integrated with a new bag feature augmentation strategy. Within this MIL framework, we initially extract features from WSI patches using an ImageNet-trained encoder, forming a bag representation. Subsequently, we augment the bag with the features obtained from our pre-trained diffusion model. By doing so, we maintain the generality of the ImageNet pre-trained features while incorporating the IHC-guided features learned by our model.To summarize, our main contributions have four folds:  
 
 Appendix:  


Decision: This paper proposed a multimodal denoising diffusion pre-training method for whole slide image classification. After rebuttal, the reviewers rated as 1 borderline accept, 1 borderline reject and 1 weak accept. The reviewers confirmed the merits of this paper and they also raised major concerns on the complexity of method and systematic experimental evaluation. After reconciling all the comments, I tend to suggest a decision of accept, but suggest the authors revise the manuscript by taking all the reviewers’ comments into consideration.

Reviews: review: paper_summary: paper_summary: In this paper, a multimodal denoising diffusion pre-training framework is introduced that leverages IHC staining to learn visual representations. It involves training with H&E-to-IHC re-staining and IHC image reconstruction tasks, enhancing structural and staining consistency between modalities. This framework outputs IHC-guided features from H&E images and employs a class-constrained contrastive loss for semantic consistency. Additionally, this paper integrates a bag feature augmentation strategy for WSI classifiers, enhancing feature diversity with the pre-trained model.
strength_weakness: strength_weakness: 1.This paper proposes to use a diffusion model to generate IHC modal images and use the information in them to add additional semantics, which is somewhat innovative.
2.Extensive experiments were conducted on multiple datasets.
limitations: limitations: 1.The information from IHC modality and HE modality is complementary, so it makes sense to combine both types of information for downstream task analysis. However, generating IHC modality from HE might be unreliable, because certain structures, such as the epithelia of lymphatic or blood vessels, are indeed indistinguishable in HE, and can only be accurately identified with CD-31 and D2-40 staining. I believe that using a small amount of proprietary data for pre-training, as mentioned in your experiment, may be unreliable.

2.To demonstrate the effectiveness of your approach, you can undertake an experiment similar to the example of lymphatic and blood vessels I described. Specifically, you can compare the identification and characterization of these structures using both HE and IHC modalities.
paper_summary: paper_summary: In this paper, the authors introduce a pretraining method for WSI data. Specifically, they employ a denoising diffusion model coupled with a contrastive loss, which guarantees semantic consistency between the retained and reconstructed features. Experimental results indicate that the proposed method outperforms existing pre-training models, thus demonstrating its effectiveness in handling WSI data.
strength_weakness: strength_weakness: - **S1:** The research on WSI pre-training  is a trending direction.

- **S2:** The method proposed by the author achieves improved results across multiple datasets and against various baselines.
limitations: limitations: - **W1:**  The proposed method appears to be a straightforward combination of dilution and contrastive techniques.
- **W2:** The author has introduced an excessive number of hyperparameters, which may complicate the training and deployment of the model.
paper_summary: paper_summary: The paper introduces a novel multi-modal denoising diffusion pre-training framework which leverages IHC staining for visual representation learning. Additionally, a class-constraint contrastive loss is introduced to ensure semantic coherence among dual-modal features. To integrate with WSI classifiers employing multi-instance learning, a bag feature augmentation strategy is designed to enrich the bags with features extracted by the proposed pre-trained model.

The paper demonstrates the effectiveness of the method by comparing it with current state-of-the-art methods across multiple datasets.
strength_weakness: strength_weakness: 1. The method proposed in the paper is innovative. It is worth exploring to learn feature representations through the task of virtual staining from H&E images to IHC, as well as the reconstruction task of IHC images.

2. The paper is well-organized and easy to follow.

3. The experimental design of the paper is rigorous.
limitations: limitations: 1. For the virtual staining from H&E to IHC, current methods seem unable to produce biologically meaningful IHC images. Even in the generated IHC images shown in Fig. 5 of the paper, there are numerous discrepancies compared to the corresponding ground truth IHC images. Is there any value in learning feature representation from such an "incorrect staining transformation"?

2. When creating the dataset, did you evaluate the extent of tissue sample damage caused by first using H&E staining and then washing off the dye before applying IHC staining?

3. In Sec. 4.4, do you mean "we evaluate the classification performance using each of the multi-scale feature maps from the decoder of the denoising U-net $\textbf{as}$ the synthetic IHC virtual-stained feature map"?

I believe this is a very meaningful and systematic work. Are you planning to open-source the code?