ThickV-Stain: Unprocessed Thick Tissues Virtual Staining for Rapid Intraoperative Histology

Authors: lulin  shi
xingzhong  hou
ivy h m wong
simon c k chan
zhenghui  chen
claudia t k lo
terence t w wong
x  shi
i h m hou
s c k wong
z  chan
c t k chen
t t w lo
  wong
Venue: MIDL_io_2024_Conference
Published: 2024-01-26 03:25:31

Abstract: Virtual staining has shown great promise in realizing a rapid and low-cost clinical alternative for pathological examinations, eliminating the need for chemical reagents and laborious staining procedures. However, most of the previous studies mainly focus on thin slice samples, which still require tissue sectioning and are unsuitable for intraoperative use. In this paper, we propose a multi-scale model to virtually stain label-free and slide-free biological tissues, allowing hematoxylin- and eosin- (H&E) staining generation in less than a minute for an image with 100 million pixels. We name this ThickV-Stain model, specifically developed to virtually stain intricated and unprocessed thick tissues. We harness the ability of a multi-scale network to encourage the model to capture multiple-level micromorphological characteristics from low-resolution images. Experimental results highlight the advantages of our multi-scale method for virtual staining on unprocessed thick samples. We also show the effectiveness of ThickV-Stain on thin sections, showing generalizability to other clinical workflows. The proposed method enables us to obtain virtually stained images from unstained samples within minutes and can be seamlessly integrated with downstream pathological analysis tasks, providing an efficient alternative scheme for intraoperative assessment as well as general pathological examination.

Main paper: 
 introduction: 
 Histological examination is regarded as the gold standard for cancer diagnoses, which aims to determine whether there are any lesions or analyze the tissue microenvironment (Gurcan et al., 2009) . However, routine histological staining is chemical reagent-dependent and usually takes several days for sample preparation, heavily limiting their applications on rapid histopathology, especially intraoperative tissue examination (Maloney et al., 2018) .Although the frozen section has been considered as an alternative for rapid pathological diagnosis to guide tumor resection in surgery, frozen samples are unstable and easily introduce some artifacts during rapid freezing, making diagnosis difficult or even misdiagnosed (Taxy, 2009) . With the development of section-free imaging techniques, unprocessed thick tissue samples can be visualized in grayscale images without sectioning procedures (Fereidouni et al., 2017; Glaser et al., 2017) . The use of label-free or stain-free microscopes is not a routine diagnostic workflow for pathologists, and it is difficult for pathologists to interpret the biological changes with the grayscale images acquired through this method.The emergence of virtual staining has provided a novel solution for achieving rapid histology. It refers to the digital transformation or generation of histological stains through computer algorithms (Bai et al., 2023) . There are some studies on the image transformation from H&E staining to special or immunohistochemistry (IHC) staining (Liu et al., 2021; de Haan et al., 2021; Lahiani et al., 2019; Vasiljević et al., 2021; Lahiani et al., 2020; Xu et al., 2020) and image generation from unstained thin sections (Rivenson et al., 2019b,a; Zhang et al., 2020; Li et al., 2020 Li et al., , 2021;; Meng et al., 2021; Shi et al., 2023) . Although the existing virtual staining studies have achieved impressive performance, they mainly rely on well-prepared thin slices, and the acquisition and preparation of these thin slices are still cumbersome and time-consuming (as shown in Figure 1a ), which is unsuitable for intraoperative tissue diagnosis. Therefore, this paper does not focus on image translation from the existing staining or unstained thin sections but focuses on virtual H&E staining from sectioning-free and label-free tissues.In this paper, we investigate directly using autofluorescence (AF) microscopy (Zhang et al., 2022) , to obtain AF images of unprocessed thick samples, which does not necessitate tissue embedding, sectioning, and dewaxing (Figure 1c ). Our objective is to convert the acquired grayscale images into pathologist-interpretable images. Additionally, we evaluated that the virtually stained images can further be used for automatic tumor diagnosis. The whole procedure of the proposed pipeline can be completed within several minutes with minimal tissue preparation. In comparison to the frozen section (Figure 1b ), our method is faster and non-destructive to tissues. In brief, the key contributions of this paper are as follows: 1). We propose a ThickV-Stain model1 for complicated unprocessed bio-samples.2). The first proposal to use a multi-scale generator for virtual staining from low-quality input images. 3). SOTA performance on both thick tissue and thin section data. 
research problem: 
 The research problem is the automatic translation from grayscale AF images of thick samples to histologically stained images, which can be easily integrated with other downstream tasks (e.g., tumor detection, and report generation), enabling rapid intraoperative diagnosis.The first challenge here is the low image quality of AF input. In slide-free microscopy, AF images from thick specimens suffer lower molecular contrast and are blurrier. This is due to the lack of optical sectioning capability in the slide-free imaging system and multilayer information would be captured at the same time. From Figure 2a , we can see that the image quality and resolution of the thick samples are relatively poor when compared with that of thin tissue slices. Therefore, it is much more challenging to recognize and transform those indistinguishable components from images of unprocessed thick specimens. As shown in Figure 2a , cell nuclei arranged in adenocarcinoma acinar structure can be observed in the AF of the thin section while no obvious related features are shown in the AF of thick tissues. Hence, it poses a huge challenge to learn the correct mapping between two domains.Another barrier is that it is not possible to obtain aligned chemically stained images with unstained images for model training as histological staining will only be carried out on a thin slide. In this situation, after acquiring scanned images of unprocessed and unstained thick samples, only the scanned surface would be sectioned and stained for reference. Since AF images of thick samples contain multiple-layer information, there will be a considerable morphological difference between the AF images and the chemically stained versions (Figure 2b ). Unsupervised learning is required when there is no paired ground truth. Current unsupervised virtual staining methods mainly exploit cycleGAN as the basic architecture (Zhu et al., 2017) . (Li et al., 2021) shows that the cycleGAN can hardly distinguish the background and tissue context towards the virtual staining of complicated human samples. This is because the cycle-consistency loss cannot guarantee semantic rationality. Even though the semantic change on the model output image is incorrect, the reconstructed one can be recovered perfectly due to the loss. Some work also reported that cycle consistency loss plays an indispensable role in key structure preservation of input images (Pumarola et al., 2018) , but weak constraints between the translated output and ground truth (Liu et al., 2021) . To resolve this ambiguity, some researchers have tried to use saliency mask (Li et al., 2021) and region labels (Shi et al., 2022) as semantic guidance for virtual staining.In our case, it is quite challenging to extract the consistent saliency mask or region label due to the limited image contrast. We address the above challenges with the multiscale generator. We claim to use a multi-scale model to explore sufficient information for a better understanding of the complex data, which does not require additional supervision or manual annotations. We aim to use different scales focusing on different histological patterns according to the receptive field size and capturing multi-level representations.To have special attention to learning multi-structural features, we propose a novel multiscale generator as shown in Figure 3 . It consists of three U-Net modules with the same convolution kernel size (4 × 4) but with different receptive fields due to different input image resolutions. The top branch in Figure 3 captures more detailed histological features, such as cell nuclei. This is because the convolution kernel of this module will perceive small objects in the input resolution. Conversely, the bottom one can perceive a large field as the original image is scaled down, such as global information of edges and background.The U-Net architecture used in this paper consists of an encoder and a decoder network. The encoder involves 5 convolution layers and the decoder consists of 5 fractionally-strided convolution layers that upsample the feature maps to the original spatial dimensions. The feature maps from the encoder with the corresponding ones from the decoder are connected by skip connections to preserve the spatial information from the input. Following the U-Net, the outputs from the two additional scales are resized to the original size of the input image and then concatenated to get the final output. The form of the output can be expressed as:G(x) = conv([U 1 (x), R(U 2 (x)), R(U 3 (x))]) (1)where [•] refers to the concatenation operator, U i represents the i-th branch. R(•) denotes the bicubic interpolation function to change the size of images. We use 2 resize blocks here to achieve the increase of two additional scales (H/2, W/2) and (H/4, W/4).To avoid mode collapse caused by the imbalance design between the generator and discriminator, we also use a multi-scale discriminator (Wang et al., 2018b) . The input image is processed at three scales, with each scale being processed by a separate network. Those three discriminator networks have an identical structure but operate at different image scales. The output of each network is then combined to produce a final score.Besides GAN loss (Goodfellow et al., 2020) , we further use identity loss (Taigman et al., 2017) for better structure preserving. The overall loss functions are shown below:L G adv = -E x [logD(G(x))](2)EQUATIONL idt = E y ∥y -G(y)∥ 1 (4) L = L adv + λL idt (5)3. Experiments and results 
dataset: 
 The data used in this paper were collected from 14 patients and involved various lung adenocarcinoma subtypes. The H&E-stained slices used for training were cut from the imaged thick sample surface. Due to the megapixels of whole slide images (WSIs, ∼20,000 × ∼20,000), we randomly sampled 10,000 small image patches per epoch with the size of 256 × 256 from seven patients for training. During testing, each WSI of the remaining seven patients' data was split into 256 × 256 images with 16-pixel overlap to avoid artifacts, and the test cohort included 31,317 image patches (thick sample data used in section 3.3) and 22,283 image patches (thin section data used in section 3.4).The overall network was implemented in PyTorch on a single NVIDIA GeForce RTX 3090 GPU. We used the architecture of U-Net in each scale forming our generator and multiscale discriminator based on the pix2pixHD. We trained our model with the Adam optimizer (with β 1 = 0.5, β 2 = 0.999). The initial learning rate was set to 2 × 10 -4 for the generator, and 1 × 10 -4 for the discriminator with a linear decay scheduled after 15,000 iterations. The different learning rate was designed to avoid mode collapse due to the fast convergence of the discriminator. The batch size was set to 16. The λ in 5 is set to 5.In this paper, we chose the commonly used unsupervised I2I architectures, including cycle-GAN, CUT, and a seminal virtual staining model (UTOM), as the baseline models. The visual comparison is shown in Figure 4 . Compared to the AF image, the H&E staining (1 st column) can only visualize some surface structure of the sample, providing a basic reference rather than an exact ground truth. In the 2 nd row (marked with blue arrow), there are obvious pathological features of fibrosis in our results. From the 3 rd row, the major tumor component consisting of solid sheets can be observed in our results (marked with yellow dashed lines). The same structure can also be observed in H&E-stained reference image. Even though UTOM also draws a correct outline, it misses cell nuclear information within the outline. Both CUT and cycleGAN mistranslate the background areas, which is consistent with (Li et al., 2021) . To evaluate the image quality quantitatively, we calculate the Fréchet Inception Distance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Bińkowski et al., 2018) . The results are shown in Table 1 , where ThickV-Stain achieves the best FID score (36.71). UTOM and cycleGAN are better than CUT, showing the same trend with visual analysis.To explore whether our virtual staining meets clinical needs, we invited two pathologists to score our virtually stained images. The evaluation metrics are from two aspects: the pathological properties and visual quality. We followed the setting from (Rivenson et al., 2019b) to evaluate nuclei details, extracellular fibrosis, and overall staining quality. Regarding the visual quality, fidelity refers to whether any artifacts in the stained image can affect the image quality and accuracy refers to whether the pathological diagnosis (e.g., the location of the tumor region) from virtually stained images is consistent with the real staining. The evaluation results in Table 2 show that our model significantly outperforms other baseline models. The results of cycleGAN and CUT provide limited useful information to the pathologist, which is consistent with our analysis of visual results.Moreover, we apply a lung adenocarcinoma detection model for automatic tumor region detection with our virtually stained results. (Wang et al., 2018a) use human lung adenocarcinoma cancer data, which is consistent with our case. Here, we apply the trained model on both generated images and real H&E images to see if our virtual stained results can show consistent diagnostic conclusions with real staining. In addition, the MSE between the two tumor probability maps, noted T-MSE, can be used to evaluate the potential of virtually stained results in real clinical practice. The comparative results can be found in Table 1 , and a visual example can be found in Appendix Figure 7 .We also validate the extensibility of the proposed method on other clinical workflows. Moreover, the ground truth is accessible for the virtual staining of thin sections, which can provide an accurate evaluation of the model performance. It can be observed that ThickV-Stain (Figure 5c ) outperforms other baseline models on unstained thin slices compared with ground truth (Figure 5b ). As for quantitative measures, we chose the structural similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), and MSE, as metrics because ground truth can be obtained for thin-section virtual staining. The numerical results are shown in Table 1 . Regarding PSNR and MSE, ThickV-Stain and UTOM are relatively close, and CUT are inferior to the other models. As for the FID and KID scores computed on the test set, ThickV-Stain significantly outperforms other models.In this paper, we claim that the virtual staining performance on complicated dataset, especially unprocessed thick tissue, is limited if the model cannot represent complex features adequately. To explore the importance of the multi-scale model for virtual staining of complicated thick samples, we carried out an ablation study on multi-scale generator.From Figure 6 , we can see that for virtual staining of thin section data, even a single-scale generator can also perform well. It achieves correct translation where the same cancerous patterns and fibrosis can be observed, with only slight degradation in image quality compared with the multi-scale generator. And there is no significant difference in the FID results (20.29 and 21.74) . We analyze this because the AF of thin sections are of such high quality that even a single scale model is sufficient to correctly transform the image features.In this case, multi-scale does not have obvious advantages when dealing with simple data. Secondly, for thick tissues, the single-scale generator cannot recover the fibrosis and quantitative results are also far from satisfactory compared with the multi-scale generator, which shows the importance of the multi-scale model for handling complicated data.Additionally, the single-scale generator on thin sections even achieves better FID than the multi-scale generator on thick tissues (21.74 vs 42.39). That means even increasing the complexity of the model cannot completely offset the impact of data complexity. Virtual staining on thick tissue is still difficult to achieve comparable results on thin slice data. 
conclusion: 
 In this paper, we present ThickV-Stain for virtual histological staining on unprocessed thick tissues. Specifically, we design a multi-scale generator to fuse large receptive field image features to achieve virtual staining on complex data. We also use a well-trained lung adenocarcinoma detection model to demonstrate the potential of our generated staining to be integrated with other downstream pathological image analyses tasks. With our proposed ThickV-Stain, the staining task of thick samples can be implemented, allowing pathologists to identify pathological features on unprocessed samples directly, thus providing an alternative to rapid pathology.  
 
 Appendix: Code and model are available at https://github.com/TABLAB-HKUST/ThickV-Stain. 


Decision: The reviewers unanimously agree to accept the submission.

Reviews: review: paper_summary: paper_summary: This paper proposed a multi-scale generator for virtual histological staining on unprocessed thick tissues. They also evaluated detection results of a pretrained model using samples processed with proposed staining method to show method can be integrated with other downstream pathological image analyses tasks.
main_review: main_review: paper clearly explains a problem and provides details on existing works and provides an approach. Then by comparing the methods with SOTA justifies the applicability of technique and its better performance.
strength_weakness: strength_weakness: The paper should improve model explanation and elaborate on chosen evaluation metrics as well as data details. Authors are advised to provide a comprehensive comparison discussion on results for better understanding of reader and reason why proposed methods performs better than others.
questions: questions: minor comments
paper_summary: paper_summary: This article addresses the issue of virtual H&E staining. The authors propose a GAN architecture dedicated to generating H&E staining images from an AF image of a thick tissue section. This task is challenging due to the blur caused by the superposition of information from the multi-layered sample. Moreover, there is no ground truth of such H&E staining as it is only performed on a thin tissue section. One of the goal of such approach is to perform rapid virtual staining in an intra-operative setting.

The novelty of the approach lies in a multi-scale generator based on three UNets. The first UNet is a classic one, while the other two use downsampled versions of the input sample to increase their receptive fields and focus on larger-scale structures.

Experiments were conducted on a private dataset of 14 patients with lung adenocarcinoma. A comparison with three baseline methods was performed on both thick and thin sections. A blind qualitative analysis was conducted by two pathologists. Additionally, the usefulness of these virtual stainings was assessed by analyzing a downstream task, namely automatic tumor region detection.
The authors claim that their approach yields superior results and enables pathologists to rapidly identify pathological features from unprocessed tissue samples.
main_review: main_review: The authors have appropriately addressed the majority of my comments and subsequently revised their article. The only remaining minor point would be to include a sentence explaining the rationale behind the choice of identity loss, as articulated in their response. 
I keep my rating and believe that the article is suitable for publication in MIDL.
strength_weakness: strength_weakness: - The description of the proposed method is rather brief in the article, lacking several key details, particularly regarding the architecture of the generator. The resizing factor of the blocks and the specifics of the backbone UNet architecture are not explained. Additionally, the "shared Unet" highlighted in green in Figure 3 requires clarification. The rationale behind incorporating the identity loss should also be explained. Similarly, a concise description of the discriminator architecture is required.

- The quantitative results lack standard deviations, which could lead to questioning the significance of the findings. Including standard deviations would improve the robustness and interpretability of the results.

- Regarding the qualitative analysis, there appears to be a significant level of inter-expert variability (see Table 3), which may diminish the significance of the experiment. It would be beneficial for the authors to quantify this variability and provide commentary on its implications within the article.
questions: questions: - Add more details on the generator and discriminator architectures
- Add standard deviation on the quantitative results
- Address comments of the "detailed comments" section.
paper_summary: paper_summary: This study introduces ThickV-Stain, an innovative method for performing virtual histological staining on unprocessed, thick tissue specimens. The approach involves the creation of a multi-scale generator that integrates image features across large receptive fields, enabling effective virtual staining on complex datasets. Furthermore, the inclusion of a sophisticated lung adenocarcinoma detection model illustrates the compatibility of their virtual staining technique with further pathological image analysis tasks. ThickV-Stain represents an advancement in the staining of thick samples, permitting pathologists to directly observe pathological features in unprocessed specimens and offering a rapid alternative for pathology evaluations.
main_review: main_review: I would keep my rating. The paper excels in innovation, methodology, presentation, and potential impact, making it a valuable contribution to its field and warranting a strong accept recommendation. Thanks
strength_weakness: strength_weakness: + The study does not compare the proposed method against a comprehensive array of baseline methods. This extensive comparison would improve the robustness and rigor of the analyses.
Otherwise, I don't see any other major weakness.
questions: questions: N/A

Comments: comment: Dear Review QhRu:

Thank you for your thoughtful review and for recognizing the importance of our work.
Dear Review yRfX:

Thanks for your very positive review, and constructive suggestions, especially regarding the detailed model explanation and chosen evaluation metrics, we will improve it in the next version. 

Below is our response to your comments:

1.	Regarding **tissue thickness**:

We can image the sample directly no matter how thick it is because we only image the sample surface. For microscopy, only the information within the penetration depth can be captured by the microscope and the depth of light penetration is limited (no more than 100μm). Therefore, no matter how thick the sample is, light cannot penetrate deeper, deeper information is difficult to obtain. 

If we only focus on the surface information of the sample, then the thickness of the sample does not matter. So it will not affect the model performance. At the same time, we do not need to process the sample (tissue embedding, tissue sectioning, etc.) and can directly image the sample, enabling rapid imaging. 

2.	Regarding the **chosen evaluation metrics**:

FID and KID are generally used to evaluate the image quality for generative models which does **not require paired ground truth**. This is exactly the case for this paper where there are morphological differences between the AF images (model input) and the stained versions (ground truth).  Therefore, we chose FID and KID and calculated the similarity between the two datasets of images.

But for virtual staining on a thin section (~5 μm), we can prepare the ground truth because the chemical stain procedures can be performed on the same thin section that was imaged. With ground truth as reference, we further chose SSIM, PSNR, and MSE. Those three metrics are commonly used to assess the quality or fidelity of generated images when **the ground truth is provided**.

Moreover, the above metrics are used to evaluate the quality of the generated images or whether the generated image is similar to the real one. To see whether the generated images meet the real clinical needs, we invited pathologists to grade images based on their expertise.


We hope this response has addressed your concerns effectively. We are grateful for the chance to discuss our work and thank you again for your valuable input.
Dear Review 94hM: 

Thank you for your thorough review and valuable feedback on our work. We address each of them below.
1. Regarding the details of the model design: 

**Generator**: The resize block is a bicubic interpolation function to change the size of images. We use 2 resize blocks here to achieve the increase of two additional scales (H/2, W/2) and (H/4, W/4). The original image and two resized images are input to the corresponding convolution block for channel adaptation and then to the same U-Net backbone. The U-Net architecture consists of an encoder and a decoder network. The encoder network is 5 convolutional layers that downsample the input image by decreasing its spatial dimensions while increasing the number of channels. The decoder network consists of 5 fractionally-strided convolution layers that upsample the feature maps to the original spatial dimensions while decreasing the number of channels. The contracting and expanding paths are connected by skip connections, which concatenate the feature maps from the encoder network with the corresponding feature maps from the decoder network. These skip connections help preserve the spatial information from the input image and improve the image generation accuracy. Following the U-Net, the outputs from the two additional scales are resized to the original size of the input image. These three outputs are then concatenated to get a feature map with dimensions (3C, H, W). Finally, a convolutional layer is used to map the feature map from (3C, H, W) to (C, H, W) to produce the final result. 

**Discriminator**: we use a multi-scale discriminator which is proposed in pix2pixHD. In a multi-scale discriminator, the input image is processed at multiple scales, with each scale being processed by a separate discriminator network. Those 3 discriminator networks have an identical network structure but operate at different image scales. We downsample the real and generated images by a factor of 2 and 4 to create an image pyramid of 3 scales. The discriminators are then trained to differentiate real and synthesized images at 3 different scales, respectively. The output of each discriminator network is then combined to produce a final discriminator score.

We have revised the model design in the main text and highlighted all changes . 

2. Regarding the **standard deviation**:
We added some experiments and the standard deviation results shown as below:
|    | CycleGAN | CUT | UTOM | ThickV-Stain (ours) |
| --------| :-----------:|:--------:| :-----------: |:-----------:| 
| FID   |97.24$\pm$5.88| 97.26$\pm$24.37| 62.00$\pm$6.88|**36.71$\pm$3.27**|
| KID   |0.057$\pm$0.011|0.063$\pm$0.016|0.034$\pm$0.005|**0.015$\pm$0.005**|

3. The rationale of **identity loss**

For unsupervised image-to-image translation, the generally used loss function is cycle consistency loss which has been confirmed to play an indispensable role in key structure preservation of input images. However, for virtual staining, some research and our previous work (see below) show that cycle consistency loss cannot guarantee the correct translation between tissue content and background. Therefore, we omit this loss for model efficiency. However, without this loss, the model will lose some original structure, and then we add the identity loss, which also can help the model preserve the input structure and will not increase the model parameters.

Shuting Liu, et al. Unpaired stain transfer using pathology-consistent constrained generative adversarial networks. IEEE Transactions on Medical Imaging, 40(8):1977–1989, 2021. 

Xinyang Li, et al. Unsupervised content-preserving transformation for optical microscopy. Light: Science & Applications, 10(1):1–11, 2021

Lulin Shi, et al. One-side virtual histological staining model for complex human samples. In 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pages 1–4. IEEE, 2022.

4. Regarding **Fig 7**:
This is a quite good point.  Here the pathologist annotation is directly requested from our laboratory partners. Due to the different parameters set when exporting H&E WSI, there are some color and rotation differences between the image in the bottom right and the left one. We have unified those two images in the revised pdf.

5. Regarding **Fig 6**: 
The annotations in Fig. 6 are manually labeled for interpretation of the figure. Thanks for your advice and we have emphasized this in the main text.

We once again thank you for the helpful comments and suggestions. We hope we have addressed all your concerns. Please don't hesitate to let us know if we can clarify anything else!