Deep-Learning Based Virtual Stain Multiplexing Immunohistochemistry Slides – a Pilot Study

Authors: oded  ben-david
elad  arbel
daniela  rabkin
itay  remer
amir  ben-dor
sarit  aviel-ronen
frederik  aidt
tine  hagedorn-olsen
lars  jacobsen
kristopher  kersch
anya  tsalenko
©  oded
Venue: MICCAI_org_2024_Workshop_COMPAYL
Published: 2024-06-27 14:25:29

Abstract: In this paper, we introduce a novel deep-learning based method for virtual stain multiplexing of immunohistochemistry (IHC) stains. Traditional IHC techniques generally involve a single stain that highlights a single target protein, but this can be enriched with stain multiplexing. Our proposed method leverages sequential staining to train a model to virtually stain multiplex additional IHC on top of a digitally scanned whole slide image (WSI), without requiring a complex setup or any additional tissue sections and stains. To this end, we designed a novel model architecture, guided by the physical sequential staining process which provides superior performance. The model was optimized using a custom loss function that combines mean squared error (MSE) with semantic infor-mation, allowing the model to focus on learning the relevant differences between the input and ground truth. As an example application, we consider the problem of detecting macro-phages on PD-L1 IHC 22C3 pharmDx NSCLC WSIs. We demonstrated virtual stain multiplexing CD68 on top of PD-L1 22C3 pharmDx stained slides, which helps to detect macrophages and distinguish them from PD-L1+ tumor cells, which are often visually similar. Our pilot-study results showed significant improvement in a pathologist's ability to distinguish macrophages when using the virtually stain multiplexed CD68 decision supporting layer.

Main paper: 
 methods: 
 Sequential staining is a method for adding an additional immunohistochemical (IHC) stain (stain 2), to tissues which were already stained (stain 1) Moreover, according to the Beer-Lambert law for absorption of light passing through a medium [15] , the optical density of stained tissue is linearly dependent on the local concentration and absorption coefficient spectrum of chromogen.We designed a model architecture Figure1(a) which corresponds to these physical properties of sequentially stained tissues. First, the model is optimized using optical density. Second, learning the virtual stain concentration map is separated from learning the virtual stain color, determined by the absorption coefficient vector.The first step of our model involves transforming an input patch stained with stain 1 to the optical density domain, followed by a U-Net neural network (Ronneberger et al., 2015) for inferring a single-channel stain 2 concentration map for the patch. Subsequently, the concentration map is multiplied by a learnt absorption coefficient vector to derive an optical density virtual stain. For brightfield (BF) WSIs, the absorption coefficient is length three RGB vector, hence we call this model a 3x1 architecture. The optical density virtual stain is then added to the optical density input patch, which yields an optical density virtually stain-multiplexed patch. This patch is subsequently inverse-transformed to a virtually stainmultiplexed bright-field-like output. It is important to note that although the RGB vector is learnt during training, it is a global parameter vector and not dependent on the input patch.The physical properties of sequential staining have also informed the development of our loss function for model optimization, as depicted in Figure. 1 (b) . Given the additive nature of sequential/virtual staining, semantic masks, denoted as M ask GT /M ask out , are constructed for the ground-truth and output patches, respectively. These masks are obtained by taking the difference between the optical density ground-truth/output patch and the input patch; any pixel with an optical density difference exceeding a threshold value is considered positive for sequential/virtual stain. The semantic masks are then used to compute a semantic pixel-wise binary cross entropy loss (BCE) between the corresponding ground-truth and output patches.The semantic loss is combined with a mean squared error (MSE) loss between the optical density ground-truth and output patches. To focus the training on relevant differences, M ask GT is used as a mask for the MSE loss. The parameter α allows for tunable weighting of the relative strengths of these two loss functions, resulting in a combined loss function:Loss = M SE(output, GT ; M ask GT ) + αBCE(M ask out , M ask GT ) + θ(-v)v 2 (1)Where the last term is a regularization term, reflecting the additive nature of sequential staining, keeping the inferred virtual stain output positive,θ is the Heaviside step-function and v denotes the virtual stain output. We evaluate the model's performance using the intersection over union (IOU) metric between M ask GT and M ask out , which is used for both hyperparameter tuning and computational evaluation.  
experimental results: 
 To demonstrate the utility of virtual stain multiplexing, we applied it to the problem of detecting macrophages in PD-L1 IHC 22C3 pharmDx stained NSCLC WSIs.Out of 200 NSCLC PD-L1 IHC 22C3 stained slides, 49 slides were chosen based on their TPS values, that were near to the clinically relevant thresholds of 1% and 50%. Following selection, the tissues were sequentially stained with CD68 PG-M1 (GA613), yielding sequentially stained paired WSIs. Each WSI pair was aligned; since paired WSIs are of the same tissue section, the obtained alignment was pixel-perfect. The study pathologist annotated tumor regions for each WSI pair which were used for model training. Of these, 26 regions where macrophages were difficult to distinguish from tumor cells, or where macrophages were identified as infiltrating PD-L1 positive tumor areas, were selected as validation-set regions for method evaluation and excluded from model training. See full dataset curation, preparation and annotation details in appendix A.We implemented the model described in section 2 using Pytorch. See full training and implementation details in appendix B. To evaluate the performance of our proposed virtual stain-multiplexing method in wholeslide images (WSIs), we conducted a qualitative assessment. We present several representative examples of our model outputs generated on challenging validation set regions in Figure 2 (left), alongside the corresponding input and sequentially stained ground truth patches. Notably, the virtual CD68 staining demonstrated a high degree of visual consistency with the actual CD68 stain, as evaluated by an expert pathologist. Additionally, our virtual stain method effectively highlighted macrophages, as evidenced by the staining of most macrophages in the validation set regions. Qualitative effect of combined lossIn Figure 3 , we present an illustrative example that compares a ground truth patch and the corresponding output of a full model trained with combined MSE and BCE loss, to the output of an ablation model that is solely trained using MSE loss. The comparison shows that the outputs of the ablation model possess two distinctive features in contrast to those of the full model. Specifically, the ablation model acquires faint and non-specific background staining artifacts that are present in the ground-truth patches but are not present in input patches. Nevertheless, these non-specific stains do not appear in the outputs of the full model. Moreover, the outputs of the ablation model exhibit fainter overall virtual staining in comparison to the outputs of the full model and the ground truth patches. We observed that for cells with explicit membranal staining, the full model tends to stain the entire cell cytoplasm, while the ablation model often displays partial staining patterns, similar to those found in the ground truth patches.Several ablation tests were conducted to investigate the contributions of different parts of our proposed method. For full ablation tests details see appendix C. The aim of this evaluation was to directly test the potential of the suggested model as an assistive tool for pathologists in detecting macrophages. This evaluation scheme differs from a pixel-level evaluation, as it takes into account that the virtual stain may not always match the sequential ground truth stain pixel for pixel, but may still accurately stain the correct cells.To carry out this evaluation, individual cells from 13 regions of interest, selected from the validation set, were annotated as macrophage/not-macrophage. Pathologist 1 generated cell-level ground-truth annotations based on matching pairs of PD-L1 & PD-L1 + sequentially stained CD68. Pathologist 2 first generated baseline annotations, based on PD-L1 WSI only. Then, pathologist 2 generated annotations based on matching pairs of PD-L1 & PD-L1 + virtually stain multiplexed CD68, viewed side-by-side. These annotations were then compared to the ground truth annotations by pathologist 1. The model performance was evaluated by measuring the change in pathologist 2's precision, sensitivity, and accuracy when assisted by the model com-pared to when not using the model.Confusion matrices comparing the pathologist 2's annotations are presented in Figure 4 . The addition of virtually stained multiplexed CD68 improved the performance of pathologist 2's annotations, increasing precision and sensitivity from 0.44 and 0.32 to 0.67 and 0.79, respectively, while specificity remained unchanged at 0.92. Paired McNemar's test (McNemar, 1947) yielded a p-value of less than 10 -30 . 
conclusions: 
 In this pilot-study, we propose the method of virtual stain multiplexing, which combines virtual staining and virtual multiplexing. Virtual stain multiplexing of immunostains in the same section offers a promising avenue for the development of more accurate and reliable scoring methods for cancer diagnosis, without incurring additional reagents or tissue.We presented a deep learning based model for CD68 virtual stain-multiplexing that can identify and virtually stain macrophages on an internal validation set. By virtually staining these cells, our model provides a useful tool for analyzing and interpreting PD-L1 22C3 pharmDx IHC WSI. Notably, we found that pathologist 2's performance in detecting macrophages improved significantly when assisted by the model.In section 3.4, we demonstrated the importance of incorporating semantic loss in the ablation test. This addition was essential in enabling the model to focus on learning the relevant differences between the input and ground truth. The semantic BCE loss served as a guide for the model by providing information on where to add virtual staining, while the masked MSE loss directed the model on how to add it.Interestingly, we observed that the addition of the semantic loss caused the model to virtually stain the entire cell cytoplasm, at the expense of visual fidelity to the ground truth staining patterns. However, this characteristic of the model was found to be advantageous in the context of macrophage detection. The study pathologists found it easier to interpret the virtual stain CD68 compared to the ground-truth sequential stain, owing to this behavior of the model.In Section 3.5, we presented a quantitative pilot-study evaluation of the use of our model for macrophage detection. The results were encouraging, showing significant improvement in both precision and sensitivity of pathologist 2 when assisted by the model. In an ongoing study, building on this pilot-study, we will test the efficacy of the method with multiple pathologists.Our proposed method can be extended to virtually multiplex additional stains, including those for other types of immune cells. As shown in Figure 1 (c), the 3x1 architecture allows for arbitrary hue settings at inference time, facilitating such multiplexing. Moreover, our method opens up avenues for detailed exploration of the spatial arrangement of tumor infiltrating immune cells within the tumor microenvironment. Since our model utilizes the same input as the standard PD-L1 IHC 22C3 pharmDx stained tissue scans that are used in clinical practice, such investigations can be retrospective, employing clinically obtained data. This could lead to valuable insights into the relationship between immune cells and tumor cells, ultimately aiding in the development of new cancer therapies.  
: 
 1 Introduction "Tissue is the issue" is a theme that serves as a guideline in nowadays pathology practice, that requires ever-growing information deduced from tissue samples, while their size keeps getting smaller. Extracting maximum information from tissue samples is a common challenge in pathology in the era of personalized medicine that necessitates the use of increasing number of immunohistochemical (IHC) stains. Combined with the increasing use of small biopsies as the main tissue material, this puts a significant limit on the number of different IHC stains which can be run. While this challenge can be addressed in research using fluorescent IHC multiplexing or mass spectrometry IHC, in routine diagnostic pathology the common practice is to stain multiple consecutive sections with different immunostains at the cost of labor, reagents and tissue.Digital pathology offers an array of alternative solutions as it enables to digitally unmix (Ruifrok and Johnston, 2001) and superimpose staining from one section to another, to create virtual multiplexed layers (Huss and Grunkin, 2022; Visiopharm, 2023) or otherwise to project AI inferred specific cell density maps (Bloom et al., 2022) . Virtual multiplexing methods, however, suffer from the same two main draw-backs as traditional pathology. First, the method requires additional tissue sections and reagents, adding to cost and complexity. Second, the consecutive tissue sections are at least 3-5 microns apart resulting in an increasingly growing distance between slides with each additional stained section. Thus, sections do not contain the exact same cells.With recent advances in deep learning based models, a different approach called virtual staining has emerged (Owkin, 2020; de Haan et al., 2021) . Virtual staining methods use generative adversarial networks (GANs) to unlock information from existing Hematoxylin-Eosin (HE) stained tissue patterns without the need for physically staining additional tissues. Pushing this approach even further enables the inference of immunohistochemical staining patterns using unlabeled tissue (Rivenson et al., 2019; Zhang et al., 2020; Pradhan et al., 2021) . For example, Bai et al. (2022) developed a method that generates a virtual HER2 IHC whole slide image (WSI) by recording several auto-fluorescence images of an unlabeled tissue section.A prime example of the benefits of combining information from multiple stains is the detection of macrophages in PD-L1 22C3 pharmDx stained non-small cell lung cancer (NSCLC) tissues. PD-L1 IHC 22C3 pharmDx (GE006) is an FDA approved qualitative immunohistochemical assay intended for use in the detection of PD-L1 protein in formalin-fixed, paraffinembedded (FFPE) non-small cell lung carcinoma NSCLC tissue. PD-L1 protein expression in NSCLC is determined by using Tumor Proportion Score (TPS) (Herbst et al., 2016) , which is the percentage of viable tumor cells showing partial or complete membrane staining at any intensity. The TPS scoring protocol calls for the exclusion of the staining of immune cells including macrophages (Technologies, 2021) . Since macrophages may show morphological similarity to NSCLC cells, differentiating positive PD-L1 staining of macrophage from that of tumor cell is often challenging (Paces et al., 2022) . Although some morphological features distinguish macrophages from tumor cells, combined staining of PD-L1 IHC 22C3 pharmDx with a separate IHC stain such as CD68 can highlight the macrophages and assist clinical pathologists in scoring more accurately. Beck et al. (2019) developed an automatic PD-L1+ detection method for urothelial carcinoma, achieving strong correlation (0.837) with pathologist consensus scores on tumor cells. Although not required for clinical purposes, they found that the detection of macrophages is still a challenging task, with inter-observer agreement/correlation as low as 0.287.In this paper, we present a novel method called virtual stain multiplexing, which combines virtual staining and virtual multiplexing. We apply this method to the problem of detecting macrophages on PD-L1 IHC 22C3 pharmDx NSCLC WSIs.The novel architecture of our proposed method is guided by the physical sequential staining process and is trained using a loss function that combines mean squared error (MSE) with semantic information. Our approach of incorporating semantic information into the training process proved to be a crucial component in achieving the desired performance.We trained an AI-based model using sequentially stained NSCLC WSIs to infer virtual CD68 IHC from an input stained with PD-L1 IHC 22C3 pharmDx. The inferred virtual stain is then combined with the input to yield the virtual stain multiplexed output. As a pilot-study, we tested the effectiveness of our model by comparing the performance of a pathologist in detecting macrophages in PD-L1 IHC 22C3 pharmDx stained NSCLC with and without using the CD68 virtual stain as a decision-supporting layer. We found that the addition of the virtual stain significantly improved the performance of the pathologist in detecting macrophages. 
 
 Appendix: Ben-David, Arbel, Rabkin, Remer, Ben-Dor, Aviel-Ronen, Aidt, Kersh and Tsalenko 



Reviews: review: title: title: Paired image translation between PD-L1 and CD68
main_review: main_review: The paper introduces a method for stain translation from PD-L1 to CD68 to allow the visualization of macrophages on PD-L1 stained images and the removal of these towards a more accurate estimation of the Tumor Proportion Score. The transformation between PD-L1 to CD68 is modeled by a so-called 3x1 model architecture that separates the learning of the virtual stain concentration and the learning of the stain absorption coefficients. This is a alternative to other similar works relying on Pix2pix or CycleGAN. The method employs guidance through semantic segmentation through the minimization of a binary cross entropy loss on thresholded virtual / real images in addition to the use of a regression loss on the RGB channels between the virtual and ground truth PD-L1 / CD68 combined images. 

- The paper is well written, the methods and the experiments clearly presented. 
- The proposed method is showed to increase concordance between pathologists when classifying cells. 
- It would be interesting to investigate whether this translates in an increase in the concordance of the TPS scores as well. Also, given the data at hand it would be interesting to develop and include an automatic detection of macrophages on PD-L1 stained images into an existing method for potentially improving on the automatic assessment of TPS / TC scores on these images. 

- Given that paired images with near pixelwise accuracy are available, it is unclear why a pix2pix approach is not used to generate a in silico CD68 image from the co-registered PD-L1 image.
- The paper misses to cite previous works on virtual staining for PD-L1 scoring, in particular [Kapil et al., IEEE TMI 2021] and translation guidance based on semantic segmentation [Mahapatra et al, MICCAI 2020].
title: title: Interesting approach to virtual staining, but the method is not properly validated
main_review: main_review: Summary of paper:

The manuscript proposes a new method of virtual staining using a U-Net to predict an optical density map based on an input optical density map of another immunohistochemistry stain. In this work, the authors use PDL1 IHC as input stain and predict CD68 to detect macrophages and reduce the number of false positives for PDL1 counting.

Pros:
- The manuscript is well written, with great figures explaining the model and results.
- Their proposed solution including a mask for virtual staining is a nice way of creating cleaner output images.
- The authors include 2 pathologists to do a small practical evaluation demonstrating the usefulness of their approach.


Major concerns:
- The model is trained and validated on the same cohort, where staining and tissue preparation are similar. If the authors want to show that their approach really works, they must include an additional test set from a different source. Moreover, it is not clear if the validation regions are from WSI that were also used for training (in other regions).
- The validation areas are selected to be challenging regions. How does the model perform on less “challenging” regions where it should not predict false information? 
- A threshold is mentioned to create the stain mask. How was this threshold determined? How does changing this threshold affect the model (during training and in general?) Does this threshold generalize?


Minor concerns:
- How much do the pathologists agree if they both use the real sequential staining?


Other comments:
- The manuscript does not follow the submission guidelines and is not using the correct template. Please use the provided template.


Code, model weights, and/or data availability:
- unavailable


Conclusion 

The authors propose an interesting method for virtual staining and include a pathologist validation of their application which is always nice. However, the validation of their proposed approach is restricted to a single cohort on which the model was also trained on, and only on ROIs that were manually selected for this study.
title: title: Virtual staining of mIHC using DL
main_review: main_review: The authors introduce a DL-based method for virtually staining mIHC slides using a U-Net based model with a custom loss function. The model have been qualitatively evaluated with two pathologists. 

Below are my comments:
1. Firstly, the paper needs a proper proofread to improve the clarity of prose and succinctness of sentences. 
2. GE0006 should be cited when first mentioned.
3. Literature review can be significantly extended and improved as only one related work is mentioned.
4. The contributions of the work are not clearly outlined and how novel is the solution is not specified.
5. The motivation behind virtual staining should be more clearly described.
5. I think claiming that the architecture is novel and mentioning that multiple times in the paper is an over-statement. The model uses a U-Net and modified MSE loss function. I'm not sure how novel is that. Also, in the abstract, the novelty is exactly specified: "To this end, we designed a novel model architecture, guided by the physical sequential staining process which provides superior performance." Then the mention of a custom loss function comes next: "The model was optimised using a custom loss function that combines mean squared error (MSE) with semantic in-formation, allowing the model to focus on learning the relevant differences be-tween the input and ground truth." When I read the two former sentences, I perceive that the architecture is novel plus that the loss function is custom. But from reading the paper, it seems that this is not the case. 
6. The conclusion section is too long and can be partially moved to a discussion section.
7. In the results section, there is a sparsity of numerical metrics through quantitive evaluation. I think this should be worrying.