Stromal Tissue Segmentation in Multi-Stained Serial Histopathological Sections of Pancreatic Tumors

Authors: david  montalvo-garcía
juan e ortuño
ana d ramos-guerra
subhra s goswami
pablo  santiago diaz
maria  evangelina patriarca-amiano
lidia  estudillo
rosa  noguera
nuria  malats
maría j ledesma-carbayo
Venue: MICCAI_org_2024_Workshop_COMPAYL
Published: 2024-06-30 13:58:42

Abstract: In this work we propose and compare different deep learning algorithms for the segmentation of stromal regions in pancreatic histopathological image using three consecutive tissue sections, each uniquely stained with Hematoxylin and Eosin (H&E), Masson’s Trichrome, and Alcian Blue. After a non-rigid registration process, variations in tissue distribution between consecutive slides still persist, which leads to distinct desired segmentations of tissues for each stain, thus underscoring the need for a specific segmentation and co-segmentation approaches to achieve higher accuracy. We compare single stain models, with respect to multi-stain techniques that either consider the multiple stains all at once in training or are based on multi-branch siamese and co-segmentation techniques. We demonstrate superior performance in identifying stromal regions with the multi-stain approaches in comparison to the segmentation techniques applied to individual stains, by effectively utilizing the complementary information each staining technique provides. This advancement is poised to enhance the further evaluation of tumor microenvironment and stromal characteristics in patients with pancreatic cancer.

Main paper: 
 introduction: 
 Stroma, the supportive tissue surrounding neoplastic cells, is not merely a passive scaffold but actively participates in tumor progression and metastasis (Grünwald et al., 2021) . Accurate segmentation of stromal regions in tissue samples enables to quantitatively assess the tumor microenvironment, which is crucial for understanding tumor behavior, the interaction between cancer cells and the surrounding stroma, and the overall architecture of the tumor (Grünwald et al., 2021) .The quantification of stromal components varies markedly with the staining method used. Hematoxylin and Eosin (H&E), the most common stain used in histopathology, allows for the identification and quantification of general stromal features like cellularity and inflammatory elements, essential for assessing cell density and overall stromal morphology (Grünwald et al., 2021; Sanegre et al., 2020) . Other stains offers complementary information, like Masson's Trichrome, offering detailed insights into collagen fibers, enabling the quantification of their density, distribution, and orientation, which are crucial for understanding the stiffness and architecture of the tumor microenvironment (Sanegre et al., 2020) . Alcian Blue staining is key for detecting and quantifying glycosaminoglycans (GAGs) in the stroma, providing valuable information about the extracellular matrix composition and its potential influence on tumor invasion and metastasis (Sanegre et al., 2020) .This complementary analysis can be conducted by re-staining a single sample with several techniques, but this approach poses challenges, including potential alterations in tissue integrity and staining quality (Ozawa and Sakaue, 2020) . As a solution, staining consecutive sections with each technique is often preferred. Yet, this method introduces its own set of complications. Notably, tissue types and structure distribution changes across sections due to micrometer-scale differences which impede perfect registrations between different stains. Therefore, applying the same segmentation across various stains is not ideal. Instead, employing distinct segmentation masks for each stain is recommended to ensure precise delineation of tissue types across all stains, enhancing the reliability and validity of the analysis.The field of histopathology has seen significant advancements in the automatic segmentation of tissues, including stroma, using Deep Learning techniques, a development that has effectively circumvented the laborious task of manual labeling. This automation has primarily been focused on H&E stained samples, which dominate the published literature (Xu et al., 2016; Huang et al., 2017; Du et al., 2018; Al-Milaji et al., 2019) . While in theory, these models could be extended to other staining methods, there exists the notable gap in the specific developme of specific methods (Huang et al., 2017; Mei et al., 2020) , among other reasons due to the scarcity of labeled data and pre-trained models for stains other than H&E.In applying advanced models to stroma segmentation in consecutively multi-stained histological samples, two paths emerge: using a distinct model for each stain or a unified model trained on all stains, yet still segmented independently. In this context Generative and self supervised methodologies are typically used to perform domain adaptation (Mei et al., 2020; Liu et al., 2024) . However, there exists a third option: the co-segmentation approach, which merits consideration. Image co-segmentation refers to the process where two or more images, each containing shared objects of interest, are segmented jointly, ex-ploiting their interdependencies and synergies. These strategies have been mainly proposed and exploited in natural imaging (Chen et al., 2019; Li et al., 2019b) , and has been scarcely used in biomedical imaging (Zhong et al., 2019; Zou and Shi, 2021) .In this work we analyze the potential of different segmentation and co-segmentation approaches, which can be particularly relevant when analyzing sequential tissue sections stained with various dyes, such as H&E, Alcian Blue, and Masson's Trichrome, to comprehensively understand tissue structures and compositions in pancreatic cancer. 
data: 
 We compiled data from 39 patients across two different institutions, utilizing three consecutive tissue sections per patient stained with H&E, Masson's Trichrome, and Alcian Blue or Alcian Blue PAS (Periodic Acid-Schiff), each 4 microns thick. These sections were scanned at a resolution of 0.22 microns by pixel with a Zeiss scanner. Following the non-rigid registration for precise alignment described in Section 2.2, four expert pathologists selected 3 Regions Of Interest (ROIs) of 1x1 mm from each Whole Slide Images (WSIs) for every staining technique, resulting in a total of 351 ROIs.The first step in our methodology involves the non-rigid registration of WSIs. This process is crucial for aligning the images accurately, taking into account potential deformations and inconsistencies inherent in tissue sections. The registration algorithm is based on (Gatenbee et al., 2023) and adapts to the unique characteristics of each WSI, ensuring that the ROIs in consecutive slides are precisely aligned. This alignment is essential for accurate comparison and analysis in subsequent segmentation steps. Our approach carefully considers the microvariations in tissue structure and staining differences, ensuring that the registration is both robust and sensitive to the subtle details in the histopathological images.To address the task of stromal tissue segmentation in multi-stained serial histopathological sections, we have compared different state-of-the-art deep learning-based segmentation architectures, as well as custom architectures tailored to this problem, which incorporate the spatial information from the three serial sections stained with HE, Masson's Trichrome, and Alcian Blue or Alcian Blue PAS.nnUNet. The nnU-Net framework was used as our base configuration and model as the demonstrated state-of-the-art in semantic segmentation over other architectures as visual transformers (Isensee et al., 2021 (Isensee et al., , 2024)) . It adapts its configuration to each dataset's imaging modality and characteristics, considering image size, spacing, intensity properties, and hardware constraints, enhancing segmentation performance and generalization. To allow each branch the freedom to extract features specific to each stain and to construct each segmentation in the decoder, the first and last blocks of the branches are unique to the stain they represent (depicted in the Figure 1 with the representative colors of each stain). To ensure that the feature maps of the deepest blocks of the network learn features common to all three stains, certain blocks (marked in green in Figure 1 ) have been included following a Siamese network approach, sharing their weights between the different branches (Ji et al., 2022) . The number and position of shared blocks can be varied to adjust the degree of adaptation to different stains. The architecture has been designed to include a co-attention mechanism between branches, allowing them to share contextual information extracted from the feature maps of the three stains. Among the various coattention mechanisms available in the literature, the Co-Attention Recurrent Unit (CARU) (Li et al., 2019a) has been adapted for this problem. As shown in Figure 2 , the feature maps of the three branches are fed into the recurrent neural block in the physical order in which the sections of the three stains were taken. The final output of the recurrent network is then concatenated with each of the input feature maps. The recurrent block has a state feature map (G n ) that allows storing contextual information among the input feature maps. After noise removal with the reset gate (g z ), contextual information is extracted in the update gate (g d ) by combining spatial and channel attention maps using two parallel attention modules (Woo et al., 2018) .  
experiments and results: 
 Of the 39 available patients, 25 were selected for training, with 5 of these used to validate the different proposed techniques, resulting in 75 ROIs for training and 15 for validation.The remaining 14 patients with a total of 42 ROIs were reserved as an independent test set. As a first approach, an nnUNet model has been trained for each stain, encompassing the three models under the label nnUNet singleStain. During inference, the corresponding model is chosen from the three available models depending on the stain to be segmented. As an alternative to using three independent models, one per stain, a single nnUNet model was trained with all stains mixed together, without discerning during training which stain was received as input. This model, labeled nnUNet multiStain, showed improvements when evaluated on the validation set (see Table S1 of the supplementary material). Model hyperparameters, such as input image resolution, patch size, loss function, and network architecture, were optimized through initial experiments in the validation set. Dice and Cross Entropy was the best performing Loss function. Data augmentation techniques like rotations, mirroring Gaussian noise, Gaussian blur, brightness, contrast and gamma corrections were applied during training. The models have been trained for 400 epochs, with one epoch being defined as iteration over 250 mini-batches. Different image resolutions have been tested, and the best results have been obtained by taking patches of 1024 and halving the resolution to train the models with final patches of 512. For inference, the standard nnUNet sliding windowing procedure has been used, with window size equal to the training patch size and a 50% overlap, weighting the predictions by a Gaussian, increasing the weight of the predictions of the central patch pixels. Test time augmentation by mirroring along all axes are also applied. The models were trained on a workstation with an NVIDIA RTX A6000 (48 GB) GPU.Regarding the multi-staining models, two alternatives have been implemented: a base model without any co-attention mechanism, with shared weights between the three stain branches in the central blocks (as displayed in Figure 1 ), labeled as Siamese UNet, and a second model analogous to the previous one, but adding the CARU block as a co-attention mechanism after the blocks with shared weights, labeled as CARU UNet. These models have been trained with the same hyperparameters and using the same data augmentation and inference techniques detailed earlier, with three exceptions: during each epoch, all training images pass through the network; a histopathological-specific data augmentation technique has been added (Shen et al., 2022) ; and a further post-processing step was applied to the model output to improve the definition of the stroma tissue border, based on the morphological operations of opening and closing.A model analogous to the nnUNet multiStain model has been trained to ensure fair comparison of the multi-stain architectures with a base architecture as some small details of nnU-Net are not implemented (such as deep supervision or oversampling techniques). This model, labeled as UNet multiStain, was used to pre-train the multi-stain models to account for the increased complexity and number of trainable parameters limited by the size of the training sample. This pre-trained model was used to initialize the weights of two new multi-stain models, on which fine tuning was applied to train the weights but using the co-registered input of the three stains: a first model based on Siamese nets, with all blocks shared between stains and no co-attention mechanism, labeled as Siamese UNet FT ; and a second model, analogous to the previous one but with a CARU-based co-attention mechanism, keeping all the blocks shared between stains except for the block following the co-attention mechanism (since this block varies its structure as it must be able to receive as input the concatenation of the CARU output with the previous attention map of the stain). This second model has been labeled as CARU UNet FT.To measure the algorithm's performance for each model we used the Dice score and the Intersection over Union metrics of the segmented stromal areas vs the ground truth. This metrics were assessed for all the ROIs selected by the pathologists in the validation (Table S1 of the supplementary material) and the testing datasets (Table 1 ). The metrics were computed with respect to the ground truth of each stain independently ("Area per Stain" in Tables S1 and 1 ) as well as comparing the common stromal area among the three stains in ground truth vs the corresponding resulting area for each of the models ("Common Stromal Area" in Tables S1 and 1 ). Additionally we computed Dice score and IoU for the different stains in the testing set to assess the performance of the different models independently for each stain (Table 3 and Figure 3 ).  
discussion: 
 Results show that Muli-Stain approaches including the nnU-Net trained with all the stains perform better than the single strain approaches. Comparison of Multi-Stain methods in this data resulted in a challenging task given the good performance of all the methods and the known limitations of integrative metrics such as Dice and IoU metrics (Maier-Hein et al., 2024) . Given the multiple subregions nature of stromal segmentation, distance based metrics were not considered as they could be significantly affected by segmentations that exclude small subregions, that may not be clinically relevant. The multi-stain architectures based on three branches present slightly better Dice and IoU than the base architecture UNet multiStain and show less standard deviation of the metrics. Specifically the CARU UNet FT scheme is the one with higher metrics in the testing dataset showing the contribution of pre-training and co-attention mechanisms in the context of multi-stain serial segmentation. These results are also shown in the analysis per stain type where CARU UNet FT also improves the base architecture UNet multiStain for Masson Trichorme and Alcian Blue based stains. In Figure 3 , the top ROI shows comparable results between models, and the bottom ROI shows how CARU improves segmentation. The nnUNet multiStain and the UNet multiStain schemes show very good performance and could be an option easy to implement for multi stain studies with very good performance.In this study, we have explored different multi-stain methods for segmenting consecutive regions in histopathological samples stained with different dyes. Multi-stain approaches improved single stain approaches specifically for all the stains and more notably for Alcian Blue. The variation in tissue characteristics between slides presents challenges to balance the imposed consistency vs the independent segmentations of serial slides. Some trends in the contribution of the co-segmentation techniques are observed, however metrics that allow to measure the specific gains in multi-stain studies require further research to better assess these gains and the setups in which they would be more suitable. Schemes based on self-supervised techniques and foundational models could enable the extraction of the most meaningful features from the different stains (Chen et al., 2024) .  
compliance with ethical standards: 
 This research study was conducted retrospectively using anonymised patient data following ethical approval by the Ethics Committees of the contributing institutions and CiberONC. 
 
 Appendix:  



Reviews: review: title: title: Review of submission 25
main_review: main_review: Summary: 

The paper investigates deep learning approaches for stromal segmentation in pancreatic histopathology. It compares various models using three consecutively tissue sections with different stains (H&E, Masson's Trichrome, Alcian Blue) after non-rigid registration with VALIS. The authors propose a novel multi-branch architecture with a co-attention mechanism for multi-stain co-segmentation. Additionally, they evaluate variants like UNet vs. nnUNet, pre-training strategies, and Siamese vs. co-attention methods.


Pros:
- The paper presents an interesting approach for co-segmentation of multi-stained tissue sections.
- The paper is well written, with a clear organization.
- The proposal of a novel multi-branch architecture with a co-attention mechanism is a valuable contribution.
- The evaluation includes experiments with different model variants (UNet vs. nnUNet, pre-training, Siamese vs. co-attention).

Cons:
- While the authors claim superior performance for multi-stain approaches, Table 1 results for the test set show minimal differences between methods, particularly for nnUNet_singleStain  which differs from the best performer by 0.003. Figure 3 examples also exhibit minimal segmentation mask variations across models.
- Statistical tests are needed to assess significant differences between models.
- The dataset size appears limited. Evaluating model behavior with more training data would be beneficial.
- The models are trained using ground truth masks generated independently for the three stains. It would be interesting to conduct some experiments to check if a single ground truth mask (for example for the HE image) could be used to co-segment stroma tissue in the three sections after the registration stage.
title: title: Small performance / variance gain with shared encoder
main_review: main_review: In general sharing an encoder to solve related tasks can lead to learning a better representation. The performance gain however is quite small. Since the dataset size is also modest, the performance delta could be due to the fact that the multistain model simply sees (significantly) more unique data. That said, the contribution is straightforward and makes sense. The argument that sharing an encoder among related tasks generally improves performance is also mostly uncontroversial.
title: title: good methodological comparison of several methods for stromal segmentaiton
main_review: main_review: In this paper, the authors compared different deep learning algorithms for the segmentation of stromal regions in pancreatic histopathological image using three consecutive tissue sections, stained with H&E, MT, and AB.

Pros:
1. This is an important work on comparing different segmentation models; 
2. The clinically relevant task of quantifying stromal tissue is explored;
3. Addressing a challenging task of multi-stain slides and methods to utilise the complimentary information in the stainings.

Cons:

It is hard to see any statistical difference between the compared approaches so it's hard to make a conclusion that a particular approach is better than others, especially considering that the comparison was performed only for a specific dataset and a specific cancer type, only for a single scanner. Nevertheless, it's an important step towards building knowledge through comparison studies like this.

Also, it'd be beneficial to mention the computational cost of the chosen approaches, including the number of parameters in the models, training times, as well as the inference times/costs per image/dataset.

Finally, it would be great, if possible, to release the dataset that could greatly benefit the research community for further comparison studies especially self-supervised approaches, for example.