Lymphocytes subtyping on H&E slides with automatic labelling through same-tissue stained ImmunoFluorescence images

Authors: etienne  pochet
luis  cano
roger  trullo
Venue: MICCAI_org_2024_Workshop_COMPAYL
Published: 2024-06-22 12:38:07

Abstract: Accurate identification and classification of immune cells within tissue samples are critical for understanding disease mechanisms and predicting treatment responses as a cornerstone for personalized medicine. Traditional histopathology relies on hematoxylin and eosin (H&E) staining, which provides structural context but lacks specificity for immune cell sub-types, preventing pathologists from more precise identification. In contrast, immunofluorescent (IF) staining enables precise targeting of specific markers, but this recently developed technology is very costly and not widely applied in clinical practice yet.
In this work, we propose a method to leverage registered pairs of H&E and IF stained images from the same tissue to automatically generate cell type labels for H&E from IF marker expression, allowing for precise identification. In particular, we demonstrate the feasibility of lymphocyte sub-typing from H&E images by training cell-level classifiers to accurately distinguish T-cells subtypes (CD45 / CD3e / CD4 / CD8a).

Main paper: 
 related works: 
 In recent years, significant progress has been made in the classification of immune cell types using H&E stained images. Nuclei segmentation in WSI has been extensively studied Hayakawa et al. (2021) . Models like HoverNet Graham et al. (2019) and StarDist Schmidt et al. (2018) have achieved high performance, being widely used in numerous applications. Kumar et al. (2017) and PanNuke Gamper et al. (2019) provide classification annotations for multiple cell types and nuclei segmentation masks. These datasets allow segmentation models to specialize in cell classification on H&E, with available labels including broad families like epithelial cells, lymphocytes, macrophages, and neutrophils for MoNuSAC, and inflammatory, dead, non-neoplastic, neoplastic, connective for PanNuke. These categories can be reliably distinguished by trained human experts when examining H&E stained samples, as they exhibit distinct morphological characteristics Neumann and Neumann (2021) .Few works have enriched H&E cell classification models with cell-level labels from paired IF images. Reddy et al. Reddy et al. (2022) demonstrated lymphocyte prediction in colorectal cancer (CRC) patients using algorithmically generated annotations from paired H&E and IF images. Similarly, Wee et al. (2023) identified specific sub-types of CD8+ cells on H&E images using labels from co-registered IF images, applied to a private dataset of mouse tissue samples.Our work innovates by identifying and classifying fine-grained lymphocyte sub-types directly from routine H&E stained images, leveraging a large publicly available dataset. This is achieved through a computational approach that generates granular single-cell training labels using paired multimodal data from H&E and IF imaging. Previous efforts focused on broader immune cell categorizations discernible by human observers based on H&E morphology or suggested lymphocyte subtyping potential using proprietary datasets. In contrast, our methodology studies the ability to automatically estimate lymphocyte subsets, which is not feasible through manual annotation. 
dataset: 
 We use the data shared by Lin et al. (2023) , who introduced a new approach for highplex IF and H&E imaging of the same tissue sample. Their platform, ORION, enables the simultaneous capture of 16-18 markers, surpassing the typical clinical research standard of five or six plex. ORION facilitates high-quality IF and H&E imaging, validated through rigorous qualitative and quantitative assessments. The authors shared a dataset of 41 image pairs (H&E and IF) with a resolution of 0.325 microns per pixel from 40 colorectal cancer (CRC) patients. The dataset includes 16 markers relevant to CRC (e.g., CD45, CD3e, CD4, CD8a, CD68, Pan-CK). They demonstrated their approach's potential by using cell-level information from both modalities to model progression-free survival. They also released initial pre-processing results, including image registration between IF and H&E and cell segmentation based on the nuclei marker on IF. An example of the imaging data is shown in Figure 1 and interactive visualizations of all 41 image pairs can be found at https://www.tissue-atlas.org/atlas-datasets/lin-chen-campton-2023/.We leverage the IF expressions to generate labels for the H&E images. Visual inspection showed that the registration of both images did not reach pixel-level alignment. Hence, we When analyzing IF images, pathologists visually classify cells based on their marker expressions, effectively applying a thresholding algorithm manually. We hypothesize that this workload can be greatly reduced for the annotator and that many of those cell annotations could be obtained automatically. Instead of selecting a single binary threshold as it is commonly done, we chose to define multiple ranges, to allow for cells to be labelled as positive, negative or unknown, In particular, we define Negative range: include expressions low enough that cells can be considered negatives with high confidence. In this range, marker expression is of the same order of magnitude than the background noise; Positive range: expression is high enough that cells can be considered positives with high confidence; Uncertainty range: Cell expression is in between the background noise and the clearly positive cells. Cells with marker expression in this range are not labelled by the algorithm; Over exposure range: cell expression is too high to say with confidence that it is not due to staining artefacts. Cells with expression above this value are not labelled.Trained pathologists can distinguish cells more accurately by considering local context. However, algorithmically labeling extensive data involves trade-offs. To minimize mislabeling risk, we chose not to label all cells, setting cautious thresholds to enhance dataset signal-to-noise ratio. Whole Slide Images (WSIs) contain millions of cells; labeling even a fraction yields numerous annotations. In this work, an anatomical pathologist established marker thresholds, enabling semi-automatic labeling of image datasets with minimal human intervention. An example of these thresholds is shown in Fig. 2 One of the objectives of our work is to facilitate the analysis of cell populations that contribute to providing data on disease prognosis. Various studies have been published, demonstrating how the population of CD3e+ and CD8a+ cells play an important role in determining the prognosis of various types of tumors Kasurinen et al. (2022) ; Zeng et al. (2022) . Additionally, the creation of immune scores with these subgroups of cell populations is increasingly advocated in different solid tumors as well as in new guidelines for personalized patient treatment Trabelsi et al. (2019) ; Panahi et al. (2023) . Therefore, we decided to focus on T-cells to demonstrate sub-type classification feasibility from H&E images. These sub-types follow a hierarchical structure Lin et al. (2023) . In particular, CD4 and CD8a cells are a sub-type of CD3e cells, which are a sub-type of CD45 cells. To further increase the quality of annotations, we curate the labels we generate by filtering out cells that do not match with those rules i.e., allow cells to be labeled as CD3e positive if they are CD45 negative.We applied the gating and filtering technique to all 41 available image pairs, resulting in the labeling of numerous cells. The analysis revealed a significant imbalance, particularly in the distribution of CD8a and CD4 labels within the dataset, with each representing a very small proportion of positive samples. This disparity is expected due to their characterization as specialized sub-types, which are less prevalent in tissue samples. Specifically, the positive to negative ratios for CD45, CD3e, CD4, and CD8a markers are approximately 0.22, 0.04, 0.006, and 0.006 respectively. These ratios highlight the challenges faced in a task that is already unachievable for trained humans.We study this cell sub-typing problem as an image classification task. To classify a given cell, we crop a patch around its centroid coordinates and feed it to an image classifier network. We leverage state-of-the-art (SOTA) pre-trained backbones as feature extractor, and complement them with Multi-Layer Perceptron (MLP) classification head. As we are trying to predict multiple classes that are not mutually exclusive, we can not use a single model trained with multi-class cross-entropy. The simplest formulation of the problem is to train independent binary classifiers for each one of the target types (left Fig. 3 ). These models are trained with binary cross-entropy loss (BCE). We experiment with convolutional neural networks (CNNs) backbones, in ConvNext Liu et al. (2022) and HoverNet. The latter is the standard model in cell segmentation for H&E images. It is composed with a U-Net Ronneberger et al. (2015) feature extractor and pixel-level prediction heads. Therefore, we leverage the features generated by the U-Net and and replace the pixel classification heads with an average pooling followed by the MLP head. We also experiment with Vision Transformers (ViT) Dosovitskiy et al. (2020) , using the Phikon weights Filiot et al. (2023) . This very recent model has been trained with Self-Supervised Learning (SSL) on thousands of H&E WSIs. It represents a state-of-the-art foundation model for pathology images. The cell sub-types that we selected for this work are closely related, as they are part of the T-cell category. Hence, we hypothesize that joint training on the multiple classes would be beneficial. Therefore, we also experimented with a multi-task learning formulation. In this case, we train models with a shared backbone, in hope that common features can be used to enhance the sub-typing. The common backbone generates the patches features, which are fed to independent classification heads (right Fig. 3 ). Each marker prediction is evaluated with a BCE loss. The total loss is a combination of equally weighted losses of each markerL = 1 4 (L CD45 + L CD3e + L CD4 + L CD8a )4 Experiments 
experimental setup: 
 These experiments are run on a single 16GB V100 GPU. We used AdamW Loshchilov and Hutter (2017) optimizer with a learning rate of 5e-5 and weight decay of 1e-5. unbalanced nature of the sub-typing classification task, we decided to resample the training and validation sets to ensure a minimum ratio of positive cells for each markers. In practice, we sampled 20,000 cells for each patient. Half of them are not positive to any markers. In the other half, we make sure that there is at least 1 4 of positive cells for each of the 4 markers. For the testing patients, we randomly sample 20,000 cells for each patient to match with the real distribution of labels.To determine an appropriate patch size, we built a binary classifier of CD45 cells using the ConvNeXt backbone, initial experiments were conducted with various input patch sizes. Results showed that AUROC values improved with increasing patch size until 112 pixels, reaching a peak AUROC of 0.96. Beyond this size, AUROC values remained stable at 0.95 for a patch size of 96 pixels, and gradually decreased to 0.92 for a patch size of 32 pixels. Notably, a patch size of 112 pixels corresponds to approximately 36µm in the images, encompassing a neighborhood of about 3-4 cells around the central cell of interest, which aligns well with the typical size of T-cells (10µm diameter) Lin et al. (2023) . This patch size provides a balance between capturing sufficient local context and maintaining focus on the central cell information. All following results are obtained with a patch size of 112 pixels.We explore various models to subtype T-cells, comparing state-of-the-art backbones: Con-vNeXt Li et al. (2022 ), HoverNet Graham et al. (2019) , standard instance segmentation models using the U-Net extractor as described in Sec. 3.3.1, and Phikon, a ViT model pretrained on pathology images. Additionally, we contrast training independent models for each marker with independent backbone tuning against a single multi-task model using a shared feature extractor and independent classification heads. We also mitigate the impact of high class imbalance by weighting positive samples in the BCE loss, with weights set inversely proportional to the ratio of positive samples in the training set for each marker.Given the sub-type hierarchy, predicting broader categories like CD45 from H&E is expected to be feasible. CD3e, representing a wider T-cell sub-type, may also be predictable. However, identifying finer sub-types such as CD4 and CD8a is highly challenging due to their scarcity in the dataset. Results in Table 1 confirm this pattern: models achieve better accuracy for CD45 and CD3e compared to CD4 or CD8a. Performance aligns with sub-type hierarchy, with CD45 showing the highest discrimination, followed by CD3e, and the least for CD4 or CD8a.We observe that CNN-based models consistently outperform ViT models across all markers, indicating convolutional layers better suitability for fine precision tasks and robustness with fewer positive examples. Despite having half the parameters, HoverNet competes well with ConvNeXt, likely benefiting from pre-training on H&E cell segmentation. Moreover, jointly trained models using a shared backbone slightly outperform independently trained counterparts. For instance, with the HoverNet backbone, AUROC increases from 0.91 to 0.94 and BAcc from 0.83 to 0.86 for the CD3e task. This suggests potential in leveraging multi-task learning to enhance training efficiency, despite modest improvements that highlight room for further enhancement in exploiting class similarities.Balancing category weights in the Binary Cross Entropy loss significantly impacts performance. This adjustment has limited impact on CD45 and CD3 tasks due to their lower dataset imbalance. However, for CD4 and CD8, using weighted BCE dramatically improves balanced accuracy: from 0.5 to 0.86 for CD4 and 0.53 to 0.80 for CD8 with multi-task Hov-erNet. A balanced accuracy of 0.5 indicates a model assigning the same label to all samples (TPR=0, TNR=1 or vice versa). Weighted BCE effectively enhances learning by emphasizing rare examples. The high imbalance challenges evaluation, where even minor error rates can drastically affect real-world performance. Addressing the trade-off between False Positive and False Negative examples depends heavily on the application. For instance, a tool identifying regions of interest with higher marker expression may prioritize fewer false negatives over more false positives.Fig. 4 visualizes two regions for all markers using the multi-task HoverNet trained with weighted BCE, offering insights into the model behavior. High accuracy is observed for CD45 prediction, while CD3e predictions show good correlation with marker expression despite numerous false positives. CD4 and CD8 exhibit high recall with visible correlation, yet with considerable false positives due to their rarity. Post-processing, as described in Section 3.2.2, could potentially refine these predictions by enforcing alignment with T-cell sub-type hierarchy. 
conclusion: 
 This paper presents a methodology for cell classification in H&E images across a broader range of cell types than previously explored in the literature. By leveraging paired H&E and IF images, we semi-automatically generate a large volume of cell-level annotations, ensuring high quality through pathologist input limited to three slide-level thresholds per image pair. Using this algorithmically generated dataset, we train image classifiers to distinguish cells expressing specific markers. We demonstrate promising results in sub-typing T-cells on H&E images, where morphological variations are imperceptible to a trained pathologist. While our methodological approach is simple, it highlights the feasibility of this task. 
: 
 1 Introduction Histopathological analysis relies on H&E stained whole slide images (WSI) as the gold standard for visualizing tissue structures Gurcan et al. (2009) . Deep learning research has made great advances in this domain Dimitriou et al. (2019) ; Komura and Ishikawa (2018) ; Tizhoosh and Pantanowitz (2018) , particularly in cell segmentation Schmidt et al. (2018) ; Stringer et al. (2021) where recent developments have enabled multiple downstream tasks in digital pathology for diagnosis and prognosis Saltz et al. (2018) ; Echle et al. (2021) ; Angell et al. (2020) .While H&E staining is very effective for some tasks, it lacks the discriminatory power needed for precise cell type identification beyond general categories. This limitation makes challenging accurate ground truth labeling and the development of specialized cell classification models. On the other hand, advanced machine learning models may exploit humanimperceptible patterns in H&E images to achieve finer-grained cell type classification.Accurate identification of precise cell types in H&E images would streamline histopathological workflows, reducing reliance on costly and time-consuming alternatives like Immunohistochemistry (IHC) and IF Duraiyan et al. (2012) . IF imaging offers higher precision and multiplexing capabilities but typically is applied to consecutive cuts which makes cell level alignment with H&E not possible.In this work, we propose a methodology leveraging a publicly available H&E and IF stained dataset Lin et al. (2023) on same tissue images, enabling semi-automatic generation of cell phenotype labels. Our approach facilitates training classifiers for cell types not extensively studied in literature, such as T-cell lymphocyte sub-types (e.g., CD45-CD3e-CD4-CD8a), critical in immune function and disease. Our work demonstrates feasibility in lymphocyte sub-typing on H&E slides using convolutional neural networks and vision transformers, representing a novel contribution to the field. 
 
 Appendix:  



Reviews: review: title: title: Review of Lymphocytes subtyping on H&E slides with automatic labelling through same-tissue stained ImmunoFluorescence images
main_review: main_review: The paper presents evaluation of how well DL can detect different subtypes of T-cells on H&E stained WSIs. The method involves using immunofluorescence (IF) images to produce semi-automatic labels for the four subtypes of T-cells (CD45, CD3e, CD4, and CD8a) which would be challenging to do for a pathologist. However, the step involves a pathologist to manually select threshold on IF images for the 41 WSI and IF pairs used in the study, hence the generalisation of the results is uncertain.  Furthermore, this setup brings a question of how reliable are the semi-automatic annotations since the evaluation is done on the same semi-automatically prepared ground truth labels.

Three algorithms are tested with two different regimes: a binary classification and multi task learning.  The results indicate that transformer performs worse than CNN methods which is not surprising given the low number of image pairs in the study. The contributions of the paper include:
* evaluating several cell classification setups, 
* using IF for semi-automatic labelling,
* classifying more subtypes of T-cells  than previously. However the results for additional subtypes are quite poor. 

Overall, the paper is well written and easy to follow.
title: title: Timely use of IF to predict HE but novelty questionnable
main_review: main_review: This paper investigates the feasibility of predicting lymphocyte populations (CD45, CD3, CD4 and CD8) on H&E images, using multiplexed immunofluorescence images, as a source. 41 images of colorectal cancer from Lin et al are used. Authors employ a multi-task learning approach to perform joint training on multiple lymphocyte classes, compare patch sizes, test CNN and ViT and compare model performance on all cell subtypes. It is also unclear from Figure 1 whether all CD3+ are CD45+ as well (which must be the case). There may be an error in the figure legend with reference to the colors of the cell types. Since CD45 + cells include T-cells and B-cells, the more critical distinction to make is between these 2 classes first, before subtyping T-cells further. It is well-known that lymphocytes can be detected accurately from H&E images, and in colorectal cancers, T-cells are the predominant lymphocyte type found in proximity to the cancer, so the high accuracy for CD3+ cell subtyping explains these observations.  What the cytological aspects are that may help to distinguish CD4 and CD8 from H&E are not hypothesized in this paper.
title: title: Review for Etienne Pochet
main_review: main_review: Quality:
The quality of the work is overall good.
Pros:
The results obtained from semi-automatically sub-typing T-cells on H&E images are promising.
Significant potential impact on personalized medicine and digital pathology  
Cons:
Immediate clinical applicability is not fully demonstrated.
Not very well explained why ViT with higher performance in other domains shows lower performance in this work against the two CNNs.

Clarity:
The clarity of the work is overall okay.
Pros:
The dataset, models, and hyperparameters used are clearly explained.
Cons.
The acronyms are not explained in the text. For example: MTL seems important as it is used in the results table

Originality.
The originality of the work is very good as it presents a novel integration of H&E and IF images using state-of-the-art models.

Significance:
Overall, the significance of this work lies in its potential to transform diagnostic pathology practices, contribute to the advancement of personalized medicine, and improve accessibility to advanced immune profiling techniques.