HoVer-NeXt: A Fast Nuclei Segmentation and Classification Pipeline for Next Generation Histopathology

Authors: elias  baumann
bastian  dislich
josef  lorenz rumberger
iris d nagtegaal
maría  rodríguez martínez
inti  zlobec
b  baumann
j l dislich
i d rumberger
m r nagtegaal
i  martínez
  zlobec
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 17:56:02

Abstract: In cancer, a variety of cell types, along with their local density and spatial organization within tissues, play a key role in driving cancer progression and modulating patient outcomes. At the basis of cancer diagnosis is the histopathological assessment of tissues, stained by hematoxylin & eosin (H&E), which gives the nuclei of cells a dark purple appearance, making them particularly distinguishable and quantifiable. The identification of individual nuclei, whether in a proliferating (mitosis) or resting state, and their further phenotyping (e.g. immune cells) is the foundation on which histopathology images can be used for further investigations into cellular interaction, prognosis or response prediction. To this end, we develop a H&E based nuclei segmentation and classification model that is both fast (1.8s/mm2 at 0.5mpp, 3.2s/mm2 at 0.25mpp) and accurate (0.84 binary F1, 0.758 mean balanced Accuracy) which allows us to investigate the cellular composition of large-scale colorectal cancer (CRC) cohorts. We extend the publicly available Lizard CRC nuclei dataset with a mitosis class and publish further validation data for the rarest classes: mitosis and eosinophils. Moreover, our pipeline is 5× faster than the CellViT pipeline, 17× faster than the HoVer-Net pipeline, and performs competitively on the PanNuke pan-cancer nuclei dataset (47.7 mPQTiss, +3% over HoVer-Net). Our work paves the way towards extensive single-cell information directly from H&E slides, leading to a quantitative view of whole slide images. Code, model weights as well as all additional training and validation data, are publicly available on github.

Main paper: 
 introduction: 
 Histopathological assessment of tissues is a cornerstone for the diagnosis and prognosis of diseases, including cancer. Among the most frequent and deadly is colorectal cancer (CRC), for which the overall 5-year survival rate is only around 65% (Siegel et al., 2023) . Tissue biomarkers play a crucial role in improving prognostication and designing more personalized treatments for individual patients. In recent years, deep neural networks have shown promising results in biomarker prediction (Kather et al., 2019; Bulten et al., 2019) and discovery (Zheng et al., 2023) , as well as other tasks such as segmentation and classification of biological structures (Ronneberger et al., 2015; Havaei et al., 2015) . One of the emerging tasks is the identification, classification, and segmentation (i.e. panoptic segmentation) of cells or more commonly their nuclei directly on routine diagnostic slides (Gamper et al., 2019; Graham et al., 2021a) . Most cell types can only be differentiated on H&E images at high magnifications (i.e. 20x or 40x), because identification is based on nuclear morphology, texture and tissue context. However, Whole Slide Images (WSI) at high magnification are large with sizes above 100000 × 100000 pixels, and inference runtimes of currently available models make it infeasible to run them for clinical routine or investigations on large-scale cohorts. Methods for panoptic segmentation include HoVer-Net (Graham et al., 2018) , which uses an encoder-decoder architecture with three decoders, one for semantic and two for watershed-based instance segmentation. In comparison, the panoptic segmentation version of StarDist (Weigert and Schmidt, 2022 ) has a similar architecture, but only one decoder for instance segmentation which predicts star-shaped polygon mask proposals for the individual cells and post-processes them to instances by means of a non-maximum suppression algorithm. CellViT (Hörst et al., 2023) then further improves state-of-the-art by using a SAM (Kirillov et al., 2023) encoder combined with HoVer-Nets decoders. On the other hand, Tommasino et al. (2023) propose a simplified HoVer-UNet for 3× speedup over HoVer-Net. Recently, the CoNiC challenge tried to find new best practices in nuclei segmentation and classification (Graham et al., 2023 (Graham et al., , 2021b)) . They found that the top three submissions, which included our own, are based on newer encoders such as EfficientNet-v2 (Tan and Le, 2021) , and tackled class imbalance via class distribution-based importance sampling and loss weighting (Weigert and Schmidt, 2022; Rumberger et al., 2022; Zhang and Zhang, 2022) . The CoNiC challenge dataset (Lizard) (Graham et al., 2021a) , which is based on H&E CRC images at ∼0.5mpp, includes six classes: lymphocytes, neutrophils, plasma cells, eosinophils, epithelial cells and connective-tissue cells. The post-challenge analysis Graham et al. (2023) , demonstrates the value of the dataset and such methods by successfully applying them to tumor grading and patient survival prediction tasks. However, Lizard does not consider mitoses as separate objects of interest. Rather, they are classified as epithelium, lymphocyte, neutrophil, or not at all (c.f. Figure 1 D ). Mitoses are indicators of the proliferative activity of tumors and have an impact on treatment decisions for e.g. breast or pancreatic neuroendocrine tumors ( Lukasiewicz et al., 2021; Kim and Hong, 2016) . In Summary, long inference times of publicly available models, incompatible with large-scale investigations, coupled with the absence of biologically relevant class annotations such as mitosis, motivate the subsequent contributions:1. We developed HoVer-NeXt (HN), an updated model based on our CoNiC challenge submission, which retains high performance on the Lizard cell types while also predicting mitoses.2. We provide an additional mitosis training dataset, modify Lizard to include mitosis, and publish both together with additional validation sets for mitoses and eosinophils.3. We provide a model trained on the PanNuke (Gamper et al., 2019) pan-cancer panoptic segmentation dataset, which shows competitive performance.4. We construct a highly efficient WSI inference pipeline, which achieves a 17× speedup over the HoVer-Net pipeline and a 5× speedup over the CellViT pipeline on whole slide inference.Code for training, inference, weights for all models as well as links to data can be found here: github.com/digitalpathologybern/hover next train, and here /hover next inference.  
summary of the conic 2022 submission: 
 Starting from the HoVer-Net model setup, we propose several simplifications and extensions. Firstly, we combine the two instance segmentation decoders into a single decoder. The binary nuclei segmentation map is replaced by a 3-class nuclei boundary, nuclei center, background prediction map (BCB-map), which showed improved results in other modalities (Caicedo et al., 2018) , and can directly be used for watershed-based instance segmentation, reducing the need for additional post-processing steps. The HoVet-Net HV maps or center-point vectors are thus only used as an auxiliary task (Hirsch and Kainmueller, 2020) .As the architecture, we use a U-Net (Ronneberger et al., 2015) with an EfficientNet-V2 encoder (Tan and Le, 2021) . To tackle the class imbalance in Lizard, we employ classbased importance sampling using per-pixel class statistics as weights and use focal loss with class weighting proportional to the inverse of the exponential moving average class prior (Araslanov and Roth, 2021) . Finally, model outputs are post-processed with class-specific hyperparameters, as the classes differ in average object size and shape. For more details we refer to our previous publication for the CoNiC Challenge (Rumberger et al., 2022) . While the setup and processing steps were feasible within the scope of optimizing for the challenge metrics, they do not scale well to WSI. Therefore, we optimize the model and embed it in a pipeline for efficient WSI inference.We updated the model with a ConvNeXt-v2 (Woo et al., 2023) encoder, which shows competitive results on a variety of benchmarks (Roy et al., 2023) . ConvNeXt-v2 uses a larger pooling operation which we accommodate for by adding an additional upsampling step to maintain the same U-Net depth. In our experiments, we use ConvNeXt-v2 Tiny, Base and Large. We further simplify the model by replacing class based loss weighting with a standard focal loss (Lin et al., 2017) , since data sampling is already sufficient to treat the label imbalance (Ablation: See Supp. C.2). A convex-hull-based post-processing step is also removed as individual convex hull computations are computationally expensive. Tilebased normalization leads to artifacts in out-of-distribution tiles and is removed as well (See Supp. Figure 6 ). The training setup can be found in Supp. A.1. Beyond these changes, we setup an easy-to-use WSI inference pipeline.Relevant foreground area on the WSI is first identified using a threshold on the gray scale representation of the WSI thumbnail (Details: Supp. A.3). The model then processes tiles with overlap (8px/0.5mpp, 16px/0.25mpp) with classmap and BCB-map being compressed and stored on disk. For test-time-augmentations (TTAs), we only include HED color augmentation, mirroring and 90°rotation to avoid negative effects of augmentations (Details: Supp. A.4, Augmentation Parameters: Supp. C.1). Tiles are center-cropped and stitched to large regions for parallel processing. Then, based on individual class thresholds, foreground area and seed points are generated in the BCB map and then processed with a watershed algorithm to get nuclei instances. Small holes in instances are removed and false merges are resolved. Classes are assigned based on majority vote and instances are filtered based on class-specific size thresholds, which are determined via hyperparameter search on the validation set. Finally, overlaps between large ROIs are resolved (Details: Supp. A.2).Optimizing for speed: To further optimize the inference time, storage, and memory requirements of the pipeline, the model runs at half-precision and the class output is quantized by mapping softmax outputs to values between 0-255. Outputs are written to disk without post-processing by a separate process to ensure high GPU utilization. Data is stored as LZ4 compressed Zarr arrays, allowing for fast compression and concurrent writes and reads. Finally, the pre-stitching of raw inference tiles to large regions allows us to avoid resolving large numbers of overlaps, retain the option to parallelize watershed, and keep the peak memory consumption low.Lizard and PanNuke Lizard and PanNuke are publicly available H&E panoptic nuclei segmentation datasets, one CRC specific and one pan-cancer (More details: Supp. A.5).To be able to compare HoVer-NeXt to our own challenge submission, we use the same 80% train, 20% validation split with the "GlaS" subset as an out-of-distribution test set.Conclusions drawn in the challenge evaluation (Graham et al., 2023) are therefore likely to be translatable to our new models. To compare our model with HoVer-UNet (Tommasino et al., 2023) , CellViT (Hörst et al., 2023) as well as HoVer-Net (Graham et al., 2018) , we also include the PanNuke dataset as a benchmark.We create our own dataset specifically for mitoses and extend the Lizard dataset with mitoses. To achieve this, we select 48 ROIs (8192 × 8192px) from 11 H&E stained CRC WSI, create a mitosis specific pHH3 immunohistochemistry restain,register the images and generate ground truth by thresholds on the stain deconvolved DAB channel (see Supp. A.6). To generate panoptic segmentation labels for this dataset beyond mitoses, as well as re-labeling Lizard for mitoses, we adapt a self-training routine proposed by Yang et al. (2021) (see Supp. A.7).Further validation: MitEval and EosEval Additionally, we create two holdout test sets, one with eosinophils manually annotated by a board-certified pathologist in 11 ROIs of CRC resection WSI respectively, and one with 3 board-certified pathologists annotating 13 ROIs for mitosis (Supp. A.8). For both datasets, we report WSI-level performance.Foucart et al. (2023) show that panoptic quality should be avoided for the evaluation of nuclei segmentation and classification. Panoptic quality is defined as the product of the detection F1 score and Intersection over Union (IoU) of true positives. However, the small size of nuclei makes IoU too sensitive for coarse annotations. Moreover, the aggregation of IoU and F1 score incentivizes not detecting an instance at all over misclassifying it. We therefore employ their guidelines, yet also report panoptic quality for comparison. For binary segmentation, we use F1-Score and Matthews correlation coefficient (MCC). For detection, we use distance-based matching (6µm@0.5mpp, 12µm@0.25mpp (Sirinukunwattana et al., 2016) ) and evaluate the detections using balanced accuracy and F1 Score. Detection metrics for Lizard are evaluated on 248 × 248 center crops to avoid having to detect nuclei with their center outside of the tile. For segmentation, we use Hausdorff distance, which has the advantage of also considering shape irregularities that IoU would miss (Foucart et al., 2023) . In the first experiment we compare our new HoVer-NeXt model with our CoNiC submission (referred to as HN CoN iC ) on the GlaS test set. All results shown in Figure 2 are obtained with 16TTA. For binary segmentation, all three HoVer-NeXt models achieve higher F1 score and MCC, with HN Large having +0.005 F1-Score HN CoN iC and HN T iny being on par with HN CoN iC with 0.814 binary pixel F1 score. These improvements are reflected in an increase of +0.009 for HN Large over the 0.832 baseline for binary detection F1. Considering class-specific classification, the results are more diverse, with HN CoN iC achieving the highest balanced accuracy in neutrophil (0.706) and plasma cell (0.65) classification. HN Base achieves the highest balanced accuracy on lymphocytes (0.867) and is on par with HN Large and HN CoN iC on connective tissue cells (0.794). On eosinophils, we observe the largest change with HN Large at 0.785 (+0.074) and even HN T iny (+0.064) largely increases in accuracy over HN CoN iC (0.711). Epithelial cells remain more consistent, but HN Large achieves the highest balanced accuracy with 0.796, HN CoN iC the lowest with 0.789. For segmentation quality, HN CoN iC achieves the lowest Hausdorff distances across all cell types except eosinophils (Figure 2 A ). Investigating EosEval, we find an increase in performance across all model sizes compared to GlaS. HN Large has 0.553 F1 score on GlaS, but a per region average F1 score of 0.668 on EosEval. On MitEval, HN T iny performs best with 0.553 F1 compared to 0.521 for HN Large and 0.517 for HN Base . Evaluating the effect of TTA, we observe an initial F1 score bump when using TTAs (Figure 2 C ), but less of an increase with additional views. HN Base has an increase of 2.4% in F1 from zero to four TTA views, but less from eight to sixteen (+0.0004) with standard deviation decreasing slightly (+-0.0039 to +-0.0036). Additionally, we find that the rare cell types, eosinophils, neutrophils, and mitoses gain the most from TTA (+2.16, +4.16, +4.87 % F1 Score with 4 TTA), whereas more common cell types only show increases of less than one percent or even a slight negative impact (Plasma cells: -0.35%). 
pannuke: 
 We train HoVer-NeXt on PanNuke, optimize hyperparameters, and evaluate it with the PanNuke evaluation script (Gamper et al., 2020) . HN T iny with 16 TTA has a tissue average mPQ (mPQ T iss ) of 0.477, achieves the highest PQ for inflammatory (0.418) and dead (0.154), and improves on HoVer-Net in the epithelium (+0.024) and connective (+0.027) class, yet only reaches 0.536PQ on neoplastic (Figure 3 ). CellViT sets the current state-of- 2023) note that their model was only performing on par with HoVer-Net without pretraining. Moreover, we observe qualitatively, that the performance does not seem to translate to WSI (Supp. Figure 7 ). More thorough evaluation metrics for HN T iny can be found in Supp. C.5.We utilize five publicly available TCGA WSI (Supp.  
discussion: 
 To make large-scale investigations into the cellular composition of CRCs feasible, we develop a pipeline for nuclei segmentation and classification. Our model retains the predictive performance of our original CoNiC challenge submission, improves upon the detection metrics, and successfully learns the additional mitosis class using the mitosis and Lizard-mitosis datasets. Differences in eosinophils and plasma cells between HN CoN iC and HN Large and the ablation results indicate that the loss function and sampling strategy have varied impact on rare cell types, however with no clear best configuration. We also find that particularly eosinophils and neutrophils are more sensitive to color changes, but reduce this problem with TTA. Also the removal of convex-hull-based post-processing likely leads to more segmentation outliers increasing the Hausdorff distance. One of the major improvements to further increase model accuracy on small datasets such as PanNuke are large-scale pretraining (Chen and Krishnan, 2022; Hörst et al., 2023) or semi-supervised learning approaches such as Rumberger et al. ( 2023). Also, other published implementations of ConvNeXt-based U-Net variants such as (Roy et al., 2023) could further improve results. Larger context sizes could also lead to more robust classification, in particular in the healthy vs. malignant case (Frei et al., 2023) . As demonstrated by our adaptation of ST++ (Yang et al., 2021) , automatic labeling of objects in histopathology by re-staining is a straightforward way of generating large labeled datasets and even single institute data provides sufficient variety for learning mitosis. Finally, HoVer-NeXt is 5× faster than state-of-the-art and runs inference on TCGA COAD/READ (N=576) at 0.5mpp in 50 hours. 
limitations: 
 Lizard contains wrongly annotated mitoses and not all pHH3-positive objects are visibly mitoses, thereby creating a noisy dataset. Moreover, perfect annotations on H&E are difficult, and cell types are only estimated by pathologists. Therefore, reported results will never entirely reflect the true model performance. Moreover, a more accurate approach for H&E mitosis annotations on Mit-Eval could have been chosen (Aubreville et al., 2023) . Finally, we did not use the 3-fold evaluation split for Lizard to maximize available training data and compare with our own HN CoN iC . 
conclusion: 
 We publish HoVer-NeXt, a fast and efficient H&E-based nuclei segmentation and classification pipeline, allowing for investigations into cellular compositions, spatial relationships, and morphological parameters of nuclei directly on large cohorts. While much of current research focuses on spatial technologies such as multiplex-immunofluorescence, generating such data is still expensive. Here, HoVer-NeXt can be a used for hypothesis generation and for finding interesting WSI to be further investigated using spatial technologies. Our work facilitates the next generation of histopathology and provides an important building block towards a quantitative view of WSI in clinical routine. a dilation step using a circular kernel. The size of the kernel is again determined by the image dimensions as we use the 0.01% of the size of the longest dimension as diameter.The filtering step ensures that we do not keep small fragments and small slide artifacts as relevant foreground and the dilation step avoids cut corners where some lighter tissue would be missing due to the blurring step. These threshold steps were chosen qualitatively by considering WSI from multiple cohorts and verifying that estimated foreground area was within reasonable bounds. 
 
 Appendix: Applied augmentation methods were selected during the original challenge submission with all transformation parameters being chosen such that the transformed images still appear as though they could be crops form an H&E image. However, some of these transformations remove information or make it more challenging for the model to make a correct prediction such as adding Gaussian noise or blurring the image. Therefore these transformations are removed from the set of augmentation methods during inference, but during training it is useful for the model to also learn to produce acceptable results even if the image is blurry.Including during training are also elastic deformation, rotation in a range from 0 • -45 • , shearing, as well as zooming, all of which utilize an interpolation method to transform the image, thereby changing image information. The same then applies for the model outputs, which need to be inversely transformed, where then any differences introduced by the interpolation will lead to less exact nucleus boundary predictions. Additionally, rotating by 45 • or shifting the image removes pixels completely which also leads unnecessarily worse performance. Figure 5 illustrates these concepts. rotated by less than 90°which means that an interpolation method needs to be used. The model then receives the transformed image, provides an output but naturally cannot provide an output for the now invisible area. When rotating the image back to its original orientation, a large area of the original input is actually missed. Zooming into a detail of the rotation of the model output, we can quickly observe differences where the interpolation method softens some edges and reduces differences between neighboring pixel values. While this does not necessarily have to have a negative impact on the final result, differences of a single pixel can already change the hausdorff distance for this nucleus significantly.Both datasets are available at the TIA-Warwick websiteLizard The Lizard dataset is an H&E based nuclei segmentation and classification dataset for CRC and normal colon tissue with six classes: neutrophils, epithelial cells, lymphocytes, plasma cells, eosinophils, and connective tissue cells (Graham et al., 2021a) . Raw H&E images are available at 0.5mpp both as pre-cropped (with overlap) 256 × 256 tiles as well as full ROIs. It combines multiple datasets from several institutes and has 495,179 total annotated nuclei but is highly imbalanced with neutrophils and eosinophils only accounting for 0.89% and 0.68% of all instances. Additionally, ∼84% of the dataset is background. The dataset combines multiple datasets from different institutes, one of which is the GlaS subset which we are using as an external test set.PanNuke PanNuke is another H&E nuclei panoptic segmentation dataset but with a wider focus on samples from many different cancers (Gamper et al., 2019 (Gamper et al., , 2020)) . Here the classes are neoplastic cells, inflammatory cells, connective tissue cells, dead cells, and non-neoplastic epithelial cells, again with considerable class imbalance as well as tissue type imbalance. PanNuke is only available as 256 × 256px crops and only at ∼0.25mpp.A.6. Registration and ground truth preparation for the pHH3 mitosis datasetAll whole slide images are converted to TIFFs. Then, each H&E -pHH3 pair is registered in its entirety by manually specifying an anchor point in an exactly matching tissue area (e.g. by selecting a nucleus that was clearly observable in both images) to remove any offset differences in the images. Afterwards, we use SimpleElastix to estimate rigid and non-rigid registration transforms using greyscale versions of the images down-sampled to 0.5mpp and apply these transforms to the whole slide images at full resolution. All registrations were performed on a machine with 64Cores and 512GB memory. Thresholds for ground truth masks from the DAB channel are set individually per ROI to account for intra-and inter-WSI differences. ROIs were deliberately selected to include a large area of potential mitoses and areas problematic for the pHH3 stain such as necrotic areas or clear stain artifacts were avoided. All registered images, ROIs and generated masks were verified qualitatively to ensure that accurate segmentation masks were generated.First, we train five models on just the lizard dataset with 5 cross validation folds and run ensemble inference on the the new mitosis crops. Then, a new model is trained on the combined dataset and checkpoints are saved every 50000 steps. Based on relative per sample changes in mean panoptic quality from the first checkpoint to the best (considering validation metrics) checkpoint, samples are split into easy and hard samples. Easy samples would be the ones with less than median change in panoptic quality and hard samples those with more. The same model is then re-trained from scratch only on the easy samples from the mitosis dataset and used to predict the hard samples creating the final mitosis dataset. Mitosis ground truth from the restain are always the only "nuclei" of the mitosis class and mitosis predictions are re-classified to the second most likely class. A model trained on only the mitosis dataset is then used to predict mitoses on Lizard where new mitosis annotations are only added if there is no other label on any of the pixels yet.Mitosis Evaluation For this dataset, we specify 13 ROIS from nine randomly selected CRC H&E resection WSI to ensure that each mitosis can actually be observed on the H&E which is not guaranteed when using automatic label generation from pHH3-based restains. pHH3 is also positive for cells in G2 and some other objects also sometimes pick up the antibody. Nine ROIs are on five slides from an internal cohort (0.12mpp) and four ROIs are on four publicly available TCGA Slides (0.25mpp). Annotations were done as small ellipses around the mitotic figures by three pathologists. The final dataset matches annotations by a maximum distance of 6µm and similar to Aubreville et al. (2023) , annotations with at least one matching additional annotation are added to the dataset, however we do not include an additional review step. The three observers have an ICC3 of 0.860 [0.69,0.95]. Images at ∼0.5mpp are provided both pre-tiled, and as complete ROIs in npy format.Eosinophil Evaluation Eosinophils are a comparatively easy to spot subset of immune cells discernible on H&E, yet in the lizard dataset, and during the CoNiC challenge, none of the models perform well in detecting them. Therefore, we created an additional eosinophil point annotation dataset with 11 ROIs of varied sizes from 8 Patients to further evaluate eosinophil detection performance, in particular also across different stain variations. ROI raw H&E images (at ∼0.5mpp) and annotations are provided as individual npy files.A.9. Specifications for Inference Time comparisonAll models run on a HPC Node with 2xIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, (20 cores each), 1600GB RAM with one NVIDIA A40 48GB GDDR6. We set the following parameters for CellViT, HoVer-Net and HoVer-NeXt, unspecified parameters are left to default. In our HPC environment, we need to set OMP NUM THREADS to 16 (matching the number of workers) for the PyTorch HoVer-Net pipeline (github.com/vqdang/hover net) as we otherwise could not achieve competitive speeds. For CellViT, all TCGA-AA* images had to be ran using the 20× parameter and the others with the 40× parameter set. While the images indeed have different magnifications stored, their resolution is the same (∼0.25mpp). Moreover, the CellViT Pipeline requires pre-processing WSI which we included into the total processing time as the other two pipelines do this on the fly. All experiments are run on only 16 cores. For future evaluation, we also specify the commit and repository used for the comparison: HoVer-Net (67e2ce5e3f1a64a2ece77ad1c24233653a9e0901) CellViT (4bc42811c9841805ef0984b3ec0daf159312323a) HoVer-NeXt (c5bf99fdc2d8bd5129d780c5f19ee83a4babb0d4 To evaluate this dataset, we use the same distance based matching using the centroid of the annotated ellipse.  


Decision: The work introduces HoVer-NeXt, an advanced model for nuclei segmentation and classification in histopathology, improving upon a previous challenge submission. It offers extensions, simplifications, and new training/validation data, validated on the PanNuke dataset. The model, focusing on H&E-stained images, is efficient for WSI analysis, promoting hypothesis generation and offering competitive speed and accuracy. An expanded dataset, Lizard, is provided for mitosis and eosinophils study. HoVer-NeXt, with varying sizes, and its source code are publicly available on GitHub, under GPL-3.0, enhancing its application in biomedical analyses.

The paper is notable for its comprehensive contributions to biomedical research, particularly through its commitment to openness and transparency by providing all code and data necessary for model training and inference. Its development of an efficient WSI inference pipeline, which outperforms existing methods in both time savings and performance, is a significant strength. The clarity and accessibility of the manuscript enhance its impact, with resources including ConvNextV2 weights being made publicly available, fostering reproducibility and further research. Innovations such as the updated HoVer-NeXt model and a faster processing pipeline, alongside extensive evaluation and methodological refinements, underscore the paper's robustness and its forward leap in computational efficiency and accuracy in the field of biomedical imaging.

Reviewers provide unanimously positive ratings.

Reviews: review: paper_summary: paper_summary: The paper introduces a segmentation- and classification-based model for use in histopathology, specifically focused on H&E-stained images. The paper shows the evolution to the current iteration of the pipeline with the updated model HoVer-NeXt. The pipeline is particularly focused on H&E, given it is the most cost-effective of the staining technologies. The authors suggest the pipeline can be used for hypothesis generation. The pipeline performs favorably to previous works, being fast and accurate. The publication also offers an extension to the existing dataset, Lizard, for mitosis and eosinophils.
main_review: main_review: The authors have produced a robust body of work and provided comprehensive responses to the review set out to them, including answering questions related to future modifications. I maintain my original score of a strong accept.
strength_weakness: strength_weakness: -	While the authors note that generating other data, such as immunofluorescence, is considerably expensive (and suggest their pipeline can be used on H&E for hypothesis generation), the adaption of this pipeline to immunofluorescence would be interesting and very useful. Particularly for those who work in spatial transcriptomics. 
- Additionally, I would be interested to see extensions (given the focus here has been on H&E) to other staining techniques, such as CD31 or CA9.
questions: questions: The biggest item to address would be future plans for this work and adapting it to other staining techniques.
paper_summary: paper_summary: The authors introduce HoVer-NeXt, an updated version of their previous model, designed for scalable segmentation and classification of large whole slide images (WSI). This new model, along with its source code, has been made publicly available on GitHub. It includes the weights for ConvNextV2 trained on the Lizard-Mitosis dataset, with sizes detailed as Large (842.2MB), Base (391.6MB), and Tiny (144.4MB), in addition to a Tiny version trained on the PanNuke dataset. All the code and models are released under the GPL-3.0 license. Furthermore, the authors provide additional validation sets specifically for mitoses and eosinophils, enhancing the model's utility and accuracy in relevant biomedical analyses.
main_review: main_review: Thank you for the thorough revisions and additional details provided in response to the feedback, particularly on pipeline enhancements and model clarifications, which have addressed my initial concerns. The introduction of new studies and speed improvements significantly deepens the understanding and impact of your work.

Your efforts to differentiate your pipeline from existing models and the focus on efficiency for whole slide image analysis are commendable. The updates regarding the batch size and the implementation of automatic mixed precision during inference highlight the strength of your evaluation process.

I recommend a strategy to optimize your computational workflow to address the high memory demand, suggesting the use of dependency graphs and automated job arrays with OpenSlide's delayed reading feature. This would allow for more efficient memory management and flexibility in computational resource usage.

Given the comprehensive revisions and the significant contributions of your manuscript to the field, I have revised my decision to a strong accept. The advancements made in your work are commendable, and I look forward to its impact on future research and applications.
strength_weakness: strength_weakness: The approach presented can be seen as a refinement of previously published works, positioning it more as a paper focused on optimization rather than introducing fundamentally new methodologies. It primarily enhances aspects related to cell segmentaion and classification, building upon the foundation laid by earlier research. This indicates that the contribution of the paper lies in its ability to optimize existing methods, rather than in the introduction of entirely new techniques or concepts in the field.

This work incorporates tunable hyper-parameters that are adjusted in a post hoc manner based on the validation sets. By including these adjustments as part of the end-to-end learning process, the overall performance and efficiency of the model are significantly enhanced.

The authors need to moderate their claims regarding the speed of their models, as the inference time comparison outlined in their 'A.6. Specifications for Inference Time Comparison' section does not constitute a strict apples-to-apples comparison. The discrepancies in the experimental setup (HoVer-NeXt is clearly favored), such as the unequal adjustment of hyper-parameters, inconsistent image processing parameters, underutilization of available computational resources, and reliance on default settings that may not be uniform across models, suggest that the observed differences in speed could stem, at least in part, from these technical inconsistencies rather than purely from the models' optimization. Therefore, a more cautious interpretation of the speed results is warranted until a more equitable comparison can be conducted.
questions: questions: "... since data sampling is already sufficient to treat the label imbalance.": In section 2.2 on HoVer-NeXt, the text mentions that data sampling is adequate for addressing label imbalance. To substantiate this claim, it is recommended to present quantifiable metrics that clearly demonstrate the impact of each condition on the final results. An ablation study could be particularly insightful in this context. For example, the study could compare:

1. The combination of the model with class-based loss weighting and data sampling, illustrating how both techniques contribute together to mitigating label imbalance.
2. The model paired with class-based loss weighting alone, to assess the effect of adjusting the loss function to prioritize certain classes without altering the data distribution.
3. The model with data sampling only, to determine the effectiveness of this strategy in balancing the class distribution independently of loss function adjustments.

" ... To make HoVer-Net work at all, ...": The critique of the efficiency evaluation of HoVer-Net, particularly in its application to deep learning (DL) based predictions, warrants a more cautious interpretation. The comparison of inference times, as detailed in the section 'A.6. Specifications for Inference Time Comparison,' does not provide a straightforward comparison due to notable differences in the experimental setups (HoVer-NeXt is clearly favored). These differences encompass a range of issues, including unequal hyper-parameter adjustments, inconsistencies in image processing parameters, and the underutilization of computational resources. Such discrepancies undermine the credibility of the reported speed comparisons. Moreover, the reliance on default settings, which can vary significantly across different DL models, adds another layer of complexity to the comparison. This suggests that the differences in inference speed observed might not purely reflect the inherent efficiency of the models but could also be a result of the varied experimental conditions. Therefore, to accurately assess CellViT, HoVer-Net and HoVer-NeXt performance in DL-based predictions, it is essential to conduct comparisons under standardized conditions. This approach would ensure that any differences in speed observed are directly attributable to the models' optimization and capabilities, rather than external variables. Consequently, it becomes clear that the reported gains in speed may not solely derive from the deep learning model itself but from the entire processing pipeline in the HPC environment which is not fair.

Could you elucidate the rationale behind CellViT's implementation with a notably smaller batch size of 8, in contrast to the larger batch sizes of 64 employed by HoVer-Net and HoVer-NeXt?

For a more equitable comparison, it is suggested to incorporate architectures similar to Mask-RCNN into the analysis. Mask-RCNN is known for its optimization for handling heavy workloads effectively. Including such architectures would ensure that the comparison takes into account models that are well-suited for demanding tasks, leading to a fairer evaluation of performance across different systems.
paper_summary: paper_summary: The presented work describes the development of a model for nuclei segmentation and classification for the quantitative assessment of tissue composition in histopathology. The model is described as an updated version to a previous challenge submission that reported competitive performance. In comparison to the already published challenge submission, the authors updated the model with some extensions and simplifications, provide new training and validation data for the scientific community, validate their updated model on the PanNuke benchmark dataset for comparison and provide an efficient WSI inference pipeline, which they evaluate against other relevant methods from the literature.
main_review: main_review: The authors put significant effort into the rebuttal and the revision of the manuscript, which significantly benefitted the manuscript and removed all previous doubts. This has become a really great paper, and I am certain this needs to be presented at MIDL.
strength_weakness: strength_weakness: While the authors acknowledge that the newly added mitoses class is a noisy dataset in the limitation section, the validation of the reliability of these added labels could have been more elaborate. 
- First, quantifying mitotic figures is a very challenging task even for experienced pathologists. Therefore, evaluating the model on a hold-out test set which was annotated by a single pathologist can create a very biased result, as inter-rater agreement between pathologists can be quite low. This risk of misidentifying mitotic figures is even elevated at a resolution of 0.5mpp as small differences in the morphology are more difficult to distinguish. 
- Second, as mentioned by the authors, the creation of labels from PHH3 poses some additional challenges such as selecting a proper threshold for DAB channel, but also properly registering the PHH3 annotations to the HE image, which methodology was not mentioned by the authors in the manuscript. While the self-training strategy describes a sophisticated way of generating labels in this way, the resulting labels will still contain mislabeled mitotic figures and mitotic figures not visible in HE. 
- Additionally, mitotic figures were only added to the Lizard dataset when no other label was on any of the pixels yet, which is questionable, as it is likely that some of the mitotic figures were previously misassigned to one of the original classes, and therefore additional label inconsistencies are introduced by the new mitosis class. The new mitosis class in the Lizard-Mitosis dataset were not verified by any pathologist and its reliability is therefore questionable. 
- Finally, the authors claim that the single institute data for mitosis was sufficient to learn mitosis, however the experiments do not support this claim, as the training data included only ROIs from 11 samples and there was, e.g., no evaluation of the mitosis class on data from a different institute. Therefore, the robustness of this method is questionable and this should be acknowledged.

The authors claim to have achieved a new SOTA performance with the PanNuke benchmark dataset. However, the CellViT model has better validation metrics. The authors claimed that these results were only achieved by the CellViT model when they used pre-trained models. The use of pre-trained models for the encoder should not invalidate the performance. Especially for transformers, pre-training strategies on large datasets are of high importance and the use of publicly available models should be encouraged. Therefore, the CellViT model should not be denied SOTA performance.
questions: questions: - How reliable are the mitosis labels in the Lizard-Mitosis dataset? 
- How often was the mitosis label predicted for a cell that was previously assigned to another class? 
- Is it therefore sensible to only include the new mitosis label, if no labels were assigned to the pixels yet. 
- How reliable are the mitosis annotations created by the pathologist? 
- Is it possible to create a multi-rater consensus for the hold-out test set or use another dataset which provides labels as multi expert consensus?

Comments: title: Reviwer KHyf final decision
Response to the authors
Response Part 2
Response Part 1
comment: I thank the authors for their comprehensive response to myself and all reviewers. I echo their sentiments in complimenting the authors for producing a robust body of work. The proposed suggestions to augment the capabilities of the existing pipeline (e.g., semi-supervised training to adapt to alternate modalities, such as immunofluorescence) and the ongoing efforts to adapt the pipeline for spatial transcriptomics are implementations I am looking forward to seeing in the future. I maintain my original score of a strong accept.
Thank you for the comprehensive response and the meticulous revisions made to the manuscript based on the feedback provided. Your detailed explanations and the additional information included, particularly concerning the pipeline enhancements, threshold values for model inference, and the foreground background estimation, have significantly clarified my initial concerns. Moreover, the added chapters and studies, such as the ablation study and the augmentation strategies, alongside the modifications for speed improvements, have contributed to a deeper understanding of your work's value and its advancements in the field.

I appreciate the effort to clarify the distinctions between your pipeline and previously published models, and the emphasis on making the model more efficient and usable on whole slide images. The inclusion of supplementary chapters and the correction of errors further demonstrate your commitment to transparency and rigor in your research.

The updates regarding the batch size for CellViT and the correction of previous oversights, like the enabling of automatic mixed precision during inference, are particularly noteworthy. These changes not only address my concerns but also showcase the robustness of your evaluation process.

Considering the authors' familiarity with high performance computing environments, I propose a strategy to improve your computational workflow and address the challenge of requiring 1600GB of RAM. This high memory demand often leads to prolonged job initialization times and restricts your operations to a single, high-capacity computing node. My recommendation is to incorporate dependency graphs and automated job arrays that operate independently. This approach would utilize OpenSlide's capability for delayed reading, allowing for data to be loaded into RAM only as necessary. Specifically, it would enable the loading of particular tile sets on demand, rather than the entire slides, ensuring that only sufficient tiles are available to maintain GPU activity during inference through the data loader. By adopting delayed reading, you could dynamically manage memory usage, thus reducing the need for extensive hardware and increasing the flexibility of job scheduling across various computing resources. Implementing this modification could significantly streamline your computational processes, making them more versatile and user-friendly within different high-performance computing contexts.

Given these substantial revisions and clarifications, I am convinced of the manuscript's significant contributions to the field. Therefore, I have decided to change my decision from borderline to strong accept. The advancements you've made in developing a more efficient pipeline for whole slide image analysis, as well as the thoroughness of your experimental validation, are commendable. I look forward to seeing the impact of your work on future research and applications in the domain.
Thank you for your thoughtful response to my review. I appreciate the time and effort you invested in addressing my concerns.

1. Missing Image Registration

The addition of a new section (A.6) in the supplementary material explaining image registration and DAB label extraction is a valuable addition. This context will undoubtedly benefit readers in understanding your methodology.

2. CellViTs SOTA

I acknowledge your point that the strengths and potential of your work will be evident to attentive readers, despite revisions made.

3. Quality of Mitosis Labels

I appreciate the significant effort taken to improve the validity of the test set results by incorporating annotations from two additional pathologists and achieving a strong inter-rater agreement (ICC3  = 0.860). While acknowledging the limitations of using unverified labels in the lizard dataset, I recognize your emphasis on these limitations within the manuscript. This transparency allows readers to interpret the findings appropriately.

Overall, the revisions you implemented effectively address the concerns I raised. Thank you for strengthening your manuscript.
Dear Review committee, 

We are grateful for the opportunity to submit a revised version of our manuscript entitled “HoVer-NeXt: A Fast Nuclei Segmentation and Classification Pipeline for Next Generation Histopathology”. We would like to thank the reviewers for their comments and feedback, which we feel have now significantly improved the paper. Please find our each of our replies in a point-by-point manner below, where we address each comment. All changes made to the manuscript have also been indicated in the rebuttal. All changes in the manuscript have been highlighted in teal. We hope that you find this revision satisfactory and that our work will be of great interest to the readers and attendees at MIDL 2024.
We would like to thank the reviewer for the time invested in reviewing our manuscript. We feel the paper is now significantly improved and we have done our best to address them in changes in the manuscript as well as the following responses.

Regarding the missing information on the image registration

To address these questions, we have added an additional section in the supplementary material (A.6.) explaining the process used for image registration and DAB label extraction. Image registration is done with the entire WSI using SimpleElastix and a single manually defined anchor point to pre-align images to ensure a correct registration. Given the size of the images, registration is done at 0.5mpp and then applied to the full resolution images.

Regarding CellViT’s SOTA:

We agree with the arguments you put forward in that CellViT should not be denied SOTA and have therefore updated the tables and re-worded our manuscript to reflect these changes.

“How reliable are the mitosis labels in the Lizard-Mitosis dataset?”
And
 “How often was the mitosis label predicted for a cell that was previously assigned to another class?”

The reason we included an additional mitosis test set is to avoid the most likely impossible task of quality controlling each mitosis label. Lizard is only available in form of small ROIs, there is no context which can be used to generate a general impression of how mitotic figures look like in these samples and all ROIs are only available at 0.5mpp. 

There are 331 newly added mitoses in the lizard dataset. The predicted labels contained 407 mitoses, which means 76 objects were predicted as mitosis but already had a different label in lizard. 55 were labelled as lymphocyte, 11 as epithelium, and 5 each as neutrophil, plasma cell and connective tissue cell. That means 76 objects might be mitoses, but a pathologist decided differently, with 331 objects remaining as “potential” mitoses. Correcting and verifying this manually suffers from the exact issues the reviewer is describing in that these images are only available at 0.5mpp with varying stain and scanning quality.

Our method using self-training was intended to adjust for the noisy mitosis labels by using phh3 ground truth in another dataset, which in turn has noisy (i.e. not verified) labels for the other classes. During training a batch is drawn randomly from one or the other dataset.
In ST++ paper, the authors suggest that their method even improves the model’s results on the test-set which is not necessarily something we could verify, but we note that this in turn means that the noise introduced by this self-training process does not have a negative effect on the final results. Of course, a nuclei segmentation and classification dataset with ground truth IHC labels for all classes would be more optimal, but we show that we can add this rare class using a separate dataset.


"Is it therefore sensible to only include the new mitosis label, if no labels were assigned to the pixels yet."

If we overwrote an existing label, we would have to make sure the model is correct, however the authors of the lizard dataset report that annotations were verified by pathologists, which we would consider a more accurate annotator than our own model. As already mentioned, the images are only available at low resolution and potentially more compressed than the original slides which the original annotators could have used to make a better decision. On a more fundamental note, we believe that creating a large scale dataset for H&E object annotations based on nuclei/object specific antibodies with restains for all classes might be a necessary next step to further improve H&E based nuclei classification and to better validate the results.


Regarding
    “How reliable are the mitosis annotations created by the pathologist? “
And
   “ Is it possible to create a multi-rater consensus for the hold-out test set or use another dataset which provides labels as multi expert consensus?”

We have added annotations by two more pathologists, updated the supplementary material accordingly and updated the mitosis detection results by taking majority vote labels from the three pathologists. Their ICC3 is 0.860 [0.69,0.95]. We have added to our limitations section, that a more optimal way of annotating could have chosen as e.g. shown by Aubreville et al. (2023), but given the limit time scope of this rebuttal period, we at least have a majority consensus on all labels now. 

Additionally, we moved the MitEval dataset to the same ROI-WSI-aggregation evaluation as the eosinophil dataset, such that metrics from each different WSI contribute equally to the final metric and we can show between slide variability. Plots and descriptions have been updated accordingly.

Lastly, we have fixed the wrong label for the mitosis dataset evaluation.
Regarding the ablation study to compare loss weighting with sampling and the combination thereof:

We added the requested ablation study as an additional chapter in the appendix C.2., which indeed does not provide sufficient evidence for differences in this dataset with this model between the methods for dealing with class imbalance. Only the model with no strategy to deal with class imbalance performs consistently worse than all other models. Therefore, we conclude from this limited information, that reducing overall complexity of the method by only using e.g. sampling or loss weighting, as suggested in our first draft, is still a reasonable approach with our claim that data sampling is sufficient still holding.

Regarding your raised concerns that speed improvements might not solely be coming from improvements to the model or the HPC environment being tailored to our approach:

We appreciate the reviewer’s concern. In this paper, we simplify and improve our previously published model, but mainly focus on creating a pipeline that can then be used for large scale investigations in existing H&E cohorts. In the context of nuclei segmentation and classification, the whole slide pipeline has the crucial step of resolving tile overlaps which we solve in a faster way than HoVer-Net (or CellViT). Moreover, tile extraction, memory management, and maximization of CPU and GPU utilization are all part of improvements of the pipeline that indeed are model independent. In contrast to other works, where algorithms are developed on pre-extracted tiles, we generate here an end-to-end pipeline for whole-slide inference.

An apples-to-apples comparison in this setting is very difficult, as the reviewer correctly describes, as pipelines rely on different strategies for fast inference, with e.g. CellViT using larger tiles for inference and our method using less overlap than HoVer-Net. However, most end-users of such pipelines will be applying these models “as is” on WSI and we therefore aimed to create a comparison that will be valuable to end-users deciding on which pipeline to use for their project.

Regarding your question on why we chose batch size 8 for CellViT:

CellViT has a recommended input tile size of 1024x1024 compared to 256x256 of HoVer-Net and HoVer-NeXt. Batch size 8 is the default specification given by CellViT and we assume, similar to how we would specify a reasonable default implementation, that this batchsize results in a good performance. However, we did miss enabling automatic mixed precision during inference in the previous evaluation as this was indeed not enabled by default, as you correctly pointed out. As CellViT uses a SAM-based vision transformer encoder, the maximum possible batch size with mixed precision on a NVIDIA A40 GPU was 30, which is why the new evaluation now runs on this batch size. As this sped up the inference time, we have corrected our manuscript with updated timings and time comparisons. Moreover, we also noticed that we wrongly still included specifying omp_num_threads=1 and setting workers to 0, which we had already fixed prior to our latest results.

Regarding your suggestion to compare with Mask-RCNN

For a proper comparison with Mask-RCNN, we therefore would also need to create a full pipeline for Mask-RCNN which is not in the scope of this paper. Moreover, we note that past publications, such as HoVer-Net already include a comparison with Mask-RCNN in terms of model prediction quality and inference speed and conclude that HoVer-Net improves on it by a large margin:
“Regarding the processing time, the average time to process a 1000×1000 image tile over 10 runs using Mask-RCNN for segmentation and classification was 106.98 seconds. Meanwhile, HoVer-Net only took an average of 11.04 seconds to complete the same operation; approximately 9.7×  faster.”

-	Graham, S., Vu, Q.D., Raza, S.E., Azam, A.S., Tsang, Y., Kwak, J.T., & Rajpoot, N.M. (2018). Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image analysis, 58, 101563 .

We have also reworded some parts of the abstract and introduction to better reflect the focus of this paper.
We thank the reviewer for the thorough review of our paper. The provided comments and feedback have helped to significantly improve the manuscript even further and are greatly appreciated. Please allow us to address each concern below.
Importantly, we want to stress that indeed we present a pipeline here in this paper, and not the previously published model. While we have also made considerable changes to the model itself, the focus on many improvements lies on making our model more efficient and usable on whole slide images. Therefore, investigations and comparisons focus on the pipeline, and not the model.

Regarding your specific raised issues and questions:

“Additionally, providing the specific threshold values used in the experiments would greatly aid in understanding and replicating the study's methods.“

Thank you for the comment. All threshold values for further postprocessing during model inference are specified in files that are included in the model weights zip file, as they are specifically optimized for each model. Hence, we believe an additional table summarizing these would be unnecessary. 

Regarding the questions on how foreground background estimation was performed:

We have added a supplementary chapter on this topic to explain how FGBG estimation is performed in our pipeline, which can be found on page 14 of the revised manuscript (Supp. A.3.). We also corrected a mistake where we specified that the image is tresholded in HSV space, but rather it is done on the greyscale image using OpenCVs greyscale transform matrix. We note that the specific thresholds chosen are arbitrary in that, after trying a range of values, with the current values, we qualitatively could not observe any issues in missing foreground, and quantitatively could not observe outliers resulting from a bad foreground estimation on over 3000 slides from multiple different cohorts and institutions. Published works such as:

Abbet, C., Studer, L., Zlobec, I., & Thiran, J. P. (2022, April). Toward automatic tumor-stroma ratio assessment for survival analysis in colorectal cancer. In Medical Imaging with Deep Learning.

 use the same method. As this is not consistent with the method employed by HoVer-Net, we have also added a foreground area comparison into Supp. Table A.10.

Regarding your points on the negative effects of augmentations

Thank you for these comments. We have added another supplementary chapter (A.4) including a figure that further explains what we mean by reducing the augmentations during inference. Specifically, we want to highlight that some augmentations such as rotating the image by 45° removes information from the image which is necessary for a good segmentation. Naturally, during training, the ground-truth can be adapted accordingly to avoid this issue. Additionally, we have included a new table (Supp. C.1.). specifying all parameters for the augmentation methods, which was previously missing in the paper. The training augmentation strategy has been selected already during the CoNiC challenge submission and remains unchanged.
We would like to thank the reviewer for taking the time to read and review our manuscript as well as for the follow-up questions, which we address below. 

We believe switching to a semi-supervised training regimen will increase options to switch to other modalities such as immunohistochemistry or immunofluorescence images. Our model currently requires many accurate manual annotations and is focused on classification, which in the context of spatial transcriptomics might be interesting but should always be extendable to give all information and not just the reduction to a single class. However, replacing the model and training process is rather straightforward and there is an ongoing effort in our lab to adapt the pipeline to spatial transcriptomic data. Regarding CD31 and CA9 specifically, as both are membrane markers and not located near the nucleus, the challenge here will be to correctly attribute the stain to the right cell. We are positive that this can be easily solved with labelled data using our pipeline, but at the same time also believe that a carefully engineered method using just nuclei instance segmentation methods will be sufficient.