Advancing Multiplex Immunofluorescence Imaging Cell Detection using Semi-Supervised Learning with Pseudo-Labeling

Authors: yasin  shokrollahi
karina pinao gonzales
maria esther salvatierra
simon p castillo
pingjun  chen
b leticia rodriguez
sara  ranjbar
luisa m solis soto
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 23:04:05

Abstract: Accurate cell detection in multiplex immunofluorescence (mIF) is crucial for quantifying and analyzing the spatial distribution of complex cellular patterns within the tumor microenvironment. Despite its importance, cell detection in mIF is challenging, primarily due to difficulties obtaining comprehensive annotations. To address the challenge of limited and unevenly distributed annotations, we introduced a streamlined semi-supervised approach that effectively leveraged partially pathologist-annotated single-cell data in multiplexed images across different cancer types. We assessed three leading object detection models, Faster R-CNN, YOLOv5s, and YOLOv8s, with partially annotated data, selecting YOLOv8s for optimal performance. This model was subsequently used to generate pseudo labels, which enriched our dataset by adding more detected labels than the original partially annotated data, thus increasing its generalization and the comprehensiveness of cell detection. By fine-tuning the detector on the original dataset and the generated pseudo labels, we tested the refined model on five distinct cancer types using fully annotated data by pathologists. Our model achieved an average precision of 90.42%, recall of 85.09%, and an F1 Score of 84.75%, underscoring our semi-supervised model's robustness and effectiveness. This study contributes to analyzing multiplexed images from different cancer types at cellular resolution by introducing sophisticated object detection methodologies and setting a novel approach to effectively navigate the constraints of limited annotated data with semi-supervised learning.

Main paper: 
 introduction: 
 The introduction of multiplex immunofluorescence (mIF) imaging techniques marks a significant advancement in tissue analysis, enabling the in situ quantification of multiple proteins (Amitay et al., 2023) . Cell detection from mIF images is a fundamental task, facilitating the quantification of the tumor microenvironment and spatial configuration with identified cell locations and types, thereby providing insight into tumor progression. Integrating deep learning into medical imaging and pathology has significantly led to advancements in disease diagnosis and treatment strategies. Previous methods mainly formulated cell identification as segmentation under the fully supervised fashion, and cell phenotyping primarily relied on the expression of proteins. Schmidt et al. presented StarDist, a novel method for cell segmentation in microscopy images, utilizing star-convex polygons for segmentation and effectively handling crowded cellular environments (Schmidt et al., 2018) . Complementing this, Greenwald et al. developed Mesmer, a deep learning algorithm for whole-cell segmentation in tissue images, which achieved human-level performance and addressed the critical challenge of cell segmentation in tissue imaging (Greenwald et al., 2022) . Also, Stringer et al. introduced Cellpose, a generalist algorithm for precise cell segmentation across various microscopy image types without requiring retraining (Stringer et al., 2021) . Moreover, groundbreaking advancements have been made by Amitay et al. with the development of CellSighter, a neural network demonstrating over 80% accuracy in cell classification from multiplexed images (Amitay et al., 2023) . Additionally, unsupervised methods were also incorporated for cell phenotyping. Bortolomeazzi et al. introduced SIMPLI, a versatile tool for multiplexed image analysis by integrating cell segmentation and classification with unsupervised and supervised methods (Bortolomeazzi et al., 2022) .Despite these advancements, the field still faces challenges concerning the limited availability of comprehensively annotated datasets, heterogeneity across cancer types, and the intricate task of cell segmentation and classification in histology images. Several innovative approaches have been introduced to address the need for improved object detection methods in natural scene images. The study by Abbasi et al. proposed enhancing YOLO's performance on partially labeled datasets by creating pseudo-labels for unlabeled instances, significantly improving generalization performance (Abbasi et al., 2020) . Niemeijer et al. developed a method for fusing datasets with partially overlapping classes, employing pseudo-labeling with uncertainty quantification to enhance model robustness (Niemeijer et al., 2023) . However, these advanced techniques have not been fully investigated for mIF images.Tackling the challenges of cell detection posed by tumor heterogeneity and lack of comprehensive annotations in mIF images, our study underscores the critical role of precise quantification of complex cellular patterns in predicting immune cells within the tumor microenvironment-immunotherapeutic responses. We initially trained three detectors on partially annotated datasets and selected the best one based on the performance. Next, we used pseudo labels produced by this model to train a new detector. Furthermore, we comprehensively compared our method's performance across different portions of partial annotations. To finalize the validation of our pipeline, we performed evaluations on five specific cancer types: Papillary Urothelial Carcinoma (PUC), Penile Squamous Cell Carcinoma (PSCC), Urothelial Carcinoma (UC), Cholangiocarcinoma (CC), and Rectal Squamous Cell Carcinoma (RSCC), all fully annotated by pathologists. It further highlights our approach's robustness, effectiveness, and generalization, establishing a new benchmark in medical image analysis and pathology. 
methodology overview: 
 To generate fully annotated images, this study delves deeply into medical image analysis, focusing on evaluating the performance of three leading object detection models: Faster R-CNN (Ren et al., 2015) , YOLOv5s (Jocher, 2020) , and YOLOv8s (Ultralytics, 2023) . These models are assessed for their proficiency in detecting and classifying cells within mIF images, a process integral to advancing the precision and speed of object detection, pivotal factors in early and accurate cancer diagnosis. Initially, all detectors were trained using a dataset that was partially annotated. To create this dataset, 10 whole mIF images were used, and patches were extracted from these images for training. Additionally, the dataset was expanded through the use of augmentation techniques. Then, the best performing detector, determined by the comparative analysis, was selected to generate pseudo labels (additional labels generated by the detector with the confidence level greater than 90%) in an iterative training loop. These pseudo labels were then used to retrain a new detector, enhancing the model's learning process and improving its detection capabilities. Figure 1 provides a schematic overview for generating pseudo labels and employing a semi-supervised learning approach for the detection and classification of cells, including immune cells (CD45+), epithelial and cancer cells (panCK+), and others (CD45-panCK-).Figure 1 : Schematic for generating pseudo-labels and utilizing a semi-supervised learning approach for the detection of cells, including immune cells (CD45+), epithelial and cancer cells (panCK+), and others (CD45-panCK-).This study employs an mIF dataset of histology images from papillary urothelial carcinoma tumors of 10 different patients, obtained through our Patient Mosaic initiative following IRB approval, and was generated in line with the GeoMx DSP manufacturer's recommendations (nanoString). Recognizing the labor-intensive nature of annotating large histology images, the study employed a partial annotation strategy. In this study, pathologists annotated approximately 10% of the cells in each of the 10 images, totaling 11,643 cells labeled as CD45, 15,228 cells labeled as panCK, and 9,489 cells labeled as Others. Annotations were made by marking points at the centers of cells for rapid yet accurate identification of each cell type. Subsequently, we drew a fixed square bounding box with a length of 50 pixels around each center point. This step was crucial for converting the point annotations into appropriate inputs for object detection models like YOLO and Faster R-CNN, which necessitate bounding box information. To manage the extensive size of these images, a patch extraction method was applied, creating non-overlapping patches of 640x640 pixels that retain essential cellular features critical for accurate detection (Figure 2 ). The dataset was further enriched through data augmentation techniques such as flipping, zooming, rotating, and blurring, expanding the dataset to 51,924 patches, of which 70%, 15%, and 15% were used for training, validation, and testing, respectively. The 15% testing portion was carefully chosen from different samples and did not include any augmented images to ensure the model was tested on completely new data.The pseudo labels generation phase involved pretraining the YOLOv8s model with partially annotated data, guided by semi-supervised object detection strategies to effectively utilize a limited number of labeled images (Gao et al., 2019; Jeong et al., 2019) . To further enhance our model's performance and prevent overfitting, we incorporated dropout with a rate of 0.3, extensive data augmentation including mixup and mosaic techniques with ratios of 0.5 and 0.1 respectively, and regularization with an L2 penalty coefficient of 0.01, alongside early stopping with a patience of 5 epochs to ensure generalization during the training process. Moreover, the training was conducted for a shorter span, capped at 50 epochs, to avoid excessive fitting to the partial dataset. This was followed by the generation of pseudo labels, a process inspired by methods such as CSD (Jeong et al., 2019) , specifically, we borrowed the idea of using image augmentations, such as flipping, to enforce consistency. In our approach, pseudo labels were first generated by selecting predictions with a confidence level above 90%. These high-confidence pseudo labels were ensured to contain almost all initial partial annotations from pathologists. This comparison validated the accuracy of all pseudo labels, acting as an additional verification step. Consequently, the method used a confidence threshold for the initial generation and ground truth comparison for validation, ensuring that only the most reliable pseudo labels were added to the dataset.Although the quality of pseudo labels was not directly evaluated by pathologists, the final model trained with these labels was assessed on the fully annotated dataset. After the pretraining phase with the partial dataset from papillary urothelial carcinoma, we merged the original labels with the pseudo labels (Initially, the annotated dataset by pathologists contained 11,643 cells labeled as CD45, 15,228 cells labeled as panCK, and 9,489 cells labeled as Others, after adding pseudo-labels generated through semi-supervised learning, the dataset expanded to include approximately 140,000 CD45 cells, 200,000 panCK cells, and 120,000 cells labeled as Others). This enriched dataset was then utilized to retrain the YOLOv8s model. We also analyzed the model's performance at different annotation levels during this retraining phase. Specifically, we considered 100% of the annotations for each class. Subsequently, we experimented with excluding 50% and 75% of the annotations for each class to assess the model's performance under varying levels of annotation scarcity (results presented in the Appendix Section A). The study also included fully annotated datasets of five distinct cancer types, utilized in the final stages of validation to assess the model's performance across varied cancer cell appearances and histology conditions, as illustrated in Figure 2 . These datasets were derived from five distinct cancer types, four of which are unseen types during the training process. For each case, four patches of images were fully annotated by two independent pathologists, ensuring high data accuracy. Our primary computational resource was a Linux server equipped with dual NVIDIA GeForce RTX 4090 GPUs, 256GB of random access memory, and a 32-core CPU, ensuring optimal performance and efficiency.  
initial phase: selection of best object detection model: 
 This section encapsulates the comparative performance analysis of various object detection networks, specifically focusing on their capabilities in identifying distinct cell types.After the initial phase, we evaluated the performance of these models, ultimately selecting YOLOv8s for further processes based on its optimal performance. In the final phase, we enhanced YOLOv8s by integrating pseudo labels with the original dataset, which involves semi-supervised learning and iterative refinement. To validate the performance, we used four patches of size 640x640 pixels for each cancer type, which have been fully annotated by two pathologists. The involvement of pseudo labels significantly improved the model's ability to detect cells (Figure 3A , all the p-values < .0001), validating the effectiveness of the strategy and ensuring the quality of pseudo labels. Figure 3B exemplifies four annotated patches from the PSCC, UC, CC, and RSCC, illustrating the comparison of the cell counts between the pathologists' annotations and the predictions made by YOLOv8s. The comprehensive validation of the YOLOv8s model across various cancer types is presented in Table 2 . The model was tested against five cancer types: PUC, PSCC, UC, CC, and RSCC. The model showed high precision, recall and F1 score rates in most categories, with CD45 cells in RSCC and CC demonstrating perfect recall (100%). In the case of PSCC, there were no CD45 cells present in the patches examined, as indicated by the dashes in the table. The model also performed strongly in panCK and Others cell types, especially in PSCC and PUC, showing high precision and recall. Additional details regarding the Area Under the Curve (AUC) metrics ,and the average number of annotations per class for these five types of cancers are provided in the Appendix, Section C, and Section B.In our study, the annotations provided by pathologists were centered on each cell rather than including segmentation masks. Still, we aspired to compare our results using state-ofthe-art cell segmentation methods. Specifically, we applied StarDist (with its "Versatile" model for fluorescent nuclei and the "DSB 2018" model from their 2D paper) (Schmidt et al., 2018) and Cellpose (using their latest "cyto3" model) (Stringer and Pachitariu, 2024) to predict segmentation masks on our validation datasets. For evaluation, we matched the centers of the predicted segmentations to the bounding boxes of our annotated cell centers. The StarDist method yielded a precision of 0.450, a recall of 0.210, and an F1 score of 0.284, while Cellpose demonstrated a precision of 0.862, a recall of 0.230, and an F1 score of 0.363. Throughout this evaluation, we considered all cells as a single category and utilized only the blue channel of the images.  
interpretation of results: 
 This study addressed the challenge of partially annotated mIF images through a semisupervised deep learning approach, focusing on detecting three cell classes: CD45, panCK, and Others. Cell detection in mIF is necessary to quantify and analyze the intricate spatial distribution of cellular patterns within the tumor microenvironment, which is crucial for understanding immunological studies in oncology. The model's validation on fully annotated patches across five cancer types, despite being trained on a dataset with partial annotations (approximately 10% of full annotations), not only underscores its capability to detect the majority of cells within the patches but also demonstrates its versatility in addressing different cancer types. This adaptability and reliability are evident in the consistent performance across various histology conditions, ranging from Papillary Urothelial Carcinoma to Rectal Squamous Cell Carcinoma. The model's adeptness in various contexts reflects findings similar to those of Amitay et al., who underscored the significance of accurate cell classification in computational pathology (Amitay et al., 2023) . Furthermore, the ability of YOLOv8s to deliver high performance, even with a limited set of annotations, indicates its robust learning mechanism and capacity to generalize from sparse data. This characteristic is particularly valuable considering the resource-intensive nature of manual annotations in medical imaging. This robustness and adaptability make our model a promising tool in digital pathology, opening new avenues for efficient and accurate cancer cell detection and classification.While the study presents promising results, certain limitations and opportunities for future research are apparent. This study performed annotations by marking points at the centers of cells rather than using bounding box techniques, it could limit the model's ability to fully understand the spatial context of each cell, which could be critical for certain analytical tasks like cells segmentation. Future endeavors could focus on refining the model's performance under sparse annotations, potentially employing advanced semi-supervised learning techniques, as highlighted by Xu et al. (Xu et al., 2021) . Furthermore, extending the model's detection capabilities to encompass a broader range of cell types and pathological conditions could offer a more holistic tool for pathologists, possibly integrating with comprehensive analysis software like SIMPLI, as suggested by Bortolomeazzi et al. (Bortolomeazzi et al., 2022) . Also, the absence of CD45 marker values in PSCC samples due to the non-availability of CD45 in PSCC patches points to a limitation in evaluating the model's capability to detect the CD45 marker in PSCC, reflecting a need for more comprehensive data to validate the model's performance for this specific marker. Moreover, while our primary objective was to identify cell centers and cells detection, comparisons with state-of-the-art methods such as StarDist and Cellpose highlight the potential for future research to enhance cell annotations by incorporating segmentation masks, thereby potentially improving the performance metrics against state-of-the-art segmentation methods.In summary, our study contributes to mIF imaging analysis by introducing a semisupervised learning model specifically designed for cell detection across different cancer types. By integrating a novel pseudo-label generation strategy, we aim to improve detection accuracy. This method showcases the potential of the YOLOv8s model to address the unique challenges presented by mIF datasets, potentially moving us toward enhanced precision and efficiency in healthcare diagnostics.  
 
 Appendix:  


Decision: The majority of the reviewers render positive ratings.

The evaluation of the work revolves around an innovative method for detecting cell types in MxIF imaging, leveraging a semi-supervised learning approach with YOLOv8. The authors addressed a challenge in cell detection within mIF imaging, focusing on improving model training with limited annotations through the generation of pseudo-labels. This method was tested across various cancer types, showcasing its potential to enhance the accuracy of cell detection in complex biological samples.

[pros]
The semi-supervised learning model using YOLOv8 and the generation of pseudo-labels for training with limited annotations is novel in the context of MxIF imaging for detecting different cell types across various cancer tissues. Conducting experiments across a diverse range of cancer subtypes demonstrates the method's robustness and adaptability to tissue heterogeneity. The authors' approach improved the F1 scores across different cell types, demonstrating the efficacy of incorporating pseudo-labels into the training process. 

[cons]
The paper falls short in providing a comprehensive comparison with SOTA methods, limiting the ability to fully assess the novelty and efficacy of the proposed method against existing solutions. The paper lacks crucial details about the experimental setup, including the total number of pseudo-labels generated and the methodology for tuning this parameter. The concern of overfitting, especially with semi-supervised methods that rely heavily on pseudo-labels, was not adequately addressed.

Reviews: review: paper_summary: paper_summary: In this paper, the authors introduce a method for detecting cell types in multiplex immunofluorescence (mIF) imaging. The approach first involves training a model, specifically YOLOv8, using a limited set of annotations. This trained model is then used to generate pseudo-labels, which are in turn used to retrain the model. The authors have conducted extensive experiments across various images representing different types of cancers and tissues.
main_review: main_review: The authors have responded to my concerns about the previous lack of comparisons by presenting baseline values derived from established cell segmentation methods, and about reproducibility by sharing their code. As pointed out by another reviewer, incorporating additional experiments that explore different semi-supervised strategies could further enhance the manuscript's quality.
strength_weakness: strength_weakness: - The paper lacks a comparative analysis with state-of-the-art (SOTA) methods. For instance, the method could be compared with models such as Stardist, or Cellpose, pre-trained on immunofluorescence (iF) data and fine-tuned with the annotations provided here, with a simple thresholding method for cell classification. Furthermore, the performance of YOLOv8 trained solely on partial annotations is not discussed for all five cancer types in the evaluation dataset, which limits the ability to directly assess the interest of the proposed method compared to using the model trained on limited annotated data alone,  which appears to already exhibit satisfactory performance.
- The description of the experimental setup is missing critical details, such as the total number of pseudo-labels generated, the methodology used to tune this parameter, and the assessment of pseudo-label quality. The paper mentions strategies like Consistency-based Semi-supervised learning for object Detection (CSD) to prevent overfitting but fails to clarify if they were implemented or how they were adapted. The potential for overfitting is a significant concern with methods of this nature, and the results indicate some degree of overfitting, particularly in the "others" cell type across the four additional cancer types, raising important questions about the method's robustness.
- The clarity and structure of the paper could be improved, as it is currently challenging to follow. The blending of experimental settings with methods and results, coupled with a literature review that is not organized chronologically, hinders the reader's comprehension of the paper, and makes it difficult to identify clearly the contribution of the authors.
questions: questions: To qualify for publication at MIDL2024, the authors must improve the overall clarity and structure of the manuscript, clearly highlighting the novel contributions of their work. Additionally, it is imperative to undertake further experiments that compare their approach with state-of-the-art (SOTA) methods and evaluate the effectiveness of the model when trained solely on partial annotations on the evaluation datasets. These steps are essential to demonstrate the value and efficacy of the proposed semi-supervised learning method.
paper_summary: paper_summary: This paper discusses a semi-supervised approach developed to improve cell detection in mIF within the tumor microenvironment. The challenge arises from limited and unevenly distributed annotations for training cell detectors. The authors tested three object detection models with tremendous partially annotated data from different cancer types. An enriched dataset was created using pseudo labels generated by YOLOv8s. The fine-tuned model achieved high accuracy on fully annotated data from five cancer types, demonstrating the effectiveness of the semi-supervised approach in cellular analysis of mIF.
main_review: main_review: My initial comments were addressed satisfactorily; however, the experimental approach to testing the mitigation strategies for pseudo label inaccuracies remains partially unexplored. To more rigorously evaluate these strategies, I recommend comparing the pseudo labels with pathologist-generated labels in a small, targeted region. This comparison could provide deeper insight into the effectiveness of the pseudo labeling process. Although this suggestion might extend beyond the current scope of your study, it is crucial to understand how discrepancies in pseudo labeling could potentially compromise the entire training process. Detailed exploration and documentation of such scenarios would significantly strengthen the manuscript.
strength_weakness: strength_weakness: * The representation could benefit from some improvement. For instance, the description accompanying Figure 1 may require enhanced clarity to better convey the process of training all detectors on partially annotated datasets and the subsequent selection of the separate top-performing model for pseudo label generation through an iterative loop.

* The paper repeatedly claims that the improvement is significant. However, there is no statistical significance test provided as evidence.
questions: questions: Please review the weaknesses and detailed comments sections. Cross-validation is encouraged but not mandatory during the rebuttal phase.
paper_summary: paper_summary: In this paper, the authors propose to enhance cell detection in low-data regimes for multiplex immunofluorescence imaging by using self-distillation. They train three different detection models on a small manually annotated dataset and chose YOLOv8s of these as the best-performing one. In the next step, they predict unseen data by the model to generate pseudo-labels. Another YOLOv8s model is trained using the combination of manual and pseudo-labels. The authors evaluate the resulting model on fully, manually annotated test data of 5 different cancer types.
main_review: main_review: My 2 major concerns were addressed in the last revision. Regarding the contributions: Thank you for pointing out the differences between your apporach and Abbasi et al. This is now more clear to me. Regarding the comparison to SOTA: Thank you for the comparison with CellPose and StarDist. While the the comparison with these methods is useful (although being segmentation instead of detection, but requested by another reviewer) it is not a fair comparison as the Yolo model was trained with the specific training dataset while StarDist and CellPose were pretrained on different data. Now that your contribution is an advancement over Abbasi et al., it would be necessary to also compare with this method. Otherwise we cannot tell whether your approach actually is an improvement.
strength_weakness: strength_weakness: * the paper is not very well structured. Examples are
  * in the introduction, related work for cell segmentation is discussed (Greenwald et al), then cell classification (Amitay et al. etc) and then segmentation again (Schmidt et al. etc). 
  * The results section contains a lot of description on the evaluation process, which would belong to an evaluation section and a lot of result interpretation, which would belong to the discussion section
* There is quite some redundancy in the paper
* The experiment of the different annotation levels is entirely in the appendix, still its results are referred to in the discussion
* In the whole paper it is not clear whether the model performed only detection of cells or also classification. Are there 3 different classes the model has to decide between or is just the evaluation separated into those classes?
* The results of the proposed approach are not compared to any other approaches. Comparison to state-of-the-art methods or comparison to the YOLOv8s model without training on the pseudo-labels would have been useful
* The paper lacks novelty, as the methods are already state of the art (as the authors explain in the introduction). Also, the validation is very limited such that there are only limited new findings on the application of the method on mIF images
questions: questions: The authors should be very clear on the contributions of their work and support their claims by performing the required evaluations. A rewrite of their paper following a clear structure would then help to get this message across

Comments: title: The response details improvements to the manuscript, including comparative analysis with SOTA methods and clarifications of unique contributions, highlighting methodological in pseudo-label generation and domain-specific training for mIF data analysis.
The response highlights the inclusion of comparative analysis with SOTA methods and clarifies the generation and validation of pseudo-labels to address reviewers' concerns.
Clarifying of Our Approach: Comparative Analysis, Pseudo-Labels, Methodological, and Manuscript Structure revised
Clarification and Enhancement of Methodology for Cell Detection in mIF Imaging
Addressing Feedback for Improved Cell Detection in mIF Imaging
comment: We are thankful for your great detailed review and constructive feedback, we address the raised concerns with modifications and clarifications implemented in our new revised manuscript regarding to SOTA and clarification of our contributions:

1. Lack of Comparative Analysis with SOTA Methods:Thanks for the great feedback and suggestions. We have improved our manuscript with additional experiments and discussions to address the concerns raised. We have included results from StarDist and Cellpose in Section 3.2 of our manuscript. These experiments were conducted applying their pre-trained models to our validation dataset. Specifically, we utilized only the blue channel for prediction and treated all cells as a single class to align with our research context. The performance metrics for these methods were evaluated by aligning the centers of predicted segmentation masks within the bounding boxes of annotated cell centers, acknowledging the absence of segmentation masks in our ground truth. 
The results from StarDist and Cellpose are now detailed in the manuscript, indicating a precision of 0.450, a recall of 0.210, and an F1 score of 0.284 for StarDist, whereas Cellpose achieved a precision of 0.862, a recall of 0.230, and an F1 score of 0.363. These findings confirm our decision to focus on detection methods, like YOLO, optimized through our training pipeline with only partial annotations. Furthermore, they underscore the challenges and concerns when applying segmentation methods to our specific dataset. 
In the discussion and limitations sections, we have expanded upon the implications of these comparisons. We acknowledge our primary aim of identifying cell centers and detection and juxtaposing our approach with state-of-the-art methods,  StarDist and Cellpose, validateing our methodology's efficacy and opening routes for future research. This includes integrating segmentation masks to enhance cell annotations and improving comparative performance against leading segmentation methods. 
| Method   | P     | R    | F1    | 
|----------|-------|------|-------| 
| Stardist | 0.450 | 0.210| 0.284 | 
| Cellpose | 0.862 | 0.230| 0.363 | 
| YOLOv8s | 0.904 | 0.850| 0.847 | 

2. clarification of contribution:
Thanks for great feedback, we appreciate the opportunity to clarify the technical distinctions between our approach and the work of Abbasi et al. on enhancing object detection models with semi-supervised learning and pseudo-label generation. Our methodology, while inspired by the broad principles outlined by Abbasi et al., incorporates several key innovations and domain-specific adaptations that distinguish it significantly: 
- Pseudo-Label Generation Strategy: Abbasi et al. employed a framework that relied on object detectors pre-trained (proxy network) on individual datasets to generate a set of pseudo-labels for unlabeled instances. Our method differentiates with this by using an iterative pseudo-labeling strategy tailored for mIF data. This involves dynamically generating pseudo-labels based on model confidence (> %90 confidence rate) and validation on fully annotated dataset annotated by pathologists. 

- Training Process: Our training process incorporates domain-specific augmentations and optimizations to improve detection performance across diverse cancer types, which is critical for applications in precision medicine and pathology research. 

- Also based on your suggestion we changed that sentence “this research contributes to the field of mIF image analysis by introducing an advanced semi-supervised learning model for cell detection, paving the way for future explorations and practical implementations” to “our study contributes to mIF imaging analysis by introducing a semi-supervised learning model specifically designed for cell detection across different cancer types. By integrating a pseudo-label generation strategy, we aim to improve detection accuracy.”
We thank Reviewer for your great feedback. Our response aims to address the concerns raised about SOTA and Pseudo label generating:

1. Lack of Comparative Analysis with SOTA Methods: 
Thanks for the great feedback and suggestions. We have improved our manuscript with additional experiments and discussions to address the concerns raised. We have included results from StarDist and Cellpose in Section 3.2 of our manuscript. These experiments were conducted applying their pre-trained models to our validation dataset. Specifically, we utilized only the blue channel for prediction and treated all cells as a single class to align with our research context. The performance metrics for these methods were evaluated by aligning the centers of predicted segmentation masks with the bounding boxes of annotated cell centers, acknowledging the absence of segmentation masks in our ground truth.  
The results from StarDist and Cellpose are now detailed in the manuscript, indicating a precision of 0.450, a recall of 0.210, and an F1 score of 0.284 for StarDist, whereas Cellpose achieved a precision of 0.862, a recall of 0.230, and an F1 score of 0.363. These findings confirm our decision to focus on detection methods, like YOLO, optimized through our novel training pipeline with only partial annotations. Furthermore, they underscore the challenges and conserns when applying segmentation methods to our specific dataset. 
In the discussion and limitations sections, we have expanded upon the implications of these comparisons. We acknowledge our primary aim of identifying cell centers and detection and juxtaposing our approach with state-of-the-art methods,  StarDist and Cellpose, validateing our methodology's efficacy and opening routes for future research. This includes integrating segmentation masks to enhance cell annotations and improving comparative performance against leading segmentation methods.  
This addition provides a comprehensive justification for our approach, emphasizing its advantages over existing solutions while acknowledging the constructive potential for methodological enhancements in future studies. 

| Method   | P     | R    | F1    | 
|----------|-------|------|-------| 
| Stardist | 0.450 | 0.210| 0.284 | 
| Cellpose | 0.862 | 0.230| 0.363 | 
| YOLOv8s  | 0.904 | 0.850| 0.847 |

2. generation of pseudo-labels: hank you for your feedback. To clarify and address the concerns of potential contradictions as you highlighted, we have revised our explanation as follows: Upon generating new labels, we initially applied a confidence threshold of 90%. Among these labels, we ensured the inclusion of all labels that align with the ground truth or partial annotations provided by pathologists, in addition to the new labels generated. This verification process assures us of the reliability of our pseudo labels, ensuring we incorporate only the most dependable ones into our dataset. Here is full modified sentences in our manuscript: 

 “In our approach, pseudo labels were first generated by selecting predictions with a confidence level above 90\%. These high-confidence pseudo labels were ensured to contain almost all initial partial annotations from pathologists. This process maintained pseudo labels with high accuracy, acting as an additional verification step. Collectively, the method used a confidence threshold for the initial generation and ground truth comparison for validation, ensuring that only the most reliable pseudo labels were added to the final training dataset” 

3. data availability:
Thanks for asking about our code and data. The data from our study cannot be fully shared right now and can be deposited into the public repository only after the publication of the clinical study because of some internal policies we need to follow. But we have still managed to do a couple of things:  
- We have included some samples of our dataset in GitHub. Highlighting our patches and annotations formats.  
- We have put all the related code on GitHub for anyone to look at and use.  

 
Thanks for understanding. We are all for sharing our science as openly as possible within these guidelines.  
Our GitHub repository: https://github.com/idso-fa1-pathology/semi-supervised-cell-detection
Thank you for your revision. Several of my concerns have been addressed. However, there are still two major issues: Firstly, the paper lacks a comparison with other state of the art. Secondly, the contributions are still not clear. They are stated in 2 places in the paper and read like if the authors have invented ("[...] introducing an advanced semi-supervised learning mode [...]") this semi-supervised approach while at the same time crediting Abbasi et al. for this. In my view, the authors have only applied this existing technique to a new dataset. However, for a validation paper, more evaluation would have been required (such as comparison to SOTA)
Thank you for addressing my concerns and the additional experiments that compare the performance of the YOLOv8 model trained solely with partial annotations to your proposed method, as well as for detailing the steps taken to prevent overfitting. These additions indeed underline the merit of your approach. However, I maintain certain concerns regarding the comparative effectiveness of your strategy against state-of-the-art (SOTA) methods and the methodology behind constructing the pseudo-labels.

The inclusion of experiments with Stardist is appreciated, yet it does not fully negate the potential relevance of segmentation methods for cell detection in this context. Despite Stardist's shortcomings in your tests, other methods mentioned in your introduction, like Cellpose or CellProfiler[1], applied on the blue channel, might offer better generalization to mIF imaging. Additionally, employing a basic approach with QuPath[2], followed by straightforward thresholding or leveraging a simple machine learning classifier trained on partial annotations for classification, could serve as a practical baseline. 
Moreover, the Stardist results add value to the paper by demonstrating that the application of such methods is not always direct in your setting and underscores the rationale for focusing on detection-based methods like YOLO, which benefit from your novel training pipeline by requiring only partial annotations. However, further comparison with other baselines is necessary to fully justify the advantages of your proposed approach compared to currently used solutions.

Regarding the generation of pseudo-labels, your explanations have clarified several points, yet some aspects remain confusing. In particular, you mention that the pseudo-labels were curated using a confidence threshold, however, Section 2.3 suggests a direct comparison of these pseudo-labels with ground truth annotations from pathologists ("We evaluated the generated pseudo labels through a designed loop by comparing them with the ground truth") which seems contradictory.

Lastly, could you provide information on the availability of the code and data used in your experiments? This would greatly aid in the reproducibility of your results.

[1]: Stirling DR, Swain-Bowden MJ, Lucas AM, Carpenter AE, Cimini BA, Goodman A (2021). CellProfiler 4: improvements in speed, utility and usability. BMC Bioinformatics, 22 (1), 433. . PMID: 34507520 PMCID: PMC8431850
[2]: https://qupath.readthedocs.io/en/stable/docs/tutorials/multiplex_analysis.html
We thank Reviewer  for your constructive feedback. Our response aims to address the concerns raised, underlining the revisions made to enhance our manuscript's clarity, rigor, and comparative analysis depth.

1. Lack of Comparative Analysis with SOTA Methods: 
We thank the reviewer for the comment. Since the objective of this work was cell detection and classification, we focused on comparing against models developed for detection, such as Fast R-CNN, YOLOv5, and YOLOv8. Cellpos and Stardist are cell segmentation models and not directly applicable here. Stardist expects full cellular masks and it doesn't match our annotations. Without finetuning, the trained nuleus detection Stardist model applied to the blue channel in our mIF images (makers for nucleus), obtains an F1-Score of 0.35 in the validation dataset. We associate this poor performance with characteristics of mIF imaging such as poor definition of cell boundaries as well as small sizes of cells in our data. For finetuning starDist, we adjusted our labels by expanding the centroids of annotations by 50 pixels and creating circular masks for each annotation and achieved the average precision and recall scores of 0.284 across cell types. Given this poor performance compared to Yolov8 (Table 2, 0.848), we decided to not include these results in the manuscript as we believe this quick data preparation approach may have not been sufficient to provide StarDist a fair chance.  

2. Performance of YOLOv8 with Partial Annotations: 
Thank you for your feedback. As suggested by the reviewer, to evaluate the contribution of pseudo labels to improve the task performance across the five cancer types, in this version we compare the performance of models with and without pseudo labels. We show that incorporating pseudo labels generation significantly improves F1 score (CD45: 0.656 vs. 0.949, panCK: 0.417 vs. 0.880, Others: 0.359 vs.0.740) with p values (Wilcoxon test) of 3.0517578125e-05, 1.9073486328125e-06, 3.0517578125e-05, respectively. We have incorporated these results in Figure 3A and section 3.2.
3. Experimental Setup and Overfitting Concerns:
Thank you for this feedback. We have clarified the increase in dataset size after incorporating pseudo-labels: from initial counts of 11,643 for CD45, 15,228 for panCK, and 9,489 for Others, to 140,000, 200,000, and 120,000 for CD45, panCK, and Others, respectively. Also, we have updated Section 2.3 to address concerns about parameter tuning, pseudo-label quality, and overfitting. This procedure includes our method for creating pseudo-labels. We guaranteed the accuracy of these pseudo-labels by only accepting those with high confidence, that is, predictions with a confidence level of over 90%. This means new predicted labels were added only if their prediction confidence threshold was above 90%. As for mitigating overfitting we adopted strategies including dropout, data augmentation, regularization, and early stopping.  Furthermore, the model's performance was evaluated across five fully annotated cancer types not encountered during the training phase, resulting in average precision, recall, and F1 scores of 0.904, 0.851, and 0.848, respectively, for cell detection (calculated based on table 2 in manuscript). Also, the evaluation was conducted on a limited sample of four patches per cancer type. Considering the heterogeneity of tumors, this sample size may not have been sufficient to capture the full spectrum of variability present in each cancer type, potentially leading to an imbalanced testing set. This imbalance might have contributed to the lower performance observed in the 'Others' category for Urothelial Carcinoma and cholangiocarcinoma. 
4. Clarity and Structure of the Paper: 
Acknowledging the feedback on the need for improved clarity and structure, we have undertaken a thorough revision of the manuscript. This includes a clearer separation of experimental settings, methods, and results, and a reorganized literature review to better highlight the novelty and contributions of our work. Also we removed some redundancy and These changes aim to facilitate a better understanding of our approach and its significance in the field of cell detection in mIF imaging.
We are thankful to Reviewer 3 for their detailed review and constructive feedback. Below, we address the raised concerns with modifications and clarifications implemented in our revised manuscript:

1. Manuscript Structure and Clarity:
Thank you for your valuable feedback. 
- We have carefully restructured the introduction section to ensure that all related literature is presented cohesively and consistently. 
- The results section has been restructured to clearly separate the evaluation process and interpretation of results, aligning content more appropriately with the sections.
- Redundancies identified across the paper have been meticulously removed to streamline content and improve readability.
- Details on the experiment of different annotation levels, previously in the appendix, have been refined to ensure a more integrated discussion within the main text.
2. Model's Functionality (Detection and Classification):
Thanks for your comments, our model is designed for both detection and classification of cells. As illustrated in Figure 1, the model differentiates among three categories: immune cells (CD45+), epithelial and cancer cells (panCK+), and others (CD45-panCK-). This clarifies that the evaluation encompasses these specific classes, detailing our method's capacity to not only detect but also accurately classify cells into these predetermined categories. 
3. Comparison to Other Approaches and with partial annotaions:
- Thank you for your feedback. As suggested by the reviewer, to evaluate the contribution of pseudo labels to improve the task performance across the five cancer types, in this version we compare the performance of models with and without pseudo labels. We show that incorporating pseudo labels generation significantly improves F1 score (CD45: 0.656 vs. 0.949, panCK: 0.417 vs. 0.880, Others: 0.359 vs.0.740) with p values (Wilcoxon test) of 3.0517578125e-05, 1.9073486328125e-06, 3.0517578125e-05, respectively. We have incorporated these results in Figure 3A and section 3.2, Also, we have added the Area Under the Curve (AUC) performance metrics into Appendix C and referenced it in Section 3.2 of the results for a more thorough evaluation of the model's discriminative capacity. 
4. The references to Hinton et al, is at the wrong position, it should be after the term "Self-distillation" and not after its application to immune infiltration. 
Thanks, the reference location has been modified 
5. Evaluation metrics are not explained. mAP50 should at least have one sentence of explanation or a reference to one. 
 Thank you. We have added this part (mean Average Precision at 50% Intersection Over Union (IoU)) to the sentence: “The table presents key measures of model performance, recall, and mAP50 (mean Average Precision at 50% Intersection Over Union (IoU)), which are central to our evaluation criteria.” 
6. The manual annotations:
Thank you for your valuable feedback. To clarify the annotation and model training process further, we have updated the methods section of our manuscript as follows: After pathologists annotated the center of cells by marking approximately 10% of the cells across 10 images, we proceeded to draw 50*50 pixels bounding box around each point. In total, pathologists annotated 11,643 cells as CD45+, 15,228 as panCK+, and 9,489 as Others. This step was crucial for adapting the point annotations for training object detection models, which require bounding box inputs. For the evaluation of our model, we compared the predicted bounding boxes with this ground truth bounding boxes to accurately measure false positives rate and calculate precision, ensuring the model's effectiveness in detecting and classifying cell types. 
7. Contributions and Structure:
- We have revised the sentence for clarity: "The study also incorporated fully annotated datasets from five distinct cancer types during the final validation phase. This was to evaluate the model's ability to perform across a range of cancer cell appearances and histological conditions." 
-  Thanks for great feedback, we have updated Section 2.3 to address concerns about parameter tuning, pseudo-label quality, and overfitting. Our approach ensures reliable pseudo labels by including only high-confidence predictions (over 90%). The final model's performance across five distinct cancer types, which were fully annotated by expert pathologists not seen during training, attests to our method's robustness and effectiveness against overfitting. 
- Thanks for feedback, kindly we revised to this sentence to avoid contradiction: “YOLOv5s also demonstrated high mAP50 and competitive recall scores, nearly matching those of YOLOv8s in certain aspects. However, the comprehensive analysis highlights YOLOv8s's slight edge in performance, especially in terms of recall, marking it as the most accurate and reliable among the tested models.” 
- Thanks for the feedback; we removed the repetitive reference.
We are grateful for Reviewer 2’s comprehensive review and constructive suggestions. Our revised manuscript has been updated to address the points raised, as outlined below.

1. Clarity and Representation Improvements: 
In response to the feedback on enhancing the clarity of Figure 1, we have revised the description to more effectively convey the process of training detectors on partially annotated datasets and the selection process for the top-performing model used in pseudo label generation. This clarification aims to improve the reader's understanding of our methodology and its iterative nature.
2. Statistical Significance of Improvements: 
Thank you for your feedback. As suggested by the reviewer, to evaluate the contribution of pseudo labels to improve the task performance across the five cancer types, in this version we compare the performance of models with and without pseudo labels. We show that incorporating pseudo labels generation significantly improves F1 score (CD45: 0.656 vs. 0.949, panCK: 0.417 vs. 0.880, Others: 0.359 vs.0.740) with p values (Wilcoxon test) of 3.0517578125e-05, 1.9073486328125e-06, 3.0517578125e-05, respectively. We have incorporated these results in Figure 3A and section 3.2, Also, we have added the Area Under the Curve (AUC) performance metrics into Appendix C and referenced it in Section 3.2 of the results for a more thorough evaluation of the model's discriminative capacity. 
3. Inclusion of AUC Performance Metrics: 
Thank you for your valuable feedback. As suggested, in this version we have incorporated the AUC metrics (0.90, 0.80, 0.83 for CD45, panCK, and Others, respectively) into Appendix C and referenced it in Section 3.2 of the results for a more thorough evaluation of the model's discriminative capacity. 
4.Mitigating Error Propagation from Pseudo Labels:
Thank you for your great feedback. In this version we have clarified the details related to parameter tuning, pseudo-label quality, and overfitting. We ensured reliability of pseudo labels by including only high-confidence predictions (outputs with > 90%). To evaluate if errors during pseudo labeling generation would propagate to the model’s performance, we compare the performance of two approaches, with and without pseudo labels, in classifying cells. In this version (Figure 3A) we show that in data unseen during training, the addition of pseudo labels boosted the model performance; hence, resulting very unlikely a propagation of error during the pseudo label generation step that would have a significant impact of cell detection and classification.  
5. Clarification of Scale in Figure 1: 
Thanks for feedback, unit is added as requested.