Boundary-aware Contrastive Learning for Semi-supervised Nuclei Instance Segmentation

Authors: ye  zhang
ziyue  wang
yifeng  wang
hao  bian
linghan  cai
hengrui  li
lingbo  zhang
yongbing  zhang
z  zhang
  wang
y  zhang
Venue: MIDL_io_2024_Conference
Published: 2024-01-27 12:16:30

Abstract: Semi-supervised segmentation methods have demonstrated promising results in natural scenarios, providing a solution to reduce dependency on manual annotation. However, these methods face significant challenges when directly applied to pathological images due to the subtle color differences between nuclei and tissues, as well as the significant morphological variations among nuclei. Consequently, the generated pseudo-labels often contain much noise, especially at the nuclei boundaries. To address the above problem, this paper proposes a boundary-aware contrastive learning network to denoise the boundary noise in a semi-supervised nuclei segmentation task. The model has two key designs: a low-resolution denoising (LRD) module and a cross-RoI contrastive learning (CRC) module. The LRD improves the smoothness of the nuclei boundary by pseudo-labels denoising, and the CRC enhances the discrimination between foreground and background by boundary feature contrastive learning. We conduct extensive experiments to demonstrate the superiority of our proposed method over existing semi-supervised instance segmentation methods.

Main paper: 
 introduction: 
 Nuclei instance segmentation is essential in the quantitative analysis of pathological images. The characteristics of nuclei, including their size, morphology, and distribution, can provide valuable insights into the tumor microenvironment, thereby offering crucial support for cancer diagnosis, staging, and grading processes (Khened et al., 2021; Hollandi et al., 2022) . In recent years, deep learning techniques have made remarkable advancements in nuclei segmentation (Zhang et al., 2024a,b) . DCAN (Chen et al., 2016) adopts a dual-branch decoder architecture to predict semantics and contours simultaneously to enhance the instance distinguishing. HoverNet (Graham et al., 2019) incorporates distance and gradient constraints to split individual instances effectively. Similar methods such as CDNet (He et al., 2021) , and CellPose (Stringer et al., 2021) are also proposed to address overlapping nuclei challenges. However, these supervised methods typically rely on pixel-level annotations, which are time-consuming and labor-intensive and need professional guidance, hindering the development of models. Therefore, developing a technique that can effectively address the dependency on manual annotation for nuclei instance segmentation is crucial.A common approach to address the problem of scarce labeled data is semi-supervised learning (Reddy et al., 2018; Van Engelen and Hoos, 2020) . During the training process, abundant unlabeled and insufficient labeled data are used to train the network. The existing semi-supervised methods mainly leverage prior information to improve the pseudo-label quality. For example, ShapeProp (Zhou et al., 2020b) combines the information from bounding boxes and partially annotated masks to improve the segmentation accuracy of target regions based on Mask R-CNN (He et al., 2017) . PAIS (Hu et al., 2023) uses a dynamic alignment loss to address the misalignment problem between classification and segmentation results, and then a new threshold filtering method for pseudo-labels is proposed. PointWS-SIS (Kim et al., 2023) balances false negative and false positive errors by utilizing point supervision prior information. However, due to the low color contrast differences between the nuclei and tissues, these methods still have defects in generating nuclear pseudo-labels, limiting the application of semi-supervised instance segmentation in pathological images.Some methods use pseudo-label optimization strategies to enhance nuclei segmentation accuracy in semi-supervised scenarios. MMT-PSM (Zhou et al., 2020a) integrates multiple data-augmented segmentation results to construct reliable predictions and enhance pseudolabels' confidence. CDCL (Wu et al., 2022) uses feature contrastive learning to promote feature consistency between the teacher and student networks, thus improving the quality of pseudo-labels. PG-FANet (Jin et al., 2024) employs a pseudo-label guided module that aggregates multi-scale, multi-stage features to enhance segmentation performance. However, nuclei exhibit diversity in morphology and size, and in cases with limited annotations, it is challenging for the teacher network to capture the complete range of nuclei shape features. Consequently, the generated pseudo-labels often contain edge noise because existing pseudo-label optimization methods lack specific designs for denoising nuclei boundaries, which always leads to inaccurate nuclei boundary predictions.In this paper, to address the issue of boundary noise in nuclei segmentation, we propose a coarse-to-fine boundary-aware contrastive learning network for semi-supervised nuclei segmentation (BASS1 ). Firstly, we design a low-resolution denoising (LRD) segmentation head that promotes boundary smoothness. Additionally, within this segmentation head, we use a low-weight loss for the nuclei boundary region optimization, which reduces the impact of uncertain boundary prediction during training. Secondly, to minimize boundary noise further, we design a cross-RoI contrastive learning (CRC) module that finely partitions the internal, external, and boundary regions of nuclei, enhancing the discriminative capability of nuclei boundary features. To demonstrate the effectiveness of our proposed method, we conduct comparative experiments and ablation studies on two public datasets. The experimental results show that our proposed method outperforms existing semi-supervised methods, and the ablation studies demonstrate the effectiveness of the proposed modules. 𝑘 ! # 𝑘 $ # 𝑘 % # 𝑘 # # 𝑘 $ " 𝑘 % " 𝑘 # " ： ： ： ： ： ： ： ： 𝑞 ! 𝑞 % 𝑘 ! 𝑘 % 𝑘 $ 𝑘 # 𝑘 ! 𝑘 % 𝑘 $ 𝑘 # m m m Mean Operation 
framework overview: 
 To address the boundary noise problem of nuclei segmentation under a semi-supervised scenario, we propose a coarse-to-fine boundary-aware denoising model, as shown in Fig. 1 . Our whole training process can be divided into three stages. First, the labeled dataD L = {(x i , y i )} N i=1is used to train a teacher model. In this step, we employ Mask R-CNN (He et al., 2017) as the baseline, and the loss function of the teacher network is defined as follows:LossEQUATIONwhere L t seg is the loss of the segmentation head, and L t det is the loss of the detection head, which consists of the classification loss and regression loss. Then, the trained teacher network is employed to generate pseudo-labels y p j for input x j . To reduce the uncertainty of pseudo-labels, we employ box and pixel threshold filtering to generate high-confidence pseudo-labels. Finally, we combine the labeled data D L and the generated pseudo-labeled data D U = {(x j , y p j )} M j=1 to train the student network. In the student network, our developed denoising methods are shown in the green box of Fig. 1 , which consists of a low-resolution denoising (LRD) module and a cross-RoI contrastive learning (CRC) module. The LRD employs low-resolution pseudo-labels as supervision information to promote the smoothness of nuclei contours. Meanwhile, the CRC utilizes boundary-aware contrastive learning to enhance the discriminative capability of contour features. In the training process, the overall loss is designed as follows:EQUATIONwhere L nmh represents the naive high resolution segmentation loss, L lrd represents the low-resolution segmentation loss, and L cl represents the contrastive learning loss.Before training the student network, we chose high-confidence instances as pseudo-labels to reduce the uncertainty of the samples during student network training. In the pseudolabel generation stage, the teacher network outputs probability values for instances (box value) and mask probabilities (pixel value). We consider the pixel-level threshold v p to be the hyperparameter. As for the box-level threshold v b , we assume the nuclear number distribution is consistent between labeled and unlabeled data. Based on this assumption, we uniformly sample 91 values between 0.1 and 1.0. Then, we iteratively apply these probabilities to filter the instances and calculate the number distribution of nuclei in the unlabeled data. Finally, we select the threshold closest to the distribution of labeled data.We validate the effectiveness of the threshold filtering method in ablation experiments.In the naive Mask R-CNN (He et al., 2017) , the RoI head outputs a 14 × 14 feature map containing boundary noise. In the subsequent convolution process, Mask R-CNN increases the size of the feature map to capture more semantic information, but the boundary noise is also amplified. To avoid amplified noise effects, we design a low-resolution denoising module as shown in Fig. 1 (b), which utilizes the low-resolution pseudo-labels as supervision for model training. In the LRD, BASS directly performs segmentation in the 14 × 14 feature map. This approach effectively smooths the boundaries and initially reduces the noise in nuclei boundaries. Furthermore, to minimize the impact of boundary uncertainty on segmentation, we apply a weighted loss to the low-resolution segmentation head. Specifically, pixels in the boundary region are assigned a lower weight, and other areas are set to a high weight.According to previous studies (Wang et al., 2022) , although low-resolution images can reduce the boundary noise, they lose some detailed information. To preserve the details, we parallel the original segmentation head and low-resolution prediction head to perform the segmentation task simultaneously, as shown in Fig. 1(b) . In this manner, the output mask head decreases the influence of the original feature noise and keeps more details.In the subsection, we propose an elaborate denoising method named cross-RoI contrastive learning. It leverages labeled data to train a boundary feature extraction module, and then the module is applied to learn the embedding of unlabeled data, which can mitigate the impact of boundary noise caused by pseudo-labels and enhance the feature discrimination ability of foreground and background. In general, object boundaries typically correspond to hard-to-classify samples, and their embeddings are highly unstable. To avoid the impact of features from difficult samples on the representation of easy-to-classify samples, we employ region-based contrastive learning and our proposed CRC is shown in Fig. 1 (c) .First, the input image x is fed into the network for feature extraction and alignment, then we randomly sample two aligned RoI features f i and f j .Based on the contour (plotted in black line) shown in Fig. 1 (c), we split the feature maps into foreground region F and background region B. By shrinking and expanding d distances, we obtain the inner contour (plotted in red dotted line) and outer contour (plotted in blue dotted line). The regions between them and the true contour are represented as the inner boundary and outer boundary. These boundary regions correspond to challenging pixels for classification and can be expressed using the following formula:EQUATIONwhere c i represents the contour pixel closest to pixel p i . At the same time, we also sample other foreground and background pixels, which can be expressed as the following equations:EQUATIONwhere R f represents the set of pixels obtained by excluding R i from F and R b represents the set of pixels obtained by excluding R o from B.Next, we sample pixel features from the sets R i , R o , R f and R b . The sampling ratio is set to α. For feature f i , the sampled features are denoted as k i b , k i o , k i f and k i i respectively. Similarly, for feature f j , the sampled features are denoted as k j b , k j o , k j f and k j i respectively. Then, we calculate the query features of background and foreground across RoIs through:EQUATIONwhere M represents the averaged operation of vectors. Finally, to narrow the same category feature distance and expand the feature distance between foreground and background. We calculate four pairs of contrastive losses as follows:EQUATIONwhere CL represents contrastive learning loss. k b represents the concatenation of k i b and k j b . The calculation of CL is described below:EQUATIONwhere q + and k + represent a pair of positive instances, k -represents a negative instance, and τ is the temperature hyper-parameter. Different from the previous methods, PC 2 Seg (Zhong et al., 2021) extracts positive instance pair of contrastive learning from a single perspective. However, our proposed CRC performs contrastive learning cross-RoI, which enhances the feature generality. 
datasets: 
 Our method is evaluated on the Cryosectioned Nuclei Segmentation (CryoNuSeg) dataset (Mahbod et al., 2021) , the Digestive-System Pathological Segmentation (DigestPath) dataset (Da et al., 2022) , and Multiple Organs Nuclei Segmentation (MoNuSeg) dataset (Kumar et al., 2017) . CryoNuSeg contains 30 images from 10 organs, each with a size of 512 × 512. DigestPath contains 69 images of the digestive system, each with a size of approximately 1500×1200. MoNuSeg contains 30 images from 7 organs, each with a size of 1000 × 1000. Following the previous method (Graham et al., 2019) , we crop all the images to patches of 256 × 256 pixels with an overlap of 128 pixels for data preprocessing. All experiments are carried out with an RTX 3090 GPU. SGD is used as the optimizer. The learning rate, momentum, and weight decay are set to 0.02, 0.9, and 0.001, respectively. Besides, we evaluate the segmentation performance in terms of Dice (Vu et al., 2019) , aggregated Jaccard index (AJI) (Kumar et al., 2017) , and panoptic quality (PQ) (Kirillov et al., 2019) .We compare our proposed BASS against several state-of-the-art methods, including MMT-PSM (Zhou et al., 2020a) , PointWSSIS (Kim et al., 2023 ), ShapeProp (Zhou et al., 2020b) , NoisyBoundary (Wang et al., 2022) and PG-FANet (Jin et al., 2024) . Besides, to validate the improvement of our semi-supervised model, we also compare our method with supervised Mask R-CNN. We trained the models using 1/8, 1/4, and 1/2 of the labeled data on ResNet-50. A more detailed data split is presented in the appendix. Quantitative comparison results on three datasets are displayed in Table 1 , which shows that our method achieves the optimal performance at all three annotation ratios. Even with only 1/8 of the annotations, our BASS exceeds the suboptimal method approximately 1% in PQ. Fig. 2 displays the visual comparison results. We can see that MMT-PSM and NoisyBoundary mistakenly identify nuclei as tissue due to the lack of semantic discrimination between nuclei and tissues. Although ShapeProp and PointWSSIS employ weak labels to enhance the location ability of nuclei, they still have nuclear shape errors. 3 and 4 . Table 3 uses nuclear count statistics to determine the opti-mal box threshold. We can find that the model performs best when the threshold is set to 0.38. When changing the value, the model's performance deteriorated, indicating the effectiveness of using nucleus count for box threshold selection. In addition, Table 4 shows the experiment results of pixel threshold. The table shows that when choosing 0.5 as the threshold, the model achieved optimal performance in terms of Dice and PQ scores. Ablation Studies for Sampling Ratio α. We conduct sampling ratio experiments and set four sampling ratios of 0.1, 0.3, 0.5, and 0.7 in the CRC. The experimental results are shown in Fig. 3 . The table shows that as the sampling ratio increases, the performance gradually improves, indicating that the sampling ratio indeed influences the performance.When the sampling ratio is large, the model obtains more sampled pixels, resulting in better contrastive learning performance. However, as the sampling ratio increases, the computational cost of the model also increases. Therefore, we select 0.7 as the final sampling ratio, which achieves the best balance between model performance and computational cost.  
conclusions: 
 This paper proposes a boundary-aware contrastive learning model based on the teacherstudent framework for semi-supervised nuclei segmentation. The model utilizes a lowresolution feature supervision head and a cross-RoI contrastive learning module to achieve the nuclei boundary denoising. However, the model trains the teacher and student networks in separate stages, which hinders the student network from effectively utilizing the features extracted by the teacher network. Therefore, in the future, we will adopt an end-to-end training approach for both the teacher and student networks to enhance the information interaction between the teacher and student networks. 
 
 Appendix: Our code is availiable at https://github.com/zhangye-zoe/BASS. 


Decision: This paper proposes a novel boundary-aware contrastive learning model semi-supervised nuclei segmentation. The reviewers found the paper to be well motivated with good results demonstrated by comparative experiments and ablation studies. There were some initial concerns around lack of statistical results pointing to the significance of the results, lack of SOTA methods in methods compared against, issues with clarity, code not being shared which all appear to have been resolved by the rebuttal resulting in 2 weak accept and 1 strong accept recommendation by the reviewers. A fourth reviewer who gave a weak reject did not give a final rating after the rebuttal. Having taken a look at their criticisms myself and the authors rebuttal, I am satisfied in general by the authors' response.

Reviews: review: paper_summary: paper_summary: This paper proposes a method for nuclei segmentation in a semi-supervised setting, using a teacher-student model, with extra regularization involved. 

The results seem good, on two different datasets, but overall the paper is difficult to follow and to fully understand the complete method.
main_review: main_review: The authors have significantly improved the manuscript and clarity, and also shared the public code in the meantime. Congratulation for the good and hard work performed in such a short time.

This addresses most of my initial concerns about the method. DSC and AJI are still pretty much the same metric but that could easily be addressed in a journal extension.

I recommend acceptance of this paper, as a poster.
strength_weakness: strength_weakness: - The method is quite complex and not all parts are necessarily motivated when introduced
- The paper is raher difficult to follow (out of the 5 papers I reviewed this is the one that gave me the most troubles to make sense of), partly because is written as code and not as actual mathematical notation
- ironically, the code is not shared. For such a complex method, I do not think that it could be reproduced without access to the code.
questions: questions: - Not quite a question, but I think that as the rebuttal/discussion period lasts three weeks, there is plenty of time to rework and re-explain the core of the method.
- What are the box and pixel threshold precisely? Do they involve hyperparemeters to set manually?
paper_summary: paper_summary: The authors propose a way to improve the segmentation of nuclei inside glomeruli using semi-supervised learning. The proposed method uses techniques based on pseudo-labeling and denoising of nuclei edges. They comparatively evaluated the method against other ones, achieving better performances on instance segmentation.
main_review: main_review: I commend the authors for carefully addressing all the issues raised in my review. With the addition of another dataset and a more thorough experimental analysis, I am revising my final rating to a strong acceptance.
strength_weakness: strength_weakness: The authors relied on a notably limited dataset, with CryoNuSeg comprising only 30 images and DigestPath containing 69 images. Furthermore, they omitted any statistical significance study through cross-validation, which is crucial for robust evaluation.

Additionally, the conclusion appears to be merely a summary of the paper's content rather than a conclusive analysis or discussion of the findings.
questions: questions: How can you enhance the generalizability of your experiments to more accurately assess this aspect?
paper_summary: paper_summary: This paper proposes a boundary-aware contrastive learning model for semi-supervised nuclei segmentation based on the teacher-student framework. In the student network, a low-resolution denoising module and a cross-RoI contrastive learning module are proposed to ease the contour noises of nuclei from coarse and fine aspects.
main_review: main_review: 1.	The cross-RoI contrastive learning is very complicated and may be unstable due to several factors I have mentioned.
2.	Some details should be clarified as mentioned.
3.	Computation cost is also an important issue.
strength_weakness: strength_weakness: I have some concerns regarding the paper.
1.	The cross-RoI contrastive learning needs a pretrained boundary feature extraction module which is said to train on labeled data. So, does the method need a 2-stage training? If yes, this will make this method complicated, if I change the ratio of labeled data, I need to retrain a boundary feature extraction module. This will also pose another issue: the method will heavily depend on the trained feature extraction module, and the performance of this module will also depend on the quantity and quality of your selected labeled data. So, how to ensure the quality of this feature extraction module? Especially under extremely limited labeled conditions.
2.	Will the denoising only preserve easy cases? Like you have filtered out most of the pixels via box- and pixel-level thresholds. Yet, the filtered pixels can provide more information for training as they may be the hard cases. Will this lead to bias?
3.	More semi-supervised baselines should be adapted to compare. Many methods focused on pseudo-labeling.
4.	What are the weak and strong augmentation used in the paper?
5.     The computation cost should be listed and compared to previous methods.
questions: questions: See above
paper_summary: paper_summary: The study introduces a novel contrastive learning model, designed with an awareness of boundaries, for the task of semi-supervised nuclei segmentation, employing a teacher-student framework. Within the framework, the student network incorporates a low-resolution denoising module alongside a cross ROI contrastive learning module. These modules are specifically aimed at reducing the contour noise of nuclei from both macro and micro perspectives. Comprehensive comparative analyses conducted on two publicly available datasets have demonstrated the enhanced performance of this approach over existing methods in semi-supervised instance segmentation.
main_review: main_review: Thanks very much for the author's response. The paper's quality has significantly improved. My major concerns have been resolved, especially the statistical analysis. Therefore, I continue to support my positive evaluation.
strength_weakness: strength_weakness: The review highlights a significant shortfall in the evaluation's thoroughness, noting its failure to compare the segmentation approach with current state-of-the-art (SOTA) models, despite using public datasets. This gap makes it difficult to accurately assess how the proposed method stacks up against the leading techniques in the field.

Moreover, the analysis lacks a detailed statistical examination. The absence of statistical significance tests, like p-values, raises concerns about the reliability of the differences observed between the proposed method and others. Such analysis is essential in medical image research to ensure findings are not due to chance.

The critique also questions the adequacy of the training sample size for a semi-supervised contrastive learning approach, suggesting that a directly supervised learning method might have been more appropriate. Furthermore, the performance metrics presented, particularly the Dice scores, are considered low. This observation underscores the need for a comparison with fully-supervised methods, such as Mask-RCNN, to provide a clearer understanding of the proposed method's efficacy.
questions: questions: In outlining the key issues in the weaknesses section, I've identified areas that, if improved upon, could substantially elevate both the quality and influence of the paper. Addressing these points would be greatly advantageous for the authors.

Comments: title: Please read and respond to author comments
Reply to Reviewer yGzw
Reply to Reviewer mLet
Reply to Reviewer DPah
Reply to Reviewer 8Bvg
An Unified Reply to All Reviewers
comment: Thanks for the response from the author. The quality of the paper has been substantially improved. My concerns have been addressed. So I still render the positive decision.
Dear reviewers. The authors have posted responses to your reviews. Please take the time to read and respond before March 27.
Thank you for your recognition of our article and your valuable comments. We will reply to your suggestions individually in the hope that these answers can solve your confusion.

Weaknesses

(1) According to your suggestion, we added the latest semi-supervised nuclear comparison method, PG-FANet, and the experiment results are shown in Table 1 of the main text. The experiment results show that the performance of our method is superior to that of state-of-the-art, which confirms the effectiveness of our method.

[1] Jin Q, Cui H, Sun C, et al. Inter-and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation[J]. Expert Systems with Applications, 2024, 238: 122093.

(2) Thanks for your valuable advice, we did neglect statistical testing, which is very important for the nuclear segmentation task. Based on your comments, we evaluated each method for statistical significance. In Table 1 of the main text, we have added the identification of the significance test. Thank you again for your professional advice.

(3) Thanks for your constructive comments; our model is feasible with a small sample size. This is because our contrastive learning is performed on pixel features. At each iteration, the model will randomly sample a fixed proportion (α=0.7) from a mini batch for contrastive learning. Due to randomness, the samples sampled during each iteration are not the same, so a small number of training samples is enough for model training.
Your comments are valuable, full supervision is a suitable application, and our model can work. In the Table 6 of supplement materials we give a fully supervised experiment, and the experiments show that contrastive learning is effective for fully supervised learning. In the future, we will enhance our contrastive learning program to achieve better performance in both fully supervised and semi-supervised tasks.

(4) The low Dice index is caused by two reasons: (1) First, we adopted stricter data partitioning when training the model. We split the training, validation, and testing sets on a 6/2/2, while many articles report results on 8/1/1 or 7/1.5/1.5. (2) Second, it is challenging to conduct a semi-supervised model on CryoNuSeg and DigestPath datasets, which not only have a small size nuclei morphology, but also have a very significant difference in nuclear morphology (CryoNuSeg id a pan-cancer dataset) between the training set and the test set.
In addition, according to your suggestion, we conducted supervised experiments on Mask R-CNN (shown in Table 1 of main text) and the experimental results also showed that the Dice value on CryoNuSeg data was low, but on the whole, our semi-supervised method was still greatly improved compared with the supervised model.

Questions To Address In The Rebuttal

Thank you for your constructive comments and professional guidance on this article. According to your comments in weakness, we have explained these problems one by one in our reply. At the same time, we also added some comparison experiments and revised the methods in the main body and supplementary materials to make the paper read more smoothly. We hope these replies will satisfy you. If you have any other questions, you are welcome to point out, we will make further positive reply.

Justification Of The Preliminary Rating

Thank you again for your recognition of the method and pointing out the shortcomings of our work. Your suggestions have positive guiding significance for our future work, and we will continue to improve this work according to your suggestions.
We sincerely appreciate your suggestions regarding the methods and experiments presented in this paper. We have thoroughly analyzed your questions and provided detailed responses to address them. 

Weaknesses

(1) We sincerely apologize for any confusion caused. We would like to clarify that the two-stage training is not necessary for calculating cross-RoI CL. Our CL is trained in an end-to-end manner. However, during the loss calculation, we only utilize labeled data to compute the contrastive loss, even though the unlabeled (pseudo-labeled) data also generate RoIs. 
In our cross-RoI CL, the boundary feature extraction is carried out on the generated RoIs, which is the outputs of the Mask R-CNN detection step. As described in 'unified reply', the Mask R-CNN is trained in an end-to-end manner and the detection results can be revised in the second step. Therefore, a different labeled ratio don't need train the network from scratch and a small number of samples will not directly impact the feature extraction. Because the segmentation network will continue to correct the detection results.

Also, about ``whether the method you mentioned will depend on the quantity ..."., which can be answered in two ways: (1) In the original Mask RCNN, to enhance the feature extraction capability and consider the size change of the object, the author adopts FPN in the feature extraction network. (2) Since the model be trained end-to-end, and total loss is weighting multiple loss, so the network can continue to optimize objects with inaccurate location in RPN. Therefore, the process will adjust those ROIs to make the prediction closer to the actual object position.

To provide you with a better understanding of our training method, we would like to further elaborate on the details.  Mask-RCNN is a two-stage detection framework. Under mmdetection, detectron or other frameworks, Mask-RCNN can be trained end-to-end. Firstly, an RPN network is used to locate the objects, in which a large number of RoI regions are generated. Secondly, a segmentation network makes segmentation predictions for each extracted RoI feature. Our cross-RoI contrastive learning module is improved based on the existing structure, which does not require an additional training process but only uses the features extracted from RoI. We sincerely hope that these answers have addressed your concerns and cleared up any confusion.

(2) Thank you for your question. In the original article, we lacked a description of this part. Now, we add box and pixel thresholds in Section 2.2 of main text, we hope it can help readers understand our approach. For question "the filtered pixels can provide more information ...", which does exist. But, we can only minimize the influence of threshold selection on the generation of pseudo-labels, so we use the iteration optimization strategy. To facilitate your understanding, we have added the threshold setting in Section 2.2. Besides, to verify the effectiveness of our method, we supplement ablation experiment in Table 3-4. The experiment proves that the threshold optimization is better than the method of setting hyperparameter in box threshold.

(3) This is a good suggestion. We have added the latest state-of-the-art method PG-FANet in the Table 1 of main text.

(4) Following UniMatch, CCVC, and other methods, in SSL, data augmentation and consistency constraints always are used to improve the segmentation performance. Generally, weak data augmentation includes image shape changes, while strong data augmentation includes image color jitter. In this article, we use image resize and rotation for weak data augmentation; strong data augmentations are Gaussian noise and image blur. 

(5) Thank you for your professional advice. We have added the parameters quantity comparison experiments in the Table 9 of supplement materials.

Justification of XX 

(1) Contrastive loss instability does affect training. For this, we visualize the contrastive loss in Fig. 5 of supplement materials, which shows the loss has apparent downward and convergence trend. That's because we only sample part pixels to calculate the CL and take the average operation on the query vector, which can ensure the feature better represent the general features of foreground and background. Therefore, the loss curve is smooth.

(2) According to your suggestions, we answered these detailed questions individually as shown above. 

(3) We compared the computation costs of baseline model in the Table 9 of supplementary material. If any other problems need to be addressed please point out we will further explain or improve.

We hope our responses can address your questions and aided in your understanding of the content presented in this article. If you still have any concerns or issues regarding the model design and experimental procedures, please bring them to our attention. Your feedback is valuable to us, and we are committed to addressing any remaining challenges or inquiries.
Weaknesses

(1) Thank you for your reply and pointing out the shortcomings of our experiment, which is extremely important for presenting our experimental results.
About the amount of data, though the amount of DigestPath data is small, the size of each labeled region is large. In data preprocessing, the overlapping strategy is adopted to split images, and about 5000 patches of 256*256 are obtained on the DigestPath dataset. We consider that this data is enough for model training and validation. At the same time, according to your suggestion, we added a new MoNuSeg dataset to perform experiments, and the results are shown in Table 1 of the main text, which further confirmed the effectiveness of our proposed method. In addition, we supplemented the p-value statistical significance experiment in the Table 1 of mian text, and the experimental results showed that our method met the statistical significance hypothesis. Furthermore, we performed a 4-fold cross-validation experiment and calculated the standard deviation of cross-validation. Due to the limited length of the text, we present the experimental results in the Table 7 of supplementary materials. Thank you again for your suggestions. You are welcome to point out any inadequacies and we will actively revise them.

(2) Thank you for your valuable suggestions. We have read the conclusion carefully, and there are indeed problems you mentioned. According to your suggestions, we have rewritten the conclusion section and expounded the possible improvement of the article in the future. If there is still any need for improvement, please point it out, and we will actively revise it according to your comments.

Detailed Comments

Thank you for your constructive comments. Based on your comments, we have carried out two generalized verification aspects.
First, we conducted a generalization experiment of the model. Specifically, we performed experiments and comparisons on the new data MoNuSeg, and the results are shown Table 1 of main text. From this result, we can see that our method still performs best on the latest data set, confirming that our model has good generalization. 
Secondly, we conducted generalization experiments on the parameters of the model. Specifically, we mutually verified the model performance on CryoNuSeg and DigestPath. Due to the limited length of the text, we present the experimental results in the Table 8 of supplementary materials. The experimentresults show that when our model is test on other dataset, the performance decreases. This phenomenon is because our method does not consider distribution differences among datasets, resulting in a poor cross-data set validation effect.
Finally, according to your suggestion, we modified the conclusion and marked it with blue font in the body. Specifically, we briefly summarize our method and illustrate the shortcomings of our method. This is also the improved direction of our model in the future. Thank you again for your valuable suggestions in our work.

Questions To Address In The Rebuttal

(1) For the generalization problem of the model, we performed generalization experiments on the model and parameters, and the experimental results showed that our model had good generalization performance on the data set. Still, it did have shortcomings in generalization across the data set. We will improve this problem in the future. For example, strong data augmentations help the model train on richer images and feature consistency constraints are used to enhance the generalization performance of features extracted by the model.

Justification Of The Preliminary Rating

(1) Thank you for your comments on the writing and experiments of this paper. We have made careful revisions based on your comments. If there are still other problems in the paper, please point them out, and we will continue to improve them. Thanks again for your comments on this article.
Weakness

(1) Thank you for your questions regarding the method described in the article. We have had a thorough discussion after carefully considering your comments. The problem you have raised may be caused by our inadequate description of the baseline method Mask R-CNN, for which we have added a description of the teacher model baseline in the revised vision in the Section 2.1 Line 90-92. 

The modules proposed in this paper build upon the existing structure of Mask R-CNN. The method itself is designed to be straightforward and approachable. We have taken care to ensure that the proposed modifications and additions are implemented in a manner that maintains clarity and simplicity. 

In the teacher model, we use Mask R-CNN as the baseline, a two-stage detection and segmentation method, which can be trained end-to-end. The first stage of the model is responsible for detecting and outputting some candidate box positions and corresponding confidence scores. The second stage of the model performs segmentation using the detection results, and our LRD and CRC approaches are designed based on the existing framework.

In the student model, we retrain a Mask R-CNN model using data with annotated labels and pseudo-labels. We introduce low-resolution denoising and contrastive learning modules into the segmentation head to achieve boundary denoising.
In the paper, the generation of pseudo-labels does not require training, so the model only needs to train a teacher network and a student network. As for training methods, many wrapped deep learning frameworks can directly serve the segmentation or detection training, such as Mask R-CNN, FCOS, and so on. It is easy to implement the method by directly invoking these libraries (e.g. mmdetection and detectron2).

In addition, based on your comments, we reviewed the method described in this article again. Just as you said, there are some redundant expressions in the opening sentence of each paragraph. We have revised the statement in blue font. Please let us know if you have any questions, and we will actively respond to them. The revised version has been uploaded to openreview. 

(2) Thank you for your patient guidance and advice. Based on your suggestions, we have carefully revised the formula and made it as clear as possible. Specifically, we use more concise and distinguishable symbols to represent the four areas of CL, hoping this will reduce the difficulty of reading. In addition, you can also see the modification result in the main text Eq.3 and Eq.4 of Section 2.4.

(3) Due to our oversight, we did not open source the code in the initial release. Now, we have uploaded the code to GitHub: https://github.com/zhangye-zoe/BASS. You can refer to the instructions in the README.md to run our work. For the introduction of each section, we also added specific explanations in the corresponding implementation parts of the code.

Detailed Comments

(1) Thank you for pointing out the problem in the article in detail. We have simplified the expression of the weight of the loss function, eliminating the w1, w2, and w3 weights.

(2) According to your suggestion, we have improved this part of the formula. You can see the modified result in the annotations in the main text. Thank you for pointing out the problems in the article's expression.

(3) Your suggestions have been beneficial to the revision of our article, so in the body, we have simplified some formula symbols, which will be easier for other readers to read.

(4) As two important indexes for evaluating nuclear instance segmentation, Dice and AJI are similar to some extent. However, their emphasis is different. Dice pays more attention to pixel-level segmentation performance, while AJI tends to evaluate the quality of connected domain segmentation. Following previous works HoverNet and TripleU-Net, we also adopted Dice and AJI as evaluation criteria.

(5) There are indeed many redundant expressions in the previous version, and we have corrected these repeated expressions. These changes have been highlighted in blue for your convenience.

Questions To Address 

(1) Thank you for listing the shortcomings of this article. We have carefully read your suggestions and made detailed revisions, such as method description and formula expression. In addition, we added many comparison experiments and ablation experiments. 

(2) For methodological details of box and pixel threshold, because of limited page of reply, we added them Section2.2 of body. Besides, we validate the effectiveness of this threshold filtering in Section 3.4 ablation experiments. 

(3) Thank you for your feedback on our methods and experimental details. We have provided responses to address these issues, but there may still be some areas that need to be clarified in our replies. If there are any points that you still do not understand, please let us know, and we will be more than happy to provide further clarification.
Thank you for your kind and valuable suggestions. We greatly appreciate your feedback, as it will contribute to the improvement of our article. In the part, I would like to give a unified reply to the questions that have been raised many times in the review. We hope that this unified response can help reviewers o have a general understanding of our modifications.

Firstly, regarding the code, previously, due to our negligence, we did not upload it. Now, we have uploaded the code to GitHub: https://github.com/zhangye-zoe/BASS. We believe that accessing the code will provide you with a better understanding of the methods discussed in the article. Meantime, based on comments of reviewers, we also uploaded the revised text to the following link: https://openreview.net/forum?id=rVx9DiR5Ha&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DMIDL.io%2F2024%2FConference%2FAuthors%23your-submissions)where we highlighted the revised text in blue.

Secondly, about the two questions: (1) box(pixel) threshold choose method (if it is chosen like hyperparameter) and (2) training method (when we conduct contrastive learning whether we need a single training or not). To answer these questions, we need to introduce the baseline Mask R-CNN. 

In this paper, we use Mask R-CNN as the baseline model, and the network can be end-to-end training to achieve detection and segmentation. In the process of object detection, Mask-RCNN outputs some candidate boxes (RoIs) and corresponding confidence scores (box scores). In general, the model uses 0.5 as a threshold to filter RoIs with high confidence. However, given the characteristics of less data annotation and a large number of nuclei in semi-supervised tasks, the fixed threshold may not suitable for all scenarios of nuclear segmentation. Therefore, we used optimization strategies to choose the threshold values $v_b$ and $v_p$.  For your understanding, we added the method description of the box and pixel threshold in the Section 2.2 of main text. This process is simple but effective and the code is uploaded to the following file to facilitate your understanding: https://github.com/zhangye-zoe/BASS/blob/main/scripts/kumar/filter_pl.py.

For the cross-RoI contrastive learning, based on the extracted RoI features in the detection stage, we directly conduct contrastive learning on these RoI features. Thus, they don’t need a single training stage. If you have any further questions or require additional clarification, please let us know, and we will be glad to reply you.

Next, we genuinely appreciate the valuable feedback provided by the reviewers regarding the equation symbols and the conclusions of our article. We have carefully considered their suggestions and made the necessary corrections to ensure the expressions are explicit and easily comprehended. The revised version can be found at the following link: https://openreview.net/forum?id=rVx9DiR5Ha&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DMIDL.io%2F2024%2FConference%2FAuthors%23your-submissions). If there are any further areas you find unreasonable or require improvement, please kindly point them out, and we will continue to enhance the clarity and readability of the paper. 

Finally, we sincerely thank all the reviewers for their insightful suggestions. We have considered their comments carefully and made significant additions to our experimental design. These additions include experiments conducted on the new dataset MoNuSeg, a new semi-supervised segmentation comparison method, cross-validation experiments, and a comparison of model parameters. To ensure that the results of these comparison experiments are accessible to everyone, we presented all the additional experiments in the supplement materials, including Table 6, 7, 8, 9 and Figure 5. Furthermore, we have addressed each reviewer's questions and concerns with thorough responses. If you have any specific questions regarding the methodology, we will happily respond comprehensively. 

Once again, we extend our appreciation to reviewers for detailed suggestions, which have significantly enhanced our article's quality. We value their contributions and remain committed to delivering a comprehensive and robust research paper.