NIS3D: A Completely Annotated Benchmark for Dense 3D Nuclei Image Segmentation
Authors: Wei Zheng
James Cheng Peng
Zeyuan Hou
Boyu Lyu
Mengfan Wang
Xuelong Mi
Shuoxuan Qiao
Yinan Wan
Guoqiang Yu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: benchmark
image segmentation
embryonic nuclei image
3D
fluorescent
TL;DR: NIS3D: A Completely Annotated Benchmark for Dense 3D Nuclei Image Segmentation

Abstract: 3D segmentation of nuclei images is a fundamental task for many biological studies. Despite the rapid advances of large-volume 3D imaging acquisition methods and the emergence of sophisticated algorithms to segment the nuclei in recent years, a benchmark with all cells completely annotated is still missing, making it hard to accurately assess and further improve the performance of the algorithms. The existing nuclei segmentation benchmarks either worked on 2D only or annotated a small number of 3D cells, perhaps due to the high cost of 3D annotation for large-scale data. To fulfill the critical need, we constructed NIS3D, a 3D, high cell density, large-volume, and completely annotated Nuclei Image Segmentation benchmark, assisted by our newly designed semi-automatic annotation software. NIS3D provides more than 22,000 cells across multiple most-used species in this area. Each cell is labeled by three independent annotators, so we can measure the variability of each annotation. A confidence score is computed for each cell, allowing more nuanced testing and performance comparison. A comprehensive review on the methods of segmenting 3D dense nuclei was conducted. The benchmark was used to evaluate the performance of several selected state-of-the-art segmentation algorithms. The best of current methods is still far away from human-level accuracy, corroborating the necessity of generating such a benchmark. The testing results also demonstrated the strength and weakness of each method and pointed out the directions of further methodological development. The dataset can be downloaded here: https://github.com/yu-lab-vt/NIS3D.


Reviews:
[{"review_id": "QWMnYHwCwY", "reviewer": "Reviewer_x8hy", "summary": "", "questions": "", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This is by far the largest data set in this specific problem space (light microscopy 3D segmentation). A tremendous amount of thoughtful, careful annotation has been done here, which will be important in this problem space for benchmarking methods.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "knKdZE01au", "reviewer": "Reviewer_bNXG", "summary": "", "questions": "", "limitations": "\nThere are few claims/statements in the paper that are neither backed up with data/findings nor by proper references to the literature\ni) Authors state that (line 74-75), using the presented tool speeds up the annotation workflow. However the did not present any data in this direction. In some cases, cleaning the labels generated by an automated model can be more tedious and time-consuming than doing the labelling from scratch. Did the authors at least test on a few images how speed-up benefit is achieved?\nii) Line 145: Princut method can automatically generate boundaries for user-identified cells, and the method is sensitive to weak intensity changes, which can reduce the human bias in low-quality regions.  Is this claim just observational or do the authors have quantitative analysis data to support this claim?\n\n", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well-written and the ideas are well-presented. It is easy to read and understand. \nDetails on the dataset and the benchmark tests are included.\n\nAuthors provide exhaustive labels. They used multiple annotators to label the dataset and also presented how they merged the labels of the annotators. This also can be used as a guideline in future studies. \n\nThey used data from 4 different samples which increases the variation in the dataset and makes the dataset more generalizable.\n\n ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "PWpgpHvqYC", "reviewer": "Reviewer_Rsqm", "summary": "", "questions": "", "limitations": "The authors have clearly laid out a limitation of their data, namely that for some model organisms, their data only comprised a single image.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Fully annotated 3d nuclei data is a scarce resource; To this end, the novel data describes in this work constitutes a very valuable contribution. \nThe proposed evaluation metric, which takes into account human annotator agreement, is very interesting.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "V2TMaWQekx", "reviewer": "Reviewer_d7Xp", "summary": "", "questions": "", "limitations": "The authors discuss the limitations of the dataset. The main limitation is, although the dataset is larger than its counterparts, it still is relatively small and may not be an ideal candidate to both train and test methods.  ", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The dataset has both 3D data and 3D annotation labels.\n- It is much larger than other similar datasets. \n- The cell segmentations were carried out by three annotators. Variability among the annotators is computed as a confidence score and included in the dataset. \n- The authors carry out an experimental comparison of several 3D segmentation methods, showing the utility of the dataset. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "chPL82jdXx", "reviewer": "Reviewer_W2ro", "summary": "", "questions": "", "limitations": "Limitation is thoroughly discussed.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "They collected their own data (Zebrafish 1) and made a public dataset. \n\nThis is the first benchmark that provides densely annotated cell in 3D volumes. Benchmarking 3D instance segmentation is more challenging than 2D mainly because constructing dataset is prohibitively laborious. Their proposed benchmark will be useful in developing 3D nuclei segmentation model in the future.\n\nRecognizing the inherent ambiguity and potential biases in nuclei segmentation, the researchers introduced a confidence score as a means to establish a more reliable metric for their benchmark evaluation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The reviewers all liked the paper. The authors' response clarified most points raised by the reviewers. In view of that, the authors are strongly invited to take the feedback on board for the final version.
Decision: Accept (Poster)