SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei Zheng<sup>1</sup>, James Cheng Peng<sup>1</sup>, Zeyuan Hou<sup>1</sup>,
Boyu Lyu<sup>1</sup>, Mengfan Wang<sup>1</sup>, Xuelong Mi<sup>1</sup>
Shuoxuan Qiao $^{1}$ , Yinan Wan $^{2}$ , Guoqiang Yu $^{1}$
$^{1}$ Dept. of Electrical and Computer Engineering, Virginia Tech
$^{2}$ Biozentrum, University of Basel
$^{1}$ $\{\text { zhengw, jameschengpeng, stanhou, boyu93, mengfanw, mixl18, kevinqiao, yug } \}$ @vt.edu
2{yinan.wan}@unibas.ch
3D segmentation of nuclei images is a fundamental task for many biological studies. Despite the rapid advances of large-volume 3D imaging acquisition methods and the emergence of sophisticated algorithms to segment the nuclei in recent years, a benchmark with all cells completely annotated is still missing, making it hard to accurately assess and further improve the performance of the algorithms. The existing nuclei segmentation benchmarks either worked on 2D only or annotated a small number of 3D cells, perhaps due to the high cost of 3D annotation for large-scale data. To fulfill the critical need, we constructed NIS3D, a 3D, high cell density, large-volume, and completely annotated Nuclei Image Segmentation benchmark, assisted by our newly designed semi-automatic annotation software. NIS3D provides more than 22,000 cells across multiple most-used species in this area. Each cell is labeled by three independent annotators, so we can measure the variability of each annotation. A confidence score is computed for each cell, allowing more nuanced testing and performance comparison. A comprehensive review on the methods of segmenting 3D dense nuclei was conducted. The benchmark was used to evaluate the performance of several selected state-of-the-art segmentation algorithms. The best of current methods is still far away from human-level accuracy, corroborating the necessity of generating such a benchmark. The testing results also demonstrated the strength and weakness of each method and pointed out the directions of further methodological development. The dataset can be downloaded here https://github.com/yu-lab-vt/NIS3D.
With the rapid development of live-cell microscopic imaging and genetic fluorescent reporters, researchers are able to record the time-lapse 3D images of cell nuclei during the embryogenesis process[1]. Such data are valuable for a wide range of biological research, for instance, the mechanisms and patterns of cell differentiation, the origin and diversity of cell types, and the causes and consequences of developmental defects [2-7]. In these studies, a critical step is 3D embryonic cell nuclei image segmentation, which is the foundation of subsequent analyses including cell tracking, lineage analysis, morphogenesis analysis, and morphodynamic analysis [8-10].
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27   
28   
29
Figure 1: Examples of NIS3D benchmark. Each row represents a specific data, with columns from left to right displaying raw data, the annotated ground truth, and the corresponding confidence score map, respectively. Confidence scores are assigned on a four-level scale to indicate the reliability of each annotation, from low to high: "undefined masks", "1/3", "2/3", and "3/3". These levels are labeled by the colors deep blue, light blue, yellow, and red, respectively. It's worth noting that "undefined mask" indicates regions with a group of very blurry cells that annotators can't decide their boundaries. The detection whose majority of pixels are within the undefined masks will be ignored, neither considered as true positive nor false positive.
Unlike other types of nuclei image data, 3D embryonic nuclei data possess distinctive characteristics, such as high cell density, large volumes, low signal-to-noise ratio (SNR), and a diverse range of shapes and intensities within the same volume, as exemplified in Figure 2. Consequently, the segmentation of 3D embryonic nuclei images presents great challenges. Despite the numerous 3D segmentation methods proposed, there is currently a lack of a widely accepted comprehensive benchmark for evaluating their performance. The existing benchmarks or datasets [11-16] for nuclei segmentation predominantly provide 2D ground truth, thereby overlooking the critical aspect of 3D analysis. Although some datasets [17, 18] do offer 3D annotations, they focus on the very early stages of embryo development, resulting in uncharacteristically low cell density and a very limited number of annotated cells.
Annotating large volumes of 3D embryonic nuclei is a time-consuming and labor-intensive task that requires a thorough manual inspection of the data. Unlike 2D images, where objects reside on individual planes with relatively simple morphological structures, annotating 3D images presents significantly greater challenges for the following reasons: (a) In 3D image annotation, a nucleus is captured across multiple consecutive z-slices, resulting in 2D boundaries on each slice comprising
Non-embryonic data
Figure 2: Left: 2D non-embryonic nuclei images [12, 13, 19]. Right: 3D embryonic nuclei images.
NIS3D
its surface. Consequently, annotating the same number of cells in 3D requires significantly more time compared to 2D labeling. (b) In dense object arrangements, portions of the cell surface can be parallel to the chosen visualization view and remain invisible, further complicating the annotation process. (c) The cell morphology and texture in 3D images exhibit far greater complexity than their 2D counterparts, demanding annotators to adhere to higher standards when labeling nuclei accurately. Consequently, annotating 3D images necessitates a larger investment of human labor compared to 2D cases.
To fill the gap of a completely annotated 3D embryonic cell image dataset, in this report, we present NIS3D, a 3D, high cell density, large-volume, and completely annotated embryonic Nuclei Image Segmentation benchmark. We provide examples of the benchmark in Figure 1. NIS3D provides more than 22,000 3D nuclei in the embryo images of zebrafish, drosophila, and mouse, which are the most commonly used species in the field. Each image of NIS3D is annotated by three independent well-trained annotators spending a total of  $700+$  hours. To allow more nuanced testing and performance comparison, a confidence score is computed for each cell to show its reliability. There are four levels of confidence scores in NIS3D, from the least confident score representing ambiguous annotation to the largest confident score indicating great consistency among all annotators.
To be more specific, the advantages of NIS3D are as follows:
- A good representation: NIS3D provides large-volume images of high cell densities, with nuclei whose signal-to-noise ratio, shape, and brightness vary with position. These properties make this benchmark challenging but well representative of data from real research.
- 3D complete annotation: All cells are annotated, and all labels are 3D. Compared with sparse annotation or 2D annotation, NIS3D can provide a more comprehensive evaluation, including the evaluation of false positives.
- Confidence score: A confidence score is computed for each cell, allowing more nuanced testing and performance comparison.
- Multiple species: NIS3D contains the three most commonly used species in this field (zebrafish, drosophila, and Mus Musculus) to provide enough diversity.
To facilitate the 3D annotation, we developed a semi-automatic annotation tool. It can generate a suggestive 3D boundary on all z-slices for the user-identified cell, without tedious labeling on each z-slice. The suggestive boundary can even outperform human annotation in low-quality regions, but annotators still have the authority and flexibility to further fine-tune unsatisfied cell boundaries. The tool not only significantly speeds up the annotation workflow but also reduces human bias.
45   
46   
47   
48   
49   
50   
51   
52   
53   
54   
55   
56   
57   
58   
59   
60   
61   
62   
63   
64   
65   
66   
67   
68   
69   
70   
71   
72   
73   
74   
75   
76
Table 1: Existing benchmarks
Name	Data	Label	Annotator #	Complete label	Sample #	Year
Cell tracking challenge[14]	3D	2D	3	No	3,000+	2014
BBBC039v1 [20]	2D	2D	1	Yes	23,000+	2018
2018 Data Science Bowl[12]	2D	2D	1	Yes	30,000+	2018
S-BSST265[19]	2D	2D	N/A	Yes	7813	2020
BBBC032v1[17]	3D	3D	N/A	Yes	57	2018
BBBC050[18]	3D	3D	N/A	Yes	1,814	2020
C.elegans[21]	3D	3D	1	Yes	15,000+	2022
NIS3D (ours)	3D	3D	3	Yes	22,000+	2023
*For Cell tracking challenge, we only consider the 3D embryonic data. N/A annotator number means that the work didn't mention it.
Figure 3: Representative examples of existing benchmarks. The numbers on the left top indicate the resolution of this data. All data shown here are the full data or z-slices from the corresponding dataset and are not cropped. Different colors represent different ground truth labels. The over-segment and under-segment issues in BBBC050 are indicated by white arrows.
Existing benchmarks Table 1 summarizes the existing benchmarks in this field and Figure 3 shows examples of them. The cell tracking challenge is considered as the most commonly-used benchmark in this field, but it only provides sparse annotation for embryonic data, making the false positive evaluation infeasible. BBBC039v1, 2018 data science bowl, and S-BSST265 are also popular benchmarks providing nuclei data, but they are not embryonic and have a relatively low nuclei density with nuclei well separated. As a result, they cannot be used to comprehensively evaluate the segmentation algorithms dealing with densely packed nuclei. Moreover, all these benchmarks only provide 2D annotations while various biological questions require algorithms to detect the boundary in 3D space. It should be pointed out that BBBC032v1 and BBBC050 do provide 3D annotations of animal embryos of mouse and drosophila, but they are at very early developmental stages with two critical issues: unusually low nuclei density and small sample size. Moreover, BBBC050 is not initially made as a benchmark and has some quality issues. Figure 3 shows obvious under-segment and over-segment problems for the ground truth of BBBC050. C.elegans is a better option, but it not only suffers from low diversity but also has discrepancies such as incomplete or erroneous labels, which is hard to avoid for ground truth from only one annotator. Additional information can be found in the supplementary.
Existing segmentation methods The existing segmentation methods can be classified into two groups: semantic segmentation and instance segmentation. Semantic segmentation assigns the
77   
78   
79   
80   
81   
82   
83   
84   
85   
86   
87   
88   
89   
90   
91   
92   
93   
94   
95
same label to the object of the same class. For nuclei images, it can only get foreground rather than individual cells. Some famous models, like ilastik[22], labkit[23], and 3D U-Net [24] belong to this class. Such models work well for many low cell-density data. But for embryonic data analysis, they are far from satisfactory, and thus instance segmentation is necessary. There are popular tools, such as Vaa3D[25], MorphoLibJ[26], and 3D Suite[27], providing unsupervised methods for instance segmentation. Those methods are easy to use but generally fail to deal with the complex morphological and intensity patterns of embryonic data. There are also some supervised models, for instance, methods proposed in recent years, like Mesmer[15], QCAnet[18], Cellpose[11], and StarDist [28]. Some of those methods like Mesmer can only be used for 2D images. QCAnet consists of two submodels, which can detect the foreground and nuclei center separately, then it uses the marker-based watershed to generate the segmentation result. Cellpose and StarDist are originally designed for 2D data, but they both make 3D extensions based on assumptions of 3D cell shape. The Cellpose 3D extension is still trained on 2D data, but it does 2D segmentation for xy plane, xz plane, and yz plane separately first, and then thresholds the average cell probability from 3 directions to reconstruct the 3D segmentation result. StarDist 3D extension estimates both cell probability and radial distance to the boundary for each pixel, then reconstructs the 3D segmentation result.
3 NIS3D
NIS3D collects 6 large volume embryonic nuclei images from the three most widely used species in the field with over  $22,000+$  manually annotated cells. In this section, we will give the details of data collection, annotation, and recommended evaluation metrics. Additional information can be found in the supplementary.
Zebrafish 1 (in-house dataset) Transgenic zebrafish embryos with fluorescent nuclei marker  $Tg(bactin2:H2BmCherry)$ , inside their chorions, were embedded in  $1\%$  low melting point agarose prepared in E3 medium, enclosed by glass capillary before extruded into the imaging chamber. Images were acquired with Zeiss LightSheet 7 Microscopy, with  $20\mathrm{x}/\mathrm{N.A}$ . 1.0 detection objective (additional optical zoom factor  $0.55\mathrm{x}$ ) and dual-side  $10\mathrm{x}/\mathrm{N.A}$ . 0.3 illumination objectives. Fluorescence was activated by  $561\mathrm{nm}$  laser and detected with LP585 filter. Time-lapse imaging was performed at 2-minute interval from 4 to 20 hours post fertilization. Within each time interval, four 3D volumes were acquired with 90-degree rotation in between to achieve full-embryo multiview coverage. The z-stack was set to have the voxel size of  $0.43\mathrm{um}\times 0.43\mathrm{um}\times 2.5\mathrm{um}$ , so that each cell nuclei is sectioned by at least 3 planes. We picked a time point in the middle of this time-lapse data. The nuclei in this data don't have strong texture and the nuclei shapes are consistent, but the data suffers from low SNR, especially in the first  $40\mathrm{z}$ -slices. The voxel size is  $0.43\mathrm{um}\times 0.43\mathrm{um}\times 2.5\mathrm{um}$ .
Zebrafish 2 The data are the first time-point of embryo 3 in the public dataset [7]. Zebrafish 2 recorded the tailbud of a zebrafish in a late stage of embryo development. There is a significant amount of blurring within more anterior portions of the tail, which is hard even for human to distinguish the boundary and we use the undefined mask to mask it out. This data have very high cell density and small cell size. The voxel size is  $1\mathrm{um}\times 1\mathrm{um}\times 1\mathrm{um}$ .
Drosophila (fruit fly) 1&2 The two drosophila images are selected from [10]. Drosophila images are picked at time-point 20 and 50 of the brachyenteron (byn) gene reporter data. The data are for early Drosophila embryogenesis and the cells are large and relatively sparse. This data shows complicated textures within cells and bright background noise. The voxel size is  $1\mathrm{um}\times 1\mathrm{um}\times 1\mathrm{um}$ .
Mus musculus (mouse) 1&2 The Mus musculus data are selected from [9]. The image is picked at the time-point 150 and 200 of embryo 4. It was recorded during relatively late Mus musculus embryogenesis. The center of this image is very blurry, and we use an undefined mask to mask it.
96   
97   
98   
99   
100   
101   
102   
103   
104   
105   
106   
107   
108   
109   
110   
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122   
123   
124   
125   
126   
127   
128   
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141
This data shows various cell shapes and strong textures within cells. The voxel size is  $1\mathrm{um}\times 1\mathrm{um}\times$  1 um.
The workflow consists of three steps: independent annotation, label fusion, and manual reviewing.
Independent annotation We first train our annotators on how to distinguish the nuclei, noise, background, texture within cells, and the gap between cells for nuclei images. Then we train the annotator how to use PrinCut, the semi-automatic annotation tool we developed for this project. PrinCut can automatically generate a suggestive boundary for the user-identified cell at the surface with continuous positive principal curvature. The suggestive boundary is sensitive to the weak intensity changes that humans may ignore, which can reduce human bias in low-quality regions. However, principal curvature can also be over-sensitive to cell texture and insensitive to the shape of cells. We request the annotators to merge and split the suggestive boundaries or manually draw the boundary by brush until the boundaries meet the annotator's expectations. For the region that annotators do believe there are cells but cannot identify the boundaries due to image quality, annotators will label it as an undefined mask. By the end of this step, we get three sets of independent labels for each image.
Label fusion We first match the labels from different annotators, as shown in Figure 4. We consider labels from three annotators as matched if they have an intersection over union (IoU) greater than 0.5 between each other. For labels that meet these criteria, we consider them to be associated with the same ground truth, and we calculate the boundary of this ground truth label based on the three matched labels (more details in supplementary). The same approach is applied to find all ground truths and their corresponding labels. If the ground truth is associated with  $x$  labels from different annotators, its confidence score is  $x / 3$ . About  $1.76\%$  of labels belong to conflict labels, which usually means the regions in those labels are very confusing. Those labels will be further manually reviewed.
GT with 3/3 confident score
Figure 4: The top row shows criteria for determining the labels belonging to the same ground truth, while the bottom row shows an example for this case. The three circles row represent the three labels from different annotators and we will calculate the Intersection over Union (IoU) between them. "IoU  $<0.5$  " generally means that two labels belong to different ground truths. "IoU  $>0.5$  " generally means that two labels belong to the same ground truth. The green circles in the top row represent the labels belonging to the ground truth while the red circles represent the label unrelated to this ground truth. For example, annotator A and annotator B created annotation for 2/3 confidence score ground truth while annotator C didn't create any annotation for this ground truth, and the confidence score of this ground truth is 2/3.
GT with 2/3 confident score
GT with 1/3 confident score
Conflict labels
142   
143   
144   
145   
146   
147   
148   
149   
150   
151   
152   
153   
154   
155   
156   
157   
158   
159   
160   
161   
162   
163   
164   
165
Table 2: Confidence score distribution.
Score Data	1/3	2/3	3/3
Zebrafish 1	529	1762	10423
Zebrafish 2	1668	1745	2224
Drosophila 1	13	85	1596
Drosophila 2	1	1	488
Mus Musculus 1	N/A	369	1191
Mus Musculus 2	N/A	395	660
By the end of this step, we get a set of ground truth labels with different confidence scores and a group of conflict labels for each image.
168 Manual reviewing For the conflict label with bad image quality, the label will be manually set as an undefined mask, otherwise, we pick the best candidate label as the ground truth label and set the confidence score as 1/3. For low-quality data, extra undefined masks are also drawn on specific low-quality regions. The Mus Musculus images have strong textures within cells, we manually set all ground truth with 1/3 confidence score to uncertain labels. Table 2 shows the distribution of the confidence score as a reference of human annotation variation.
174 3.3 Evaluation metrics
Choosing the correct metric that adequately reflects the biological nature is important but usually neglected [29]. The existing metric of the cell tracking challenge and the 2018 Data Science Bowl give results that are inconsistent with human intuition, thereby affecting the evaluation process. To address these issues, we have reformulated the evaluation metric to align more closely with our specific objectives. For instance, a high W-F1 score coupled with a low W-SEG score now indicates successful cell detection while indicating room for boundary enhancement. Similarly, a high W-IoU score combined with a low W-SEG score signifies accurate foreground detection, while highlighting potential over-segmentation or under-segmentation concerns.
183 Preprocessing and truth positive criteria To initiate the process, we exclude detections where more than  $50\%$  of their pixels fall within the undefined mask. Then for a given detection  $D_{i}$ , we determine it matches with ground truth  $G_{j}$  if and only if both of the two following condition holds:
$$
D _ {i} = \arg \max  _ {D _ {k}} \operatorname {I o U} \left(D _ {k}, G _ {j}\right) \quad G _ {j} = \arg \max  _ {G _ {k}} \operatorname {I o U} \left(D _ {i}, G _ {k}\right), \tag {1}
$$
where  $D_{k}$  and  $G_{k}$  are all possible choices from detections and ground truth, and  $\mathrm{IoU}(A,B)$  is the intersection over the union between  $A$  and  $B$ .
Weighted precision(W-Precision), recall(W-Recall), and F1(W-F1) The weighted scores are based on the confidence score.
190 The weighted true positive (W-TP) and false negative (W-FN) are calculated as follows:
$$
\mathrm {W} - \mathrm {T P} = \sum C _ {i} T _ {i} \quad \mathrm {W} - \mathrm {F N} = \sum C _ {i} \left(1 - T _ {i}\right), \tag {2}
$$
where  $C_i$  is the confidence score of ground truth  $G_{i}$  and  $T_{i}$  is the detection flag of  $G_{i}$ .  $T_{i} = 1$  indicates if  $G_{i}$  is detected, otherwise  $T_{i} = 0$ .
193 The weighted precision (W-Precision), recall (W-Recall), and F1 (W-F1) are calculated as follows:
$$
\mathrm {W} - \text {P r e c i s i o n} = \frac {\mathrm {W} - \mathrm {T P}}{\mathrm {W} - \mathrm {T P} + \mathrm {F P}} \quad \mathrm {W} - \text {R e c a l l} = \frac {\mathrm {W} - \mathrm {T P}}{\mathrm {W} - \mathrm {T P} + \mathrm {W} - \mathrm {F N}} \quad \mathrm {W} - \mathrm {F 1} = \frac {2 \mathrm {W} - \mathrm {T P}}{2 \mathrm {W} - \mathrm {T P} + \mathrm {F P} + \mathrm {W} - \mathrm {F N}} \tag {3}
$$
194 Weighted IoU Weighted IoU (W-IoU) is used to show the accuracy of foreground of detection.
$$
\mathrm {W} - \mathrm {I o U} = \frac {\sum_ {i \in A \cap B} f (i)}{\sum_ {i \in A} f (i) + \sum_ {i \in B / A} 1} \tag {4}
$$
166   
167   
168   
169   
170   
171   
172   
173   
174   
175   
176   
177   
178   
179   
180   
181   
182   
183   
184   
185   
186   
187   
188   
189   
190   
191   
192   
193   
194
Figure 5: Examples of baseline methods results. The raw data and ground truth are from Zebrafish 2 and are also shown in Figure 1(C). The different colors represent different detection instances.
where  $A$  is the pixel belong to ground truth,  $B$  is the pixel belong to detection, and  $f(i)$  is the confidence score of  $i$ -th pixel.
Weighted SEG Weighted SEG (W-SEG) score is used to show the average IoU score of all cells.
$$
\mathrm {W} - \mathrm {S E G} = \frac {\sum C _ {i} G _ {i}}{\mathrm {W} - \mathrm {T P} + \mathrm {W} - \mathrm {F N} + \mathrm {F P}}, \tag {5}
$$
where  $G_{i}$  is the IoU score of the  $i$ -th ground truth.
In this section, we test baseline methods on our benchmark dataset. The baseline methods we choose are: 3D Suite [27], Cellpose [11], StarDist [28], Vaa3D [25], and QCAnet [18]. The primary reason for selecting these particular methods over others is their capability to perform 3D instance segmentation and their relatively better performance, combined with the fact that the developers have provided software with a user-friendly interface accompanied by detailed instructions for usage. In this way, it is feasible for us to tune the methods on the data in NIS3D so that we can provide a fair performance analysis. In this section, we will present the experimental settings, evaluate the results, and discuss the limitation of baseline methods.