SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All experiments are conducted on our workstation with NVIDIA V100 GPU and Intel Xeon Platinum 8268 CPU in this work. We provide the correct cell size and resolution as parameters for all methods. 3D suite includes multiple unsupervised methods, where we choose iterative thresholding due to its best performance. Cellpose also provides several models, and we choose "cyto" and "nuclei" for the same reason. In this section, the results of supervised methods are based on pre-trained models because some of them can only be trained on 2D data and it is unfair to train others. Other settings for each method are summarized in the supplementary.
195   
196   
197   
198   
199   
200   
201   
202   
203   
204   
205   
206   
207   
208   
209   
210   
211   
212   
213   
214   
215
4.2 Benchmark result
The experiment results of baseline methods are summarized in Table 3 and examples of the results are visualized in Figure 5. Evaluation metrics are introduced in Section 3.3. The IoU represents the intersection over the union between foreground and background to evaluate the binary segmentation performance, while SEG represents the average IoU treating every cell individually to evaluate the instance segmentation performance. When a method's SEG score is much lower than the IoU score, it generally means that this method has a lot of instance segmentation errors, such as under-segment multiple cells or over-segment one cell. 3D suite, an iterative threshold method, tends to only detect the bright region of cells and miss many dim cells. As a result, it usually gives high precision but low recall and SEG. The performance of Cellpose cyto varies from data, which can miss the majority of cells for some data and give a very low IoU score. Cellpose nuclei tend to over-segment
Table 3: The table of experiment result.
Data name	Method name	W-F1	W-Prec	W-Recall	W-IoU	W-SEG	Time (s)
Zebrafish 1	3D Suite	0.477	0.998	0.313	0.102	0.076	718
Cellpose_cyto	0.098	0.769	0.053	0.013	0.007	1271
Cellpose_nuclei	0.492	0.363	0.761	0.339	0.073	1303
StarDist	0.586	0.770	0.473	0.263	0.192	1620
Vaa3D	0.041	0.992	0.021	0.209	0.002	1200
QCAnet	0.510	0.885	0.359	0.276	0.158	8070
Human	0.967	0.971	0.963	0.927	0.905	N/A
Zebrafish 2	3D Suite	0.462	0.985	0.302	0.110	0.066	180
Cellpose_cyto	0.661	0.949	0.507	0.492	0.237	740
Cellpose_nuclei	0.654	0.971	0.492	0.263	0.167	242
StarDist	0.529	0.977	0.363	0.380	0.183	60
Vaa3D	0.101	0.995	0.053	0.703	0.007	56
QCAnet	0.031	0.849	0.016	0.446	0.003	496
Human	0.880	0.915	0.848	0.805	0.610	N/A
Drosophila 1	3D Suite	0.879	0.859	0.900	0.265	0.207	362
Cellpose_cyto	0.550	0.397	0.894	0.815	0.257	320
Cellpose_nuclei	0.607	0.457	0.903	0.421	0.151	215
StarDist	0.924	0.881	0.972	0.691	0.586	480
Vaa3D	0.260	0.988	0.149	0.671	0.025	278
QCAnet	0.275	0.585	0.180	0.595	0.051	2334
Human	0.997	0.995	0.998	0.915	0.906	N/A
Drosophila 2	3D Suite	0.877	0.785	0.993	0.339	0.273	420
Cellpose_cyto	0.092	0.048	0.982	0.606	0.028	478
Cellpose_nuclei	0.278	0.162	0.990	0.413	0.057	236
StarDist	0.684	0.526	0.979	0.563	0.325	245
Vaa3D	0.446	0.651	0.339	0.428	0.094	307
QCAnet	0.402	0.685	0.284	0.399	0.133	2790
Human	0.996	0.992	0.999	0.909	0.903	N/A
Mus Musculus 1	3D Suite	0.463	0.824	0.322	0.175	0.102	600
Cellpose_cyto	0.536	0.509	0.566	0.339	0.142	602
Cellpose_nuclei	0.356	0.240	0.688	0.37	0.056	570
StarDist	0.594	0.789	0.476	0.256	0.191	480
Vaa3D	0.207	1.000	0.116	0.515	0.018	780
QCAnet	0.454	0.726	0.330	0.451	0.127	4133
Human	0.981	0.975	0.988	0.973	0.897	N/A
Mus Musculus 2	3d Suite	0.667	0.841	0.553	0.127	0.090	92
Cellpose_cyto	0.371	0.412	0.338	0.099	0.045	188
Cellpose_nuclei	0.378	0.285	0.561	0.235	0.054	264
StarDist	0.540	0.549	0.531	0.311	0.158	221
Vaa3D	0.270	0.967	0.157	0.510	0.032	147
QCAnet	0.298	0.689	0.190	0.391	0.067	1472
Human	0.959	0.967	0.950	0.887	0.809	N/A
216   
217   
218   
219   
220   
221   
222   
223   
224   
225   
226
and give very low precision. StarDist has the overall best performance since it has relatively fewer overall under-segment or over-segment issues. However, StarDist still gives a considerable amount of false positives and false negatives. Vaa3D can usually detect the foreground well but tends to under-segment cells, which leads to high precision and IoU but low recall and SEG scores. QCAnet tends to under-segment and gives low recall as well. For some data, QCAnet may detect a large region full of noise. The human-level performance is also evaluated and provided. For human performance, the SEG score of Zebrafish 2 is lower than other data because of their smaller cell size.
NIS3D presents a 3D, high cell density, large-volume, and completely annotated Nuclei Image Segmentation benchmark with over  $22,000+$  cells from commonly studied species in the field. To the best of our knowledge, NIS3D is the first benchmark to provide a publicly available 3D nuclei image annotation of this scale and offers method developers a valuable opportunity to comprehensively evaluate their techniques, establishing an essential foundation for further method development. For supervised models, We also provide two suggestive training/test split settings, one is an in-image split setting and one is a cross-image split setting. For the in-image split, we use  $50\%$  of the image as the training set and the other  $50\%$  as the test set. For cross-image split, we use 3 full images as the training set and the rest as the test set. Check the supplementary for more details about the supervised learning result and suggestive splitting.
It is worth noting that existing segmentation methods are partially limited by their reliance on natural image segmentation principles and place the primary focus on predicting cell foregrounds. Consequently, these methods often suffer from both over-segmentation and under-segmentation issues when objects are densely packed. Considering the fact that these issues are highly related to cell boundary detection and cell boundaries are relatively easier to be detected in nuclei data, we suggest that future method developers not only estimate the probability of cells for each pixel but also estimate the probability of cell boundaries.
[1] Long Chen, Leanne Lai Hang Chan, Zhongying Zhao, and Hong Yan. A novel cell nuclei segmentation method for 3d c. elegans embryonic time-lapse images. BMC bioinformatics, 14(1):1-12, 2013.
[2] Marcelo Cicconet, Michelle Gutwein, Kristin C Gunsalus, and Davi Geiger. Label free cell-tracking and division detection based on 2d time-lapse images for lineage analysis of early embryo development. Computers in biology and medicine, 51:24-34, 2014.
[3] Bhavna Rajasekaran, Koichiro Uriu, Guillaume Valentin, Jean-Yves Tinevez, and Andrew C Oates. Object segmentation and ground truth in 3d embryonic imaging. PloS one, 11(6):e0150853, 2016.
[4] Gopi Shah, Konstantin Thierbach, Benjamin Schmid, Johannes Waschke, Anna Reade, Mario Hlawitschka, Ingo Roeder, Nico Scherf, and Jan Huisken. Multi-scale imaging and analysis identify pan-embryo cell dynamics of germlayer formation in zebrafish. Nature communications, 10(1):5753, 2019.
[5] Akanksha Jain, Vladimir Ulman, Arghyadip Mukherjee, Mangal Prakash, Marina B Cuenca, Lokesh G Pimpale, Stefan Munster, Robert Haase, Kristen A Panfilio, Florian Jug, et al. Regionalized tissue fluidization is required for epithelial gap closure during insect gastrulation. Nature communications, 11(1):5604, 2020.
[6] Martin H Dominguez, Alexis Leigh Krup, Jonathon M Muncie, and Benoit G Bruneau. Graded mesoderm assembly governs cell fate and morphogenesis of the early mammalian heart. Cell, 186(3):479-496, 2023.
[7] BJ Steventon, Alexander van Oudenaarden, Jan Huisken, Courtney Lancaster, Martin Lenz, Leila Muresan, Gopi Shah, Maria Florescu, Tim Fulton, and Andrea Attardi. Neuromesodermal progenitors are a conserved source of spinal cord with divergent growth dynamics. 2018.
[8] Fernando Amat, William Lemon, Daniel P Mossing, Katie McDole, Yinan Wan, Kristin Branson, Eugene W Myers, and Philipp J Keller. Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data. Nature methods, 11(9):951-958, 2014.
227   
228   
229   
230   
231   
232   
233   
234   
235   
236   
237   
238   
239   
240   
241   
242   
243   
244   
245   
246   
247   
248   
249   
250   
251   
252   
253   
254   
255   
256   
257   
258   
259   
260   
261   
262   
263   
264   
265   
266   
267   
268   
269   
270   
271   
272   
273   
274
275   
276   
277   
278   
279   
280   
281   
282   
283   
284   
285   
286   
287   
288   
289   
290   
291   
292   
293   
294   
295   
296   
297   
298   
299   
300   
301   
302   
303   
304   
305   
306   
307   
308   
309   
310   
311   
312   
313   
314   
315   
316   
317   
318   
319   
320   
321   
322   
323   
324   
325   
326
[26] David Legland, Ignacio Arganda-Carreras, and Philippe Andrey. Morpholibrary: integrated library and plugins for mathematical morphology with imagej. Bioinformatics, 32(22):3532-3534, 2016.
[27] Jean Ollion, Julien Cochennec, François Loll, Christophe Escudé, and Thomas Boudier. Tango: a generic tool for high-throughput 3d image analysis for studying nuclear organization. Bioinformatics, 29(14):1840-1841, 2013.
[28] Martin Weigert, Uwe Schmidt, Robert Haase, Ko Sugawara, and Gene Myers. Star-convex polyhedra for 3d object detection and segmentation in microscopy. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3666–3673, 2020.
[29] L Maier-Hein, A Reinke, P Godau, M Tizabi, F Buttner, E Christodoulou, B Glocker, F Isensee, J Kleesiek, and M Kozubek. Metrics reloaded: Recommendations for image analysis validation. arXiv, 2023.
327   
328   
329   
330   
331   
332   
333   
334   
335   
336