Human Perception-based Evaluation Criterion for Ultra-high Resolution Cell Membrane Segmentation
Authors: Ruohua Shi
Wenyao Wang
Zhixuan Li
Liuyuan He
Kaiwen Sheng
Lei Ma
Kai Du
Tingting Jiang
Tiejun Huang
Venue:  (2021)
Keywords: Neuroscience
Connectomics
Human perception
EM dataset
Membrane segmentation
Evaluation criterion

Abstract: Computer vision technology is widely used in biological and medical data analysis and understanding. However, there are still two major bottlenecks in the field of cell membrane segmentation, which seriously hinder further research: lack of sufficient high-quality data and lack of suitable evaluation criteria. In order to solve these two problems, this paper first introduces an Ultra-high Resolution Image Segmentation dataset for the Cell membrane, called U-RISC, the largest annotated EM dataset for the Cell membrane with multiple iterative annotations and uncompressed high-resolution raw data. During the analysis process of the U-RISC, we found that the current popular segmentation evaluation criteria are inconsistent with human perception. This interesting phenomenon is confirmed by a subjective experiment involving twenty people. Furthermore, to resolve this inconsistency, we propose a Perceptual Hausdorff Distance (PHD) evaluation criterion to measure the quality of cell membrane segmentation results. Detailed performance comparison and discussion of classic segmentation methods along with two iterative manual annotation results under existing criteria and PHD is given.

Under review as a conference paper at ICLR 2021
HUMAN PERCEPTION-BASER EVALUATION CRITE- RION FOR ULTRA-HIGH RESOLUTION CELL MEM- BRANE SEGMENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Computer vision technology is widely used in biological and medical data analysis and understanding. However, there are still two major bottlenecks in the field of cell membrane segmentation, which seriously hinder further research: lack of suf- ficient high-quality data and lack of suitable evaluation criteria. In order to solve these two problems, this paper first introduces Ultra-high Resolution Image Segmentation dataset for the Cell membrane, called U-RISC, the largest annotated Electron Microscopy (EM) dataset for the Cell membrane with multiple iterative annotations and uncompressed high-resolution raw data. During the analysis pro- of the U-RISC, found that the current popular segmentation evaluation criteria are inconsistent with human perception. This interesting phenomenon is confirmed by subjective experiment involving twenty people. Furthermore, resolve this inconsistency, we propose a new evaluation criterion called Perceptual Hausdorff Distance (PHD) measure the quality of cell membrane segmentation results. Detailed performance comparison and discussion of classic segmentation methods along with two iterative manual annotation results under existing evalu- ation criteria and PHD is given.
INTRODUCTION
Electron Microscopy (EM) is powerful tool to explore ultra-fine structures in biological tissues, which has been widely used in the research areas of medicine and biology (ERLANDSON (2009); Curry et al. (2006); Harris et al. (2006)). In recent years, EM techniques have pioneered an emerg- ing field called "Connectomies" (Lichtman et al. 2014 ) which aims to sean and reconstruct the whole brain circuitry at the nanoscale. "Connectomics" has played key role in several ambitious projects, including the BRAIN Initiative [Insel et al. (2013)) and MICrONS (Gleeson & Sawyer 2018 in the U.S., Brain/MINDS in Japan (Dando 2020)), and the China Brain Project (Poo et al. 2016 ) Because EM scans brain slices at the nanoscale, produces massive images with ultra- altra-high resolution and inevitably leads to the explosion of data. However, compared to the advances of EM, techniques of data analysis fall far behind. In particular, how to automatically extract information from massive raw data reconstruct the circuitry map has growingly become the bottleneck of EM applications.
One critical step in automatic EM data analysis is Membrane segmentation. With the introduction of deep learning techniques, significant improvements have been achieved in several public available
EM datasets ISBI 2012 and SNEMI3D (SSII 2012 2012) [SBI 2013 (2013); Arganda-Carreras et al. (2015b); Lee et al. (2017). 7. One of the earliest works Ciresan et al. (2012) used a succession of max- max-pooling convolutional networks a pixel classifier, which estimated the probability of pixel is membrane. [Ronneberger et al. (2015) presented U-Net structure with contracting paths, which captures multi-contextual information. Fully convolutional networks (FCNs) proposed by Long al.] (2015) led to breakthrough in semantic segmentation. Follow-up works based U- net and FCN structure ( (Xie & Tu 20: Drozdzal et al. (2016); Hu et al. (2018); Zhou et al. (2018): Chaurasia & Culurciello/ (201)); : Yu et al. (2017): Chen al. (2019b)) have also achieved outstanding results near-human performance.
Despite much progress that has been made in cell membrane segmentation for EM data thanks to deep learning, one risk to these popular and classic methods is that they might be "saturated" at the current datasets their performance appear be "exceedingly accurate" (Lee et al. (2017)). How

---

Under review as a conference paper at ICLR 2021
can these classic deep learning based segmentation methods work on new EM datasets with higher resolution and perhaps more challenges? Moreover, how robust of these methods when they are compared with human performance such EM images?
To expand the research of membrane segmentation on more comprchensive EM data, we first es- tablished dataset "U-RISC" containing images with original resolution (10000 10000 pixels, Fig. D To ensure the quality of annotation, it also costs us over 10,000 labor hours to label and double-check the data. To the best of our knowledge, U-RISC is the largest uncompressed annotated and EM dataset today. Next, we tested several classic deep learning based segmentation methods on U-RISC and compared the results human performance. We found that the performance of these methods was much lower than that of the first annotation. To understand why human perception is better than the popular segmentation methods, we examined in detail the Membrane segmentation results by these popular segmentation methods. How to measure the similarity between two image segmentation results has been widely discussed (Yeghiazaryan & Voiculescu (2018); Niessen et al. (2000); Veltkamp & Hagedoorn (2000): Lee et al. (0117). Varduhi Yeghiazaryan (Yeghiazaryan Voiculescul (2018)) discussed the family of boundary overlap metrics for the evaluation of medi- cal image segmentation. Veltkamp. etc (Veltkamp & Hagedoorn (2000)) formulated and summed up the similarity measures in a more general condition. In some challenges, such as ISBI2012 [Arganda-Carrent et al. (2015a)), they also considered multiple metries like Rand score on both original images and thinned images. However, we found there was a certain inconsistency between current most popular evaluation criteria for segmentation(e.g g.. FI score, IoU) and human perception: while some figures were rated significantly lower in Fl score or IoU, they were "perceived" better by humans (Fig.)
Such inconsistency motivated us to propose a human-perception based criterion, Perceptual Haus- dorff Distance (PHD) to evaluate image qualities. Further, we set up subjective experiment to collect human perception about membrane segmentation, and we found the PHD criteria is more consistent with human choices than traditional evaluation criteria. Finally, we found the current popular and classical segmentation methods need to be revisited with PHD criteria.
Overall, our contribution in this work lies mainly in the following two parts: (1) we established the largest, original image resolution-based EM dataset for training and testing; (2) we proposed human-perceppion based evaluation criterion, PHD, and verified the superiority of PHD by subjective experiments. The dataset we contributed and the PHD criterion proposed may help researchers to gain insights into the difference between human perception and conventional evaluation criteria, thus motivate the further design of the segmentation method catch up with the human performance on original EM images.
2 U-RISC: ULTRA-HIGH RESOLUTION IMAGE SEGMENTATION DATASET FOR CELL MEMBRANE
Supervised learning methods rely heavily on high-quality datasets. To alleviate the lack of training data for cell membrane segmentation, we proposed an Ultr--hih Resolution Image Segmentation dataset for Cell membrane, called U-RISC The dataset was annotated upon RC1, a large scale retinal serial section transmission electron microscopic (sSTEM) dataset, publically available upon request and described in the work Andeennn al. (201I). I). The original RCI dataset is 0.25mm diameter, 370 TEM slices volume, spanning the inner nuclear, inner plexiform, and ganglion cell layers, ac- quired at 2.18 nm/pixel across both axes and 70nm thickness in z-axis. From the 370 serial-section volume, we clipped out 120 images in the size of 10000 x 10000 pixels from randomly chosen sections. Then, manually annotated the cell membranes in iterative annotation-correcte procedure. Since the human labeling process is very valuable for uncovering the human learning process, during the relabeling process, we reserved the intermediate results for public release. The U-RISC dataset will be released on https:// /Ancnymouss com on acceptance.
2.1 COMPARISON WITH OTHER DATASETS
ISBI 2012 Cardona et al. 2010)) published a set of 30 images for training, which were captured from the ventral nerve cord of Drosophila first instar larva at a resolution of 4x4x50 nm/pixel through ssTEM (Arganda-Caan Carreras et al. (2015b): [SBI 2012 2012). Each image contains 512x512 pixels, spanning realistic area of 2x2 pm approximately. In the challenge of SNEMI3D (Kasthuri
2

---

Under review as a conference paper at ICLR 2021
Number (1,8)
6.3
(c) Pixel Number
Number of Images
200
200
350
10
(a) U-RISC Example
(b) ISBI 2012 Example
(d) Number images
Figure 1: Dataset examples and statistical comparisons. (a) and (b) are examples of U-RISC and ISB12012. The first line shows the original image, and the second line shows its annotation. (c) shows the pixel number of U-RISC, ISB12012, and SNEMI3D datasets. (d) shows the image number of the three datasets.
(a) Example 1
(b) Example
Figure 2:
Examples of images with their labels. The left of (a) (b) are two original image parts, and the right are the images covered by their labels (red lines).
al. 2015); ; ISBI 2013 2013)). ) the training data is 3D stack of 100 images in the size of 1024x 1024 pixels with the voxel resolution of 6x x29 nm/pixel. The raw images were acquired the resolution of 3x3x20 29 nm/pixel using serial section scanning electron microscopy (ssSEM) from mouse somatosensory cortex (Kasthuri et al. (2015): [SBI 2013 (2013). U-RISC contains 120 pieces of annotated images (10000x 10000 pixels) the resolution of 28x 70 nm/pixel from rabbit retina.
Due the difference of species and tissue, U-RISC can fill in the blank of annotated vertebrate retinal segmentation dataset. Besides that, U-RISC has some other characteristics which can be focused on the future segmentation study. The first one that the image size and realistic size of U-RISC is much larger, specifically, the image size of U-RISC is 400 and 100 times of ISBI2012 and SNEMI3D respectively, and the realistic size 100 and 9 times of them respectively (Fig. T (c)), which can be applied in developing deep learning based segmentation methods according to various demands. And along with the iterative annotation procedure U-RISC actually contains sets of annotation results with increasing accuracy, which could serve ground truth at different level standard. And the total number of annotated images 12 and 3.6 times the public annotated images of ISBI2012 and SNEMI3D respectively (Fig. [ (d)). An example of the image with its label shown in the Supplementary. Due to the limitation of the size the supplementary material, we only uploaded quarter (5000 5000 pixels) size of the original image with its label.
3

---

Under review as a conference paper at ICLR 2021
(a) Original Image
(b) First Iterative Correction
(c) Second Iterative Correction
Figure 3: Example of iterative labeling. (a) shows the original EM image. (b) and (c) shows the first and second round of annotation processes. The red arrows point to the deleted parts during the inspection processes, and the blue arrows point to the added parts.
2.2 TRIPLE LABELING PROCESS
The character of high resolution TEM image can display much more detailed sub-cellular struc- ture, which requests be affected by many factors, such as section thickness or sample staining (Fig, ((b) And low imag- ing quality also requests more labeling efforts. Therefore, inereasing labeling efforts is essential to completely annotate U-RISC. To guarantee the labeling accuracy, we set an iterative correction mechanism in the labeling process (Fig. B). Before starting the annotation, labeling rules were troduced to all annotators. 58 qualified annotators were allowed to participate in the final labeling process. After the first round annotation, experienced lab staff with sufficient background knowl- edge were responsible to point out labeling errors pixel by pixel during the second and the third rounds annotation. Finally, the third round annotation results were regarded as the final "ground truth". And previous two rounds of manual annotations are also saved for later analysis. Fig. shows an example of the two inspection processes. We can see that there are quiet few mislabeled and missed labeled cell membranes in each round. Therefore, the iterative correction mechanism is very necessary.
patience to label out the cell (Fig. 2(a). Besides, the imaging quality can
3 PERCEPTION-BASED EVALUATION
In the analysis of EM data, membrane segmentation is generally an indispensable key step. How- ever, in the field of cell membrane segmentation, most of the previous studies, such as Zhou et al. (2018); Chaurasia & Culurciello (2017); Drozdzal et al. (2016), were not specifically designed for high resolution datasets such as U-RISC In addition, although many researchers discussed various evaluation criteria for medical and general tasks, few researchers actually incorporate them into the design of the architectures of cell membrane segmentation methods.
By comparing the segmentation results of the popular and classic segmentation methods, we found that the widely used evaluation criteria of segmentation were inconsistent with human perception in some cases, which is further discussed through the perceptual consistency experiment (details in Sec. 3.2). To address this issue, we proposed a new evaluation criterion called Perceptual Haus- dorff Distance (PHD). The experimental results showed that it was more consistent with human perception.
3.1 INCONSISTENCY BETWEEN EXISTING EVALUATION CRITERIA AND PERCEPTION
Many researchers have proposed various metrics for segmentation evaluation Yeghiazaryan & Voiculescu (2018 8); Niessen et al. (2000); Veltkamp & Hagedoorn 2000); Arganda- Eiganda--Careere Carreras et al. (2015b) Lee et al. (2017)) ). Some of them, which are the most popular, such as score, Dice Coefficient and loU [Sasaki et al. (2007): [ice (1955): Kosubl (2019)
used as the evalua- tions in most segmentation methods [Ronneberger et al. 2015); Zhou et al. (0088); Chaurasia & Culurciello (2017); Yu et al. (017): Chen et al. (2019b)). ISBI2012 cell segmentation challenge used Rand scores (V-Rand and V-Info) (|Arganda- Carreras et al. (2015b)) on thinned membrane for evaluation. Recently, researchers made discussions various boundary overlap metries for the
4

---

Under review as a conference paper at ICLR 2021
evaluation of medical image segmentation (Yeghiazaryan & Voiculescu (2018) The most popular evaluation criteria, such as Fl score, are based on the statistics of the degree to which pixels are clas- sified correctly. There also some metrics designed based point set distance, such as ASSD (Yeghiazaryan & Voiculescu )2018). which is not widely used in recent deep learning researches.
However, quality of segmentation should be judged with respect the ultimate goal. When need to use segmentation to reconstruct the whole structure of membranes and connect them, such statisties may not be consistent with human perception in cell membrane segmentation tasks. In the process of segmentation experiments, some interesting phenomena were found. Fig. shows an example of the original image with its manual annotation and segmentation results by two methods GLNet [Chen et al. (2019b)) and U-Net (Ronneberger et al. (2015)). The scores indicated that (d) was more similar to (b) than (c).
It should be noted that if these segmentation results are used for reconstructing the structure of cells, the mistakes and loss of structure will be more noticeable when subjects inspect the area surrounded by the red dashed lines in the images, Therefore consider that (c) is better prediction, because (d) misses some edges. The reason for the three scores (c) are lower was that the predicted cell membrane of (c) was thicker than manual labeling. Therefore, it can be inferred that the existing evaluation criteria might not sufficiently robust to variations in the thickness and structures of the membrane, and the evaluation result was more likely inconsistent with human perception.
F1 score IoU Dice Fl score IoU Dice 0.6798
0.4866 0,3162 0.4804
0,5149 0.6798
(a) Original Image (b) Manual Annotation
(c) GLNett
(d) U-Net
Figure 4: Examples of some segmentation results. (a) is the original EM image. (b) is the manual
annotation of cell membrane. (c) and (d) are segmentation results from GLNet and U-Net.
3.2 PERCEPTUAL CONSISTRNCY EXPERIMENTS
In order to verify the above conjecture, a subjective experiment was designed to explore the con- sistency with the existing evaluation criteria and human subjective perception. Six popular and classical segmentation methods were used to generate cell membrane segmentation results on U- RISC: U-net (Ronneberger al. (2015). LinkNet Chaurasia & Culurciello (2017)), CASENet (Yu et al. (2017)) SENet ( (Hu et al. (2018), U-Net++ ( ([houu et al. (2018)) GLNet ( (Chen) al. (0T9b)). Using these segmentation results, 200 groups of images were randomly selected. Each group contained 3 images: the final manual annotation (ground truth) and two automatically generated segmentation results for the same input cell image.
20 subjects were recruited to participate in the experiments. They had either a biological background or experience in cell membrane segmentation and reconstruction. For each group, each of the 20 subjects had three choices. If the subject can tell which segmentation result is more similar to the ground truth, or she can choose which one. Otherwise, the subject can choose "Difficult to choose". The experiment interface is shown in the Appendix I.
Before the experiment, the subjects were trained on the purpose and source of the images. During the experiment, 200 groups of images were divided into four groups average in order to prevent the subjects from choosing randomly due to fatigue. For each batch of groups, the subjects needed to complete the judgment continuously without interruption.
After the experiment, for each group, if there were more than 10 votes of the same number, was called a valid group. Otherwise, it was invalid and discarded. There were total of 113 valid groups. Then, based on these valid groups, the consistency of the Fl score, IoU, and Dice with human choices was calculated.
5

---

Under review as a conference paper at ICLR 2021
According experimental results, the consistency of F1 score, IoU, and Dice with human choice was only 34.51%, 35, 35,40% and 34.51%, respectively. Therefore, it can be inferred that the three criteria are not consistent with human subjective perception in most cases. More results and design of subjective experiments are shown in the Appendix II, and IV,
3.3 PERCEPTUAL HAUSDORFF DISTANCE
Based on the subjective experimental results, it was verified that the widely used evaluation criteria for general segmentation were inconsistent with human perception of cell membrane segmentation. This paper proposes a new evaluation standard based on human perception, namely, Perceptual Hausdorff Distance (PHD for short), considering the structure but ignoring the thickness of cell membrane.
An Overview of PHID
As Fig. 4sowws from the perspective of neuronal reconstruction, the thickness of the cell membrane not the key for evaluation. In fact, when the goal is reconstruct the structure of cells, humans will pay more attention on structure changes, instead of thickness changes. Hence, when measuring the similarity of cell membrane segmentation results, in order to eliminate the influence of thickness, the segmentation results of two cell membranes were skeletonized, and then the distance between two skeletons was calculated to measure the difference. Since the skeleton a collection of different points, and Hausdorff distance is a common distance to calculate the difference between two sets of points, the proposed PHD built upon Hausdorff distance.
On the other hand, through subjective experiments, it was found that people tend to ignore the slight offset between the membrane. Therefore, based on the above two considerations, the Perceptual Hausdorff Distance (PHD) based Hausdorff distance( HHttenlocher et al. (1993); AAsert et al.
2002); ; Rachasingho Tasena (2020)) with modification was designed. Fig. shows the overview of PHI. The details are as follows.
Sogmentation
Skeleton
Calculate Distance
ipkd
PHLD
Skelotonize
Tolerance Distance
(a)
Segmentation
Skeleton 2
pisel
6yisel
Skeletonize
PHD
Tolerance Distance
(b)
Figure 5: Overview of PHD. The PHD evaluation criterion takes two segmentation results input. Then, the two inputs are skeletonized. Finally, a distance can be calculated between two skeletons with different tolerance distances. (a) and (b) are two toy cases for intuitively understanding the influence of tolerance distance in PHD.
Step 1. Skeletonize the cell membrane. Zhang-Suen thinning algorithm used ( (Zhang & Suen 1984); Saudagar & Mohammed (2016)) and re-implemented to obtain the skeleton of the cell mem- brane.
Step 2. Calculate the distance between skeletons. Hausdorff distance is common distance used calculate the difference between two point sets. Consider two unordered nonempty sets of points X and Y and the Fuclidean distance d(xr, y) between two point sets. The Hausdorff distance between and Y is defined as
du(X, (X, W) = max dx W. dy, x max
max min d(x, ((x,y)} v)) max { min d(x, y)}, x X y
y E Y x E

---

Under review as a conference paper at ICLR 2021
which can be understood as the maximum value of the shortest distance from point set to another point set. It is easy to prove that the Hausdorff distance a metric (Choi (2019)).
In the task of cell membrane segmentation, we should pay attention to the global distance between two point sets, while Hausdorff distance is sensitive to outliers in two point sets. Therefore, the average distance of the two point sets is obtained naturally by using the average operation instead of all the max operations.
Furthermore, it was found that people have tolerance for the small offset between segmentation results. Specifically, if the distance between two points is very small, people tend to ignore it. Therefore, concept called Tolerance Distance is defined, which represents human tolerance for small errors.
The Perceptual Hausdorff Distance (PHD) is defined Eq.
2
dPHD (X, Y)
min d" (a y)
+
Σ
min d' (2, E
(2)
Y
x
X
x
y
>
d" d= y)
(3)
0, x yl <t
To intuitively understand the influence of tolerance distance in PHD, toy cases (a) and (b) as shown in Fig. [3] are taken as examples. In case (a), the blue skeleton scored 19 points while the orange one scored 18 points. Two skeletons are close in the Euclidean Space but do not coincide. Among all the Euclidean distance d(æ y) of x: E X and y E Y. the max distance is pixels, and the most common distance is 1.
When & = O, which means no mistake can be tolerated, and the PHD is high. If = 1, the PHD value drops lot. When the 2, PHD becomes In case (b), there a large offset between two skeletons. When the t is set to [2, 4]. the decline of PHD value is slow. When t = 6, it drops to O, which is the max distance between two point sets of skeletons. The Different settings of represent the degree of tolerance to the distance between the two skeletons. In practical applications, different tolerance distances can adopted according to different situations.
Consistency between PHD and human perception.
The consistency with human perception of PHD and existing related criteria (TPVF, TNVF, Prec, RVD. Hausdorff, ASSD, V-Rand, and V-Info) based on the subjective experimental results were also calculated (as described in Section 3.2, the formulas can found in Appendix V). The result showed that compared with other criteria, PHD with appropriate tolerance distance was more consistent with human perception.
As shown in Fig. [ while tolerance distance t of PHD increasing from 0 to 800, PHD's 's consistency to human perception rose first and then dropped slowly to O, suggesting human vision does have tolerance for certain offset. Specifically, the maximum value can be reached at 65.48%, when tol- erance distance was set to 3, suggesting that our perceptions prefer to tolerant small perturbations. It was worthy to note that the optimal PHD score (65.48%) was nearly double of the consistency scores obtained by pixel-error based metrics, such as F1 score.
Our experiment shows that most of these compared criteria in color bars be improved by skele- tonizing the segmentation results before evaluation to a certain extent. In Fig. 6 the consistency of these criteria with human perception calculated based on original images can only reach about 30%, while they can improve about 10% on skeletons. Even the best of these metrics (ASSD) on skeletons can only achieve 52.43%, which significantly lower than the score of PHD performance with t = 3. Therefore, it can be concluded that the PHD performs more consistently with human.
4 RE-EXAMINING PHD ON CLASSIC DEEP LEARNING BASED SEGMENTATION METHODS WITH U-RISC
In the previous two sections, we proposed new ultra-high resolution cell membrane segmentation dataset U-RISC and a new perceptual criteria PHD to help solve the two bottlenecks in the field of cell membrane segmentation. The subjective experiment small-scale dataset demonstrated that PHD is more consistent with human perception for the evaluation of cell membrane segmentation than some widely used criteria.

---

Under review as a conference paper at ICLR 2021
Consistency with Human Penception
TWAT
Coe
RYD-SC
[D-5c
TPyT
10
Figure 6: Consistency with human perception. The color bars show the consistency results of FI score, IoU, TPVF, TNVF, Prec, RVD, Hausdorff. ASSD, V-Rand, and V-Info without and with skeletonization (-SK). The gray bars show the consistency result of PHD. The numbers on X-axis denote the tolerance distance settings.
In order to understand the performance of deep learning methods on the U-RISC dataset, we con- ducted in depth investigation on U-RISC with representative deep learning based segmentation
methods and different evaluation criteria. To be specific, we chose representative algorithms ( U-net (Ronneberger al. (2015)), LinkNet(Chaurasia & Culurciello (2017), CASENet YYu et al. (2017). ). SENet (Hu et al. (2018), U-Net++ (Zhou et (2018)))) GL (Chen et al. (2019b)) and re-implemented them U-RISC dataset. Theen four evaluation criteria were used to compare the segmentation results: Fl score, IoU, TPVF, TNVF, Prec, RVD, Hausdorff, ASSD, V-Rand, and V-Info, and PHD.
As mentioned in Sec. 2, the results of the first two rounds of manual labeling results are retained. Therefore, the results of manual annotation under different evaluation criteria will also be analyzed.
Experiment Settings, All the six methods use same training data and testing data to compare the performance. And the parameters and loss functions are same as they are proposed in their references. The parameters for each method and other details are shown in Appendix V.
Experiments Results. The experiment results were shown in Table. The table showed the scores of different evaluation criteria on the first two rounds of manual annotation results and six segmen- tation results with the ground truth.
Our first finding was that U-RISC was a challenging dataset in the field of cell membrane segmen- tation. As shown in Table. the performance of deep learning based methods gained around 0.6 in F1-scores, far below the human level (0.98-0.99) (the first annotation performace) on U-RISC dataset, by contrast, they all exceeded 0.95 on the ISBI 2012. Despite possible improvements by parameter tuning, to such ultra-high resolution images, there was clearly a huge gap between the current popular segmentation methods and human performance.
Our second finding was that evaluation rankings for Fl-score, IoU, V-Rand-sk, and V-Info-sk were more consistent with each other, but different from PHD-basee rankings. Specifically, while PHD tended choose CASNet, none of the other metrics chose CASNet as the best choice. Accord- ing to the subjective experimental results of Sec. 3.2, PHD was much closer to human perception. Therefore, the change of ranking led by PHD may also inspire researchers re-consider the eval- uation criteria for cell membrane segmentation algorithms. It also provides a new perspective for promoting the development of segmentation algorithms.
Discussion. Based on the results of these six algorithms, it can be seen that LinkNet and CASENet are better than other methods. From the perspective of network design, LinkNet makes full use of the low-level local information and directly connects the low-level encoder to the decoder of corresponding size. This design pays more attention to the capture of local information, which leads to a more accurate local prediction. CASENet takes full account of the continuity of the edge and makes the low-level features strengthen the high-level semantic information by jumping links between the low-level feature and high-level feature, which pays more attention to structural information. Therefore, the design of LinkNet might be preferred by the traditional evaluation criteria, while CASENet might be preferred by the PHD. This also explains why the two methods rank differently under these two types of evaluation criteria. More local segmentation results of different algorithms are shown in the Appendix III
8

---

Under review as a conference paper at ICLR 2021
Table 1: Experiments on U-RISC Dataset. This table shows the different evaluation results of the first two rounds of human annotations (H.L.1, H.L.2) and six segmentation results: U-Net (Ron- neberger et al.] (2015), LinkNet(Chaurase & Culurciello (2017), CASENet (Yu et al. (0177). )
SENet (Hu et al. (2018), U-Net++ (Zhou al. (2018)).am J.add GLNet (Chen et al. (2019b)). PHD-t means the PHD score with tolerance distance I. HD means Hausdorff. -sk means evaluating on
skeleton. Note that the ground truth is the third round of human annotation.
Methods
H.L.1
H.L.2
GLNet
CASENet
LinkNet
SENet
U-Net
U-Net++
FIf
0.9212
0.9933
0.4883
0.6007
0.6070
0.5810
0.5212
0.6030
Fl-skf
0.7958
0,9852
0.0891
0.0995
0.1002
0.0955
0.0871
0.1003
loU f
0,8616
0.9901
0.3233
0.4307
0.4371
0.4107
0.3541
0.4329
IoU-sk
0,8760
0,9883
0.0466
0.0524
0.0528
0.0502
0,0456
0.0528
Dice
0,9212
0.9933
0.4883
0.6007
0.6070
0.5810
0.5212
0,6030
Dice-sk f
0.7958
0,9852
0.0891
0.0995
0,4883
0.0955
0.0871
0.1003
V-Rand-sk f
0.9398
0.9910
0.4938
0.5921
0.6310
0.5341
0.5288
0.6211
V-Info-sk f
0,9410
0,9913
0.5120
0.6013
0.6239
0,5433
0.5178
0.6234
TPVF
0,9262
0,9877
0.7657
0.8974
0.9396
0.8970
0.5350
0.9403
TPVF-sk
0,9260
0.9770
0.1752
0.1965
0.1971
0.1899
0.1542
0.2016
TNVF f
0,9993
0.9996
0.8862
0.9622
0,9602
0,9572
0.9768
0,9592
TNVF-sk
0,9493
0.9666
0.9950
0.9950
0.9950
0.9949
0.9961
0.9948
Prec f
0,9640
0,9949
0.3524
0.5604
0,5562
0,5339
0.5991
0,5493
Prec-sk
0.8526
0,9849
0.0914
0.1016
0.1028
0,0971
0,1015
0.00
RVD L
4.3e-06
6.6e-08
4.0c-04
4.0c+07
6.0e-05
2.0c+08
7,8e+08
6.4e-05
RVD-sk 4
3.1e-08
1.0e-08
6.2c+08
5.5e+08
6,4e+08
3.3c408
7.3c+08
2.9c+08
ASSD-sk sk
0.1354
0.0621
1.4970
1,2954
1,2045
1.6575
2,0074
1.7187
HD-sk
3,6441
2,0060
171.52
166.10
152.94
199.16
247.25
213.98
PHD-0
0,2102
0.0209
1.7646
L.7522
1.7254
1.7404
1.7533
1,7251
PHD-1
0.0576
0.0145
1.6311
1.5495
1,5508
1.5744
1.5892
1,5535
PHD-3
0,0456
0.0112
1.2905
1.1345
1.1357
1.1694
1.2009
1.1448
PHD-5 0.0419
0.0084
1.0018
0.8207
0.8175
0.8643
0.9101
0.8351
In addition, an example, we add experiments using U-Net, CASENet, and LinkNet on ISBI2012
and SNMI3D datasets (Appendix VI, VIII, IX). The results in Appendix VI show that U-Net with our chosen parameters can perform close SOTA on ISBI2012 (ours: (tamd=0.96899 V-Info=0.9723; SOTA: V-Rand=0.9837, --nnf==0.8888 (on skeleton)) and SNEMI3D (ours: V-w.nd--.nndn333 SOTA: V-Rand=0.9751 (on skeleton)), although we made little efforts parameter tuning. However, with the same parameter setting, U-Net gets poor scores (V-Rand=0.528% V-Info0..518) Such a big gap in its performance between U-RISC and previous datasets suggests the challenge from U-RISC dataset, which hopefully will motivate novel designs of machine learning methods in the future. And the results in Appendix VIII and IX show that evaluation rankings for F1-score, IoU, V-Rand-sk, and V-Info-sk are more consistent with each other, but they are different from PHD-based rankings.
U-RISC.
5 DISCUSSION AND CONCLUSION
This paper aims solve the two bottlenecks the development of cell membrane segmentation. Firstly, we proposed I-RISC,UUtra-hi Resolution Image Segmentation dataset for Cell membrane, the largest annotated EM dataset for the Cell membrane far. To our best knowledge, U-RISC is the only uncompressed annotated EM dataset with multiple iterative annotations and uncompressed nigh-resolution raw image data. During the analysis process of the U-RISC, inconsistency between current evaluation criteria for segmentation (e.g. F1 score, IoU) and hu- man perception. Therefore, this article secondly proposed a human-perception based evaluation criterion, called Perceptual Hausdorff Distance (PHD). Through a subjective experiment scale dataset, experiments results demonstrated that the new criterion more consistent with human perception for the evaluation of cell membrane segmentation. In addition, the evaluation criteria of PHD and existing classic deep learning segmentation methods are re- re-examined.
found certain
small-
In future research, we will consider how to improve deep learning segmentation methods from the perspective of cell membrane structure and apply PHD criterion for connectomics research. More disccusions are shown in Appendix VII.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Roberto Alejo, Juan Monroy-de Jesús, Juan H Antonio- Antonio-Velázquez Velázquez. A selective dynamic sampling back-propagation approach for handling the two-class imbalance problem. Applied Sciences, 6(7):200, 2016.
Erika López- and Juan A
James Anderson, Bryan W Jones, Carl B Watt, Margaret V Shaw, Jia- Hui Yang, David DeMill, James $ Lauritzen, Yanhua Lin, Kevin D Rapp, David Mastronarde, et al. Exploring the retinal connectome. Molecular vision, 17:355-79, 2011.
Ignacio Arganda-Carreras, Srinivas C Turaga, Daniel R Berger, Dan Ciresan, Alessandro Giusti, Luca M Gambardella, Jürgen Schmidhuber, Dmitry Laptev, Sarvesh Dwivedi, Joachim M Buh- mann, et al. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in neuroanatomy, 9:142, 2015a.
Ignacio Arganda-Caurn Carreras, Srinivas C. Turaga, Daniel R. Berger, Dan Cireşan, Alessandro Giusti, Luca M. Gambardella, Jürgen Schmidhuber, Dmitry Laptev, Sarvesh Dwivedi, Joachim M. Buh- mann, Ting Liu, Mojtaba Seyedhosseini, Tolga Tasdizen, Lee Kamentsky, Radim Burget, Va- clav Uher, Xiao Tan, Changming Sun, Tuan D. Pham, Erhan Bas, Mustafa G. Uzunbas, Albert Cardona, Johannes Schindelin, and H. Sebastian Seung. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in Neuroanatomy, 9:142, 2015b. ISSN 2tt.9///aaaaa2255.000000
1662-5129. doi: article/10. 3389/ fnana 10111.00141
URL https: /ttp::/wwww.. www www.w.froaee front ierttiieeiiiii... org/
Nicolas Aspert, Diego Santa-Cruz, and Touradj Ebrahimi. Mesh: Measuring errors between surfaces using the hausdorff distance. In Proceedings. IEEE international conference on multimedia and expo, volume 1, pp. 705-708. IEEE, 2002.
Albert
Cardona, Stephan Saalfeld, Stephan Preibisch, Benjamin Schmid, Anchi Cheng, Jim Pulokas, Pavel Tomancak, and Volker Hartenstein. An integrated micro-and macroarchitectural analysis of the drosophila brain by computer-assisted serial section electron microscopy. PLOS Biol, 8(10): c1000502, 2010.
Abhishek Chaurasia and Eugenio Culurciello. Linknet: Exploiting encoder representations for efficient semantic segmentation. In 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1-4, IEEE, 2017.
Wuyang Chen, Ziyu Jiang, Zhangyang Wang. Kexin Cui, and Xiaoning Qian. Collaborative global- local networks for memory-efficient segmentation of ultra-high resolution images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8924-899 9933-,33,, 2019a.
Wuyang Chen, Ziyu Jiang, Zhangyang Wang, Kexin Cui, and Xiaoning Qian. Collaborative global- local networks for memory-efficient segmentation of ultra-high resolution images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8924 8933, 2019b.
Jiwon Choi. Gromov-hausdorff distance between metric graphs, 2019.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204, 2015.
Dan Ciresan, Alessandro Giusti, Luca M Gambardella, and Jürgen Schmidhuber. Deep neural net- works segment neuronal membranes in electron microscopy images. In Advances in neural infor- mation processing systems, pp. 2843-2851, 2012.
Alan Curry, Hazel Appleton, and Barry Dowsett. Application of transmission electron microscopy to the clinical study of viral and bacterial infections: present and future. Micron, 37(2):91-106, 2006.
Malcolm R Dando. Japan's 's brain/minds project. In Neuroscience and the Problem of Dual Use, pp. 127-147. Springer, 2020.
llke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes. Devis Tuia, and Ramesh Raska. Deepglobe 2018: A challenge to parse the earth through satellite images. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 172-17209. IEEE, 2018
10

---

Under review as a conference paper at ICLR 2021
Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3): 297-302, 1945.
Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. The im- portance of skip connections in biomedical image segmentation. Deep Learning and Data Labeling for Medical Applications, pp. 179-187. Springer, 2016.
ROBERT A ERLANDSON. Role of electron microscopy in modern diagnostic surgical pathology. Modern Surgical Pathology, pp. 71, 2009.
Christopher Genovese, Marco Perone-Pacifico, Isabella Verdinelli, Larry Wasserman, et al. Man- ifold estimation and singular deconvolution under hausdorff loss. The Annals of Statistics, 40(2): 941-963, 2012.
Stefan Gerl, Johannes C. Paetzold, Hailong He, Ivan Ezhov, Suprosanna Shit, Florian Kofler, Amirhossein Bayat, Giles Tetteh, Vasilis Niziachristos, and Bjoern Menze. A distance-based loss for smooth and continuous skin layer segmentation in optoacoustic images. In Anne I.. Mar- tel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2020, pp. 409-319, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59725-2.
Alfie Gleeson and Abigail Sawyer. Mapping brain, 2018.
Kristen M Harris, Elizabeth Perry, Jennifer Bourne, Marcia Feinberg, Linnaca Ostroff, and Jamie Hurlburt. Uniform serial sectioning for transmission electron microscopy. Journal of Neuro- science, 26(47):12101-12100 2006.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. Comparing images using the hausdorff distance. JEEE Transactions on pattern analysis and machine intelligence, 15(9): 850-863, 1993.
Thomas R Insel, Story C Landis, and Francis S Collins. The nih brain initiative. Science, 340(6133): 687-688, 2013.
ISBI 2012. Segmentation of neuronal structures em stacks challenge ISBII 2012. brainiac2. mit hww.iaaa...mi...ddiiia.aa..aaa wwwhchllllgeeeccccccccmcoooallll 2012.
http://
ISBI 2013. Segmentation of neuronal structures em stacks challenge ISBI 2013. brainiac2 www.:/mittttiiteeemmmmmmmmmee SAOi:/SNEEM3D// 2013.
http://
Davood Karimi and Septimiu E Salcudean. Reducing the hausdorff distance in medical image segmentation with convolutional neural networks. IEEE Transactions on medical imaging, 39(2): 499-513, 2019.
Narayanan Kasthuri. Kenneth Jeffrey Hayworth, Daniel Raimund Berger, Richard Lee Schalek, José Angel Conchello, Seymour Knowles-Barley Dongil Lee, Amelio Vázquez-Reing Reina, Verena Kaynig, Thouis Raymond Jones, et al. Saturated reconstruction of volume of neocortex. Cell, 162(3):648-661) 2015.
Sven Kosub. A note on the triangle inequality for the jaccard distance. Pattern Recognition Letters, 120:36-36 38. 2019.
Kisuk Lee, Jonathan Zung, Peter Li, Viren Jain, and H Sebastian Seung. Superhuman accuracy on the snemi3d connectomics challenge. arXiv preprint arXiv: 0ttl::/70000/2000 2017.
Der-Chiang Li, Chiao-Wen Le Liu, and Susan C Hu. A learning method for the class imbalance problem with medical data sets. Computers in biology and medicine, 40(5):509-518, 2010.
Jeff W Lichtman, Hanspeter Pfister, and Nir Shavit. The big data challenges of connectomics.
Nature neuroscience, 17(11):1448-144 1454, 2014.

---

Under review as a conference paper at ICLR 2021
PL Lin, PW Huang, PY Huang, and HC Hsu. Alveolar bone-loss area localization in periodontitis radiographs based on threshold segmentation with a hybrid feature fused of intensity and the h- value of fractional brownian motion model. Computer methods and programs in biomedicine, (2133):1771266 2015.
Shuo Liu, Wenrui Ding, Chunhui Liu, Yu Liu, Yufeng Wang, and Hongguang Li. Em: edge loss reinforced semantic segmentation network for remote sensing images. Remote Sensing, 10(9): 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440 2015.
Wiro J Niessen, Carolien J Bouma, Koen I. Vincken, and Max A Viergever. Error metries for quan- titative evaluation of medical image segmentation. In Performance Characterization in Computer Vision, pp. 275-284. Springer, 2000.
Mu-ming Poo, Jiu-lin Du, Nancy Y Ip, Zhi-Qi Xiong, Bo Xu, and Tieniu Tan. China brain project: basic neuroscience, brain diseases, and brain-inspired computing, Neuron, 92(3):591-596, 2016.
Jumpol Rachasingho and Santi Tasena. A metric space of subcopulas-an approach via hausdorff' distance. Fuzzy Sets and Systems, 378: 144-166,,, 2020.
Javier Ribera, David Guera, Yuhao Chen, and Edward J Delp. Locating objects without bounding boxes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479-6489, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Yutaka Sasaki et al. The truth of the f-measure. 2007, 2007.
Abdul Khader Jilani Saudagar and Habeeb Vulla Mohammed. Openev based implementation of zhang-suen thinning algorithm using java for arabic text recognition. In Information Systems Design and Intelligent Applications, pp. 265-271. Springer, 2016.
Fred A Spiring. The reflected normal loss function. Canadian Journal of Statistics, 21(3):321-330, 1993.
Carole H Sudre, Wengi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 240-248. Springer, 2017.
Remco C Veltkamp and Michiel Hagedoorn. Shape similarity measures, properties and construc- tions. International Conference on Advances Visual Information Systems, pp. 467-476. Springer, 2000.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference computer vision, pp. 1395-1403, 2015.
Varduhi Yeghiazaryan and Irina Voiculescu. Family of boundary overlap metries for the evaluation of medical image segmentation. Journal of Medical Imaging, 5(1):015006, 2018
Zhiding Yu, Chen Feng, Ming- Yu Liu, and Srikumar Ramalingam. Casenet: Deep category aatagory-aware aware semantic edge detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 19964-973,,0 2017.
Kun Zhang, Zhiyong Wu, Daode Yuan, Jian Luan, Jia Jia, Helen Meng, and Binheng Song. Re- weighted interval loss for handling data imbalance problem of end-to-end keyword spotting. Proc. Interspeech 2020, pp. 2567-2571, 2020.
12

---

Under review as a conference paper at ICLR 2021
TY Zhang and Ching Y. Suen. A fast parallel algorithm for thinning digital patterns. Communica- tions of the ACM, 27(3):236-22 239, 1984.
Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 405 4200, 2018.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 3-11. Springer, 2018.
A APPENDIX
I. Fig. 7 is the interface of perceptual consistency experiments.
Which of the left and right pictures on the screen is more similar to the middle picture?
1/50
Left
Difficult to choose
Right
Figure Interface of perceptual consistency experiments.
II. Fig. and Fig. 9 are some examples of subjective experiment images.
III. Fig. 10 and Fig. are some examples of segmentation results of different algorithms.
IV. Experiment Details.
4.1 Subjective Experiment
1) Firstly, the 20 human raters were introduced the value of cell membrane segmentation to connec- tivity and the importance of structure before testing. And then we used several simple examples to teach them about the experimental process.
2) During the formal experiment, the distribution and selection data were random. The subjects only need to choose one of the two images that they think is more similar to the ground truth.
3) The 200 groups of images for subjective experiment were randomly selected from the segmen- tation results produced by the above six methods. Therefore, the training data from the same dataset as those that were used to create the 200 groups of images that the 20 humans evaluated the results.
4) In order to ensure the continuity of the experiment, each subject was asked to judge each group of images within specified time (less than 10 minutes).
13

---

Under review as a conference paper at ICLR 2021
5) In order to prevent the subjects from fatigue in a long period of experiments, 200 groups of images were distributed to the subjects four times on average.
4.2 Experiments U-RISC dataset.
All the six methods use same training data and testing data to compare the performance. And the parameters and loss functions same they proposed in their references.
In the training stage, 60% of the dataset was used as the training data, and then the original image was randomly cut into 1024 1024 patches to generate 50,000 training images and 20,000 validation
images.
Random flipping and clipping were used for data augmentation. Four V100 GPUs train each algorithm. In the testing stage, the original image was cut into the same size of training image, and the patch tested. These patches were eventually spliced back to the original size for evaluation.
used to
Table 2: Paremeters in methods.
Paremeters
GLNet
U-Net
SENet
CASENet U-Net+ LinkNet
Random Crop
True
True True
True
True
True
True
Random Crop
True
True
True
True
True
Learning Rate
5e-5/2e-5/5e-5
1e-3
1e-7
1e-7
1e-7
5e-4
2/1/1 Sub Batch Size
1
Num Epochs
30/20/30
100
150
300
200
300
Workers
12/12/12
16
16
16
12
V. Formulas of criteria mentioned in the texture.
The formulas of metrics we compared are shown in Table The symbols in formulations are explained as follows.
Precision (also called positive predictive value) the fraction of relevant instances among the retrieved instances, while recall (also known sensitivity) is the fraction of the total amount of relevant instances that were actually retrieved.
TP (true positives), TN (true negatives), FP (false positives), and FN(false negatives) compare the results of the classifier under prediction with ground truth. The terms positive and negative refer to the classifier's prediction, and the terms and false refer whether that prediction corresponds to the ground truth.
and Y are two point sets. d(x: y) are the points in and Y respectively.
In V-Rand (Arganda-Carreras Carreras et al. (2015)), suppose that S is the predicted segmentation and T' is the ground truth segmentation. Define Pi,j as the probability that randomly chosen pixel belongs to segment i in S and segment j in T. This joint probability distribution satisfies the nor- malization condition Egy Piş 1. The marginal distribution s; E, Pij is the probability that
a randomly chosen pixel belongs to segment in S, and the marginal distribution t, = Σ, Ptj is defined similarly.
V-Info (Arganda-Carme Carreras et al. (015b),, the mutual information 1(S:T) 7)
Egg ig Pij log Pij -E, 8, log 8; E, tú logt, ty measure of similarity between S and T. H(S)
Σ, 8t log is the entropy function.
VI. Experiments on ISBI2012 and SNEMI3D with U-Net. (Table, 4)
VII. Further Discussion.
Although the content of this paper is mainly involved in EM cell segmentation, we think the signifi-
cance is beyond.
14

---

Under review as a conference paper at ICLR 2021
Table 3: Formulation of criteria.
Criteria
Formulation
Fl score
x Expecisioon Xrecall
precision-recod
Dice Coefficient
2x EXTP+FPFFN
IoU
TP
TP+FP+FN
TPVF
TP
TP+EN
TNVF
TN
FP4TN
Prec
TP
TP4FP
RVD
FP-FN
TP+FN
Hausdorff
max {max {minger d(x, y)} maxyer {minmex d(x, y)}}
ASSD
(Ewxx min d( y) Eyer minxex d(x,)
|*++
V-Rand
RRand
Eg
21 1/ 21
V-Info
Vinfo
1(S:T) (I-a)H())aaH(T
PHD
Emex minyer dl (xx (x: + T.T Eye: min of (x,
X
| 1o-ype y/1 x y t
d= y)
0, r y St t
According to experimental results, the popular methods do not perform well on our dataset (ex- ceeded 95% on ISBI2012, while about 60% U-RISC), It shows that U-RISC challenging dataset that can promote the development of related machine learning and deep learning methods.
The U-RISC dataset may reveal several classic challenges in the field that haven't 't been solved: One challenge might be the "imbalance problem of samples" (Alejo et al. (0116); Li et al. (2010); Zhang al. (2020)) ). Due to ultra-high resolution images, the pixels labeled cell membranes only account for 5.64% of total pixels in training sets, in contrast to 21 1.96% in ISBS2012 2 and 33.23% in
SNEMI3D. The future design of deep learning methods on U-RISC will have to solve this issue.
Some other challenges might include, e.g. ultra high-resolution image segmentation (Demir et al.
(2018); Zhao el al. (2018); Chen al.a).211)) (2019a), appropriate loss function design (Sudre et al. (2017); Spiring (1993); Choromanska et al. (2015)).and the issues related to "unclosed" edges suggested by the reviewer 3.
Taken together, strongly believe that the U-RISC dataset will have great contribution in technique novelty, by revealing defects in the existing popular methods and promoting novel algorithms for solving classic challenges in machine learning deep learning community.
In addition, the design of evaluation criteria has been widely concerned in the field of computer science (Gerl al. (2000) Lin et al. (2015); Liu et al. (2018)). The PHD we proposed may inspire researchers from new perspective and further promote the developments of algorithms. The tech- nical novelties of the PHD metric lie in many aspects. To list few, (1) It can be potentially used
15

---

Under review as a conference paper at ICLR 2021
Table 4: Experiments ISBI2012 and SNEMI3D.
Methods
F1
IoU
TPVF1
TNVF1
PrecT
ISBI2012
0,9701 0.9380
0.9625
0.9261
0,9461 0.9324
0.9196
SNEMI3D
0,9368
0.9013
0.8980
Methods
F1-sk f
IoU-sk f
TPVF-sk
TNVF-ski
Prec-skf
ISBI2012
0.3942 0,2236
0.3581
0.7097 0.5466
0.9760 0.9709
0.3614
SNEMI3D
0.1598
0.1928
Methods
RVD.
ASSD
HDJ
V-Rand
V-Infot
ISBI2012
5.7e-05 4.8e-04
0.3751 1.4869
43.501 108.93
0.9689 0.9389
0.9723
SNEMI3D
0.9376
Methods
RVD-skj
ASSD-sk
HD-sk]
V-Rand-skt
V-Info-skf
ISBI20T2
9.3e-06 8.4e-06
1.1573
45.941
0.9633
0.9601
SNEMI3D
1.4210
130.95
0.9201
0.9224
Methods
PHD-O
PHD-1 +
PHD-3
PHD-5 +
ISBI2012
1.0495
0.4650
0.1043
0.0480
SNEMI3D 1,4012
1.0730
0.7666
0.6124
in other tasks, such as vascular segmentation (Gerl et al. (2020)), bone segmentation ( (Lin et al. 2015)), edge detection ( [Liu et al. (2018)), and other tasks related to structural and shape informa- tion. For example, (Geerl et al. (2020) successfully used a distance-based criterion improve skin layer segmentation in optoacoustic images. (2) It can be modified into loss functions which is also part of our on-going work. It is worthy to note that some works have successfully integrated Haus- dorff distance into the loss function (enoveee al. (2012); Karimi & Salcudean (2019); Ribera
et al. 2019).
VIII. Experiments ISBI2012 with U-Net, CASENet, and LinkNet. (Table.
IX. Experiments on SNEMI3D with U-Net, CASENet, and LinkNet. (Table.
16

---

Under review as a conference paper at ICLR 2021
Table 5: Experiments ISBI2012 Datasets. This table shows the different evaluation results of three segmentation results: U-Net (Ronneberger et al. (2015). CASENet (Yu et al. (2017), with tolerance distance HD means Hausdorft. -sk means evaluating on skeleton. Note that the ground truth is the third round of human annotation.
LinkNet(Chauraiia & Culurciello (2017)) PHD-/ means the PHD
Methods
U-Net
CASENet
LinkNet
FIf
0.9701
0.9713
0.9724
F1-skt
0.3942
0.4103
0.3807
IoU
0,9625
0,9700
0.9701
IoU-sk f
0.3581
0.3377
0.3810
V-Rand-sk
0.9633
0.9653
0.9699
V-Info-sk f
0.9601
0.9627
0.9656
Dice f
0.9701
0.9713
0.9724
Dice-sk f
0.3942
0.4103
0.3807
TPVF
0.9261
0.9326
0.9311
TPVF-sk
0.7097
0.6999
0.7036
TNVF
0.9461
0.9312
0.9308
TNVF-sk f
0.9760
0.9532
0.9650
Prec
0.9196
0.9177
0.9180
Prec-sk
0.3614
0.3310
0.3198
RVD +
5.7e-05
7.9e-05
3.2e-05
RVD-sk
9.3e-06
1.0e-05
4.9e-06
ASSD-sk +
1.1573
0.9211
1.0362
HD-sk
45.941
39.865
63.120
PHD-0 +
1.0495
1.0013
1.0235
PHD-1
0.4650
0.3892
0,4018
PHID-3 0.1043
0.0997
0.1006
PHD-5
0.0480
0.0413
0.0438
Table 6: Experiments on SNEMI3D Datasets. This table shows the different evaluation results
of three segmentation results: U-Net (Ronneberger et al. (2015).. CASENet (Yu et al. (2017). ). LinkNet(Chaurasia & Culurciello (2017)). PHD-t means the PHD score with tolerance distance t. HD means Hausdorff. -sk means evaluating on skeleton. Note that the ground truth is the third round of human annotation.
Methods
U-Net
CASENet
LinkNet
FIf
0.9380
0,9389
0.9401
F1-skf
0.2236
0.2913
0.2830
loU
0.9368
0.9373
0.9389
loU-sk
0.1598
0.1732
0.2001
V-Rand-sk
0.9201
0.9210
0.9276
V-Info-sk t
0.9224
0.9226
0.9264
Dice f
0,9380
0.9389
0,9401
Dice-sk
0.2236
0.2913
0.2830
TPVF
0.9013
0.9006
0,9103
TPVF-sk
0.5466
0.5349
0.5382
TNVF
0.9324
0.9321
0,9299
TNVF-sk t
0,9709
0.9743
0.9750
Prec
0.8980
0.8992
0.8793
Prec-sk
0.1928
0.1917
0.1876
RVD
4.8e-04
4.0e-04
8.9e-05
RVD-sk J
8.4e-06
8.0e-06
3.6e-06
ASSD-sk 1
1.4210
1.368I
1.2785
HD-sk
130.95
156.34
129.02
PHD-0
1,4012
1.3792
1,3988
PHD-1
1.0730
0.9989
1.0215
PHD-3
0.7666
0.6392
0.7313
PHD-5
0.6124
0.5821
0.5988
17

---

Under review as a conference paper at ICLR 2021
FI score 0.5209 IoU 0.3375
FI 0.6228
IoU 0.4523
Dice 0.5047
Dice 0.6228
Fl score 0.5347
Fl score 0.6232
IoU = -0,9993
IoU = 0,4526
Dice 0.4608
Dice 0.6232
score 0.5084
0.5566
ToU 0.3408
IoU 0.3856
0.5084
0.5566
score 0,3880
FI score 0.5337
IoU 0.2038
IoU = 0.3640
0.3387
Dicc 0.5337
FI 0.5525
FI 0.5622 loU 0.3910
IoU 0.3817
Diee 0.5525
Dice 0.5622
Manual Annolation
(b) Segmentation
(e) Segmentation 2
Figure 8: Examples of subjective experiment images (1). The figure in green box is the choice of most subjects. The scores of Fl score, IOU and Dice only used to illustrate the inconsistency between the three criteria and human perception, which are not shown to the subjects during the subjective experiments.
18

---

Under review as a conference paper at ICLR 2021
FI score = 0,4494
FI 0,5273
IoU = 0.2770
IoU 0.3571
Dice 0.4338
Dice 0.5263
0,5482
F1 0.5547
ToU -0.3776
ToU: - 0.3838
Dice 0.5482
Dice 0.5547
Fl score 0.5084
Fl score 0,5566
IoU = 0.3408
loU = -038856
Dice 0.5084
Dice 0.5566
Fl score = 0.5386
Fl score = 0.6145
loU 0.3642
IoU = 0.4423
Dice 0.5340
Dice 0.6133
score 0,5224 0.3519
Fl 0.7869 IoU 0064877
Dice 0.5205
Diec 0.7869
Manual Annotation
(b) Segmentation
Segmentation
Figure 9: Examples of subjective experiment images (2). The figure in green box is the choice of most subjects. The scores of F1 score, IOU and Dice are only used to illustrate the inconsistency between the three criteria and human perception, which are not shown to the subjects during the subjective experiments. For the example in the last line, most of the subjects chose "Difficult to choose"
19

---

Under review as a conference paper at ICLR 2021
Manual Annotation
CASENet
LinkNet
GLNet
SENet
U-Net
U-Net++
(a) Case
(b) Case 2
(c) Case 3
(d) Case 4
Figure 10: Examples of segmentation results (1).
20

---

Under review as a conference paper at ICLR 2021
Manual Annotation
CASENet
LinkNet
GLNet
SENet
U-Net
U-Net++
(a) Case
(b) Case 2
(c) Case 3
(d) Case 4
Figure 11: Examples of segmentation results (2).
21


Reviews:
[{"review_id": "l8gAcrXvpBn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# pros:\n\n- To the author's and reviewer's best knowledge, this paper includes the largest annotated public EM data set for cell membrane segmentation (in case it is published with this paper). Until now, the ISBI 2012 challenge (http://brainiac2.mit.edu/isbi_challenge/) dominates the evaluation of cell membrane segmentation in EM data, even though the performance is nearly saturated. New datasets can identify potential weaknesses in similar domains, that are not covered in current datasets and by state of the art methods, yet.\n\n- The discussion about suitable segmentation metrics for cell membrane segmentation is important and must be continued.\n\n- The article is written in a clear and comprehensive manner.\n\n\n# cons:\n\n- The discussion about appropriate metrics for cell segmentation, that do not depend on the thickness of the segmented cell membrane, has extensively been elaborated in \"Crowdsourcing the creation of image segmentation algorithms for connectomics\" by Ignacio Arganda-Carreras et al., Frontiers in Neuroanatomy 2015 (9) 142: pp. 1-13. However, this paper is not referenced and the therein proposed metrics are not mentioned or compared.\n\n- The evaluated \"state-of-the-art\" methods are not \"state-of-the-art\". They do not correspond to the top entries of the current ISBI Segmentation Challenge Leaderboard. Additionally, no parameters of the methods were adapted.\n\n- It is left unclear how the 20 human raters were instructed to evaluate the segmentation results. For the correct evaluation, not (only) intuitive human perception must be taken into account, but also the usability of the resulting segmentation. The segmentation results on high resolution EM data presented in this paper display many \"unclosed\" edges, which lead to severe problems, when using the segmentation as a basis for connectivity analysis. To the reviewer's understanding, the proposed Perceptual Hausdorff distance will hardly penalize these errors.\n\n- The dataset is not published in the format of a challenge, which would allow benchmarking on a private test set.\n\n- Spelling should be revised.\n\n\n# Summary\n\nThe presented new high-quality dataset is highly valuabl to the community in order to improve and develop methods for instance segmentation, specifically cell membrane segmentation. The segmentation of thin cell boundaries imposes different challenges and includes different priors, than in other domains of instance segmentation.\n\nThe use of appropriate evaluation metrics is crucial to identify suitable und successful methods in experiments and must be critically discussed including domain knowledge. However, in the presented paper, the discussion about suitable metrics is not appropriately linked to the existing literature. A metric is proposed, that is (more) consistent with \"human perception\". This is an interesting aspect, but its contribution to the successful analysis of neuronal connectivity from EM data remains unclear.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "efYk86zOz33", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength: (1) This work proposed an ultra-high-resolution image segmentation dataset for the cell membrane, named U-RISC. The proposed U-RISC is the largest annotated Electron Microscopy (EM) dataset for the cell membrane with multiple iterative annotations and uncompressed high-resolution raw data. Given the uniqueness of the proposed dataset, it is likely to contribute to the future EM based research, such as membrane segmentation. \n(2) For membrane segmentation in EM images, the authors develop a human-perception based evaluation criterion, called Perceptual Hausdorff Distance (PHD). Based on the experiments on a small-scale dataset. the proposed PHD metric is better consistency with human perception than the traditional ones.\n \nWeakness: (1) The first concern is on the proposed PHD metrics. The reviewer thinks that there is a lack of comparison analysis between the proposed PHD with the traditional Hausdorff distance. According to Equation 2, the PHD metric is build based on the Hausdorff distance. Therefore, it is necessary to include comparison analysis of using the traditional Hausdorff distance. This will further highlight the novelty of the PHD metric proposed in this paper. \n(2) The second concern is the limit technique novelty. The overall contributions of this paper have two parts: establishing a new dataset, and proposing a new PHD metric for the EM membrane segmentation tasks. However, there is no contributions based on the machine learning or deep learning based methods. The reviewer agrees that this manuscript has made some contributions on biomedical image analysis. However, the reviewer thinks this paper cannot meet the requirement of the ICLR conference. \n(3) The third concern is the limit validation experiments on the PHD metric. Since the proposed PHD metric is particularly for membrane segmentation, it is supposed to be effective on other EM datasets, such as the ISBI2012 and SNEMI3D challenges. However, the authors did not conduct comparison methods on any other EM datasets. It would be more convincing to conduct experimental analysis on various EM membrane segmentation tasks. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OVHCJC8hAL8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work presents two contributions towards cell membrane segmentation. First, it introduces a new labelled database for this purpose. The authors claim that this is the largest labelled database of high resolution Electron-Microscopy images for this purpose. Second, the work tackles the issue that the F1Score, Dice and IoU scores that evaluate segmentation performance by quantifying the overlap of 2 segmentations are not adequately describing quality of a segmentation with respect to what human experts would prefer for the task. This has been evaluated via employing humans (experts on the task of cell segmentation) to grade which segmentation they prefer, and analysed how their preference correlates with these scores. As a solution, the authors propose a metric, PHD, that can be described as the average Haussdorf distance between the skeletons of two segmentations, together with a threshold tolerance, which they show correlates better with the preference of experts with respect to segmentation quality on this task.\n\n###################################\nReasons for score: \n\nI recommend a rejection of this work for the following reasons. On one hand, constructing a database and releasing it to the community is a great contribution. I am sure that this would be very well accepted. But, on the other hand, I don’t think the article adequately describes the dataset or compare it adequately with existing databases (which makes less of a “database article”). Instead, half the article discusses a metric that is essentially an adaptation of the haussdorf distance (actually, of the “average symmetric surface distance”), adapted in a manner specific to the cell-segmentation task (applied on skeleton, and with a tolerance, the importance of which is questionable). This 2nd contribution has not been accompanied by a literature review on metrics (e.g. only discusses IoU/F1/Dice, missing related distance based metrics like ASSD completely), nor adequately evaluated with such related metrics (besides IoU/Dice/F1). Finally, the modifications, along with many claims in the article, are only relevant to the specific task of cell segmentation (and quite subjective).\n\n###################################\n\nPros: \n \n1. Great contribution by releasing publicly a new labelled database of high quality. Seems there was a lot of effort to construct good quality ground truth on a number of images much larger than the existing publically available databases. This is definitely interesting for the community that works in this problem.\n\n2. Interesting human-based evaluation of the usefulness of IoU/Dice/F1 scores for the cell-segmentation problem (by 20 humans, this is nice). It is known in the broader community that overlap metrics (IoU/Dice etc) are not perfect, hence there is a lot of work on other metrics [1,2 etc], but this substantiates/quantifies it very nicely.\n\n###################################\n\nCons:  \n\n1. If I would judge the paper focused on the 1st contribution (releasing a database), I would say that it does not contain a sufficient analysis of the database itself, and especially not an adequate description and comparison with other public databases. I think this point could be sufficiently addressed in the rebuttal.\n \n2. From the technical viewpoint, the claimed 2nd contribution is the derivation of a new metric, but the work has not performed any literature review on related work on metrics except IoU/Dice/F1. In fact, the work produces a metric (Eq.2) that seems to me the same as Average Assymetric Surface Distance (ASSD, see [1]), applied to the skeleton (thinned) segmentation, with a tolerance (task-specific modifications). I note that both skeleton-like operations have been previously performed for computing metrics in the cell-segmentation domain (e.g. for evaluation of ISBI2012 challenge: http://brainiac2.mit.edu/isbi_challenge/evaluation, notice the “after thinning” operation). Tolerance-based modifications have also been applied to various other metrics (e.g. [2] below) and are task-specific modifications (and not necessary for the metric to be appropriate in the general sense). In my opinion, this makes the value of the 2nd contribution very low. I think this point cannot be sufficiently addressed in the rebuttal, as I basically think that the contribution of the derived metric is low in comparison to existing literature.\n\n3. The “skeleton” part of PHD is not individually evaluated whether it actually adds substantially. I note that without it, the metric is essentially ASSD with tolerance (tolerance being task-specifically motivated in this context).\n\n4. Evaluation of the proposed metrics is limited, because it does not contain other metrics except IoU/Dice. E.g., it should have been compared with Haussdord, ASSD, etc.\n\n5. The scope of the paper is limited to the cell-segmentation problem.\n\n6. The work contains a number of statements that are not true and would need significant text alterations to reduce them.\n\nThe above Cons are described in detail below, with the detailed comments I raise in the “Questions for rebuttal period” section.\n\nReferences:\n\nPaper with some review on related metrics (there are many more such reviews):\n[1] Yeghiazaryan and Voiculescu, Family of boundary overlap metrics for the evaluation of medical image segmentation, Journal of Medical Imaging, 2018\n\nPaper implementing tolerance (which motivates it, and is subjective to the specific task and needs):\n[2] Nikolov et al, Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy, arxiv 2018\n\n\n###################################\n\nQuestions/points to address during rebuttal period: \n\nThe abstract “proposes” a database with “multiple iterative annotations”. However, the actual database released only contains the last 3rd annotation. Please rephrase the abstract so that this is clear to the reader that only 1 annotation should be expected. \n\nSimilar to the above, Sec 2.1 claims the existence of multiple (3) annotations as an advantage of the proposed database over other databases (Sec.2.1, “Besides that, U-RICS produced 3 sets of annotations…, all of which can be applied in developing depe learning”). But, these 3 annotations are not released, hence their existence is irrelevant to the reader and this claim/advantage simply does not hold.\n\nThe authors claim that the intermediate annotations are “are very valuable for learning” (Sec.2) and therefore will not be released. In this case, if they are valueable, perhaps consider releasing them? Otherwise, I would suggest rephrasing the article, reducing the emphasis on these 3 labels across the whole manuscript, as they are of little relevance to the reader. You can spend the saved space to extend on more related points (e.g. related work etc, see below).\n\nSec 1. claims: “We found the human performance is far superior to these methods”. I think this statement is very strong and not supported adequately by the current evaluation. I think the authors refer to experiments in Table 1., where the labels from the 1st and 2nd iteration where compared with the results in the 3rd iteration. But, naturally, image after 3rd iteration is conditioned (related) very closely to those from the 1st and 2nd iterations. Of course they will have very high agreement. Also, we note, the 2nd iteration is not a result of 1 human but of multiple (5 experts + annotator). Hence, for these 2 reasons, they cannot support the statement. For a correct assessment for such a claim, segmentations from a single annotator, who is not the same contributing to making the ground truth for the image, should be evaluated against the ground truth. Additionally, inter-rate segmentation performance (multiple humans) could also be evaluated to make such claims. I would recommend this claim to be altered, as well as state explicitly these factors (conditioning of 1st, 2nd, 3rd iterations) in sec 4 to explain how come agreement is so high in Table 1. \n\n\nThe work has performed no literature review on related metrics for segmentation. It discusses and evaluates solely overlap based metrics (Dice/F1 & IoU), and then proposes a distance-based measure. The authors should have discussed and evaluated distance based measures too, and especially have a look at the Average Symmetric Surface Distance (ASSD) metric, which is very similar to what they propose (Eq.2). See [1] as a starting point for related metrics, but there are many more. Many of these papers raise the argument that no metric is enough for all tasks, and that “quality” of a segmentation is subjective to the task. Hence, for each task, one should chose the correct metric, while for “objective” and all-around evaluation of a “general” method (e.g. an arbitrary segmentation network), multiple complimentary metrics should be used. See [2] also with related discussion. Also see that ISBI2012 challenge itself implemented multiple metrics (http://brainiac2.mit.edu/isbi_challenge/evaluation) that are not duscussed here (and their paper also discusses appropriateness of metrics).\n\nSec.1 “Considering that image compression generally loses many texture details”: Not necessarily. Depends on how much compression, what type of compression, and the actual content. If you would like to claim this, I would suggest you analyse what type of structures disappear if you do a 2x sub-sampling (based on the fact that ISBI has 2x less resolution). Otherwise, I would suggest this is rephrased a bit less strong.\n\n\nSec.2.1 intends to do a comparison with ISBI database and SNEMI3D databases. However it still does not provide important information about them. For example, there is no mention of the actual resolution of the other databases, although the work emphasizes a lot in explaining it contains more info due to higher resolution. ISBI seems to be 4 nm x 4 nm x 50 nm / pixel. (from http://brainiac2.mit.edu/isbi_challenge/home), while the introduced one is 2.18 x 2.18 x 70 nm /pixel. Notice that ISBI resolution at z-axis actually is higher than the introduced. This should be made clear in text. Also, please add same information about SNEMI3D. Do you believe this difference in z-axis could make any difference with respect to what structures can be segmented? (In fact, in Sec 2.2, you say that thicker slice affect imaging quality.)\n\nThe current database has approximately 2x resolution than ISBI in x,y plane. What type of structures do you believe are not capable of segmenting well in 4x4nm resolution, but capable at 2x2nm resolution? To support the claim that the higher-resolution is a significant advantage (and hence the contribution of releasing such a database is strong), perhaps the work should have performed an evaluation of how useful this extra resolution is in practice, to support the main contirbution.\n\nPlease discuss in Sec 2.1 what anatomy are the images of each database coming from. As they are not coming from the same tissue (e.g. this is from retina, ISBI from Larvae cord etc), please discuss if you think this could be a factor for qualitative differences between the databases. If you think it may be, then perhaps claims about what database is more suitable should be adjusted, as perhaps the two have a bit different purpose / characteristics?\n \nSec. 2.1: “much more challenging”: What evidence is this claim based on? I could make the argument that ISBI may be more challenging to segment due to the lower resolution (hence less information). Please ensure that you back up all claims with appropriate arguments. As it currently stands, this is an unsupported claim and should be removed.\n\nSame comment as above for the “suitable in exploiting cell segmentation algorithms” claim. Why more suitable to exploit algorithms? I think this needs a rephrase.\n\nSec. 3.1: “may not be consistent with human perception… tasks”, “an natural first instinct was that”, and in Sec 3.3: “humans are more sensitive to structure changes, instead of thickness changes.”. I think these statements are not passing the correct meaning. It is not the “human perception” or the “instinct” of the humans that prioritizes thin (non-)existence of structures over thickness. In your experiments, the evaluators were clearly trained about what the task is. For example, perhaps they know that in cell-segmentation, where the ultimate goal is creating the connectomic, the connectomic can be created regardless the thickness, but a structure should not be missing. Hence, what wrong structures more important than thickness, is the task. Not human perception or instinct (in fact, for me, it’s clearly easier to identify thickness, than locating a small structure missing somewhere in the images). I think these statements pass a wrong meaning. I would suggest that they are rephrased, to emphasize that in every task, where the segmentation itself is not the ultimate goal, the quality of a segmentation should be judged with respect to what the actual ultimate goal is (e.g. here, creating the whole structure of how membranes are connected?). And this should be reflected in the evaluation metrics, where in each task, different metrics, appropriate for the specific one should be used. Please discuss your viewpoint and your recommended amendments.\n\nSame point as the above, in Sec 3.3: “ humans are more sensitive to structure changes, instead of thickness changes.”: I don’t think this statement is in general true. I would say the opposite for me. I can immediately tell that the thickness differs among segmentations, but I have to focus explicitly on certain areas to find whether a specific area has been wrong segmented. I expect the fact that the humans that performed the evaluation were specifically “trained” (Sec 3.2) that they perform *the specific task of cell segmentation* is likely what made them emphasize the actual structure and not care about the thickness. In other words, what criterion/metric is most appropriate has to do with the actual task of interest. Please discuss. I would suggest all related statements about human perception, vision or sensitivity, to be rephrased in a way that is less generic, and instead perhaps passes the message that in each task, quality of segmentation should be judged with respect to the actual ultimate goal.\n\n\nSec 3.2 does not describe on what data were the segmentation methods trained. Specifically, the paper should state explicitly if the training data were different images from those that were used to create the 200 groups of images that the 20 humans evaluated the results, or were they the same. Can you please clarify this here and in the text?\n\nSec 3.1: What is the difference between F1 score and Dice Coefficient? As far as I know (I double checked), these two scores seem the same to me. Phrasing in sec 3.1 suggests they are different. Am I wrong? Please clarify. If they are the same, then perhaps one of them should be removed.\n\nRelated to above: Figure4 c: I think that F1 score really is the same as Dice. After you double check, please check this figure. In Fig 4 c, the values of F1/Dice differ. How come? Please double check and clarify. Perhaps implementation detail? Or am I wrong? You see that in the end of Sec 3.2, F1-score and Dice also brought “Exactly” same results for correlation with human perception, agreeing with my view that F1-score/dice are the same. On the other hand, in Table 1, F1 score and Dice differs in *some* methods (humans, GLNet, Unet), but are absolutely the same for other methods (SENet, CASENet, Unet++, LinkNet). Please double check and clarify. I would suggest you check for a small implementation error? Sorry if I misunderstand something, I am happy to hear clarifications.\n\nSec. 3.3: I think there is no strong technical argument given for introducing tolerance? Without the tolerance, for small offsets/differences, the distance metric will simply have low value. Humans dont \"ignore it\", it's just small so it does not \"bother them enough to mention\". Which is exactly what a low value from a distance metric means. From Fig 5, we can see that even without tolerance (=0), the metrics (ASSD on skeleton) behaves perfectly fine, giving higher PHD for the case (b) that has larger distance than case (a). I would recommend adding such explanation and discussion in the paper. What is your view on the above?\n\nSec 3.3 & Fig 4: “suggesting human vision does have tolerance”: Sure, but this does not necessarily mean it’s the right thing, right? For example, factors for inducing human “tolerance’ can be limited vision capability (our eyes are not as good as a computer in processing pixel-by-pixel) or subjectivity with respect to the task (e.g. if the human annotators know, or they have been trained, that 1-3 pixels is not a important *for the particular task* of cell segmentation). But this does not mean that a metric that has no tolerance is a bad thing, perhaps the metric is even more objective. Please discuss.\n\n“Can F1-score… be improved… refutes this”: This statement is wrong. Skeletonizing *does* improve all these metrics, as shown in Fig 6. They simply don’t reach the result of PHD. Please rephrase.\n\nThe evaluation could/should have included other metrics, such as basic Haussdorf distance, ASSD, or metrics used in related challenges, such as Rand etc (see http://brainiac2.mit.edu/isbi_challenge/evaluation) \n\n###################\n\nMinors, or additional feedback for improving the work in the future (not subject to rebuttal):\n\nI would recommend rephrasing the phrase “Surprisingly, we found” in Sec.1, as it is actually a commonly discussed issue in the literature (see my previous comments on related work/references)\n\nSec.2: “provided by Marc’s lab (Anderson et al. (2011))”: I think this could be rephrased to a more canonical way of refering to a wold, and also be more accurately descriptive? One that clarifies whether the data are exactly those described in Anderson et al 2011? Were they made publically available together with the specific paper (Anderson et al)? Or have they been provided by Marc’s lab (author of the cited work) to you personally for this current work? E.g. a rephrase like “made publically available and described in the work of Anderson et al (2011)” or something like that is more descriptive.\n\n“at an x-y resolution of 2.18 nm/pixel”: is the resolution the same along the 2 axes? If so, clarify something like “2.18nm/pixel across both axes, and 70nm…”\n\n“from different layers”: What is a layer in this context? It has not been defined. Is it a slice? Remember that you are addressing this to a non-domain-specific audience of ICLR, so ensure to be clear about these terms.\n\nFig 1: “image number” => “number of images” reads better.\n\nSec. 3.2: “and 2 segmentation results” => 2 “automatically generated” segmentations?\n\nAbstract: “proposes” a dataset? Sounds wrong. I would suggest “introduces” a dataset.\n\nSec 1: “”how robust if these methods are compared”:  grammar\n\n\n**================== UPDATE / EDIT AFTER REBUTTAL PERIOD AND UPDATES TO PAPER ========**\n\nSummary of improvements during rebuttal and remaining concerns on the updated manuscript:\n\nMain improvements:\n\n- Improved the comparison with existing EM datasets by extending descriptions and text in Sec 2.1.\n\n- Added an experiment that shows that the same model (Unet) performs significantly better on previous databases than on the proposed (Appendix VIII, IX). This acts as a solid empirical evidence that the current database is indeed more challenging than previous ones, which was previously missing. This supports the value of the database.\n\n- The authors also clarified that they will release all 3 sets of annotations, which can serve various types of methodological developments, as databases like this are not common.\n\n- Added a short discussion of previous work on metrics for segmentation quality, which previously was entirely missing.\n\n- Extended the analysis by incorporating multiple more metrics (~10) that were previously missing, extending significantly over the first version, where only overlap-based metrics were considered (Dice, IoU). The new results support that for the specific problem of cell-segmentaiton, PHD agrees more with human perception than other metrics on this task, including the very related HD and ASSD.\n\n- Showed that the Skeletonization process, part of the proposed metric, improves various metrics (among which the very related HD-based metrics), which fulfills the previous gap of empirical evidence to support its incorporation in the proposed metric.\n\n- Rephrased most points where the text was ambiguous or incorrect.\n\n\nSummary and Reviewer’s Score adjustment:\n\nOverall, the revision has improved the document significantly. My primary remaining concerns have to do with the actual paper being of interest primarily to the audience interested in the specific task of cell-segmentation, and that the technical value of the PHD metric is relatively limited (as per my initial comments). However, the updated document supports much better the main claims of the paper, and this database could serve as a benchmark for general ML segmentation methods, benefitting the greater ML community. These improvements make me increase my score from a 3 (Clear Reject) to a 6 (Above acceptance threshold).\n\n-- new minor problems in the updated version --\n\nSome new minor points that I noticed. In case the work is accepted, please try to address them for camera ready:\n\n“ASSD… which is not widely used in deep learning researches”: Not a true statement. HD/ASSD/etc are very popular metrics in segmentation tasks (including with DL) and there have been efforts to even turn them into losses. Please rephrase/remove this statement.\n\n“the consistency of the F1 score, IoU and Dice with the human choices was calculated” (Sec. 3.2): Wasn’t this done for the other metrics as well now? Update the text.\n\nSec.3.2, “Six popular… segmentation results”: Refer readers to appendix for details on training/test config.1?\n\nSec. 4, “Then four evaluation…”: “four” is not correct after the updates.\n\nSec. 4, Discussion: “it can be seen… methods”: This no longer holds after the new metrics. Taking into account all metrics, if we naively count for how many metrics a method ranks 1st (which is what the paper did in the first place), then it seems the best is LinkNet, followed by U-net++, not CASENet (which only ranks 1 for PHD-1 and PHD-3).Please update the argument.\n\nAppendix V: “texture” => text\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DlINnUKTaJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a large high-resolution cell membrane segmentation dataset and also proposes a new evaluation metric that is more consistent with human perception.  The new metric is called Perceptual Hausdorff Distance (PHD), which first applies thinning to skeletonize the segmentation outcome, then computes the Hausdorff distance between skeletons. PHD has a hyper-parameter, i.e., the tolerance distance, to represent the human's tolerance.\n\n\nOverall, I think this is a good paper that addresses how to correctly evaluate cell membrane segmentation, which is essential for evaluation at a fair standard but has not been studied extensively. The authors provide strong reasons to illustrate the limitations of existing evaluation metrics,  and present a relatively larger scale high-quality dataset for evaluating different techniques.  The pixel number and the number of images are presented to demonstrate the advantages over ISBI 2012 and SNEMI3D. The image collection, annotation, and evaluation seem to be performed very carefully with 20 subjects involved.\n\n\nSome minor additional questions:\ni) The measurement seems to be only tailored for cell membrane segmentation. Is it possible to make the criterion more generalized?\n\nii) If I am understanding correctly, the measurement seems very hard to be modified into loss functions since it involves thinning and other heuristics. Is it possible to use PHD not only for evaluation purposes but also for improving standard training?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on a segmentation of cell imagery (as opposed to the more commonly studied domain of "natural images"). Among its contributions are a novel metric for evaluation of results and a novel dataset. These are acknowledged by the reviewers as strengths. Multiple issues raised in the initial reviews were addressed in the revision (the reviewers agree on this and most of them raised their scores). On the other hand, the concerns remaining have to do with significance and impact. The final evaluation ratings are split, with only a single score clearly in favor of acceptance. 

I tend to agree that the contributions, while without a doubt valuable, make this less of a fit to ICLR than to a more specialized venue focusing on biomedical data. 
Decision: Reject