Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a large high-resolution cell membrane segmentation dataset and also proposes a new evaluation metric that is more consistent with human perception.  The new metric is called Perceptual Hausdorff Distance (PHD), which first applies thinning to skeletonize the segmentation outcome, then computes the Hausdorff distance between skeletons. PHD has a hyper-parameter, i.e., the tolerance distance, to represent the human's tolerance.


Overall, I think this is a good paper that addresses how to correctly evaluate cell membrane segmentation, which is essential for evaluation at a fair standard but has not been studied extensively. The authors provide strong reasons to illustrate the limitations of existing evaluation metrics,  and present a relatively larger scale high-quality dataset for evaluating different techniques.  The pixel number and the number of images are presented to demonstrate the advantages over ISBI 2012 and SNEMI3D. The image collection, annotation, and evaluation seem to be performed very carefully with 20 subjects involved.


Some minor additional questions:
i) The measurement seems to be only tailored for cell membrane segmentation. Is it possible to make the criterion more generalized?

ii) If I am understanding correctly, the measurement seems very hard to be modified into loss functions since it involves thinning and other heuristics. Is it possible to use PHD not only for evaluation purposes but also for improving standard training?
review_title: Good paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This work presents two contributions towards cell membrane segmentation. First, it introduces a new labelled database for this purpose. The authors claim that this is the largest labelled database of high resolution Electron-Microscopy images for this purpose. Second, the work tackles the issue that the F1Score, Dice and IoU scores that evaluate segmentation performance by quantifying the overlap of 2 segmentations are not adequately describing quality of a segmentation with respect to what human experts would prefer for the task. This has been evaluated via employing humans (experts on the task of cell segmentation) to grade which segmentation they prefer, and analysed how their preference correlates with these scores. As a solution, the authors propose a metric, PHD, that can be described as the average Haussdorf distance between the skeletons of two segmentations, together with a threshold tolerance, which they show correlates better with the preference of experts with respect to segmentation quality on this task.

###################################
Reasons for score: 

I recommend a rejection of this work for the following reasons. On one hand, constructing a database and releasing it to the community is a great contribution. I am sure that this would be very well accepted. But, on the other hand, I don’t think the article adequately describes the dataset or compare it adequately with existing databases (which makes less of a “database article”). Instead, half the article discusses a metric that is essentially an adaptation of the haussdorf distance (actually, of the “average symmetric surface distance”), adapted in a manner specific to the cell-segmentation task (applied on skeleton, and with a tolerance, the importance of which is questionable). This 2nd contribution has not been accompanied by a literature review on metrics (e.g. only discusses IoU/F1/Dice, missing related distance based metrics like ASSD completely), nor adequately evaluated with such related metrics (besides IoU/Dice/F1). Finally, the modifications, along with many claims in the article, are only relevant to the specific task of cell segmentation (and quite subjective).

###################################

Pros: 
 
1. Great contribution by releasing publicly a new labelled database of high quality. Seems there was a lot of effort to construct good quality ground truth on a number of images much larger than the existing publically available databases. This is definitely interesting for the community that works in this problem.

2. Interesting human-based evaluation of the usefulness of IoU/Dice/F1 scores for the cell-segmentation problem (by 20 humans, this is nice). It is known in the broader community that overlap metrics (IoU/Dice etc) are not perfect, hence there is a lot of work on other metrics [1,2 etc], but this substantiates/quantifies it very nicely.

###################################

Cons:  

1. If I would judge the paper focused on the 1st contribution (releasing a database), I would say that it does not contain a sufficient analysis of the database itself, and especially not an adequate description and comparison with other public databases. I think this point could be sufficiently addressed in the rebuttal.
 
2. From the technical viewpoint, the claimed 2nd contribution is the derivation of a new metric, but the work has not performed any literature review on related work on metrics except IoU/Dice/F1. In fact, the work produces a metric (Eq.2) that seems to me the same as Average Assymetric Surface Distance (ASSD, see [1]), applied to the skeleton (thinned) segmentation, with a tolerance (task-specific modifications). I note that both skeleton-like operations have been previously performed for computing metrics in the cell-segmentation domain (e.g. for evaluation of ISBI2012 challenge: http://brainiac2.mit.edu/isbi_challenge/evaluation, notice the “after thinning” operation). Tolerance-based modifications have also been applied to various other metrics (e.g. [2] below) and are task-specific modifications (and not necessary for the metric to be appropriate in the general sense). In my opinion, this makes the value of the 2nd contribution very low. I think this point cannot be sufficiently addressed in the rebuttal, as I basically think that the contribution of the derived metric is low in comparison to existing literature.

3. The “skeleton” part of PHD is not individually evaluated whether it actually adds substantially. I note that without it, the metric is essentially ASSD with tolerance (tolerance being task-specifically motivated in this context).

4. Evaluation of the proposed metrics is limited, because it does not contain other metrics except IoU/Dice. E.g., it should have been compared with Haussdord, ASSD, etc.

5. The scope of the paper is limited to the cell-segmentation problem.

6. The work contains a number of statements that are not true and would need significant text alterations to reduce them.

The above Cons are described in detail below, with the detailed comments I raise in the “Questions for rebuttal period” section.

References:

Paper with some review on related metrics (there are many more such reviews):
[1] Yeghiazaryan and Voiculescu, Family of boundary overlap metrics for the evaluation of medical image segmentation, Journal of Medical Imaging, 2018

Paper implementing tolerance (which motivates it, and is subjective to the specific task and needs):
[2] Nikolov et al, Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy, arxiv 2018


###################################

Questions/points to address during rebuttal period: 

The abstract “proposes” a database with “multiple iterative annotations”. However, the actual database released only contains the last 3rd annotation. Please rephrase the abstract so that this is clear to the reader that only 1 annotation should be expected. 

Similar to the above, Sec 2.1 claims the existence of multiple (3) annotations as an advantage of the proposed database over other databases (Sec.2.1, “Besides that, U-RICS produced 3 sets of annotations…, all of which can be applied in developing depe learning”). But, these 3 annotations are not released, hence their existence is irrelevant to the reader and this claim/advantage simply does not hold.

The authors claim that the intermediate annotations are “are very valuable for learning” (Sec.2) and therefore will not be released. In this case, if they are valueable, perhaps consider releasing them? Otherwise, I would suggest rephrasing the article, reducing the emphasis on these 3 labels across the whole manuscript, as they are of little relevance to the reader. You can spend the saved space to extend on more related points (e.g. related work etc, see below).

Sec 1. claims: “We found the human performance is far superior to these methods”. I think this statement is very strong and not supported adequately by the current evaluation. I think the authors refer to experiments in Table 1., where the labels from the 1st and 2nd iteration where compared with the results in the 3rd iteration. But, naturally, image after 3rd iteration is conditioned (related) very closely to those from the 1st and 2nd iterations. Of course they will have very high agreement. Also, we note, the 2nd iteration is not a result of 1 human but of multiple (5 experts + annotator). Hence, for these 2 reasons, they cannot support the statement. For a correct assessment for such a claim, segmentations from a single annotator, who is not the same contributing to making the ground truth for the image, should be evaluated against the ground truth. Additionally, inter-rate segmentation performance (multiple humans) could also be evaluated to make such claims. I would recommend this claim to be altered, as well as state explicitly these factors (conditioning of 1st, 2nd, 3rd iterations) in sec 4 to explain how come agreement is so high in Table 1. 


The work has performed no literature review on related metrics for segmentation. It discusses and evaluates solely overlap based metrics (Dice/F1 & IoU), and then proposes a distance-based measure. The authors should have discussed and evaluated distance based measures too, and especially have a look at the Average Symmetric Surface Distance (ASSD) metric, which is very similar to what they propose (Eq.2). See [1] as a starting point for related metrics, but there are many more. Many of these papers raise the argument that no metric is enough for all tasks, and that “quality” of a segmentation is subjective to the task. Hence, for each task, one should chose the correct metric, while for “objective” and all-around evaluation of a “general” method (e.g. an arbitrary segmentation network), multiple complimentary metrics should be used. See [2] also with related discussion. Also see that ISBI2012 challenge itself implemented multiple metrics (http://brainiac2.mit.edu/isbi_challenge/evaluation) that are not duscussed here (and their paper also discusses appropriateness of metrics).

Sec.1 “Considering that image compression generally loses many texture details”: Not necessarily. Depends on how much compression, what type of compression, and the actual content. If you would like to claim this, I would suggest you analyse what type of structures disappear if you do a 2x sub-sampling (based on the fact that ISBI has 2x less resolution). Otherwise, I would suggest this is rephrased a bit less strong.


Sec.2.1 intends to do a comparison with ISBI database and SNEMI3D databases. However it still does not provide important information about them. For example, there is no mention of the actual resolution of the other databases, although the work emphasizes a lot in explaining it contains more info due to higher resolution. ISBI seems to be 4 nm x 4 nm x 50 nm / pixel. (from http://brainiac2.mit.edu/isbi_challenge/home), while the introduced one is 2.18 x 2.18 x 70 nm /pixel. Notice that ISBI resolution at z-axis actually is higher than the introduced. This should be made clear in text. Also, please add same information about SNEMI3D. Do you believe this difference in z-axis could make any difference with respect to what structures can be segmented? (In fact, in Sec 2.2, you say that thicker slice affect imaging quality.)

The current database has approximately 2x resolution than ISBI in x,y plane. What type of structures do you believe are not capable of segmenting well in 4x4nm resolution, but capable at 2x2nm resolution? To support the claim that the higher-resolution is a significant advantage (and hence the contribution of releasing such a database is strong), perhaps the work should have performed an evaluation of how useful this extra resolution is in practice, to support the main contirbution.

Please discuss in Sec 2.1 what anatomy are the images of each database coming from. As they are not coming from the same tissue (e.g. this is from retina, ISBI from Larvae cord etc), please discuss if you think this could be a factor for qualitative differences between the databases. If you think it may be, then perhaps claims about what database is more suitable should be adjusted, as perhaps the two have a bit different purpose / characteristics?
 
Sec. 2.1: “much more challenging”: What evidence is this claim based on? I could make the argument that ISBI may be more challenging to segment due to the lower resolution (hence less information). Please ensure that you back up all claims with appropriate arguments. As it currently stands, this is an unsupported claim and should be removed.

Same comment as above for the “suitable in exploiting cell segmentation algorithms” claim. Why more suitable to exploit algorithms? I think this needs a rephrase.

Sec. 3.1: “may not be consistent with human perception… tasks”, “an natural first instinct was that”, and in Sec 3.3: “humans are more sensitive to structure changes, instead of thickness changes.”. I think these statements are not passing the correct meaning. It is not the “human perception” or the “instinct” of the humans that prioritizes thin (non-)existence of structures over thickness. In your experiments, the evaluators were clearly trained about what the task is. For example, perhaps they know that in cell-segmentation, where the ultimate goal is creating the connectomic, the connectomic can be created regardless the thickness, but a structure should not be missing. Hence, what wrong structures more important than thickness, is the task. Not human perception or instinct (in fact, for me, it’s clearly easier to identify thickness, than locating a small structure missing somewhere in the images). I think these statements pass a wrong meaning. I would suggest that they are rephrased, to emphasize that in every task, where the segmentation itself is not the ultimate goal, the quality of a segmentation should be judged with respect to what the actual ultimate goal is (e.g. here, creating the whole structure of how membranes are connected?). And this should be reflected in the evaluation metrics, where in each task, different metrics, appropriate for the specific one should be used. Please discuss your viewpoint and your recommended amendments.

Same point as the above, in Sec 3.3: “ humans are more sensitive to structure changes, instead of thickness changes.”: I don’t think this statement is in general true. I would say the opposite for me. I can immediately tell that the thickness differs among segmentations, but I have to focus explicitly on certain areas to find whether a specific area has been wrong segmented. I expect the fact that the humans that performed the evaluation were specifically “trained” (Sec 3.2) that they perform *the specific task of cell segmentation* is likely what made them emphasize the actual structure and not care about the thickness. In other words, what criterion/metric is most appropriate has to do with the actual task of interest. Please discuss. I would suggest all related statements about human perception, vision or sensitivity, to be rephrased in a way that is less generic, and instead perhaps passes the message that in each task, quality of segmentation should be judged with respect to the actual ultimate goal.


Sec 3.2 does not describe on what data were the segmentation methods trained. Specifically, the paper should state explicitly if the training data were different images from those that were used to create the 200 groups of images that the 20 humans evaluated the results, or were they the same. Can you please clarify this here and in the text?

Sec 3.1: What is the difference between F1 score and Dice Coefficient? As far as I know (I double checked), these two scores seem the same to me. Phrasing in sec 3.1 suggests they are different. Am I wrong? Please clarify. If they are the same, then perhaps one of them should be removed.

Related to above: Figure4 c: I think that F1 score really is the same as Dice. After you double check, please check this figure. In Fig 4 c, the values of F1/Dice differ. How come? Please double check and clarify. Perhaps implementation detail? Or am I wrong? You see that in the end of Sec 3.2, F1-score and Dice also brought “Exactly” same results for correlation with human perception, agreeing with my view that F1-score/dice are the same. On the other hand, in Table 1, F1 score and Dice differs in *some* methods (humans, GLNet, Unet), but are absolutely the same for other methods (SENet, CASENet, Unet++, LinkNet). Please double check and clarify. I would suggest you check for a small implementation error? Sorry if I misunderstand something, I am happy to hear clarifications.

Sec. 3.3: I think there is no strong technical argument given for introducing tolerance? Without the tolerance, for small offsets/differences, the distance metric will simply have low value. Humans dont "ignore it", it's just small so it does not "bother them enough to mention". Which is exactly what a low value from a distance metric means. From Fig 5, we can see that even without tolerance (=0), the metrics (ASSD on skeleton) behaves perfectly fine, giving higher PHD for the case (b) that has larger distance than case (a). I would recommend adding such explanation and discussion in the paper. What is your view on the above?

Sec 3.3 & Fig 4: “suggesting human vision does have tolerance”: Sure, but this does not necessarily mean it’s the right thing, right? For example, factors for inducing human “tolerance’ can be limited vision capability (our eyes are not as good as a computer in processing pixel-by-pixel) or subjectivity with respect to the task (e.g. if the human annotators know, or they have been trained, that 1-3 pixels is not a important *for the particular task* of cell segmentation). But this does not mean that a metric that has no tolerance is a bad thing, perhaps the metric is even more objective. Please discuss.

“Can F1-score… be improved… refutes this”: This statement is wrong. Skeletonizing *does* improve all these metrics, as shown in Fig 6. They simply don’t reach the result of PHD. Please rephrase.

The evaluation could/should have included other metrics, such as basic Haussdorf distance, ASSD, or metrics used in related challenges, such as Rand etc (see http://brainiac2.mit.edu/isbi_challenge/evaluation) 

###################

Minors, or additional feedback for improving the work in the future (not subject to rebuttal):

I would recommend rephrasing the phrase “Surprisingly, we found” in Sec.1, as it is actually a commonly discussed issue in the literature (see my previous comments on related work/references)

Sec.2: “provided by Marc’s lab (Anderson et al. (2011))”: I think this could be rephrased to a more canonical way of refering to a wold, and also be more accurately descriptive? One that clarifies whether the data are exactly those described in Anderson et al 2011? Were they made publically available together with the specific paper (Anderson et al)? Or have they been provided by Marc’s lab (author of the cited work) to you personally for this current work? E.g. a rephrase like “made publically available and described in the work of Anderson et al (2011)” or something like that is more descriptive.

“at an x-y resolution of 2.18 nm/pixel”: is the resolution the same along the 2 axes? If so, clarify something like “2.18nm/pixel across both axes, and 70nm…”

“from different layers”: What is a layer in this context? It has not been defined. Is it a slice? Remember that you are addressing this to a non-domain-specific audience of ICLR, so ensure to be clear about these terms.

Fig 1: “image number” => “number of images” reads better.

Sec. 3.2: “and 2 segmentation results” => 2 “automatically generated” segmentations?

Abstract: “proposes” a dataset? Sounds wrong. I would suggest “introduces” a dataset.

Sec 1: “”how robust if these methods are compared”:  grammar


**================== UPDATE / EDIT AFTER REBUTTAL PERIOD AND UPDATES TO PAPER ========**

Summary of improvements during rebuttal and remaining concerns on the updated manuscript:

Main improvements:

- Improved the comparison with existing EM datasets by extending descriptions and text in Sec 2.1.

- Added an experiment that shows that the same model (Unet) performs significantly better on previous databases than on the proposed (Appendix VIII, IX). This acts as a solid empirical evidence that the current database is indeed more challenging than previous ones, which was previously missing. This supports the value of the database.

- The authors also clarified that they will release all 3 sets of annotations, which can serve various types of methodological developments, as databases like this are not common.

- Added a short discussion of previous work on metrics for segmentation quality, which previously was entirely missing.

- Extended the analysis by incorporating multiple more metrics (~10) that were previously missing, extending significantly over the first version, where only overlap-based metrics were considered (Dice, IoU). The new results support that for the specific problem of cell-segmentaiton, PHD agrees more with human perception than other metrics on this task, including the very related HD and ASSD.

- Showed that the Skeletonization process, part of the proposed metric, improves various metrics (among which the very related HD-based metrics), which fulfills the previous gap of empirical evidence to support its incorporation in the proposed metric.

- Rephrased most points where the text was ambiguous or incorrect.


Summary and Reviewer’s Score adjustment:

Overall, the revision has improved the document significantly. My primary remaining concerns have to do with the actual paper being of interest primarily to the audience interested in the specific task of cell-segmentation, and that the technical value of the PHD metric is relatively limited (as per my initial comments). However, the updated document supports much better the main claims of the paper, and this database could serve as a benchmark for general ML segmentation methods, benefitting the greater ML community. These improvements make me increase my score from a 3 (Clear Reject) to a 6 (Above acceptance threshold).

-- new minor problems in the updated version --

Some new minor points that I noticed. In case the work is accepted, please try to address them for camera ready:

“ASSD… which is not widely used in deep learning researches”: Not a true statement. HD/ASSD/etc are very popular metrics in segmentation tasks (including with DL) and there have been efforts to even turn them into losses. Please rephrase/remove this statement.

“the consistency of the F1 score, IoU and Dice with the human choices was calculated” (Sec. 3.2): Wasn’t this done for the other metrics as well now? Update the text.

Sec.3.2, “Six popular… segmentation results”: Refer readers to appendix for details on training/test config.1?

Sec. 4, “Then four evaluation…”: “four” is not correct after the updates.

Sec. 4, Discussion: “it can be seen… methods”: This no longer holds after the new metrics. Taking into account all metrics, if we naively count for how many metrics a method ranks 1st (which is what the paper did in the first place), then it seems the best is LinkNet, followed by U-net++, not CASENet (which only ranks 1 for PHD-1 and PHD-3).Please update the argument.

Appendix V: “texture” => text

review_title: Review - Interesting database but has issues

final_score: rating: 3: Clear rejection
review_content: # pros:

- To the author's and reviewer's best knowledge, this paper includes the largest annotated public EM data set for cell membrane segmentation (in case it is published with this paper). Until now, the ISBI 2012 challenge (http://brainiac2.mit.edu/isbi_challenge/) dominates the evaluation of cell membrane segmentation in EM data, even though the performance is nearly saturated. New datasets can identify potential weaknesses in similar domains, that are not covered in current datasets and by state of the art methods, yet.

- The discussion about suitable segmentation metrics for cell membrane segmentation is important and must be continued.

- The article is written in a clear and comprehensive manner.


# cons:

- The discussion about appropriate metrics for cell segmentation, that do not depend on the thickness of the segmented cell membrane, has extensively been elaborated in "Crowdsourcing the creation of image segmentation algorithms for connectomics" by Ignacio Arganda-Carreras et al., Frontiers in Neuroanatomy 2015 (9) 142: pp. 1-13. However, this paper is not referenced and the therein proposed metrics are not mentioned or compared.

- The evaluated "state-of-the-art" methods are not "state-of-the-art". They do not correspond to the top entries of the current ISBI Segmentation Challenge Leaderboard. Additionally, no parameters of the methods were adapted.

- It is left unclear how the 20 human raters were instructed to evaluate the segmentation results. For the correct evaluation, not (only) intuitive human perception must be taken into account, but also the usability of the resulting segmentation. The segmentation results on high resolution EM data presented in this paper display many "unclosed" edges, which lead to severe problems, when using the segmentation as a basis for connectivity analysis. To the reviewer's understanding, the proposed Perceptual Hausdorff distance will hardly penalize these errors.

- The dataset is not published in the format of a challenge, which would allow benchmarking on a private test set.

- Spelling should be revised.


# Summary

The presented new high-quality dataset is highly valuabl to the community in order to improve and develop methods for instance segmentation, specifically cell membrane segmentation. The segmentation of thin cell boundaries imposes different challenges and includes different priors, than in other domains of instance segmentation.

The use of appropriate evaluation metrics is crucial to identify suitable und successful methods in experiments and must be critically discussed including domain knowledge. However, in the presented paper, the discussion about suitable metrics is not appropriately linked to the existing literature. A metric is proposed, that is (more) consistent with "human perception". This is an interesting aspect, but its contribution to the successful analysis of neuronal connectivity from EM data remains unclear.

review_title: The proposed new public annotated dataset on EM cell membrane segmentation is highly valuable to the community, yet the value of the proposed metric and the presented experiments is questionable

final_score: rating: 4: Ok but not good enough - rejection
review_content: Strength: (1) This work proposed an ultra-high-resolution image segmentation dataset for the cell membrane, named U-RISC. The proposed U-RISC is the largest annotated Electron Microscopy (EM) dataset for the cell membrane with multiple iterative annotations and uncompressed high-resolution raw data. Given the uniqueness of the proposed dataset, it is likely to contribute to the future EM based research, such as membrane segmentation. 
(2) For membrane segmentation in EM images, the authors develop a human-perception based evaluation criterion, called Perceptual Hausdorff Distance (PHD). Based on the experiments on a small-scale dataset. the proposed PHD metric is better consistency with human perception than the traditional ones.
 
Weakness: (1) The first concern is on the proposed PHD metrics. The reviewer thinks that there is a lack of comparison analysis between the proposed PHD with the traditional Hausdorff distance. According to Equation 2, the PHD metric is build based on the Hausdorff distance. Therefore, it is necessary to include comparison analysis of using the traditional Hausdorff distance. This will further highlight the novelty of the PHD metric proposed in this paper. 
(2) The second concern is the limit technique novelty. The overall contributions of this paper have two parts: establishing a new dataset, and proposing a new PHD metric for the EM membrane segmentation tasks. However, there is no contributions based on the machine learning or deep learning based methods. The reviewer agrees that this manuscript has made some contributions on biomedical image analysis. However, the reviewer thinks this paper cannot meet the requirement of the ICLR conference. 
(3) The third concern is the limit validation experiments on the PHD metric. Since the proposed PHD metric is particularly for membrane segmentation, it is supposed to be effective on other EM datasets, such as the ISBI2012 and SNEMI3D challenges. However, the authors did not conduct comparison methods on any other EM datasets. It would be more convincing to conduct experimental analysis on various EM membrane segmentation tasks. 

review_title: This paper conducted studies on Electron Microscopy image segmentation. The authors built an EM dataset U-RISC with the original high resolution, which is the currently largest annotated EM dataset, and found the current evaluation metrics for membrane segmentation are inconsistent with human perception. The authors therefore proposed a human-perception-based metric, called Perceptual Hausdorff Distance (PHD), which better follows the manual annotation choices than the traditional metrics.

Meta-review: This paper focuses on a segmentation of cell imagery (as opposed to the more commonly studied domain of "natural images"). Among its contributions are a novel metric for evaluation of results and a novel dataset. These are acknowledged by the reviewers as strengths. Multiple issues raised in the initial reviews were addressed in the revision (the reviewers agree on this and most of them raised their scores). On the other hand, the concerns remaining have to do with significance and impact. The final evaluation ratings are split, with only a single score clearly in favor of acceptance. 

I tend to agree that the contributions, while without a doubt valuable, make this less of a fit to ICLR than to a more specialized venue focusing on biomedical data. 
Decision: Reject