Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new method for segmenting images with better topological accuracy. Specifically, a new algorithm utilizing distance transform is used to identify the topologically critical locations, based on which a new homotopy warping loss is used to measure the difference between the predicted and ground-truth topology and train the segmentation network. Experiments on four 2D datasets and one 3D dataset validated the efficacy of the proposed method. Besides pixel-wise accuracy, better topology accuracy is obtained.
strengths_and_weaknesses: Strengths
1. This paper introduced an efficient heuristic algorithm to identify topologically critical locations. Compared with a previous work on topology-preserving image segmentation method which runs in cubic time complexity, the proposed algorithm's complexity is linear to image size, thus making the training more efficient.

2. A new homotopy warping loss is proposed to train the segmentation network to preserve the topology. The homotopy warping loss penalizes errors on topologically critical locations and results in models with better topological accuracies.

3. Comparisons with existing topology-preserving segmentation methods show that the proposed method achieves the best performance. Ablation studies provide some guidance regarding the choices of loss weights and loss functions.

Weaknesses
1. The proposed distance-ordered homotopy warping algorithm is possible to miss some simple pixels during the flipping although very rare.

2. In the previous work TopoNet, Variation of Information (VOI) is used as a metric to measure topological correctness. It would be better if VOI is also compared in the experiments.

3. Grammar/spelling errors:
a) Line 92: "Except for the methods mentioned above, UNet has also been one of the most popular methods for image segmentation". It seems "besides" is the proper preposition, not "except for".
b) Line 131: "efficiently identity" -> "efficiently identify"
c) Line 382: "we'd like the investigate" -> "we investigate"

questions: Please see the comments above
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new method to improve the topology of image segmentation.  The method finds topologically critical locations by comparing the predicted segmentation mask with the ground truth, and proposes a novel homotopy warping loss which guides the training to improve topological accuracy. Experiments are conducted in both 2D and 3D settings to segment satellite and biomedical images, and results show the proposed method outperforms other existing methods.
strengths_and_weaknesses: Strengths:

1.	Although previous work such as TopoNet also emphasize the importance of focussing on topologically critical locations, the warping approach taken in this paper appears to be novel and more efficient, computationally.

2.	The warping loss is intuitive and seeks to transform the topology of the predicted segmentation mask to match the ground truth (and vice versa).  The proposed algorithm focusses on simple points, which are described clearly in terms of their connectivity. 

3.	The experimental results demonstrate the method is capable of improving segmentation not just at the critical points but overall in the image.

Weaknesses:

1.	The segmentation performance obtained from this method mainly relies on the standard UNet results. If the segmentation output from UNet is noisy, the warping operation may take many iterations, perhaps more so initially as the network is starting to train.  Experiments with other architectures is left for future work.  

2.	While results are improved, the improvement may be subtle; for example 1% improvement in Dice score for most experiments in Table 1.  Nonetheless, visual results in Figure 6 show good improvement.

3.	It wasn’t clear to this reviewer if or how the method generalises to problems where there are more segmentation classes (e.g. three or more).  All the experiments shown in the paper are binary segmentation problems.  However, the paper makes it clear that it is focussing on binary segmentation.

questions: 1.	All the used cases in the paper address the segmentation error in between the segmented areas. What if some False Positives or False Negatives exist at the edges of segmentation regions? 

2.	The datasets used in the proposed study have straight or curved lines as an area of interest. Would the method be useful for datasets with more irregular shapes as a region of interest—for example, the Kvasir SEG - Simula dataset?  https://arxiv.org/abs/1911.07069 

3.	How might it be possible to apply the method for image segmentation problems with more than two classes (e.g. a problem with two foreground classes and one background)?

4.	What are the limitations of the proposed method?

limitations: Limitations are not discussed in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors implement the Warping Error metric, introduced by Jain et al. (CVPR 2010) as a loss for topology aware segmentation. To be able to efficiently calculate the warping, the authors introduce a heuristic algorithm based on the distance transform to choose the pixels to "flip" or warp to achieve a better topology. 
strengths_and_weaknesses: 
**Strengths:** 

Overall I find the proposed warping idea to be innovative and clever. It improves topology awareness in segmentation models in an efficient manner. It is significantly faster to compute than methods based on barcode-matching and persistent homology and achieves competitive performance to such methods. 


**Weaknesses:**

1) Terminology: Introduction l. 24-26 "pixels near the peripheral of the object of interest can generally be challenging, but not relevant to topology." I think this statement is problematic. When considering the inverse e.g. in the case of a surface or vessel, that a foreground pixel changes to background. Such a scenario would immediately lead to a topology mismatch (Betti error 1). 

2) Terminology: "topologically critical location" -->  I find this terminology to be not optimally chosen. I agree that the warping concept appears to help with identifying pixels which may close loops or fill holes. However, considering the warping I do not see a guarantee that such "locations" (as in the exact location) which I understand to refer to individual or groups of pixels are indeed part of the real foreground, nor are these locations unique. A slightly varying warping may propose a set of different pixels. 

3) The identified locations are more likely to be relevant to topological errors. --> this statement should be statistically supported. Compared to what exactly? Does this rely on the point estimate for any pixel? Or given a particularly trained network? 

4) Theorem 1: The presentation of a well known definition from Kong et al. is trivial and could be presented in a different way. 

5) Experimentation, lack of implementation details: In Table 2 and a dedicated section, the authors show an ablation study on the influence of lamda on the results. Lamda is a linear parameter, weighting the contribution of the new loss to the overall loss. Similarly, the studied baseline methods, e.g. TopoNet [24], DMT [25], and clDice [42] have a loss weighting parameter. It would be important to understand how and if the parameters of the baselines were chosen and experimented with. (I understand that the authors cannot train ablation studies for all baselines etc.) However, it is an important information to understand the results in Table 1.

6) Terminology: l. 34 "to force the neural network to memorize them" --> I would tone down this statement, in my understanding, the neural network does not memorize an exact "critical point" as such in TopoNet [24]. 



**Minor:** 

- I find the method section to be a bit wordy, it could be compressed on the essential definitions.

- There exist several grammatical errors, please double-check these with a focus on plurals and articles. E.g. l. 271 "This lemma is naturally generalized to 3D case."

- l. 52 "language of topology" I find this to be an imprecise definition or formulation. 

**Note:** 

After rebuttal and discussion I increased the rating to 5. 
questions: How does the filtering using the DT compare to the calculation of image persistence computationally? 
limitations: yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A topology-aware learning objective is proposed for semantic segmentation models. This objective relies upon a warping operation that transforms a predicted mask to one that is close to the ground truth mask in pixel space without changing the topology, and vice versa. The objective is a pixelwise cross-entropy computed on the points of disagreement between the target and warped source masks. Evaluation on 2D and 3D satellite and medical imagery segmentation datasets shows the algorithm is efficient and delivers strong results.
strengths_and_weaknesses: Strengths:
- Simple, intuitive algorithm and clear writing. The algorithm is well-motivated and mathematically sound, and care is taken to make it efficient using the heuristic for computing the points of interest.
- Strong experimental evaluation on diverse datasets where preservation of connectivity structure has been an important challenge.
- Ablations and comparisons are appropriately selected.

Weaknesses:
- The paper would benefit from more discussion about what may be responsible for the better performance of the algorithm relative to baseline methods of selecting critical points. The comparison with TopoNet's points of interest in Figure 7 gives some intuition about the question, but there are no comparisons of outputs of models trained using different critical point selection methods that would show the impact of a noisy point selection on the model output.
- Similarly, I would have liked to see some examples of the failure modes of the proposed algorithm, as well as images of outputs on all the datasets studied.
questions: - For what other kinds of data could this approach be useful? How could the proposed method be generalized to multiclass problems?
- Importance of the distance sorting heuristic: should drops in performance (accuracy or computation time) be expected if a simpler method of warping the masks is used, such as iteratively selecting a random non-simple point that borders the current mask? As another, simpler ablation study, I would suggest to evaluate warping only in one direction (ground truth to prediction or prediction to ground truth).
- L185: This line is somewhat confusing or misleading, since the proposed heuristic only finds a mask that is locally optimal, i.e., no simple points can be flipped to decrease the Hamming distance. (Is it known whether finding the global optimum simply slow (cf. L239-242) or computationally intractable?)
- L219: What would happen if each pixel's on/off state is sampled from the output distribution instead of bring thresholded?
limitations: Please see the first two points of Weaknesses above.
review_title: null

Meta-review: metareview: The paper proposes a topology-aware learning objective for semantic segmentation models based upon warping masks. The loss is used for training satellite data and medical segmentation datasets and provides benefits in these domains. Reviewers acknowledge that the approach is simple, intuitive, has thorough empirical evaluation and ablations. Reviewers note a few presentation issues, e.g. related to terminology. These must be fixed in a final revision of the paper. Overall all reviewers vote for acceptance and so do I.
confidence: Certain
Decision: Accept