Structure-Aware Image Segmentation with Homotopy Warping
Authors: Xiaoling Hu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Structure-Aware
Image Segmentation
Homotopy

Abstract: Besides per-pixel accuracy, topological correctness is also crucial for the segmentation of images with fine-scale structures, e.g., satellite images and biomedical images. In this paper, by leveraging the theory of digital topology, we identify pixels in an image that are critical for topology. By focusing on these critical pixels, we propose a new \textbf{homotopy warping loss} to train deep image segmentation networks for better topological accuracy. To efficiently identify these topologically critical pixels, we propose a new algorithm exploiting the distance transform. The proposed algorithm, as well as the loss function, naturally generalize to different topological structures in both 2D and 3D settings. The proposed loss function helps deep nets achieve better performance in terms of topology-aware metrics, outperforming state-of-the-art structure/topology-aware segmentation methods. 

Structure-Aware Image Segmentation with Homotopy Warping
Xiaoling Hu
Department of Computer Science Stony Brook University xiaolhuêcs cawolluuesssttoybrrooooee edu
Abstract
Besides per-pixel accuracy, topological correctness is also crucial for the segmen- tation of images with fine-scale structures, e.g., satellite images and biomedical images. In this paper, by leveraging the theory of digital topology, pixels in an image that are critical for topology. By focusing on these critical pix- els, we propose new homotopy warping loss to train deep image segmentation networks for better topological accuracy. To efficiently identify these topologically critical pixels, we propose a new algorithm exploiting the distance transform. The proposed algorithm, as well as the loss function, naturally generalize to different topological structures in both 2D and 3D settings. The proposed loss function helps deep nets achieve better performance in terms of topology-awan metrics, outperforming state-of-the-art structure/topology-awee aware segmentation methods.
identify
Introduction
Image segmentation with topological correctness is challenging problem, especially for images with fine-scale structures, c.g., satellite images, neuron images and vessel images. Deep learning methods have delivered strong performance in image segmentation task 33
However, even with satisfying per-pixel accuracy, most existing methods are still prone to topological errors, ie., broken connections, holes in 2D membranes, missing connected components, etc. These errors may significantly impact downstream tasks. For example, the reconstructed road maps from satellite images can be used for navigation broken connections, causing incorrect navigation route. See Fig. [ for an illustration. In neuron reconstruction [18 49 54 53), incorrect topology of the neuron membrane will result in erroneous merge or split of neurons, and thus eπrrrs in morphology and connectivity analysis of neuronal circuits.
22
7
g
3 5 A small amount of pixel errors will result in
Topological emrrr usually happen at challenging locations, c.g., weak connections or blurred locations. But not all challenging locations are topologically relevant; for example, pixels at the boundary of the object of interest generally be challenging, but not relevant topology. To truly suppress topological errors, we need to focus on topologically critical pixels, i.e., challenging locations that topologically relevant. Without identifying and targeting these locations, neural networks that optimized for standard pixel-wise losses (e.g., cross-entropy loss or mean-square-error loss) cannot avoid topological errors, even we increase the training size.
Existing works have targeted these topologically critical pixels. The closest method to our work is TopoNet (24). which is based persistent homology [15] [1].. The main idea is to identify topologically critical pixels corresponding to critical points of the likelihood map predicted by the neural network. The selected critical points reweighed in the training loss force the neural network to focus on them, and thus to avoid topological errors. But there are two main issues with this approach: 1) the method is based the likelihood map, which can be noisy with large amount
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

of irrelevant critical points. This leads to inefficient optimization during training. 2) The computation for persistent homology cubic the image size. is too expensive recompute at every iteration.
In this paper, we propose a novel ap- proach identify topologically criti- cal pixels in a more efficient and ac- curate manner. These locations are penalized in the proposed homotopy warping loss achieve better topolog- ical accuracy. Our method is partially inspired by the warping error previ- ously proposed to evaluate the topo- logical accuracy (26). Given a binary predicted mask s' and ground truth mask 9. we "warp" one towards an- other without changing its topology. From the view topology, to warp mask fn towards 9. we find mask fi that is homotopy equivalent to fB and is close to 9 possible [21). The difference between the warped mask f& and g constitutes the topo- logically critical pixels. We can also warp towards fa and find another set of topologically critical pixels. See Fig. 4 and Fig. | for illustrations. These locations directly correspond topological difference between the prediction and the ground truth, tolerating geometric deformations. Our homotopy warping loss targets them to fix topological errors of the model.
B
B
(a) Image
(b) GT mask
(c) UNet pred.
Figure 1: An illustration for the importance of topological correctness. If one wants to go to point B from A, the shortest path in the GT is illustrated in (b) (the green path). However, in the result predicted by UNet, though only a few pixels are misclassified, the shortest path from A to B is totally different, which is illustrated by the green path in (c). Zoom-in for better viewing.
The warping of a mask is achieved by iteratively flipping labels at pixels without changing the topology of the mask. These flippable pixels are called simple points/pixels in the classic theory of digital topology [30]. Note that in this paper, we focus on the topology of binary masks, simple points and simple pixels can be used interchangeably. To find the optimal warping of mask towards another mask is challenging due to the huge search space. To this end, we propose a new heuristic method that is computationally efficient. We filter the image domain with the distance transform and flip simple pixels based on their distance from the mask. This algorithm is proven efficient and delivers high quality locally optimal warping results.
Overall, our contributions can be summarized as follows:
We propose a novel homotopy warping loss, which penalizes errors on topologically crit- ical pixels. These locations are defined by homotopic warping of predicted and ground truth masks. The loss can be incorporated into the training of appoloyy-rreeeving deep segmentation networks.
By exploiting distance transforms of binary masks, we propose a novel homotopic warping algorithm identify topologically critical pixels in an efficient manner. This is essential in incorporating the homotopy warping loss into the training of deep nets.
Our loss is plug-and-play loss function. It can be used to train any segmentation network to achieve better performance in terms of topology. We conduct experiments both 2D and 3D benchmarks to demonstrate the efficacy of the proposed method. Our method performs strongly in multiple topology-relevant metrics (c.g., ARI, Warping Error and Betti Error). We also conduct several ablation studies to further demonstrate the efficiency and effectiveness of the technical contributions.
2 Related Works
Deep Image Segmentation. Deep learning methods (CNNs) have achieved satisfying performances
for image segmentation 333
7
36 39). By replacing fully connected layer with fully con-
volutional layers. FCN
33 transforms a classification CNNs (e.g.. AlexNet (31). VGG [43], or
ResNet (23) to fully-convolutional neural networks. In this way, FCN successfully transfers the success of image classification [3
43 47 to dense prediction/image segmentation. Instead of using Conditional Random Field (CRF) as post- post-procsssing. Deeplab (v1-v2) α7 methods add another fully connected CRF after the last CNN layer to make use of global information. Morcover, Deeplab

---

v3 19 introduces dilated/atrous convolution to increase the receptive field and make better use of context information to achieve better performance.
Besides the methods mentioned above, UNet [3 has also been one of the most popular methods for image segmentation, especially for images with fine structures. UNet architecture is based on FCN with two major modifications: 1) Similar to encoder-decoder, UNet is symmetric. The output is with the same size as input images, thus suitable for dense prediction/image segmentation, and 2) Skip connections between downsampling and upsampling paths. The skip connections of UNet are able to combine low level/local information with high level/global information, resulting in better segmentation performance.
Though obtaining satisfying pixel performances. these methods are still prone structuraa/topollgiiaal errors, as they are usually optimized via pixel-wise loss functions, such as mean-square-ente loss (MSE) and cross-entropy loss. As illustrated in Fig T small amount of pixel errors will affect or even damage the downstream tasks.
Topology-Aware Segmentation. Topology-awar segmentation methods have been proposed to segment with correct structure/topology. By identifying critical points of the predicted likelihood maps, persistent- howttttoooooooomooeeeooooo
losses [24]] 11 penalize topologically critical pixels. However, the identified critical points can be very noisy and often are not relevant to the topological errors. Illustrations are included in Sec. A.1. Moreover, the computation of persistent homology is expensive, making it difficult to evaluate the loss and gradient at every training iteration.
Other methods indirectly preserve topology by enhancing the curvilinear structures. VGG-UNet [35] uses the response of pretrained filters to enhance structures locally. But it does not truly preserve the topology, and cannot generalize to higher dimensional topological structures, such as voids. Several methods extract skeletons of the masks and penalize heavily on pixels of the skeletons. This ensures the prediction to be correct along the skeletons, and thus are likely correct in topology. clDice [42 extracts the skeleton through min/max- ppolling operations over the likelihood map. DMT Loss [25 uses the Morse complex of the likelihood map the skeleton. However, these skeletons are not necessarily topologically critical. The penalization on them may not be relevant to topology.
We also note that many deep learning techniques have been proposed to ensure the segmentation output preserves details, and thus preserves topology implicitly 33 3K
One may also use topological constraints postprocessing steps once we have the predicted likelihood 55 6
maps 20 46
19 50
37
38 [6]. Compared to end-to-end methods,
postprocessing methods usually contain self-defined parameters or hand-crafted features, making it difficult generalize to different situations.
we propose to use the warping of binary masks identify the topologically critical pixels. The identified locations more likely to be relevant to topological errors. Penalizing on these locations ensures the training efficiency and segmentation quality of our method. Another difference from previous methods is that our method rely on purely local topological computation (i.e., checking whether a pixel is simple within a local patch), whereas previous methods are mostly relying on global topological computation.
Instead of relying on the noisy likelihood maps 24 11 42 25
3 Method
By warping the binary predicted mask to the ground truth or conversely, we can accurately and efficiently identify the critical pixels. And then we propose novel homotopy warping loss which targets them to fix topological errors of the model. The overall framework is illustrated in Fig. 2
This section is organized follows. We will start with necessary definitions and notations. In Sec. [3.]] we give a concise description of digital topology and simple points. Next, we analyze different types of warping errors in Sec. 3..2 The proposed warping loss is introduced in Sec. 3.3 And finally, we explain the proposed new warping algorithm in Sec. 3.4
3.1 Digital Topology and Simple Points
In this section, we briefly introduce simple point definition from the classic digital topology 30 We focus on the 2D setting, whereas all definitions generalize to 3D. Details on 3D images will be provided in the Sec. A.2.
3

---

Connectivities of pixels. To discuss the topology of a 2D binary image, we first define the connectivity between pixels. See Fig. for an illustration. A pixel p has pixels surrounding it. We can either consider the 4 pixels that share edge with p as p' neighbors (called 4-adjacency), or consider all 8 pixels as p's neighbors (called 8-adjacency). To ensure the Jordan closed curve theorem hold, one has to use one adjacency for foreground (FG) pixels, and the other adjacency for the background (BG) pixels. In this paper, we use 4-adjacency for FG and 8-adjacency for BG. For 3D binary images, we use 6-adjacency for FG and 26-adjacency for BG. Denote by (p) the set of 4-adjacency neighbors of p, and (p) the set of 8-adjacency neighbors of p.
Simple points. For a binary image (2D/3D), pixel/voxel is called simple point if could be flipped from foreground (FG) to background (BG). or from BG FG, without changing the topology of the image [30], The following definition can be used to determine whether point is simple:
Mook
Homotopy
Topological
Warping
Critical Mask
Definition 1 (Simple Point Condition (30
Mart
Let be a point in 2D binary image. De- note by F' the set of FG pixels. Assume 4- adjacency for FG and 8-adjacent for BG. a simple point if and only if both of the fol- lowing conditions hold: 1) is 4-adjacent to just one FG connected component in Ng(p); and 2) p is 8-adjacent to just one BG con- nected component in Na(p).
Figure 2: The illustration of the proposed homotopy warping loss Lwarp- The homotopy warping algo- rithm tries to identify the topological critical pixels via the binary mask instead of noisy likelihood maps. These identified topological critical pixels/mask are used to define new loss which is complementary to standard pixel-wise loss functions. The details of Homotopy Warping and Topological Critical Mask M can be found in Sec. 3.2 and Sec. 3.3 respectively.
See Fig. 3 for an illustration of simple and non-simple points in 2D case. It is easy to check if a pixel is simple or not by inspect- ing its 3 x neighboring patch. The Defini-
tion can also generalize to 3D setting with 6- and 26-adjacencies for FG and BG, respectively.
3.2 Homotopic Warping Error
In this section, we introduce the ho- motopic warping of one mask towards another. We warp a mask through a sequence of flipping of simple points. Since we only flip simple points, by definition the warped mask will have the same topology[ The operation is called homotopic warping. It has been proven that two binary images with the always be warped into each other by flipping a sequence of simple points 40).
p
(a) adjaceet
(b) 8-adjacent (c) Simple Point(d) Non-simple
Figure 3: Illustration for 4, 8-adjacency. simple and non- simple points. (a): 4-adjacency. (b): 8-adjacency. (c): a simple point p. White and grey pixels are FG and BG, respectively. Flipping the label of p will not change the topology. (d): non-simple point p. Flipping will change the topology.
same topology
Consider two input masks, the predic- tion mask and the ground truth mask. We can warp one of them (source mask)
into another (target mask) in the best way possible, i.e., the warped mask has the minimal number of difference with the target mask (formally, the minimal Hamming distance). Once the warping finished, the pixels at which the warped mask is different from the target mask, called critical pixels, are a sparse set of pixels indicative of the topological errors of the prediction mask.
We will warp in both directions: from the prediction mask to the ground truth mask, and the opposite.
They identify different sets of critical pixels for a same topological error. In Fig. 4 we show
Note that it is essential to flip these simple points sequentially. The simple/non-singe status of a pixel may change if other adjacent pixels are flipped. Therefore, flipping set of simple points simultancously is not necessarily topology-preserving

---

(a)
(b)
(c)
(d)
(c)
(f)
Figure 4: Illustration of homotopic warping between two masks, red and white. red is the FG the prediction, this is false negative topological enror; if red is the FG of the ground truth, this is false positive error. (a-c): warping the red mask towards the white. (a): arrows show the warping direction. (b): the final mask after warping. Only single-pixel wide gap remains in the middle of the warped red mask. The non-simple/critical pixels are highlighted with red crosses. They correspond to the topological error and will be penalized in the loss. (c): at the beginning of the warping, we highlight (with green crosses) simple points that can be flipped according to our algorithm. (d-f): warping the white mask towards the red mask. Only single-pixel wide connection remains ensure the warped white mask is connected. The non con--imple/crttical pixels are highlighted with the red crosses.
A
GT
(b) Pred.
(c) Warp GT
Zoom-in
Zoom-in
Figure 5: Illustration of warping in real world example (satellite image). (a) GT mask. (b) The prediction mask. The red box highlights a false negative connection, and the green box highlights false positive connection.
(c) Warped GT mask (using the prediction mask as the target). (d) Zoomed-in view of the red box (c). (e) Zoomed-in view of the green box (c).
synthetic example with red and white masks, well warping in both directions. Warping the red mask towards the white mask ((a) and (b)) results in a single-pixel wide gap. The pixels in the gap (highlighted with red crosses) critical pixels; flipping any of them will change the topology of the warped red mask. Warping the white mask towards the red mask ((d) and (e)) results a single pixel wide link connecting the warped white mask. All pixels along the link (highlighted with red crosses) are critical; flipping any of them will change the topology of the warped white mask.
Here the red mask is the prediction mask, then this corresponds to false negative connection, i.e., connection that is missed by the prediction. If the red mask is the ground truth mask, then this corresponds false positive connection. Note the warping ensures that only topological errors are represented by the critical pixels. the synthetic example (Fig. 4). the large area of error the bottom left corner of the image is completely ignored is not topologically relevant.
In Fig. 5 we show a real example from the satellite image dataset, focusing on the emrors related to ID topological structures (connection). In the figure we illustrate both false negative connection error (highlighted with a red box) and false positive connection error (highlighted with a green box). If we warp the ground truth mask towards the prediction mask (c), we observe critical pixels forming a link for the false negative connection (d), and gap for the false positive connection (e). Similarly, we can warp the prediction mask towards the ground truth, and get different sets of critical pixels for the same topological errors (illustrations will be provided in Sec. A.3).
Note that for 2D images with fine structures, errors regarding ID topological structures are the most crucial. They affect the connectivity of the prediction results. For 3D images, errors ID or 2D topological structures are both important, corresponding to broken connections for tubular structures and holes in membranes. We will provide more comprehensive characterization of different types of topological structures and enrors in Sec. A.4.
3.3 Homotopy Warping Loss
Next, we formalize the proposed homotopy warping loss, which is evaluated on the critical pixels due to homotopic warping. As illustrated the previous section, the warping can be both directions, from the prediction mask to the ground truth mask, and the opposite.
5

---

Formally, we denote by f the predicted likelihood map of segmentation network, and fB the corresponding binarized prediction mask (i.e., f thresholded at 0.5). We denote by g the ground truth mask. First, we warp y towards SB, so that the warped mask, g" has the minimal Hamming distance from the target fB-
g" arg mi IIʃ -9"
(1)
where <I is the homotopic warping operation. The pixels at which g" and I' disagree are the critical pixels and will be penalized in the loss. We record these critical pixels due to the warping of g with a mask Mg. Mg = fB @ g*, in which @ is the Exclusive Or operation.
We also warp the prediction mask fB towards g.
s = arg min 4fe afe 11
glla
(2)
We use the mask My to record the remaining critical pixels after warping fa. M, g fé.
The union of the two critical pixel masks is the complete set of critical pixels corresponding to topo-
logical errors. M = My U My. M contains all the locations directly related to topological structures. Note this is different from persistenthhmoolgyy
method [24), DMT based method [25 or skele-
ton based method [42. which extract topological locations/structures on the predicted continuous likelihood maps. Our warping loss directly locates the topological critical piees/structures/locatoosss
based on the binary mask. The detected critical pixel set is sparse and less noisy.
Lpiintt denotes the pixel-wise loss function (e.g., cross-entropy), then Learp can be defined as:
Learyp = Ipinnt (J.g) @ M
where @ denotes Hadamard product. I Lwarp penalizes the topological critical pixels, forcing the neural network to predict better at these locations, and thus are less prone to topological errors.
The final loss of our method, Liouat, is given by:
Tttaaat Todice + Xwary Twwary
where Ldice denotes the dice loss. And the loss weight Awarp is used to balance the two loss terms.
3.4 Distance-Ordered Homotopy Warping
Even though checking whether pixel is simple not is easy, finding the optimal warping in Eq. ①1 and (2 is challenging. The reason is that there are too many degrees of freedom. At each iteration during the warping, we have to choose simple point to flip. It is not obvious which simple point will finally lead to a global optimum.
In this section, we provide efficient heuristic algorithm to find warping local optimum. We explain the algorithm for warping towards fp. The algorithm generalizes to the opposite warping direction naturally. Recall the warping algorithm iteratively flips simple points. But there too many choices at each iteration. It is hard to know which flipping choice will lead to the optimal solution. We need good heuristies for choosing a flippable pixel. Below explain our main intuitions for designing our algorithm.
First, we restrict the warping it only sweeps through the area where the two masks disagree. In other words, at each iteration, we restrict the candidate pixels for flipping to not only simple, but also pixels which g and fa disagree. In Fig. 4 (c) and (f), we highlight the candidate pixels for flipping at the beginning. Notice that not all simple points are selected as candidates. We only choose simple points within the difference set Diff 9) = fa θ g.
Second, since we want to minimize the difference of the warped and target masks, we propose to flip pixels within the difference region Diff 9). To implement this strategy efficiently, we order all pixel within Diff (B: g) according to their distance from the FG/BG, and flip them according to this order. A pixel is skipped if it is not simple.
Our algorithm is based on the intuition that far-away pixel will not become simple until nearby pixels are flipped first. To see this, we first formalize the definition of distance transform from the masks, fo and g, denoted by D and D9, For a BG pixel of g, p. its distance value D (p) is the shortest distance from to any FG pixel of g. (²) min,eFG, dist(p, s). Similarly, for FG pixel of g. q. D9 min dist(g, s). The definition generalizes to DJB

---

We observe that a pixel cannot be simple unless it has distance from the FG/BG of a warping mask. The proof is straightforward. Formally,
Lemma 1. Given a 2D binary mask m, a pixel p cannot be simple for if its distance function (p) > 1.
Proof. Assume the foreground has pixel value of and p a background pixel with index of (i.
Consider the m-adjacent (m==4) for p. Since D > 1, then we have m 1,j) = mii +,3)
any FG connected component, violating the 1) of Definition 1. Consequently, pixel (i.j) is not a simple point. This also holds for foreground pixels. This lemma naturally generalizes to 3D case.
m 1) m + 1) 0. In this case, p not 4-adjacent
Lemma D implics that only after flipping pixels with distance 1, the other misclassified locations should be considered. This observation gives the intuition of algorithm. To warp g towards fB. our algorithm is as follows: (1) compute the difference set Diff( g) as the candidate set of pixels; (2) sort candidate pixels in non-decreasing order of the distance transform D"; (3) enumerate through all candidate pixels according to the order. For each iteration, check if it simple. If yes, flip the pixel's label. It is possible that this algorithm can miss some pixels. They are not simple when the algorithm checks, but they might become simple as the algorithm continues (since their neighboring pixels get flipped in previous iterations).
One remedy is to recalculate the distance transform after one round of warping, and go through remaining pixels once more. But in practice found this is not necessary this scenario is very rare. The effectiveness of the proposed warping strategy is illustrated Sec. A.5.
Experiments
We conduct extensive experiments demonstrate the effectiveness of the pro- posed method. Sec. A.6 introduces the datasets used in this paper, including both 2D and 3D datasets. The benchmark methods are described in Sec. A.7. We mainly focus on topology-awan segmenta- tion methods. Sec. A.8 describes the eval- uation metrics used to assess the quality of the segmentation. To demonstrate the abil- ity to achieve better structuralttoolloicca performances, besides standard segmenta- tion metric, such as DICE score, we also use several topology-aware metrics to eval- uate all the methods. Several ablation stud- ies are then conducted to further demon- strate the efficiency and effectiveness the technical contributions (Sec. 4.1
Datasets. We conduct extensive ex- periments to validate the efficacy of our method. Specifically,, we use four natural and biomedical 2D datasets (RoadTrracr 4. DeepGlobe [12], Mass. (34. DRIVE (44)) and one more 3D biomedical dataset (CREMI P to validate the efficacy of the propose method. More details about the datasets and the split of the training and validation subsets are included in Sec. A.6.
(a) Original
(b) GT
(c) UNet
(d) Ours
Figure 6: Qualitative results compared with the standard UNet. The proposed warping loss can help correct the topological errors (highlighted by red circles). The sampled patches are from four different datasets.
Baselines. We compare the results of our method with several state-of-the-at methods. The standard/simple UNet (2D/3D) UNet [39 10 is used as a strong baseline and the backbone for other methods. The other baselines used in this paper include RoadTrace [4] VecRoad [8]. iCurb [52]
hhttss//cremioorr/

---

clDice 44 and DMT 225 More descriptions and DIVE 17]. VGG-UNet [351 TopoLoss 224.
implementation details of these baselines are included in Sec. A.7.
Evaluation Metrics. We use both pixel-wise (DICE) and topology metrics (ARI, Warping Error and Betti number error) to evaluate the performance of the proposed method. More details about the evaluation metries are provided in Sec. A.8.
Implementation Details. For 2D images, we use (m, n1) = (4, 8) to check if a pixel is simple not; while (m, n) (8, 26) for 3D images. We choose cross-entropy loss as wwarp/ Lyised for
Table 1: Quantitative results of different methods.
Method
DICE ARIT
Warping.
Betti
all the experiments, except the ablation studies in Tab. 3
RoadTracer
UNet 39
0.587 0.544 10.412 x10
1.591
RoadTracer
0,547
0.521 13.224x10-
2.218
For 2D datasets, the batch size is set as 16, and the initial learning rate is 0.01. We randomly
VecRoad [48]
0.552
0.533 12.819x10-3
2.095
iCurb [2
0.571
0.535
11.683x10-3
1.873
crop patches with the size of 512 x 512 and then feed them into the 2D UNet. For 3D case, the batch size is also 16, while the input size is 128 x 128 x 16. We perform the data normalization for the single patch based on its mean and standard deviation.
VGG-UNet [35]
0.576
0.536
11.231
x10
1.607
TopoNet 24
0.584
0.556 10.008 x10-3
1.378
clDice 42
0.591
0.550
9.192 x10
1.309
DMT 25
0.593
0.561 9.452 x x10
1.419
Warping
0.603
0.572 8.853 x x10**
1.251
DeepGlobe
We use PyTorch framework (Version: 1.7.1) to implement the proposed method. A sim- ple/standard UNet (2D or 3D) is used as base- line and the backbone. For the proposed method, as well the other loss function based base- lines, to make fair comparison, we use the
UNet [39
0.764
0.758 3.212 x x10
0.827
VGG-UNet
35
0.742
0.748 3.371 x10
0.867
TopoNet 24
0.765
0.763
2.908 x10
0.695
elDice 42
0.771
0.767
2,874
x10-3
0.711
DMT 25
0.769
0.772
2.751
x10 0.609
Warping
0.780
0.784
2.683x10-3
0.569
same UNet as backbone. And the training strat- egy is to train the UNet with dice loss first until converge, and then add the proposed losses to fine-tune the models obtained from the initial step.
Mass.
UNet [39
0.661
0.819
3.093 10
3.439
VGG-UNet [35
0.667
0.846
3,185x10
2.781
TopoNet
24
0.690
0.867
x.871x10
1.275
elDice 42
0.682
0.862
2.552x10
1.431
DMT
25
0.706
0.881
2.631
x10
0.995
All the experiments are performed on Tesla V100-SXM2 GPU (32G Memory), and an In- tel(R) Xeon(R) Gold 6140 CPU@2.30 GHz.
Warping
0.715
0.864
2.440x10
0.974
DRIVE
UNet 39
0.749
0.834
4.781x10
3.643
DIVE
17
0.754
0.841
4.913x10->
3,276
Qualitative and Quantitative Results. In Fig. 6, we show qualitative results from differ- ent datasets. Compared with baseline UNet, our method recovers better structures, such as con- nections, which are highlighted by red circles. Our final loss is weighted combination of the
VGG-UNet 35
0.721
0.887
4.362x10
2.784
TopoNet 24
0.761
0.902
3.895x10-3
1.076
elDice 42
0.753
0.896
4.012 x10
1.218
DMT 25
0.773
0.908
3.561
x10-3
0.873
Warping
0.781
0.911
3.419x10
0.812
CREMI
dice loss and warping-loss term Lwarp- When Awarp O, the proposed method degrades to a standard UNet. The recovered better struc- tures (UNet and Warping columns in Fig. ()) demonstrates that our warping-loss helps the deep neural networks to achieve better topolog- ical segmentations. More qualitative results are provided in Sec. A.9.
3D UNet 10
0.961
0.832
11.173
x10
2.313
DIVE 17
0.964
0.851 11.219 x x10-1
2.674
TopoNet 24
0.967
0.872 10.454 x10- 1.076
elDice 42
0.965
0.845 10.576 x 0.756
DMT 25
0.973
0.901
10.318 x10- 0.726
Warping
0.967 0.907 9.854 x 10 0.711
Tab. D shows quantitative results for three 2D image datasets, RoadTracer, DeepGlobe and The Massachusetts dataset and one 3D image dataset, CREMI. The best performances are highlighted with bold. The proposed warping-loss usually achieves the best performances in both DICE score and topological accuracy (ARI, Warping Error and Betti Error) over other topology-aware segmentation baselines. Note that we only report the mean performances here for space limitation. We also include standard deviations and use t-test to determine the statistical significance in Sec. A.10.
8

---

4.1 Ablation studies
To further explore the technical contributions of the proposed method and provide a rough guideline of how to choose the hyperparameters, we conduct several ablation studies. Note that all the ablation studies are conducted on the RoadTracer dataset.
The impact of the loss weights. As seen in Eq. our final loss function is a combination of dice loss and the proposed warping loss wawr.. The balanced term Awarp controls the influence of the warping loss term, and it's a dataset dependent nyper-parameter. The quantitative results for dif- ferent choices of Awarp are illustrated in Tab. 2 For the RoadTracer dataset, the optimal value is 1 x 10-4. From Tab. 2 we can find that different choices of Awarp do affect the performances. The reason is that, if Aworp is too small, the effect of the warping loss term is negligible. However, if Awarp is too large, the warping loss term will compete with the Litien: and decrease the performance of the other easy-classified pixels. Note that within reasonable range of Awarp- all the choices contribute to better performances compared to bascline (row '0', standard UNet), demonstrating the effectiveness of the proposed loss term. The best choices of Awarp for each specific dataset are included in Sec. A.
Table Ablation study for loss weight Xwarp.
Awary
DICEf
ARIT
Warping.
Betti
0
0.587 0,544 10,412 x x10 -3
1,591
2 10-3
0.603
0.561 9.012 x10-3
1.307
5 10
0.601
0,548 9.356 x10-
1.412
1 10
0.603
0.572 8,853 x10-3
1.251
2 x 10 10-4
0.602 0.565 9.131 x10 =
1.354
The choice of loss functions. The proposed warp- ing loss defined on the identified topological critical pixels. Consequently, any pixel-wise loss functions can be used to define the warping loss Lwwrr//ppizet. In this section, we investigate how the choices of loss functions affect the perfor- mances. The quantitative results are show in Tab. Compared with Dice loss, the cross- cessopy loss (CE) achieves best performances in terms of topological metrics. On the other hand, all these three choices perform better than baseline method (row 'w/o', standard UNet), which further demonstrates the contribution of the proposed loss term.
Table 3: Ablation study for the choices of loss.
Lpinet
DICEf
ARIT
Warpingl
Betti]
w/o
0,587
0,554 10.412 x10
1.591
MSE
0.598
0.556 9.853 x10-3
1.429
Dice loss
0.606
0.563
9,471
x10 -3
1.368
CE
0.603
0.572
8,853 x10
1.251
loss (MSE) or
Comparison of different critical pixel selection strategies. We also conduct additional experiments to demonstrate the effectiveness of the proposed critical pixel selection strat- egy. The first variation is flipping the simple pixels but removing the heuristic that uses the distance trans- form, and then use the remaining non- simple pixels as our critical pixel set, which is denoted
Table 4: Comparison of different critical pixel selection strategies.
Kemel Size
DICEf ARI
Warping
Betti]
U-Net
0.587 0.544 10.412 x10 3
1.591
w/o DT
0.586 0.547
10.256x
10->
1.473
Warping (GT - Pred)
0.594 0.567
9.171x10-3
1.290
Warping (Pred + GT)
0.598
0.562
9,124
x10-3
1.315
Warping
0.603 0.572
8.853x10
1.251
'w/o" in Tab. 4
The other two variations (warping only in one direction: ground truth to prediction or prediction to ground truth) achieve reasonably better while still slightly inferior results to the proposed version. The reason might be that the proposed critical point selection strategy contains more complete topologically challenging pixels. Both ablation studies demonstrate the effectiveness of the proposed critical point selection strategy.
Comparison with morphology post-processing. To verify the necessity of the proposed method, we also compare the proposed method with traditional morphology post- post-processing. i.e., dilation, erosion. Dilation and erosion are global operations. Though closing operation (dilates image and then erodes dilated image) could bridge some specific gaps/broken connections, it will damage the global structures. In practice, the gaps/broken will usually be more than few pixels. If the kernel size is too small, the closing operation (dilate then erode) will hardly affect the final performance; while too big kernel sizes will join the separated regions. Tab. 5 lists post- post-processing results on baseline U-Net.

---

The efficiency of the proposed loss, In this section, we'd d like investigate the efficiency of the proposed method. Our warping algorithm contains two parts, the distance transform and the sorting of distance matrix. The complexity for distance transform (n) for a 2D image where is the size of the 2D image, and H x W. H, W are the height and width of the 2D image, respectively. And the complexity for sorting is mlog (m), where m is the number of misclassified pixels. Usually
Table 5: Comparison against post- post-processing.
Kernel Size
DICE1
ARII
Warping +
Betti
U-Net
0.587 0,544
10.412 x x10
1.591
Closing (5)
0.588 0.542
10.414x10
1.590
Closing (10)
0.587 0,546
10.399x10 10
1.583
Closing (15)
0,586
0,541
10,428 x x10
1.598
Worping
0.603 0.572 8.853x10"
1.251
< n, so the overall complexity for the warping algorithm is O(n). As a comparison, [24] needs ((n²) complexity to compute the persistence diagram. And the
computational complexity for 25 42
((nlog(n)) and ((n), respectively.
The comparison terms of complexity and training time illustrated in Tab. 6 Note that for the pro- posed method and all the other baselines, we first train a simple/standard UNet, and then add the additional loss terms to fine-ture tune the models obtained from the initial step. Here, the training time is only for the fine-tune step. The proposed method takes slightly longer training time than elDice, while achieves the best performance over the others. As all these methods use the same backhone, the inference times are the same.
Table 6: Comparison of efficiency.
Method
Complexity Training time
TopoNet [24]
O(n")
23 12h
clDice 42
O(n)
2 3h
DMT 25
(nnlogn)
2 7h
Warping
O(n)
2 4h
Conclusion
In this paper, we propose a novel homotopy warping loss to learn to segment with better struc- tural/topological accuracy. Under the homotopy warping strategy, can identify the topological critical pixels/locations, and the new loss is defined on these identified pixels/locations. Furthermore, Homotopy Warping efficiently identify the
we propose novel strategy called Distance-Ordent topological error locations based on distance transform. Extensive experiments on multiple datasets and ablation studies have been conducted to demonstrate the efficacy of the proposed method.
Limitations. We mainly focus on the binary segmentation of curvilinear structures in this work. In terms of multiclass curvilinear structure segmentation, we note that it's always doable convert the multiclass segmentation task to a number of binary class segmentation tasks, which will be left for future work.
Acknowledgement. The research of Xiaoling Hu is partially supported by NSF I1S-1909038, and we would like to thank the anonymous reviews for constructive comments.
References
[1] Bjoern Andres, Jörg H Kappes, Thorsten Beier, Ulirich Köthe, and Fred A Hamprecht. Proba- bilistic image segmentation with closedness constraints. In ICCV, 2011.
[2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. TPAMI, 2017.
[3] Meir Barzohar and David B Cooper. Automatic finding of main roads in aerial images by using geometric-stochastic models and estimation. TPAMI, 1996.
[4] Favyen Bastani, Songtao He, Sofiane Abbar, Mohammad Alizadeh, Hari Balakrishnan, Sanjay Chawla, Sam Madden, and David Weti Roadtracer: Automatic extraction of road networks from aerial images. In CVPR, 2018.
[5] Anil Batra, Suriya Singh, Guan Pang. Saikat Basu, CV Jawahar, and Manohar Paluri. Improved road connectivity by joint learning of orientation and segmentation. In CVPR, 2019.
10

---

[6] Chao Chen, Daniel Freedman, and Christoph H Lampert. Enforcing topological constraints in random field image segmentation. CVPR, 2011.
[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected erfs, arXiv preprint arXiv:1412.7066 2014.
[8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2018.
[9] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:/706.05556 2017.
[10] Özgün Çiçek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In MICCAI, 2016.
[11] J Clough, Byme, I Oksuz, VA Zimmer, JA Schnabel, and A King. topological loss function for deep-learning based image segmentation using persistent homology. TPAMI, 2020.
[12] like Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: A challenge to parse the earth through satellite images. CVPR Workshops, 2018.
[13] Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary-aware feature propagation for scene segmentation. ICCV, 2019.
[14] Herbert Edelsbrunner and John Harer. Computational topology: an introduction. American Mathematical Soc., 2010.
[15] Herbert Edelsbrunner, David Letscher, and Afra Zomorodian. Topological persistence and simplification. In FOCS, 2000.
[16] Rolando Estrada, Carlo Tomasi, Scott C Schmidler, and Sina Farsiu. Tree topology estimation. TPAMI, 2014.
[17] Ahmed Fakhry, Hanchuan Peng, and Shuiwang Ji. Deep models for brain em image segmenta- tion: novel insights and improved performance. Bioinformaties, 2016.
[18] Jan Funke, Fabian David Tschopp, William Grisaitis, Arlo Sheridan, Chandan Singh, Stephan Saalfeld, and Srinivas C Turaga. A deep structured learning approach towards automating connectome reconstruction from 3d electron micrographs. arXiv preprint arXiv: wrXiv:1709002944 2017.
[19] Mingchen Gao, Chao Chen, Shaoting Zhang, Zhen Qian, Dimitris Metaxas, and Leon Axel. Segmenting the papillary muscles and the trabeculae from high resolution cardiac ct through restoration topological handles. In IPMI, 2013.
[20] Xiao Han, Chenyang Xu, and Jerry L. Prince. A topology preserving level set method for geometric deformable models. TPAMI, 2003.
[21] Allen Hatcher. Algebraic Topology. Cambridge University Press, 2002.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[24] Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. sopology-preserving deep image segmentation. In NeurIPS, 2019.
[25] Xiaoling Hu, Yusu Wang, Li Fuxin, Dimitris Samaras, and Chao Chen. Topology-aware segmentation using discrete morse theory. In ICLR, 2021.

---

[26] Viren Jain, Benjamin Bollmann, Mark Richardson, Daniel R Berger, Moritz N Helmstaedter, Kevin L Briggman, Winfried Denk, Jared B Bowden, John M Mendenhall, Wickliffe Abraham, et al. Boundary learning by optimization with topological constraints. In CVPR, 2010.
[27] Michal Januszewski, Jörgen Kornfeld, Peter H Li, Art Pope, Tim Blakely, Larry Lindsey, Jeremy
Mike Tyka, Winfried Denk, and Viren Jain. High- peeh-preciioon automated reconstruction of neurons with flood-filling networks. Nature methods, 2018.
[28] Davood Karimi and Septimiu E Salcudean. Reducing the hausdorff distance in medical image segmentation with convolutional neural networks. arXiv preprint arXiv:1904.1000 2019.
[29] Hoel Kervadec, Jihene Bouchtiba, Christian Desrosiers, Eric Granger, Jose Dolz, and Ismail Ben Ayed. Boundary loss for highly unbalanced segmentation. In International Conference on Medical Imaging with Deep Learning, 2019.
[30] T Yung Kong and Azriel Rosenfeld. Digital topology: Introduction and survey. Computer Vision, Graphics, and Image Processing, 1989.
[31] Alex Krizhevsky, Πla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[32] Carole Le Guyader and Luminita A Vese. Self-repelling snakes for topology-preserving segmentation models. TIP, 2008.
[33] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[34] Volodymyr Mnih. Machine learning for aerial image labeling. University of Toronto (Canada), 2013.
[35] Agata Mosinska, Pablo Marquez-Neila, Mateusz Koziński, and Pascal Fua. Beyond the pixel- wise loss for topology-awan delineation. In CVPR, 2018.
[36] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In ICCV, 2015.
[37] Sebastian Nowozin and Christoph H Lampert. Global connectivity potentials for random field models. In CVPR, 2009.
[38] Martin Ralf Oswald, Jan Stühmer, and Daniel Cremers. Generalized connectivity constraints for spatio-temporal 3d reconstruction. In ECCV, 2014.
[39] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[40] Azriel Rosenfeld, T Yung Kong, and Akira Nakamura. preseeng deformations of two-valued digital pictures. Graphical Models and Image Processing, 1998.
[41] Florent Ségonne, Active contours under topology control-genus preserving level sets. IJCV, 2008.
[42] Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoern Menze. cidice-a novel topology- preserving loss function for tubular structure segmentation. In CVPR, 2021.
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409 1509. 2014.
[44] Joes Staal, Michael D Abråmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Gin- neken. Ridge-based vessel segmentation in color images of the retina. TMI, 2004.
[45] Jan Stuhmer, Peter Schroder, and Daniel Cremers. Tree shape priors with connectivity con- straints using convex relaxation general graphs. In ICCV, 2013.
12

---

[46] Ganesh Sundaramoorthi and Anthony Yezzi. Global regularizing flows with topology preserva- tion for active contours and polygons. TIP, 2007.
[47] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[48] Kong-Qiang Tan, Shang-Hua Gao, Xuan- Yi Li, Ming-Ming Cheng, and Bo Ren. Vecroad: Point-based iterative graph exploration for road graphs extraction. In CVPR, 2020.
[49] Mustafa Gokhan Uzunbas, Chao Chen, and Dimitris Metaxas. An efficient conditional random field approach for automatic and interactive neuron segmentation. Medical image analysis, 2016.
[50] Sara Vicente, Vladimir Kolmogorov, and Carsten Rother. Graph cut based image segmentation with connectivity priors. In CVPR, 2008
[51] Pengxiang Wu, Chao Chen, Yusu Wang, Shaoting Zhang, Changhe Yuan, Zhen Qian, Dimitris Metaxas, and Leon Axel. Optimal topological cycles and their application in cardiac trabeculae restoration. In IPMI, 2017.
[52] Zhenhua Xu, Yuxiang Sun, and Ming Liu. icurb: Imitation learning- dearning-based detection of road curbs using aerial images for autonomous driving. IEEE Robotics and Automation Letters, 2021.
[53] Jiaqi Yang, Xiaoling Hu, Chao Chen, and Chialing Tsai. A topological-attention convlstm network and its application to em images. In MICCAI, 2021.
[54] Ze Ye, Cong Chen, Changhe Yuan, and Chao Chen. Diverse multiple prediction neuron image reconstruction. In MICCAI, 2019.
[55] Yun Zeng, Dimitris Samaras, Wei Chen, and Qunsheng Peng. Topology cuts: A novel min- cut/max-flow algorithm for topology preserving segmentation in n-d images. CVIU, 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL)? [Yes] See the Supplemental Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4ec the Supplemental Material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See the Supplemental Material.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sec.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curatinn contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "mRD_xuF-GDr", "reviewer": "Reviewer_RT7y", "summary": "The authors implement the Warping Error metric, introduced by Jain et al. (CVPR 2010) as a loss for topology aware segmentation. To be able to efficiently calculate the warping, the authors introduce a heuristic algorithm based on the distance transform to choose the pixels to \"flip\" or warp to achieve a better topology. ", "questions": "How does the filtering using the DT compare to the calculation of image persistence computationally? ", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths:** \n\nOverall I find the proposed warping idea to be innovative and clever. It improves topology awareness in segmentation models in an efficient manner. It is significantly faster to compute than methods based on barcode-matching and persistent homology and achieves competitive performance to such methods. \n\n\n**Weaknesses:**\n\n1) Terminology: Introduction l. 24-26 \"pixels near the peripheral of the object of interest can generally be challenging, but not relevant to topology.\" I think this statement is problematic. When considering the inverse e.g. in the case of a surface or vessel, that a foreground pixel changes to background. Such a scenario would immediately lead to a topology mismatch (Betti error 1). \n\n2) Terminology: \"topologically critical location\" -->  I find this terminology to be not optimally chosen. I agree that the warping concept appears to help with identifying pixels which may close loops or fill holes. However, considering the warping I do not see a guarantee that such \"locations\" (as in the exact location) which I understand to refer to individual or groups of pixels are indeed part of the real foreground, nor are these locations unique. A slightly varying warping may propose a set of different pixels. \n\n3) The identified locations are more likely to be relevant to topological errors. --> this statement should be statistically supported. Compared to what exactly? Does this rely on the point estimate for any pixel? Or given a particularly trained network? \n\n4) Theorem 1: The presentation of a well known definition from Kong et al. is trivial and could be presented in a different way. \n\n5) Experimentation, lack of implementation details: In Table 2 and a dedicated section, the authors show an ablation study on the influence of lamda on the results. Lamda is a linear parameter, weighting the contribution of the new loss to the overall loss. Similarly, the studied baseline methods, e.g. TopoNet [24], DMT [25], and clDice [42] have a loss weighting parameter. It would be important to understand how and if the parameters of the baselines were chosen and experimented with. (I understand that the authors cannot train ablation studies for all baselines etc.) However, it is an important information to understand the results in Table 1.\n\n6) Terminology: l. 34 \"to force the neural network to memorize them\" --> I would tone down this statement, in my understanding, the neural network does not memorize an exact \"critical point\" as such in TopoNet [24]. \n\n\n\n**Minor:** \n\n- I find the method section to be a bit wordy, it could be compressed on the essential definitions.\n\n- There exist several grammatical errors, please double-check these with a focus on plurals and articles. E.g. l. 271 \"This lemma is naturally generalized to 3D case.\"\n\n- l. 52 \"language of topology\" I find this to be an imprecise definition or formulation. \n\n**Note:** \n\nAfter rebuttal and discussion I increased the rating to 5. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VAZv3DQYD4T", "reviewer": "Reviewer_xhKW", "summary": "This paper proposed a new method for segmenting images with better topological accuracy. Specifically, a new algorithm utilizing distance transform is used to identify the topologically critical locations, based on which a new homotopy warping loss is used to measure the difference between the predicted and ground-truth topology and train the segmentation network. Experiments on four 2D datasets and one 3D dataset validated the efficacy of the proposed method. Besides pixel-wise accuracy, better topology accuracy is obtained.", "questions": "Please see the comments above", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper introduced an efficient heuristic algorithm to identify topologically critical locations. Compared with a previous work on topology-preserving image segmentation method which runs in cubic time complexity, the proposed algorithm's complexity is linear to image size, thus making the training more efficient.\n\n2. A new homotopy warping loss is proposed to train the segmentation network to preserve the topology. The homotopy warping loss penalizes errors on topologically critical locations and results in models with better topological accuracies.\n\n3. Comparisons with existing topology-preserving segmentation methods show that the proposed method achieves the best performance. Ablation studies provide some guidance regarding the choices of loss weights and loss functions.\n\nWeaknesses\n1. The proposed distance-ordered homotopy warping algorithm is possible to miss some simple pixels during the flipping although very rare.\n\n2. In the previous work TopoNet, Variation of Information (VOI) is used as a metric to measure topological correctness. It would be better if VOI is also compared in the experiments.\n\n3. Grammar/spelling errors:\na) Line 92: \"Except for the methods mentioned above, UNet has also been one of the most popular methods for image segmentation\". It seems \"besides\" is the proper preposition, not \"except for\".\nb) Line 131: \"efficiently identity\" -> \"efficiently identify\"\nc) Line 382: \"we'd like the investigate\" -> \"we investigate\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GRrzwUVeb4d", "reviewer": "Reviewer_6qMj", "summary": "This paper presents a new method to improve the topology of image segmentation.  The method finds topologically critical locations by comparing the predicted segmentation mask with the ground truth, and proposes a novel homotopy warping loss which guides the training to improve topological accuracy. Experiments are conducted in both 2D and 3D settings to segment satellite and biomedical images, and results show the proposed method outperforms other existing methods.", "questions": "1.\tAll the used cases in the paper address the segmentation error in between the segmented areas. What if some False Positives or False Negatives exist at the edges of segmentation regions? \n\n2.\tThe datasets used in the proposed study have straight or curved lines as an area of interest. Would the method be useful for datasets with more irregular shapes as a region of interest—for example, the Kvasir SEG - Simula dataset?  https://arxiv.org/abs/1911.07069 \n\n3.\tHow might it be possible to apply the method for image segmentation problems with more than two classes (e.g. a problem with two foreground classes and one background)?\n\n4.\tWhat are the limitations of the proposed method?\n", "limitations": "Limitations are not discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tAlthough previous work such as TopoNet also emphasize the importance of focussing on topologically critical locations, the warping approach taken in this paper appears to be novel and more efficient, computationally.\n\n2.\tThe warping loss is intuitive and seeks to transform the topology of the predicted segmentation mask to match the ground truth (and vice versa).  The proposed algorithm focusses on simple points, which are described clearly in terms of their connectivity. \n\n3.\tThe experimental results demonstrate the method is capable of improving segmentation not just at the critical points but overall in the image.\n\nWeaknesses:\n\n1.\tThe segmentation performance obtained from this method mainly relies on the standard UNet results. If the segmentation output from UNet is noisy, the warping operation may take many iterations, perhaps more so initially as the network is starting to train.  Experiments with other architectures is left for future work.  \n\n2.\tWhile results are improved, the improvement may be subtle; for example 1% improvement in Dice score for most experiments in Table 1.  Nonetheless, visual results in Figure 6 show good improvement.\n\n3.\tIt wasn’t clear to this reviewer if or how the method generalises to problems where there are more segmentation classes (e.g. three or more).  All the experiments shown in the paper are binary segmentation problems.  However, the paper makes it clear that it is focussing on binary segmentation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F9M9-kjX7MY", "reviewer": "Reviewer_4Rx6", "summary": "A topology-aware learning objective is proposed for semantic segmentation models. This objective relies upon a warping operation that transforms a predicted mask to one that is close to the ground truth mask in pixel space without changing the topology, and vice versa. The objective is a pixelwise cross-entropy computed on the points of disagreement between the target and warped source masks. Evaluation on 2D and 3D satellite and medical imagery segmentation datasets shows the algorithm is efficient and delivers strong results.", "questions": "- For what other kinds of data could this approach be useful? How could the proposed method be generalized to multiclass problems?\n- Importance of the distance sorting heuristic: should drops in performance (accuracy or computation time) be expected if a simpler method of warping the masks is used, such as iteratively selecting a random non-simple point that borders the current mask? As another, simpler ablation study, I would suggest to evaluate warping only in one direction (ground truth to prediction or prediction to ground truth).\n- L185: This line is somewhat confusing or misleading, since the proposed heuristic only finds a mask that is locally optimal, i.e., no simple points can be flipped to decrease the Hamming distance. (Is it known whether finding the global optimum simply slow (cf. L239-242) or computationally intractable?)\n- L219: What would happen if each pixel's on/off state is sampled from the output distribution instead of bring thresholded?", "limitations": "Please see the first two points of Weaknesses above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simple, intuitive algorithm and clear writing. The algorithm is well-motivated and mathematically sound, and care is taken to make it efficient using the heuristic for computing the points of interest.\n- Strong experimental evaluation on diverse datasets where preservation of connectivity structure has been an important challenge.\n- Ablations and comparisons are appropriately selected.\n\nWeaknesses:\n- The paper would benefit from more discussion about what may be responsible for the better performance of the algorithm relative to baseline methods of selecting critical points. The comparison with TopoNet's points of interest in Figure 7 gives some intuition about the question, but there are no comparisons of outputs of models trained using different critical point selection methods that would show the impact of a noisy point selection on the model output.\n- Similarly, I would have liked to see some examples of the failure modes of the proposed algorithm, as well as images of outputs on all the datasets studied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept