Title: GraphMorph: Tubular Structure Extraction by Morphing Predicted Graphs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Segmentation

Tubular Structure Extraction

Branch-level Features

Graph Representation
Abstract: Accurately restoring topology is both challenging and crucial in tubular structure extraction tasks, such as blood vessel segmentation and road network extraction. Diverging from traditional approaches based on pixel-level classification, our proposed method, named GraphMorph, focuses on branch-level features of tubular structures to achieve more topologically accurate predictions. GraphMorph comprises two main components: a Graph Decoder and a Morph Module. Utilizing multi-scale features extracted from an image patch by the segmentation network, the Graph Decoder facilitates the learning of branch-level features and generates a graph that accurately represents the tubular structure in this patch. The Morph Module processes two primary inputs: the graph and the centerline probability map, provided by the Graph Decoder and the segmentation network, respectively. Employing a novel SkeletonDijkstra algorithm, the Morph Module produces a centerline mask that aligns with the predicted graph. Furthermore, we observe that employing centerline masks predicted by GraphMorph significantly reduces false positives in the segmentation task, which is achieved by a simple yet effective post-processing strategy. The efficacy of our method in the centerline extraction and segmentation tasks has been substantiated through experimental evaluations across various datasets. Source code will be released soon.

GraphMorph: Tubular Structure Extraction by Morphing Predicted Graphs

Abstract

Accurately restoring topology is both challenging and crucial in tubular structure extraction tasks, such as blood vessel segmentation and road network extraction. Diverging from traditional approaches based on pixel-level classification, our proposed method, named GraphMorph, focuses on branch-level features of tubular structures to achieve more topologically accurate predictions. GraphMorph comprises two main components: a Graph Decoder and a Morph Module. Utilizing multi-scale features extracted from an image patch by the segmentation network, the Graph Decoder facilitates the learning of branch-level features and generates a graph that accurately represents the tubular structure in this patch. The Morph Module processes two primary inputs: the graph and the centerline probability map, provided by the Graph Decoder and the segmentation network, respectively. Employing a novel SkeletonDijkstra algorithm, the Morph Module produces a centerline mask that aligns with the predicted graph. Furthermore, we observe that employing centerline masks predicted by GraphMorph significantly reduces false positives in the segmentation task, which is achieved by a simple yet effective post-processing strategy. The efficacy of our method in the centerline extraction and segmentation tasks has been substantiated through experimental evaluations across various datasets. Source code will be released soon.

1 Introduction

Extraction of tubular structures is an essential step in many computer vision tasks [39, 13, 1, 27]. In medical applications, accurate segmentation of retinal vessels can provide crucial insights into various cardiovascular and ophthalmologic diseases [8]. In the field of urban planning and geographic information systems, the precise extraction of road networks aids in traffic management, urban development, and emergency response planning [28]. Existing deep learning-based methods model tubular structure extraction as a pixel-level classification task [34] or point set prediction task [40], without explicitly predicting the topological structures. To focus more on topology, some advanced methods design novel backbones or modules [36, 23, 33], or introduce new loss functions from the topological perspective [14, 37, 25]. However, they are still limited to the framework of pixel-level prediction.

We argue that most pixel-level frameworks have not effectively exploited the nature of tubular structures, which are inherently composed of several branches that are interconnected in complex ways. Specifically, pixel-level loss functions, like softDice Loss [26] and Focal Loss [20], struggle with subtle inaccuracies and are particularly ineffective at addressing complex topological errors.

(a) Image

(b) Label

(c) DiceLoss

(d) clDiceLoss

82.72, 92.03

Graph

Prediction

84.98, 95.17

87.39, 97.83

Dice, clDice     :

,                :

4, 5

0, 1

0, 0

Under the pixel-level frameworks, despite attempts to pay more attention to fine branches [37, 25] or topological features [14, 33], they still struggle with fully capturing the complex topological nature of tubular structures. We demonstrate this deficiency by providing an example in Figure 1. For a systematic understanding of the issues of pixel-level frameworks, we summarize them into three categories: (1) Broken branches or false negatives (FNs). (2) Redundant branches or false positives (FPs). (3) Topological errors (TEs). Therefore, understanding tubular structure extraction solely from a pixel-level perspective is fundamentally flawed.

Recognizing these limitations, we shift our focus to branch-level features , which are more essential for accurately capturing the nuances of tubular structures. Any complex tubular structure can be broken down into several branches, which distinguishes it from non-tubular objects. This perspective inspires us to extract tubular structures in two steps: (1) predicting the location of the two endpoints of each branch; (2) finding the optimal path between the two endpoints of each branch. Such a solution is intuitively aligned with human perception, and able to offer several advantages. Specifically, if the endpoints of branches are accurately predicted in the first step, redundant branches (FPs) are then potentially reduced; and the second step ensures that there is a path connecting the two endpoints of each branch, so that broken branches (FNs) and TEs are effectively suppressed. Besides, learning branch-level features during training elevates the model's focus on topology, which implicitly improves topological accuracy.

To effectively utilize branch-level features of tubular structures, we propose GraphMorph, a pipeline for obtaining topologically accurate centerline masks. GraphMorph consists of a Graph Decoder and a training-free Morph Module, which corresponds to the two steps of our solution respectively. The Graph Decoder, given multi-scale features extracted from an image patch by a segmentation network, predicts the graph G of the tubular structure. G is defined by a node set V = { ( x i , y i ) } N i =1 , which contains the coordinates of all critical points vital for maintaining topology, and an adjacency matrix A ∈ { 0 , 1 } N × N , which encodes the connectivity among nodes. Each pair of connected nodes in the graph corresponds to two endpoints of a branch of the tubular structure, thus the Graph Decoder takes full advantage of branch-level features through this graph representation. Technically, the prediction of V is addressed as a set prediction problem, solvable by our modified version of Deformable DETR [49]. To efficiently obtain the adjacency matrix A , we design a lightweight link prediction module that capitalizes on the extracted node features. Concretely, since the number of nodes in each tubular structure may be different, we generate linear weights and biases dynamically conditioned on node features, and the adjacency list of each node is obtained from its corresponding linear parameters (See Figure 2).

The Morph Module, a core contribution of this work, is intended to obtain topologically accurate centerline masks. While studies in the image-to-graph task [38, 32] also utilize graph representation,

they struggle to directly obtain accurate centerline masks due to the curved nature of tubular objects. In contrast, our Morph Module generates topologically accurate centerline masks via a novel SkeletonDijkstra algorithm. Specifically, a centerline probability map P m , together with the graph G , output by the segmentation network and the Graph Decoder respectively, serve as the input to the Morph Module. Afterwards, considering the skeleton property of centerlines, our SkeletonDijkstra algorithm finds the optimal path between each pair of connected nodes. In particular, during path finding from the start point to the end point, we always restrict the path to a single pixel width to satisfy the skeleton property of centerlines. Consequently, the topology of the resulting centerline mask is guaranteed by G , leading to a reduction in TEs. This method also minimizes the occurrence of broken branches (FNs) and redundant branches (FPs), which is a significant improvement over direct pixel-level operations on P m , such as thresholding.

We conduct the experiments by beginning with the centerline extraction task to verify the effects of the two components of GraphMorph. Experimentally, serveing as an auxiliary training module to learn the graph representation, the Graph Decoder enhances the segmentation network's focus on branch-level features, thus both volumetric metrics and topological metrics are boosted. Furthermore, employing the Morph Module at inference stage considerably improves topological metrics. For the segmentation task, we develop a streamlined post-processing strategy to refine segmentation masks via the topologically accurate centerline masks output by the Morph Module, significantly suppressing false positives of segmentation results. To verify the effectiveness of GraphMorph and the post-processing strategy, we conduct extensive experiments across four typical tubular structure extraction datasets. We have applied our methodology on three powerful backbones and achieved consistent improvements in all metrics. Moreover, compared with the state-of-the-art methods, our approach achieves the best results across all datasets.

In a nutshell, our contributions can be summarized as the following: (1) We introduce GraphMorph, an innovative framework specifically tailored for tubular structure extraction. Based on the proposed Graph Decoder and Morph Module, the branch-level features are fully exploited and the topologically accurate centerline masks are derived naturally. (2) For the segmentation task, an efficient postprocessing strategy significantly suppresses false positives via the centerline masks predicted by GraphMorph, ensuring that the segmentation results are more closely aligned with the predicted graphs. (3) Experimental results on three medical datasets and one road dataset underscore the effectiveness of our method. For both centerline extraction and segmentation tasks, GraphMorph has achieved remarkable improvements across all metrics, especially in topological metrics.

2 Related Work

Image segmentation of tubular structures. Deep learning-based methods have achieved impressive results in segmentation tasks [21, 34, 5]. To further enhance the segmentation performance of tubular structures, novel network architectures [16, 22, 36, 41, 23, 45, 40, 33] and topology-preserving loss functions [14, 29, 37, 25, 33] have been proposed. For example, in terms of network architecture, DSCNet [33] utilizes dynamic snake convolution to capture fine and tortuous local features; PointScatter [40] explores the point set representation of tubular structures and introduces a novel greedy-based region-wise bipartite matching algorithm to improve training efficiency. In terms of loss functions, clDice [37] proposes a differentiable soft skeletonization method and achieves loss calculation at centerline level, which implicitly helps model focus more on the fine branches; TopoLoss [14] and TCLoss [33] measure the topological similarity of the ground truth and the prediction via persistent homology. Despite these advancements, all of the above methods are still confined to the framework of pixel-level classification and can not entirely overcome their inherent limitations. Our method attempts to morph the predicted graphs of tubular structures to let the network focus more on branch-level features, thus ensuring the topological accuracy of predictions.

Image to graph. There are two mainstream subtasks in this area: road network graph detection [11, 43, 38, 44, 12] and scene graph generation [17, 19]. These tasks usually entail detecting key components as nodes (i.e., key points in roads, objects in scenes) and determining their interrelations as edges (i.e., connectivity in roads, interactions in scenes). Our work differs from these approaches in three ways. Firstly, we use only junctions and endpoints as nodes, which allows for explicit semantic characterization of nodes in our graph representation, unlike road network detection tasks where path points may also be regarded as nodes. Secondly, considering the curved nature of tubular objects, we propose Morph Module to obtain topologically accurate centerline masks, a goal that is not addressed

Transformer

Encoder

Bipartite matching Loss

Pixel-wise Loss

weighted-BCE Loss

Condition MLP

Value MLP

Condition weights & bias

Linear &

Sigmoid

0.87  0.52  0.11  0.36

0.57  0.43  0.94  0.75

0.08  0.88  0.64  0.23

0.39  0.78  0.31  0.89

Ground truth

Probability map

Link prediction

module

Link prediction module

Graph construction

Adjacency matrices

Position encoding

Filter

Coordinate

head

Class head

Coords & Probs

Coords & Probs

Coords & Probs

Transformer

Decoder

...

...

...

Output queries

...

Node queries

Matched queries

Multi-scale feature tokens

Input image

Modified Deformable DETR

Graph Decoder

Segmentation network

ROI Align

...

...

...

...

...

...

！

X

1

0

0

X

1

0

0

1

1

0

1

1

1

0

1

0

0

X

0

0

0

1

0

X

0

0

x

X

0.52

0.11

0.36

0.57

X

0.94

0.75

0.08

0.88

0.23

×

0.76

0.39

0.78

0.31

0.78

0.92

0.19

0.15

0.88

0.07

0.39

0.14

0.09

0.76

0.23

0.45

0.87

by these works. Finally, our dynamic link prediction module is time-efficient, compared with elaborate and time-consuming designs in these works, such as [ rln ]-token in RelationFormer [38]. For a clear understanding, we experimentally compare the differences between our approach and RelationFormer in Appendix C. These distinctions make our model not only time-efficient but also applicable to the task of tubular structure extraction with more complex topology.

3 Method

This section provides a detailed description of the training and inference procedures of GraphMorph. Figure 2 illustrates the training process of our approach, where the segmentation network and Graph Decoder are included. We detail these two components in Section 3.1 and 3.2, respectively. The training details are given in Section 3.3. Section 3.4 introduces the algorithmic flow of the Morph Module, followed by the inference processes for the centerline extraction and segmentation tasks.

3.1 Segmentation Network

The segmentation network processes an input image I with shape H × W . It serves two purposes: (1) outputting a probability map of tubular structures; (2) providing multi-scale features for the Graph Decoder. For training efficiency, we randomly sample R regions of interest (ROIs) with size H × H in the feature maps ( R = 3 in Figure 2 for illustration). The ROI is defined as any region containing centerline points. The adoption of ROIs brings two key benefits: it reduces the model's learning complexity due to simpler topological structures within each ROI, and improves training efficiency by decreasing the number of feature tokens processed in the transformer. Technically, We adopt ROI Align [10] to extract multi-scale ROI features. Note that the generality of GraphMorph allows it to be adapted to any type of segmentation network. In the experimental part, we validate the enhancement of GraphMorph on a variety of segmentation networks.

3.2 Graph Decoder

The Graph Decoder is intended to predict the graph for each ROI. Specifically, the modified Deformable DETR [49] is responsible for detecting the nodes, while the link prediction module handles predicting the connectivity among these nodes. In the following, we will dissect each component to elucidate their roles.

Modified Deformable DETR. Wehave made two adjustments to the standard Deformable DETR [49]. Firstly, since the targets are nodes with only 2-dimensional coordinates, we replace the original box head with a coordinate head that outputs 2-dimensional vectors. Secondly, considering the typically small size of ROIs, we reduce the number of layers in the transformer encoder to three while keep the decoder at six layers. Note that each ROI is treated as an independent sample and different ROIs will not interact with each other in the whole training process. Formally, the process of node prediction can be expressed as follows:

where r = 1 , 2 , ..., R . F r ∈ R L × C denotes the multi-scale features of the r -th ROI, and Q ∈ R K × C is the initial node queries, where L and K are the scale of feature maps and the number of node queries respectively. PE is the multi-scale sinusoidal positional encoding used in [49]. ClassHead is a single linear layer, and CoordHead is a 3-layer multilayer perceptron (MLP). ˆ s r ∈ R K and ˆ v r ∈ R K × 2 are the classification scores and coordinates of the nodes for the r -th ROI respectively.

Link prediction module. Since the number of nodes for each ROI may be different, we design a dynamic module generating linear weights and biases conditioned on node features to directly predict the adjacency matrix A . For the r -th ROI sample, ̂ Q r ∈ R K × C is the output queries of the Transformer decoder. The queries matched with the ground truth nodes (the number is denoted as P r ) during the bipartite matching process are preserved, and the rest queries are filtered out. We denote the kept queries as ˜ Q r ∈ R P r × C . As depicted in Figure 2, the matched queries ˜ Q r will be fed into two MLPs. The ConditionMLP generates a ( C +1 )-dimensional vector for each matched query, which serves as the weights and biases of the condition linear layer. The ValueMLP maps the queries to a value space. With the values as input, the condition linear layer of the p -th query generates the adjacency list of it. The process can be formulated as follows:

where p = 1 , 2 , ..., P r . Here, ˜ Q r p denotes the p -th item of ˜ Q r , and C is its dimension. W r p refers to the linear parameters conditioned on the p -th matched query. ˜ A r p represents the predicted adjacency list for the p -th matched query, and the concatenation of all lists forms the final adjacency matrix ˜ A r .

3.3 Training Details

Graph construction. To train the Graph Decoder, we represent the ground truth of each ROI as a graph (see Figure 2). The detailed graph construction process can be found in Appendix B.

Label assignment based on bipartite matching. Bipartite matching is widely used in solving set prediction problems [2, 49, 40]. As in [49], we first calculate the cost between the predicted and ground truth nodes. The predicted nodes are denoted as ˆ y = { (ˆ s k , ˆ v k ) } K k =1 , where we omit the index r of the ROI sample for simplicity. Under the general assumption that K is larger than the number of ground truth nodes P r , thus we pad the set of ground truth nodes with ∅ (no node) to achieve a size of K . The ground truth set can be denoted as y = { ( c i , v i ) } K i =1 , where c i is the target class label and v i ∈ [0 , 1] 2 is the coordinate of the node. For a permutation σ ∈ S K , where ˆ y σ ( i ) is assigned to y i ( i = 1 , 2 , .., K ), we define the cost between y i and ˆ y σ ( i ) as:

̸

̸

where λ class and λ coord are hyperparameters. L class (ˆ s σ ( i ) ) = L focal (ˆ s σ ( i ) , 1) - L focal (ˆ s σ ( i ) , 0) , L focal ( s, c ) is defined as -α · (1 -s ) γ log( s ) if c = 1 , and -(1 -α ) · s γ log(1 -s ) if c = 0 , where α and γ are hyperparameters. L coord is commonly used ℓ 1 loss. The optimal ˆ σ is defined as

This optimal assignment can be efficiently obtained by Hungarian algorithm [18].

Loss functions. To train the Graph Decoder, the overall loss function is comprised of three components: pixel-wise loss L Pixel between the probability map and the ground truth binary mask (in this work, we use softDice [26] and clDice [37]), Hungarian bipartite matching loss L Hungarian between the predicted and ground truth nodes, weighted-BCE loss L Adjacency between the predicted and ground truth adjacency matrices of the matched queries. For an image with R ROI samples, the last two loss functions are defined as:

̸

̸

̸

where N pos is the total number of positive locations in ground truth { A r } R r =1 , and N neg is the total number of negative locations. Thus, the overall loss function is L total = L Pixel + L Hungarian + L Adjacency .

3.4 Morph Module and Inference

The Morph Module is used to get topologically accurate centerline masks, by morphing the predicted graphs from the Graph Decoder. In this subsection, we first introduce the Morph Module, followed by the inference processes for the centerline extraction and segmentation tasks.

Morph Module. We present the algorithmic flow in Algorithm 1. In particular, G = { V, E } is the graph of an image patch (same size as an ROI sample), and P m is the probability map of centerlines obtained from the segmentation network. We iterate over each edge and use our proposed SkeletonDijkstra algorithm to find the optimal path with minimum cost. The union of these paths forms the final centerline mask.

SkeletonDijkstra is modified from Dijkstra algorithm [7]. We have made two key adaptations for centerline extraction: (1) To restrict the path to a single pixel width, ensuring the property of the skeleton, we mandate that all path points, except for the start and end points, satisfy N = 2 (where N is the number of centerline points in its eight neighbours, see Appendix B). (2) To suppress potential false-positive edges from the Graph Decoder, we exclude the paths with an average cost exceeding a threshold p thresh . These refinements optimize the algorithm to yield topologically accurate centerline masks. The detailed algorithmic flow of SkeletonDijkstra can be seen in Algorithm 2 in Appendix D.

Algorithm 1 Morph Module

Inference of centerline extraction. As depicted in Figure 3, the centerline extraction process begins with generating a centerline probability map via the segmentation network. Then, sliding window inference is employed across the entire image in Graph Decoder to obtain graphs for all split patches. Finally, the Morph Module produces the centerline mask for each patch, and the combination of these masks forms the complete centerline mask of the entire image.

Inference of segmentation. Since the segmentation probability map S m can not be used directly by the Morph Module, we first threshold S m to obtain segmentation mask S ′ m and skeletonize it into a centerline mask P ′ m . The distance from each pixel to the nearest centerline point in P ′ m is calculated

Input image

Threshold

Graph

Decoder

Morph

Module

Centerline probability

Predicted graphs

Final prediction

Segmentation network

Multi-scale features

Ground truth

Threshold-only result

(for comparison)

FN

P

772

G

M

and normalized to create a distance map D . Then the centerline probability map P m is obatined by P m = S m × (1 -D ) . Employing the Morph Module on P m yields a topologically precise mask M . To suppress false positives in S ′ m (especially isolated regions), a post-processing strategy is initiated from M 0 = M ⊙ S ′ m . This strategy involves iteratively expanding M 0 within the boundaries of S ′ m until stabilization. The stabilized mask M T is then taken as the final output. This approach, as confirmed by experiments, effectively diminishes false positives and enhances topological accuracy. The above-mentioned soft skeletonization method (from S m to P m ) and post-processing strategy introduce minimal time cost and are straightforward to implement, with detailes in Appendix E.1 and Appendix E.2.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate GraphMorph on three medical datasets and one road dataset. DRIVE [39] and STARE [13] are retinal vessel datasets commonly used in medical image segmentation. ISBI12 [1] contains 30 Electron Microscopy images to segment membranes. The Massachusetts Roads (MassRoad) dataset contains 1171 aerial images for road network extraction. We use the data splits for DRIVEand STARE provided in MMSegmentation [6]. For ISBI12, following previous works [35, 29], we split it into 15 images for training and 15 for testing. For MassRoad, we follow [40] to construct the training set, and the total 63 images of the official validation set and test set are used for testing.

Baselines. We adopt affluent baselines for comparison, including UNet [34], ResUNet [47], CSNet [30], DC-UNet [22], TransUNet [4], DSCNet [33] and PointScatter [40]. Particularly, we use LinkNet34 [3] and D-Linknet34 [48] as baselines for the MassRoad dataset. In addition, we compare with TopoLoss [14], which is a topology-based loss function.

Metrics. For the centerline extraction task, we use Dice [50], Accuracy (ACC) and AUC as volumetric metrics. For robust evaluation, we give a tolerance of a 5-pixel region around the ground truth centerline mask following [9]. We compute topological metrics following [40], including the mean absolute errors of β 0 , β 1 and the Euler characteristic. To compare fairly, we skeletonize the prediction before evaluation. For the segmentation task, we adopt Dice, clDice [37] and ACC as volumetric metrics and the same topological metrics as the centerline extraction task. Moreover, ARI (Adjusted Rand Index) [15] and VOI (Variation of Information) [24] are used to evaluate clustering similarity.

Implementation Details. For three medical datasets, we use randomly cropped 384 × 384 images for training. The size of ROI samples H is 32 and the stride of sliding window used in inference process is 30 . For MassRoad dataset, the cropped size is 768 × 768 , H = 48 and the stride is 45. For all experiments, we use 64 ROI samples per image ( R = 64 ) to train the Graph Decoder, and the number of node queries in the modified Deformable DETR is set to 100 ( K = 100 ). According to previous

experiences [38], the default hyperparameters used in loss functions are as follows: λ class = 0 . 2 , λ coord = 0 . 5 , α = 0 . 6 , γ = 2 . We use α = 0 . 75 for MassRoad due to the sparse nature of the road networks. For all types of segmentation networks, we use multi-scale features ranging from the lowest resolution to the 4 × downsampling of the original image as input of the Graph Decoder. More implementation details are introduced in Appendix A.

4.2 Main Results

We first verify the effectiveness of GraphMorph on the centerline extraction task. Then, considerable experiments are conducted on the more common segmentation task, demonstrating the powerful topological modelling capability of GraphMorph.

Centerline Extraction. In our experiments with UNet and softDice loss on four public datasets, detailed in Table 1, the inclusion of the Graph Decoder during training enables the network to learn branch-level features, leading to enhanced performance in both volumetric and topological metrics. During inference, the utilization of Morph Module results in a slight decrease in volumetric metrics; however, there is a notable enhancement in topological metrics, confirming that our network has effectively captured branch-level features of tubular structures. Overall, the combined use of the Graph Decoder and Morph Module showcases the ability to refine the segmentation network's performance, particularly in preserving the crucial topological characteristics. Our methods also beat previous SOTA Pointscatter [40] by a large margin..

Segmentation. In the initial phase of our segmentation experiments, we assessed the effectiveness of our method across different segmentation networks and datasets. As detailed in Table 2, enhancements were evident in various metrics for all dataset and network combinations. The improvements in

volumetric and distribution metrics underscore our method's effectiveness in the precision and reliability in segmenting and clustering accuracy. Most notably, the substantial advancements in topological metrics, particularly in the β 0 error, highlight our method's proficiency in capturing the intricate branch-level features of tubular structures.

Moreover, the comprehensive comparisons presented in Table 3 across various datasets underscore the superiority of our method over current state-of-the-art techniques. The results unequivocally illustrate that our approach excels in all evaluated metrics, outstripping other methodologies. The crux of this advancement lies in the exploitation of branch-level features. Compared to our approach, traditional pixel-level classification strategies, whether employing grid representations like softDice [26] or clDice [37], or point representations akin to Pointscatter [40], inherently lack in capturing the essential branch-level features. Innovatively, Our method incorporates the graph representation to capture explicit branch-level features, and morphs the predicted graphs to topologically accurate centerline masks, which are subsequently utilized for post-processing. The excellent performance of various metrics, especially topological metrics, proves the validity of our approach.

4.3 Ablation Study

Size of ROI (H). See Table 4, the experimental results on the DRIVE dataset suggest that a default ROI size of H = 32 provides a favorable balance across the evaluation metrics. Specifically, when H is reduced, an increase in β 0 error is observed, indicating diminished topological accuracy. Conversely, increasing H does not offer substantial metric improvements and leads to higher computational demands. Hence, H = 32 is adopted as the default ROI size for all three medical datasets. Similarly, the default ROI size for the MassRoad dataset has been determined to be H = 48 .

Threshold in the Morph Module ( p thresh ). We conduct experiments on the centerline extraction task. A smaller value of p thresh denotes a more stringent selection criterion, with p thresh = 1 . 0 indicating that all paths are considered without any selection filter. As detailed in Table 5, the best results were achieved at p thresh = 0 . 5 , which is adopted as the default setting across all experiments.

Post-processing on the Segmentation Task. As shown in Table 6, incorporating post-processing has led to a slight improvement in volumetric metrics and a significant elevation in topological metrics. Notably, the substantial enhancement in the β 0 metric primarily results from the successful suppression of false positives, which aligns with our initial hypothesis. For a qualitative demonstration, we direct the reader to the visual comparisons presented in the Appendix G.2.

4.4 Qualitative Results

We qualitatively analyze the role of GraphMorph in reducing FNs, FPs, and TEs on the segmentation and centerline extraction tasks, as detailed in Figure 4. The results show that our method can effectively reduce the three types of errors in both tasks. This is due to the utilization of branch-level features during training and the effective design of inference process. Furthermore, we visualize the graphs predicted by the Graph Decoder as well as the role of the Morph Module in suppressing various errors in Appendix G.1.

Image

softDice

clDice

Label

softDice

Label

Segmentation

Centerline Extraction

5 Conclusion

This paper introduces GraphMorph, a framework that diverges from traditional pixel-level prediction methods in tubular structure extraction. By integrating two core components, the Graph Decoder and the Morph Module, GraphMorph adaptly captures and leverages branch-level features. Equipped with our proposed link predcition module and SkeletonDijkstra algorithm, the training and inference processes of the network are efficiently carried out. For the segmentation task, it further employs a straightforward yet effective post-processing strategy that substantially reduces false positives in the predictions. Extensive evaluations across various datasets for medical image segmentation and road network extraction have demonstrated the superiority of GraphMorph over existing methods, particularly in terms of topological metrics. This breakthrough not only boosts precision in applicationspecific tasks but also sets a robust foundation for future research in tubular structure extraction.

References

Ignacio Arganda-Carreras, Srinivas C Turaga, Daniel R Berger, Dan Cire¸ san, Alessandro Giusti, Luca M Gambardella, Jürgen Schmidhuber, Dmitry Laptev, Sarvesh Dwivedi, Joachim M Buhmann, et al. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in neuroanatomy , 9:152591, 2015.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Abhishek Chaurasia and Eugenio Culurciello. Linknet: Exploiting encoder representations for efficient semantic segmentation. In 2017 IEEE visual communications and image processing (VCIP) , pages 1-4. IEEE, 2017.

Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 , 2021.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 801-818, 2018.

MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation , 2020.

Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische mathematik , 1959.

Muhammad Moazam Fraz, Paolo Remagnino, Andreas Hoppe, Bunyarit Uyyanonvara, Alicja R Rudnicka, Christopher G Owen, and Sarah A Barman. Blood vessel segmentation methodologies in retinal images-a survey. Computer methods and programs in biomedicine , 108(1):407-433, 2012.

Pedro Guimaraes, Jeffrey Wigdahl, and Alfredo Ruggeri. A fast and efficient technique for the automatic tracing of corneal nerves in confocal microscopy. Translational vision science & technology , 5(5), 2016.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Songtao He, Favyen Bastani, Satvat Jagwani, Mohammad Alizadeh, Hari Balakrishnan, Sanjay Chawla, Mohamed M Elshrif, Samuel Madden, and Mohammad Amin Sadeghi. Sat2graph: Road graph extraction through graph-tensor encoding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16 , pages 51-67. Springer, 2020.

Congrui Hetang, Haoru Xue, Cindy Le, Tianwei Yue, Wenping Wang, and Yihui He. Segment anything model for road network graph extraction. arXiv preprint arXiv:2403.16051 , 2024.

AD Hoover, Valentina Kouznetsova, and Michael Goldbaum. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical imaging , 19(3):203-210, 2000.

Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. Topology-preserving deep image segmentation. Advances in neural information processing systems , 32, 2019.

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification , 2:193-218, 1985.

Qiangguo Jin, Zhaopeng Meng, Tuan D Pham, Qi Chen, Leyi Wei, and Ran Su. Dunet: A deformable network for retinal vessel segmentation. Knowledge-Based Systems , 178:149-162, 2019.

Siddhesh Khandelwal and Leonid Sigal. Iterative scene graph generation. Advances in Neural Information Processing Systems , 35:24295-24308, 2022.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Sanjoy Kundu and Sathyanarayanan N Aakur. Is-ggt: Iterative scene graph generation with generative transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6292-6301, 2023.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Ange Lou, Shuyue Guan, and Murray Loew. Dc-unet: rethinking the u-net architecture with dual channel efficient cnn for medical image segmentation. In Medical Imaging 2021: Image Processing , volume 11596, pages 758-768. SPIE, 2021.

Jie Mei, Rou-Jing Li, Wang Gao, and Ming-Ming Cheng. Coanet: Connectivity attention network for road extraction from satellite imagery. IEEE Transactions on Image Processing , 30: 8540-8552, 2021.

Marina Meil˘ a. Comparing clusterings-an information based distance. Journal of multivariate analysis , 98(5):873-895, 2007.

Martin J Menten, Johannes C Paetzold, Veronika A Zimmer, Suprosanna Shit, Ivan Ezhov, Robbie Holland, Monika Probst, Julia A Schnabel, and Daniel Rueckert. A skeletonization algorithm for gradient-based optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21394-21403, 2023.

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) , pages 565-571. Ieee, 2016.

Volodymyr Mnih. Machine learning for aerial image labeling . University of Toronto (Canada), 2013.

Volodymyr Mnih and Geoffrey E Hinton. Learning to detect roads in high-resolution aerial images. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part VI 11 , pages 210-223. Springer, 2010.

Agata Mosinska, Pablo Marquez-Neila, Mateusz Kozi´ nski, and Pascal Fua. Beyond the pixelwise loss for topology-aware delineation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3136-3145, 2018.

Lei Mou, Yitian Zhao, Li Chen, Jun Cheng, Zaiwang Gu, Huaying Hao, Hong Qi, Yalin Zheng, Alejandro Frangi, and Jiang Liu. Cs-net: Channel and spatial attention network for curvilinear structure segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part I 22 , pages 721-730. Springer, 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Chinmay Prabhakar, Suprosanna Shit, Johannes C Paetzold, Ivan Ezhov, Rajat Koner, Hongwei Li, Florian Sebastian Kofler, and Bjoern Menze. Vesselformer: Towards complete 3d vessel graph generation from images. In Medical Imaging with Deep Learning , pages 320-331. PMLR, 2024.

Yaolei Qi, Yuting He, Xiaoming Qi, Yuan Zhang, and Guanyu Yang. Dynamic snake convolution based on topological geometric constraints for tubular structure segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6070-6079, 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Mojtaba Seyedhosseini, Mehdi Sajjadi, and Tolga Tasdizen. Image segmentation with cascaded hierarchical models and logistic disjunctive normal networks. In Proceedings of the IEEE international conference on computer vision , pages 2168-2175, 2013.

Seung Yeon Shin, Soochahn Lee, Il Dong Yun, and Kyoung Mu Lee. Deep vessel segmentation by learning graphical connectivity. Medical image analysis , 58:101556, 2019.

Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoern H Menze. cldice-a novel topologypreserving loss function for tubular structure segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16560-16569, 2021.

Suprosanna Shit, Rajat Koner, Bastian Wittmann, Johannes Paetzold, Ivan Ezhov, Hongwei Li, Jiazhen Pan, Sahand Sharifzadeh, Georgios Kaissis, Volker Tresp, et al. Relationformer: A unified framework for image-to-graph generation. In European Conference on Computer Vision , pages 422-439. Springer, 2022.

Joes Staal, Michael D Abràmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging , 23(4):501-509, 2004.

Dong Wang, Zhao Zhang, Ziwei Zhao, Yuhang Liu, Yihong Chen, and Liwei Wang. Pointscatter: Point set representation for tubular structure extraction. In European Conference on Computer Vision , pages 366-383. Springer, 2022.

Yan Wang, Xu Wei, Fengze Liu, Jieneng Chen, Yuyin Zhou, Wei Shen, Elliot K Fishman, and Alan L Yuille. Deep distance transform for tubular structure segmentation in ct scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3833-3842, 2020.

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2 , 2019.

Zhenhua Xu, Yuxuan Liu, Lu Gan, Yuxiang Sun, Xinyu Wu, Ming Liu, and Lujia Wang. Rngdet: Road network graph detection by transformer in aerial images. IEEE Transactions on Geoscience and Remote Sensing , 60:1-12, 2022.

Zhenhua Xu, Yuxuan Liu, Yuxiang Sun, Ming Liu, and Lujia Wang. Rngdet++: Road network graph detection by transformer with instance segmentation and multi-scale features enhancement. IEEE Robotics and Automation Letters , 2023.

Ziyun Yang and Sina Farsiu. Directional connectivity-based segmentation of medical images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11525-11535, 2023.

Tongjie Y Zhang and Ching Y. Suen. A fast parallel algorithm for thinning digital patterns. Communications of the ACM , 27(3):236-239, 1984.

Zhengxin Zhang, Qingjie Liu, and Yunhong Wang. Road extraction by deep residual u-net. IEEE Geoscience and Remote Sensing Letters , 15(5):749-753, 2018.

Lichen Zhou, Chuang Zhang, and Ming Wu. D-linknet: Linknet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 182-186, 2018.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

Kelly H Zou, Simon K Warfield, Aditya Bharatha, Clare MC Tempany, Michael R Kaus, Steven J Haker, William M Wells III, Ferenc A Jolesz, and Ron Kikinis. Statistical validation of image segmentation quality based on a spatial overlap index1: scientific reports. Academic radiology , 11(2):178-189, 2004.

Appendix

A Implementation Details

In this section, we provide more implementation details of GraphMorph.

For fair comparison to previous works like PointScatter [40], we use the ADAM optimizer with the initial learning rate 1e-3 and cosine learning rate schedule with warm-up strategy to train the network. The weight decay is set to be 1e-4 uniformly. We train the network for 3K iterations for the three medical image datasets, and 10K for MassRoad. We use batchsize=4 for all datasets. We implement GraphMorph based on PyTorch [31] and Detectron2 [42].

Our SkeletonDijkstra algorithm is designed to run solely on CPU due to its computational nature.We use p thresh = 0 . 5 across all experiments. To enhance performance and efficiency, we have implemented this algorithm in C++. For a detailed understanding of the algorithm, refer to the pseudo-code provided in Appendix D.

B Graph Construction

The process of constructing a graph can be briefly summarized in three steps as follows: (1) Generate the centerline mask of the tubular structure using skeletonization algorithm [46]; (2) Analyze each centerline point P by counting the centerline points among its eight neighbors (denoted as N , not including P ). Define P as a junction if N ≥ 3 and as an endpoint if N = 1 . Points with N = 2 are path points and are not considered as nodes. Junctions and endpoints form the node set V of the graph G . (3) If there is a pathway consisting of only path points between two nodes, then there is an edge between them. All edges form the edge set E . We use publicly accessible implementations of skeletonization 1 and graph construction. 2

However, the resultant graph may contain elements such as loops (closed paths where a node connects back to itself) and multiple edges (more than one edge connecting the same pair of nodes). Addressing both these elements is crucial; otherwise, reconstructing such structures during the inference process would be challenging. As depicted in Figure 5, to manage these complexities, we introduce new nodes in the following manner:

Loops : We insert new nodes at selected points within the loop to break the cycle.

Multiple edges : After resolving loops, we then add nodes along edges where multiple connections exist between the same pair of nodes.

These modifications ensure that the graph structure is simplified and ready for more effective processing during inference.

Loop

Multiple edges

(a)

Sum

Sum

(b)

Go

0

1

0

0

2

+

1

1

04

4

00

2

3

0

0

0

1

C Link Prediction Module

To validate the effectiveness of our dynamic link prediction module, we compare it with the approach used in RelationFormer [38].

RelationFormer learns an additional [ rln ]-token during the training of DETR to encode the relationships between node queries. In the inference stage, to predict the connection between two nodes, the method concatenates the features of the two nodes with the [ rln ]-token into a single vector. This vector is then processed by a three-layer MLP to predict the probability of connection between the nodes. Let us assume that there are P matched queries, denoted as ˜ Q r ∈ R P × C . In RelationFormer, the connection probability for each node pair is computed individually, resulting in a computational complexity of O ( P 2 × C 2 ) . In contrast, our dynamic module achieves a complexity of O ( P × C 2 ) (Equation (4) to (6)), reducing the computational burden.

Table 7 presents a comparison between the [ rln ]-token and our dynamic module on the task of centerline extraction. The metrics "Node Detection" and "Edge Detection" are reproduced from RelationFormer, which measure the accuracy of the nodes and edges extracted by the Graph Decoder. Other metrics assess the accuracy of the centerline masks output by the Morph Module. All experiments are based on UNet. The results demonstrate that both methods achieve comparable performance. However, our method significantly reduces the computational complexity, thereby shortening the inference time. This enhancement makes our approach more suitable for applications requiring fast processing speeds without sacrificing performance.

D SkeletonDijkstra Algorithm

The pseudo-code of our SkeletonDijkstra algorithm is given in Algorithm 2, which finds the optimal path satisfying the skeleton nature for two points.

E Details of Processing in Segmentation

E.1 Soft Skeletonization

See soft _ skeleton function in Listing 1. In the segmentation task, the segmentation network outputs the segmentation probability S m , which we need to soft skeletonized into the centerline probability P m for input into the Morph Module.

E.2 Post-processing to Suppress False Postives

See dilate _ with _ seg _ limit function in Listing 1. In the segmentation task, after obtaining topologically accurate centerline masks by the Morph Module, false positives can be greatly suppressed with this post-processing strategy.

F Computational Resources

Hardware Configuration. Experiments were conducted using an NVIDIA GeForce RTX 3090 with 24 GB GPU memory. The CPU used was an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, which features 28 cores.

Analysis of training process. Training on the DRIVE dataset with a UNet backbone and a batch size of 4 using "SoftDice+Ours" method requires approximately 11.8 GB of GPU memory, compared

Algorithm 2 SkeletonDijkstra Algorithm

to 5.4 GB of "SoftDice". More comparison between these two methods are show in Table 8. The increase in parameters and FLOPs in our approach primarily stems from the integration of the Graph Decoder featuring a DETR module. This component is crucial for predicting accurate topological structures of the graphs. Advancements in transformer architectures that reduce computational overhead could potentially enhance the efficiency of our model during training.

Inference time analysis. The inference times for each model component are summarized in Table 9. The data presented in the table was obtained by processing a 384 × 384 image patch from the DRIVE dataset. ROI size is H = 32 , with a sliding window stride of 30. As shown in the table, significant time is concentrated on the Morph Module. The primary time expenditure currently arises from processing the patches sequentially in our sliding window strategy. However, as each patch operates independently, there is significant potential to enhance efficiency by parallelizing the computations of all patches. Recognizing this opportunity, we plan to focus future work on optimizing the Morph Module by implementing parallel processing techniques to accelerate inference.

Listing 1 Python codes of the soft skeletonization operation and post-processing strategy during the inference process of the segmentation task.

G More Visualization Results

G.1 Visualization of Predicted Graphs

We visualize the graphs predicted by the Graph Decoder within the context of the centerline extraction task and analyze the role of the Morph Module. As shown in Figure 6, the first four rows illustrate the Graph Decoder's robust capability to predict graphs. By comparing the final predicted results obtained through the Morph Module (last column) with those obtained by thresholding P m at 0.5 (fourth column), it is evident that issues such as redundant and broken branches are effectively mitigated. However, the Morph Module also has limitations. Notably, the setting of p thresh might lead to overlooking some true-positive edges due to inaccuracies in P m , which is illustrating in the last two rows in Figure 6. This highlights areas for future improvement.

G.2 Visualization of Effect of Post-processing on the Segmentation Task

Figure 7 showcases the impact of post-processing in mitigating false positives within segmentation tasks, a procedure fully elaborated in Appendix E.2. This figure clearly reveals the elimination of isolated regions, originally predicted by the segmentation networks, across all datasets. Notably, the excision of such regions-often minute in scale-exerts a nominal effect on volumetric metrics while markedly bolstering topological metrics. This enhancement in the integrity of topological metrics through post-processing is substantiated by the data in Table 6.

H Limitations

Despite the advancements offered by GraphMorph, the method exhibits certain limitations. First, the reliance on post-processing for the segmentation task indicates a potential underutilization of branch-level features. Although false postives are significantly suppressed, segmentation results are not always topologically aligned with predicted graphs, suggesting room for improvement in

DRIVE

DRIVE

STARE

ISBI12

ISBI12

MassRoad

Image Patch

Centerline GT

Probability(      )

Thresholded

Predicted Graph

segmentation performance. Additionally, the necessity to train on relatively small ROIs, due to the complex nature of tubular structures, requires sliding window technique during inference. This technique may not fully capture comprehensive branch-level details and the global context of the entire tubular structure. Based on the limitations, future developments will aim to refine segmentation algorithms to utilize predicted graphs directly, thereby reducing dependency on post-processing. Concurrently, efforts will also focus on the capability of processing larger fields of view in a single analysis, thus preserving global context and enhancing feature consistency across the entire structure.

I Additional Experiments on 3D Dataset

The application of GraphMorph to 3D medical datasets can be initially explored for its clinical significance. Thus, we have extended GraphMorph to use the 3D UNet architecture and tested it on the pulmonary arterial vascular segmentation dataset from the PARSE challenge, which includes 100 annotated 3D CT scans. These cases were divided in a 7:1:2 ratio for training, validation, and testing.

The preliminary results, as detailed in Table 10, show that our method consistently outperforms existing baselines across all metrics, mirroring the success we observed with 2D data. This alignment

Image

w.o. Post

Label

DRIVE

STARE

ISBI12

MassRoad

Label

SoftDice

between 2D and 3D results not only underlines the effectiveness of our method but also its adaptability to 3D vessel segmentation task, which indicates the potential of GraphMorph in clinical diagnosis. Moreover, Figure I demonstrates that GraphMorph effectively suppresses false positives (FPs) and false negatives (FNs) in fine structures. Further attempts on 3D datasets will be made to validate its effectiveness.

J Broader Impacts

In this work, we present GraphMorph, a framework aimed at improving the extraction of tubular structures in medical image analysis, such as blood vessels and other elongated anatomical features. Fine-scale structures often consist of interconnected branches forming cohesive networks critical to physiological functions. By enhancing topological accuracy, GraphMorph provides more coherent and precise representations of these structures. These improved predictions with better topology in medical diagnostic scenarios related to tubular structures may assist clinical diagnosis. While our results are promising, they are based on publicly available datasets that may not fully capture the complexity and variability of real-world clinical data; therefore, further validation on more 3D datasets is necessary to confirm its applicability in clinical settings. At the present stage, we do not foresee any potential negative societal impacts arising from our work. Our goal is to contribute a useful tool for the medical imaging community, supporting efforts to improve segmentation accuracy and ultimately aiding in better healthcare outcomes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The last sentence in Abstract ; the last two paragraphs of Introduction .

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The validity of the method is demonstrated primarily through experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a detailed description of the model architecture in Section 3, including figures, textual descriptions, and algorithmic flows. Hyperparameters are mainly provided in the "Implementation Details" paragraph of Section 4.1. There are also some details although described in detail in the main text, we provide the algorithmic flow and codes in Appendix D to ensure the reproducibility of the methodology.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: However, we provide training and infernece details exhaustively in Section 3 and 4.1. Additionally, codes of processes in segmentation are supplied in Appendix E. The information we have provided is sufficient to reproduce our work. The complete code will be made publicly available soon.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 4.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experimental setup in this study involved extensive testing across multiple datasets and with various types of backbone networks, consistently demonstrating improvements through our method. This extensive validation across diverse conditions ensures the reproducibility and reliability of the results. Additionally, the sheer volume of experiments conducted makes the computation of error bars highly time-consuming and computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work does not violate the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix J analyzes the positive impacts. We observe that there are no significant negative effects of the methodology.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is proposing an AI algorithm for medical assistance that does not run such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets used in the paper have been cited in their original literature. For the utilization of publicly available code the source is also indicated.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The code and model are not publicly available at this time. We will make them available later.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper presents a technique (called GraphMorph) for extracting tubular patterns as found for instance in retinal images (veins) and aerial images (roads). 
The proposed pipeline is made of several modules: a segmentation network providing centerline probability and features, 
a graph decoder for detecting nodes and connectivity and a morph module to get centerline masks.
Experimental results demonstrate the efficiency of the proposed pipeline against state of the art methods on several benchmarks.

STRENGTHS:
The paper is well written  and clear.
Experimental results against SOTA are convincing.

WEAKNESSES:
Some hyperparameters (e.g thresholds) have to be set.
The methodology is not always clear to set these parameters e.g.  
are experiments  for setting $p_{thresh}$ (lines 560-563 )   done on test sets?

minor remarks:
- Fig 2 "Modified Deformbale DETR" -> Deformable

- Figure 3 caption: "Inferece"

QUESTIONS:
are experiments  for setting $p_{thresh}$ (lines 560-563 )   done on test sets?

LIMITATIONS:
Limitations are addressed in the paper (e.g. in appendix I line 586, and in appendix H line 576-578).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

comments: SUMMARY:
In this work, the authors tackle curvilinear image segmentation. In particular, they move away from pixel-level prediction which has limitations when predicting thin structures. They propose GraphMorph which predicts location of endpoints of each branch and finds the optimal path between them. In this way, they are able to improve upon both False Positive and False Negative errors. Their method can be applied to any segmentation backbone.

STRENGTHS:
- Their method is able to handle both FP and FN errors. Otherwise, most existing methods in literature tend to tackle only FN errors.
-  The authors compare against adequate baselines and have improvements over them.

WEAKNESSES:
- For the performance mentioned in the tables, the authors should provide standard deviation to understand whether their method is statistically significant or not. The authors should conduct t-test [1] to confirm this.
- The authors should provide the computational complexity of their method with respect to the baselines. The complexity should be about the training and inference time of their method. I suspect the algorithm is slow due to iterating over each edge.

**References**

[1] Student, 1908. The probable error of a mean. Biometrika, pp.1–25.

QUESTIONS:
See weaknesses section

LIMITATIONS:
The authors have adequately mentioned the limitations of their method, specifically the small ROI which can miss spatial context necessary for good performance.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper proposes a method to extract tubular structures from images. Specifically, the authors have integrated graph extraction into the segmentation architecture. They have proposed a link prediction part to predict graph connectivity in tandem with the segmentation network. They combined the predicted graph into a segmentation branch via a morphing module during inference.

STRENGTHS:
1. The method is well-motivated, and the paper is well-written.
2. The link prediction idea is technically sound for computationally efficient connectivity prediction.
3. The morph module, which integrates the predicted graph in the segmentation, is novel. It seems effective, as demonstrated in Table 1 and Figure 6.
4. The proposed method seems to improve segmentation results across different metrics over multiple datasets.

WEAKNESSES:
1. Is the threshold in the SkeletonDijkstra algorithm dataset-dependent?
2. Is the segmentation network trained simultaneously with the graph decoder or one before the other?
3. Are there any metrics computed on the extracted centerlines?

QUESTIONS:
1. See weaknesses
2. I was wondering whether the morph module could also be used during training to correct some segmentation errors.

LIMITATIONS:
Limitations have been discussed well.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presented a method called GraphMorph, for tubular structure extraction to achieve more topologically accurate predictions. GraphMorph consists of a Graph Decoder and a Morph Module: 1) the Graph Decoder generates/decodes/learns a graph structure from the ouput of segmentation network and segmentation image features;  2) then the Morph Module processes the graph provided by the Graph Decoder and the centerline probability map given from the segmentation network by employing a novel SkeletonDijkstra algorithm, to output the final centerline mask. Various popular datasets are used for empirical validation.\n\nSTRENGTHS:\n1, The paper is well written with good clarity.\n2, The method proposed overall is unsuprising but logical (but definately not a breakthrough).\n\nWEAKNESSES:\nMy main questions are more related to the overall technical contributions to this line of work: a) The datasets are pretty small and have been extensively tested in the past. There is an over-fitting concern on these small and widely used datasets, in analogy to the \"peek too much\" bias. b) all examples are primarily 2D patches. From the qualitative examples shown in the paper, they looks more like \"toy examples\". Does the methods work on full 3D CT scans, like extracting lung airway trees? c) The numerical improvements reported in the paper, overall are marginal. \n\nAuthors makes claims as follows: \"Broader Impacts: The GraphMorph framework significantly enhances medical diagnostics by improving the accuracy of tubular structure extraction, such as blood vessels, which is crucial in AI-assisted medical image analysis. These advancements can lead to more precise diagnostics, potentially reducing unnecessary medical procedures and costs. Furthermore, if the capability for real-time analysis is further enhanced, it could significantly impact emergency medical responses by accelerating decision-making and improving patient outcomes. Ultimately, our research not only advances technical knowledge in medical image analysis, but also holds potential for significant positive impacts in healthcare efficiency and patient care.\"  \n\nThese claims are significantly over-claimed. The examples shown here are very much toy examples on small image patches. In real world healthcare diagnosis, robust 3D tubular structures like vessels for heart and lung, or airway trees need to be precisely extracted at certian depth from 3D CT scans. Topology constrained tubular structures are parsed from medical image analysis literature (there are many previous work on vessel extraction, and lung airway extraction). The method proposed here does not seem to work directly in such real applications. So authors need to be very careful about making such claims.\n\nQUESTIONS:\n1, The SkeletonDijkstra Algorithm is proposed by authors alone, any reference?\n2, My main questions are more related to the overall technical contributions to this line of work: a) The datasets are pretty small and have been extensively tested in the past. There is an over-fitting concern on these small and widely used datasets, in analogy to the \"peek too much\" bias. b) all examples are primarily 2D patches. From the qualitative examples shown in the paper, they looks more like \"toy examples\". Does the methods work on full 3D CT scans, like extracting lung airway trees? c) The numerical improvements reported in the paper, overall are marginal. \n\n3, Authors makes claims as follows: \"Broader Impacts: The GraphMorph framework significantly enhances medical diagnostics by improving the accuracy of tubular structure extraction, such as blood vessels, which is crucial in AI-assisted medical image analysis. These advancements can lead to more precise diagnostics, potentially reducing unnecessary medical procedures and costs. Furthermore, if the capability for real-time analysis is further enhanced, it could significantly impact emergency medical responses by accelerating decision-making and improving patient outcomes. Ultimately, our research not only advances technical knowledge in medical image analysis, but also holds potential for significant positive impacts in healthcare efficiency and patient care.\"  \n\nThese claims are significantly over-claimed. The examples shown here are very much toy examples on small image patches. In real world healthcare diagnosis, robust 3D tubular structures like vessels for heart and lung, or airway trees need to be precisely extracted at certian depth from 3D CT scans. Topology constrained tubular structures are parsed from medical image analysis literature (there are many previous work on vessel extraction, and lung airway extraction). The method proposed here does not seem to work directly in such real applications. So authors need to be very careful about making such claims.  \n\n4, The results here are far from clinically useful. on smaller datasets for evaluation, error bars or std should also be reported.\n\nLIMITATIONS:\nas weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}