<figure><img src="image_1.png" /></figure>

HAL
open science

# Deep vessel segmentation based on a new combination of vesselness filters

Guillaume Garret, Antoine Vacavant, Carole Frindel

► To cite this version:

Guillaume Garret, Antoine Vacavant, Carole Frindel. Deep vessel segmentation based on a new combination of vesselness filters. 2023. hal-04479662

HAL Id: hal-04479662

https://hal.science/hal-04479662v1

Preprint submitted on 28 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Deep vessel segmentation based on a new combination of vesselness filters

Guillaume Garret*<fnref n="1" />, Antoine Vacavant†², and Carole Frindel‡<fnref n="3" />

## 1 Abstract

Vascular segmentation represents a crucial clinical task, yet its automation remains challenging. Because of the recent strides in deep learning, vesselness filters, which can significantly aid the learning process, have been overlooked. This study introduces an innovative filter fusion method crafted to amplify the effectiveness of vessel segmentation models. Our investigation seeks to establish the merits of a filter-based learning approach through a comparative analysis. Specifically, we contrast the performance of a U-Net model trained on CT images with an identical U-Net configuration trained on vesselness hyper-volumes using matching parameters. Our findings, based on two vascular datasets, highlight improved segmentations, especially for small vessels, when the model's learning is exposed to vessel-enhanced inputs.

**Keywords** — Segmentation, Deep learning, Vessels, Vesselness filters, Liver, Brain

## 2 Introduction

The vascular system, by supplying nutrients and oxygen to the body's organs and removing waste products, plays a crucial role in maintaining the body's metabolic balance [1]. Various pathologies are directly linked to the circulatory system and can have severe consequences for organs. Additionally, understanding the distribution of blood vessels is essential for planning surgical procedures. Therefore, the digital reconstruction of the vascular network offers significant benefits for both diagnosis and preoperative planning. Despite its importance, the segmentation process remains primarily manual in practice, and the automation of this labor-intensive task has been a long-standing area of interest.

Efforts to reconstruct vascular structures have traditionally relied on algorithms such as active contour methods [2, 3] and model-based approaches like graph-cut techniques [4]. A common pre-processing step involves enhancing vessel contrast using vesselness functions. For instance, in the work by [4], Frangi's vesselness function is applied prior to a graph-cut, while [3] extends Frangi's function and extracts vessels using a level-set approach based on the Chan-Vese model.

Traditional rule-based methods have now been complemented by deep learning [5]. Kitrungrot-sakul et al. [6] employ three DenseNet models to extract hepatic vessel features from different planes, namely sagittal, coronal, and axial. Rougé et al. [7] segment vessels using a topological loss that leverages vascular skeletons. Yan et al. [8] introduce LVSSNet, a 3D U-Net incorporating multi-scale

<footnote n="1">¹,² Université Clermont Auvergne, CNRS, SIGMA Clermont, F-63000, Clermont-Ferrand, France</footnote>

<footnote n="3">³ Univ Lyon, INSA-Lyon, Université Claude Bernard Lyon 1, UJM-Saint Etienne, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69100, Lyon, France</footnote>

<footnote>*guillaume.garret@uca.fr</footnote>

<footnote>†antoine.vacavant@uca.fr</footnote>

<footnote>‡carole.frindel@creatis.insa-lyon.fr</footnote>
feature fusion blocks in both encoding and decoding paths, along with spatial attention modules in
skip connections replacing conventional concatenation.

In recent research efforts, the scientific community has turned its attention to leveraging the intricate tree-like structures of vascular networks, ushering in innovative methods that integrate Graph Neural Networks into segmentation workflows. This approach, exemplified by [9] and [10], introduces a dynamic interplay between a fully connected network, a graph attention network, and the inherent branching patterns of vessels. Simultaneously, researchers, as highlighted in [11] and [12], have been investigating the impact of vesselness filters on U-Net-based models. Notably, the latter study delves deeper into filter combinations through composite learning, where vesselness filters are strategically amalgamated, and multi-model learning, employing distinct filter types with subsequent result fusion.

Building on these advances, our work introduces a new paradigm by merging the knowledge of graph-based approaches for evaluation and filter-based approaches for data pre-processing before training. We propose a unique fusion method where vesselness filters are seamlessly integrated into the learning process of a deep neural network. This approach allows our model to discern and exploit the distinctive features enhanced by various filters, such as the Jerman filter's reinforcement of responses at bifurcations [13]. Unlike previous studies limited to specific filter types, especially those based on Hessian matrices, our work pioneers a more comprehensive exploration. By conducting experiments on diverse datasets, including liver (CTA) and brain (MRA), we showcase the versatility of our approach in improving the segmentation of vessels, particularly the finer structures. Furthermore, our evaluation methodology extends beyond conventional evaluation, introducing a novel partitioning approach inspired by [8] but adapted to our unique context. Importantly, we introduce a pioneering assessment of topological areas, such as bifurcations, in deep vessel segmentation, providing a more nuanced understanding of model performance.

# 3 Materials

## 3.1 Data

### 3.1.1 IRCAD

The 3Dircadb-01 dataset, abbreviated as IRCAD, comprises 20 publicly available contrast-enhanced CT thoracic scans. The dataset maintains an equal gender distribution, with 75% of patients exhibiting liver tumors.

The original dimensions of the coronal and sagittal planes remain constant at 512 voxels, while the axial plane dimension varies from 74 to 260. Voxel sizes are not uniform across patients and axes, ranging from 0.56 to 0.87 mm in the $\vec{x}$ and $\vec{y}$ directions, and from 1.0 to 4.0 mm in the $\vec{z}$ direction.

In the work of [14], images are resampled to homogenize voxel spacing. Each data undergoes resampling to its optimal resolution using a third-order B-spline interpolation. For instance, an image with voxel spacing (0.65 mm, 0.65 mm, 1.5 mm) is resampled to (0.65 mm, 0.65 mm, 0.65 mm).

We opt for this variant of IRCAD to ensure consistency with the application of vesselness filters (see Section 3.1.3).

### 3.1.2 Bullitt

The original Bullitt dataset [15] comprises a collection of 100 (T1, MRA) image pairs from healthy volunteers. However, only a portion of the dataset has been annotated for vessel segmentation purposes. For our experiments, we have access to 33 MRA images with sparsely annotated regions that have been manually refined [14]. The voxel spacing for the MRA images is (0.51 × 0.51 × 0.80 mm), with a volume size of (448 × 448 × 128 voxels).

Similar to our approach with the IRCAD dataset, we utilize the resampled version of the Bullitt dataset, aligning with its finest resolution through third-order B-spline interpolation [14].
<figure><img src="image_3.png" /><figcaption>Figure 1: Hyper-volume where filters are concatenated to the original scan, with each $Z_i$ depth corresponding to a different filter (Original, Frangi, Jerman, Sato, Zhang, Meijering, RORPO, respectively).</figcaption></figure>

### 3.1.3 Vesselness filters

In a significant effort, [14] introduces a benchmark evaluating key vesselness filters, including Frangi, Jerman, Sato, Zhang, Meijering, and RORPO (Ranking the Orientation Responses of Path Operators). The first five filters gauge tubularity based on the Hessian matrix, while RORPO adopts a morphological approach.

Building on the insights from this benchmark, we generate a vessel-enhanced version for the two datasets mentioned earlier. Here, the filters are concatenated to their respective original scans in the channel dimension, as illustrated in Fig. 1.

## 3.2 Experimental setup

The U-Net model, sourced from the MONAI implementation, comprises 4 depth levels and trained on a workstation equipped with a Quadro RTX 8000 GPU. The number of feature maps increases by a factor of 2 at each level, ranging from 16 to 128 at the deepest level. Training employs the Adam optimizer with an initial learning rate of $10^{-4}$. The loss function integrates a weighted combination of Dice loss and Focal loss, with equal weights assigned to each term. Patches of size (64 × 64 × 64) are extracted from the input images, with a 25% overlapping ratio. To address class imbalance, we ensure that at least 85% of the patches contain a foreground voxel. Data augmentation techniques, including random rotation, random flip, and Gaussian smoothing, are also applied.

To validate the generalizability of the U-Net, we adopt a cross-validation approach. For the Bullitt dataset, 15% of the data is allocated for the test phase, while the remaining 85% is divided into 5-folds for training and validation. In contrast, due to the smaller size of IRCAD, a dedicated test set is considered impractical. Instead, the entire dataset is split into 5-folds, with one sample from the validation subset dedicated as a test volume for each fold model. Despite its unconventional nature, we believe this hybrid approach, combining k-fold and leave-one-out cross-validation, offers a reasonable compromise considering the constraints of the IRCAD dataset. To enhance the robustness of our results, we also conduct an evaluation of the U-Net on the validation sets for IRCAD.

# 4 Experiments

## 4.1 Training

To assess the influence of vesselness filters on segmentation, we conducted experiments on two distinct datasets: IRCAD and Bullitt. For each dataset, we executed two sets of experiments—one using the original data and another with the vessel-enhanced data (e.g., IRCAD, IRCAD-enhanced, Bullitt, and Bullitt-enhanced). This approach facilitated a comprehensive examination of the impact of vesselness filters on segmentation across both datasets. Our hyperparameters closely align with those employed in two prior studies [12].
## 4.2 Evaluation

To assess the effectiveness of our models, we initially employ the Dice score. While the Dice score evaluates the overlap between a segmentation map and a reference, it tends to be more sensitive to larger structures than to thinner ones, presenting challenges given the morphology of vessels.

To address this sensitivity imbalance, we refine our evaluation by considering vessel size partitions, following the methodology outlined in [14]. The ground-truth undergoes a skeletonization process through a distance transform, generating a graph representing the vascular tree. Each identified branch is labeled uniquely, and a subsequent labeling is performed, assigning the maximum size of each branch as its label. Subsequently, all branches are grouped into classes—small, medium, and large—based on specific vessel size intervals, as detailed in Table 1. Notably, in the case of Bullitt's vessels, only two groups—small and medium—are used, owing to the lower variability in vessel size.

The final masks, denoted as $M_{large}$, $M_{medium}$, and $M_{small}$, are derived through a dilation process, employing a ball structuring element whose diameter adjusts based on the vessel size. This dilation method ensures the continuity between vascular masks.

<table><caption>Table 1: Vessel partitions according to vessel size in mm</caption><thead><tr><th></th><th>Small</th><th>Medium</th><th>Large</th></tr></thead><tbody><tr><th>IRCAD</th><td>[0, 3]</td><td>[3, 6]</td><td>[6, +∞[</td></tr><tr><th>Bullitt</th><td>[0, 0.513]</td><td>[0.513, +∞[</td><td>Ø</td></tr></tbody></table>

We further evaluate segmentations at vessel bifurcations. Here, bifurcations are localized given the adjacency matrix of the skeleton, then, the $M_{bif}$ mask is obtained by dilating the bifurcation voxels and intersecting the resulting image with the ground-truth, following the method described in [14].

Additionally, in the context of the vessel segmentation task, preserving the topology of structures is crucial, a factor not considered by the Dice score. Addressing this concern, we introduce the clDice score [16]. Unlike the traditional Dice score, the clDice metric utilizes the vessel skeleton to assess the preservation of vessel connectivity.

# 5 Results

## 5.1 IRCAD

Our analysis on the IRCAD dataset, detailed in Table 2(a), reveals a notable improvement in the mean Dice for small vessels with the application of enhancement filters, with an average deviation over +9%. However, a negative impact on large vessels is observed, likely due to signal loss in our enhanced dataset (see Fig. 2). Adjusting the scale space of filters may mitigate this.

Bifurcation segmentation slightly decreased in the test set, traced back to a specific sample's drop in Dice score under our vesselness configuration.

Additionally, vessel-enhancement improved the preservation of vascular topology, reflected in increased clDice scores across partitions. On test sets, improvements ranged from an average deviation of +1.47% (medium vessels) to +16% (small vessels), and on validation sets, from +4.16% to +19.7%. Visual representation is provided in Fig. 3.

## 5.2 Bullitt

For the Bullitt dataset, we have improved segmentation with regard to our both metrics, as shown in Table 2(b). This is consistent with the previous results, as the Bullitt's vessels are significantly smaller than those of IRCAD. Hence we are not impacted by large vessels signal loss. In addition, we note that the vesselness benefit is less for Bullitt, from +1.09% to +3.88%, for bifurcations and medium vessels respectively. This smaller average deviations can be explained by the fact that the
<figure><img src="image_4.png" /><figcaption>Figure 2: Signal loss for a large vessel. White: enhanced vessels; purple:ground truth; green:mask $M_{large}$.</figcaption></figure>

Bullitt dataset is almost noise-free compared to the IRCAD, which reduces the impact of the filters. Indeed, the Peak Signal-to-Noise Ratio (PSNR) for the original Bullitt of 19.04 ± 0.43 improved to just 21.56 ± 1.12 after enhancement, compared to IRCAD, which increase significantly from 9.35 ± 1, 22 to 19.74 ± 1.60 after enhancement. An illustration of segmentation is given Figure 3.
<figure><img src="image_5.png" /><figcaption>Figure 3: Example of vessel segmentation for IRCAD (left) and Bullitt (right) datasets. Top to bottom: Ground-truth, U-Net's output using original images, and U-Net's output using our proposed hyper-volumes.</figcaption></figure>

# 6 Conclusion

In this study, we introduced a novel fusion method for vesselness filters in filter-based learning, concatenating filters into a hyper-volume. Our comparative study, utilizing a U-Net model with fixed parameters, assessed the model's efficiency when trained on original images versus our proposed hyper-volumes. The results yielded valuable insights into the efficacy of our approach.

Notably, the incorporation of vesselness-based learning demonstrated an improvement in the Dice score, particularly for small vessels, and preserved vascular structure topology, as indicated by the clDice measure. We evaluated our approach on two diverse datasets—Bullitt for cerebral MR and IRCAD for hepatic CT applications—following a partitioning approach.

Looking ahead, our future work will delve into examining the contribution of the studied filters using eXplainable AI methods. We believe this approach could reveal the most influential filters for the models, enhancing our understanding of the model's behavior.
# 7 Acknowledgments

This project receives support from the Région Auvergne-Rhône-Alpes under the DAISIES project. We thank the NVIDIA Academic Hardware Grant Program for providing the GPU resources that have been instrumental in carrying out the experiments for this work. The MR brain images utilized in this study, obtained from healthy volunteers, were collected and provided by the CASILab at The University of North Carolina at Chapel Hill. These images are distributed through the MIDAS Data Server at Kitware, Inc.

# 8 Compliance with Ethical Standards

This research study was conducted retrospectively using human subject data made available in open access by the IRCAD Institute and TubeTK project. Ethical approval was not required as confirmed by the license attached with the open access data.

# References

[1] Ann Marie Kupinski and Glen Schifrin. *The vascular system*. Lippincott Williams & Wilkins, 2022.

[2] Zeng Ye-zhan et al. "Automatic liver vessel segmentation using 3D region growing and hybrid active contour model". In: *Computers in Biology and Medicine* 97 (2018), pp. 63–73. ISSN: 0010-4825. DOI: https://doi.org/10.1016/j.compbiomed.2018.04.014. URL: https://www.sciencedirect.com/science/article/pii/S0010482518300957.

[3] Tomasz Woźniak et al. "3D vascular tree segmentation using a multiscale vesselness function and a level set approach". In: *Biocybernetics and Biomedical Engineering* 37.1 (2017), pp. 66–77. ISSN: 0208-5216. DOI: https://doi.org/10.1016/j.bbe.2016.11.003. URL: https://www.sciencedirect.com/science/article/pii/S0208521616300717.

[4] Xiaoyu Guo et al. "A novel method to model hepatic vascular network using vessel segmentation, thinning, and completion". In: *Medical & Biological Engineering & Computing* 58.4 (Apr. 2020), pp. 709–724. ISSN: 1741-0444. DOI: 10.1007/s11517-020-02128-6. URL: https://doi.org/10.1007/s11517-020-02128-6.

[5] Fabianand Jaeger Isensee et al. "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation". In: *Nature Methods* 18.2 (Feb. 2021), pp. 203–211. ISSN: 1548-7105. DOI: 10.1038/s41592-020-01008-z. URL: https://doi.org/10.1038/s41592-020-01008-z.

[6] Titinunt Kitrungrotsakul et al. "VesselNet: A deep convolutional neural network with multi pathways for robust hepatic vessel segmentation". In: *Computerized Medical Imaging and Graphics* 75 (2019), pp. 74–83. ISSN: 0895-6111. DOI: https://doi.org/10.1016/j.compmedimag.2019.05.002. URL: https://www.sciencedirect.com/science/article/pii/S0895611118304099.

[7] Pierre Rougé et al. *Cascaded multitask U-Net using topological loss for vessel segmentation and centerline extraction*. 2023. arXiv: 2307.11603 [eess.IV].

[8] Qingsen Yan et al. "Attention-Guided Deep Neural Network With Multi-Scale Feature Fusion for Liver Vessel Segmentation". In: *IEEE Journal of Biomedical and Health Informatics* 25.7 (July 2021), pp. 2629–2642. ISSN: 2168-2208. DOI: 10.1109/JBHI.2020.3042069.

[9] Donghao Zhang et al. *Graph Attention Network based Pruning for Reconstructing 3D Liver Vessel Morphology from Contrasted CT Images*. 2020. arXiv: 2003.07999 [eess.IV].

[10] Seung Yeon Shin et al. "Deep vessel segmentation by learning graphical connectivity". In: *Medical Image Analysis* 58 (2019), p. 101556. ISSN: 1361-8415. DOI: https://doi.org/10.1016/j.media.2019.101556. URL: https://www.sciencedirect.com/science/article/pii/S1361841519300982.
[11] Abir Affane et al. "Robust deep 3D architectures based on vascular patterns for liver vessel segmentation". In: *Informatics in Medicine Unlocked* 34 (2022), p. 101111. ISSN: 2352-9148. DOI: https://doi.org/10.1016/j.imu.2022.101111. URL: https://www.sciencedirect.com/science/article/pii/S2352914822002489.

[12] Shanmugapriya Survarachakan et al. "Effects of Enhancement on Deep Learning Based Hepatic Vessel Segmentation". In: *Electronics* 10.10 (2021). ISSN: 2079-9292. DOI: 10.3390/electronics10101165. URL: https://www.mdpi.com/2079-9292/10/10/1165.

[13] Tim Jerman et al. "Enhancement of Vascular Structures in 3D and 2D Angiographic Images". In: *IEEE Transactions on Medical Imaging* 35.9 (Sept. 2016), pp. 2107-2118. ISSN: 1558-254X. DOI: 10.1109/TMI.2016.2550102.

[14] Jonas Lamy et al. "Vesselness Filters: A Survey with Benchmarks Applied to Liver Imaging". In: *2020 25th International Conference on Pattern Recognition (ICPR)*. Jan. 2021, pp. 3528-3535. DOI: 10.1109/ICPR48806.2021.9412362.

[15] Elizabeth Bullitt et al. "Vessel Tortuosity and Brain Tumor Malignancy: A Blinded Study1". In: *Academic Radiology* 12.10 (2005), pp. 1232-1240. ISSN: 1076-6332. DOI: https://doi.org/10.1016/j.acra.2005.05.027. URL: https://www.sciencedirect.com/science/article/pii/S1076633205005647.

[16] Suprosanna Shit et al. "clDice - a Topology-Preserving Loss Function for Tubular Structure Segmentation". In: *CoRR* abs/2003.07311 (2020). arXiv: 2003.07311. URL: https://arxiv.org/abs/2003.07311.