# Evaluation of techniques for automated classification and artery quantification of the circle of Willis on TOF-MRA images: The CROWN challenge

Iris N Vos, Ynte M Ruigrok, Edwin Bennink, Mireille R.E. Velthuis, Barbara Paic, Maud E.H. Ophelders, Myrthe A.D. Buser, Bas H.M. van der Velden, Chen Geng, Matthieu Coupet, et al.

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Iris N Vos, Ynte M Ruigrok, Edwin Bennink, Mireille R.E. Velthuis, Barbara Paic, et al. Evaluation of techniques for automated classification and artery quantification of the circle of Willis on TOF-MRA images: The CROWN challenge. Medical Image Analysis, 2025, 105, pp.103650. 10.1016/j.media.2025.103650. hal-05448957

HAL Id: hal-05448957

https://hal.science/hal-05448957v1

Submitted on 8 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

# Evaluation of techniques for automated classification and artery quantification of the circle of Willis on TOF-MRA images: The CROWN challenge

Iris N. Vos<sup>a,b,∗</sup>, Ynte M. Ruigrok<sup>b</sup>, Edwin Bennink<sup>a</sup>, Mireille R.E. Velthuis<sup>c</sup>, Barbara Paic<sup>a</sup>, Maud E.H. Ophelders<sup>b</sup>, Myrthe A.D. Buser<sup>d</sup>, Bas H.M. van der Velden<sup>a</sup>, Chen Geng<sup>e,f</sup>, Matthieu Coupet<sup>g,h</sup>, Félix Dumais<sup>i</sup>, Adrian Galdran<sup>j</sup>, Junyi Zhang<sup>k</sup>, Wei Liu<sup>l</sup>, Ting Ma<sup>l</sup>, Madhu S. Nair<sup>m</sup>, Mathieu Naudin<sup>g,n,o</sup>, Preena K.P. <sup>m</sup>, Keerthi A.S. Pillai<sup>m</sup>, Pengcheng Shi<sup>l</sup>, Thierry Urruty<sup>g,h</sup>, Yakang Dai<sup>e</sup>, Kaiyuan Yang<sup>p</sup>, Fabio Musio<sup>p</sup>, Bjoern H. Menze<sup>p</sup>, Birgitta K. Velthuis<sup>c</sup>, Hugo J. Kuijf<sup>a</sup>

## ARTICLE INFO

**Keywords:**

Classification

Quantification

MR angiography

Circle of Willis

## ABSTRACT

Assessing risk factors for intracranial aneurysm (IA) development on images is crucial for early detection of high-risk cases. IAs often form at bifurcations within the circle of Willis (CoW), but manual assessment of these arteries is both time-consuming and susceptible to inconsistencies. Previous studies on imaging markers for IA development lack sufficient evidence for clinical implications, highlighting the need for automated methods to assess CoW morphology. No systematic approach currently exists to identify the best methodological strategies. To address this, we organized a scientific challenge to compare various techniques against a clinical reference standard. Participants were tasked with (1) automated classification of CoW anatomical variants and (2) automated prediction of CoW artery diameters and bifurcation angles. We provided 300 TOF-MRA scans for training and another 300 for testing, all manually annotated. Submissions were evaluated using balanced accuracy, mean absolute error, and Pearson correlation coefficient metrics. This paper provides a detailed analysis of the results from six participating teams. The findings show that various methods may be suitable for automated CoW assessment, but that these need further improvement to meet clinical standards. The challenge remains open for future submissions, offering a benchmark for new techniques.

<footnote><sup>a</sup> Image Sciences Institute, UMC Utrecht, Utrecht, The Netherlands</footnote>

<footnote><sup>b</sup> Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands</footnote>

<footnote><sup>c</sup> Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands</footnote>

<footnote><sup>d</sup> Princess Máxima Center for Pediatric Oncology, 3584 CS Utrecht, The Netherlands</footnote>

<footnote><sup>e</sup> Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China</footnote>

<footnote><sup>f</sup> Jinan Guoke Medical Technology Development Co., Ltd, Jinan, China</footnote>

<footnote><sup>g</sup> XLIM Laboratory, University of Poitiers, UMR CNRS 7252, Poitiers, France</footnote>

<footnote><sup>h</sup> I3M, Common Laboratory CNRS-Siemens, University and Hospital of Poitiers, Poitiers, France</footnote>

<footnote><sup>i</sup> Department of Computer Sciences, Université de Sherbrooke, Sherbrooke, Canada</footnote>

<footnote><sup>j</sup> Universitat Pompeu Fabra, Barcelona, Spain</footnote>

<footnote><sup>k</sup> School of Electronic & Information Engineering, Suzhou University of Science and Technology, Suzhou, China</footnote>

<footnote><sup>l</sup> Electronic & Information Engineering School, Harbin Institute of Technology (Shenzhen), Shenzhen, China</footnote>

<footnote><sup>m</sup> Artificial Intelligence & Computer Vision Lab, Department of Computer Science, Cochin University of Science and Technology (CUSAT), Kochi 682022, Kerala, India</footnote>

<footnote><sup>n</sup> University Hospital, CHU, Poitiers, 86000, France</footnote>

<footnote><sup>o</sup> DACTIM-MIS/LMA Laboratory University of Poitiers, UMR CNRS 7348, Poitiers, 86000, France</footnote>

<footnote><sup>p</sup> Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland</footnote>

<footnote>* Corresponding author.
E-mail address: i.n.vos-6@umcutrecht.nl (I.N. Vos).</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 1.** Maximum intensity projections for various CoW variants using 3D TOF-MRA. Left: complete and symmetrical CoW (Lippert anterior class ‘A’ and posterior class ‘A’, see Fig. 2). Middle: Third A2 segment of the anterior cerebral artery and aplasia of both posterior communicating arteries (Lippert anterior class ‘C’ and posterior class ‘E’). Right: unilateral fetal-type posterior cerebral artery (Lippert anterior class ‘D’ and posterior class ‘J’).</figcaption></figure>

# 1. Introduction

Intracranial aneurysms (IAS) often develop on arteries of the circle of Willis (CoW), an arterial anastomotic circle providing blood to the brain. Rupture of an IA most often leads to aneurysmal subarachnoid hemorrhage (ASAH), a severe type of stroke with significant socio-economic implications (Nieuwkamp et al., 2009). Unruptured IAS occur in approximately 3% of the general population, and while most do not rupture, those that do often remain undiagnosed until rupture. Efforts to prevent ASAH are therefore focused on the early detection and preventive treatment of high-risk IAS. Individuals with a family history of ASAH are more prone to having IAS with a higher risk of rupture (Zuurbier et al., 2020), which indicates that genetic factors may play a role in IA development. This has prompted recommendations for long-term screening of first-degree relatives of ASAH patients using 3D time-of-flight magnetic resonance angiography (TOF-MRA), typically at five-year intervals, since IAS can develop at any time. However, the prevalence of unruptured IAS in such screenings is 11% during initial and 8% during follow-up screening (Bor et al., 2014). This shows that the screening process is still suboptimal. The identification of risk factors capable of identifying individuals at the highest risk for IAS would improve the screening process. High-risk individuals could be screened more frequently based on an initial evaluation of the MRA scan, whereas the screening frequency for low-risk individuals could be reduced or even stopped. In addition, these risk factors could lead to the identification of other high-risk groups. Previous studies have suggested several potential imaging markers predictive of IA development, including incompleteness or asymmetry of the CoW, as well as certain variations in bifurcation angles and diameters of major CoW arteries (Vos et al., 2024; Kancheva et al., 2022). However, the level of supporting evidence for such markers remains limited.

The manual evaluation of risk factors on images is labor-intensive and susceptible to inconsistencies both within and between annotators (El-Barhoun et al., 2009). Investigations on a larger scale to either strengthen existing evidence or uncover new risk factors for IA development therefore trigger the need for automated methods to assess CoW morphology. The Circle of Willis Intracranial Artery Classification and Quantification (CROWN) challenge is designed to explore what methodological strategy is most effective for automated assessment of the CoW using 3D TOF-MRA. The challenge is composed of two tasks: classification of the CoW configuration (Task 1) and quantification of the major CoW artery diameters and bifurcation angles (Task 2).

## 1.1. Related work

Relevant literature will be discussed for each task individually. Generally, there is limited literature on methods for automatically measuring CoW morphology. While some studies have addressed (multi)class segmentation of the CoW (Nader et al., 2023; Essadik et al., 2022), the

extraction of its configuration or artery diameters and their validation against clinical standards have not been thoroughly explored.

(1) *Classification of the CoW anatomical configuration:* Various classification schemes have been proposed to categorize CoW configuration variants. One widely cited system was proposed by Lippert and Pabst (1985), which categorizes both the anterior and posterior part of the CoW into 10 different classes each. Examples of CoW variation with corresponding Lippert classes are illustrated in Fig. 1. Another notable classification system, introduced by Lazorthes et al. (1979), employs 22 variants to describe CoW configurations. Instead of adhering to these established systems, some studies use different or even no classification system (Jones et al., 2021). This lack of consistency complicates comparisons across studies and results in a wide range of reported prevalences. Moreover, definitions for hypoplasia may vary, with artery diameters considered hypoplastic ranging from below 0.5 mm to below 1.0 mm. These diameters are often manually annotated at a single artery position, further complicating study-wide comparisons.

(2) *Quantification of the major CoW artery diameters and bifurcation angles:* Automated assessment of artery diameters is a well-established research area, with numerous studies focusing on intracranial, carotid, or pulmonary arteries. Typically, this is achieved through perpendicular reconstructions based on centerline extractions (Adam et al., 2021; Sedghi Gamechi et al., 2019; Avadiappan et al., 2020). While one study has specifically addressed automatically measuring diameters of CoW arteries, this method has not been validated with reference annotations (Dumais et al., 2022). Moreover, alternative methods that do not require a segmentation step may offer similar or improved performance in an end-to-end framework. A comparison of different methods would be highly valuable for various applications in different research areas. In contrast to artery diameters, research on bifurcation angle measurements remains less extensive. Few studies have explored the relationship between bifurcation angles and neurovascular diseases, all relying on manual annotations to calculate these angles (Kancheva et al., 2022; Noh and Kang, 2019; Zhang et al., 2018). Furthermore, as there is no current golden standard these annotation protocols can vary and may be prone to errors.

## 1.2. Contributions

The CROWN challenge serves as a public benchmark of new and existing techniques for automated classification and artery quantification of the CoW. This paper summarizes the results from teams participating in the CROWN challenge throughout 2023, which was held in conjunction with the 26th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in Vancouver, Canada. The paper adheres to the Biomedical Image Analysis challengeS (BIAS) guidelines (Maier-Hein et al., 2020) for transparent reporting of biomedical image analysis challenges.
<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Overview of tasks in the CROWN challenge. Upper left: 3D TOF-MRA scan used as the input for both tasks. Red boxes indicate the anterior (upper) and posterior (lower) CoW part. Lower left: Schematic representation of the circle of Willis (CoW), with arteries denoted in black and bifurcations in red. A1 = A1-segment of the anterior cerebral artery (ACA), A2 = A2-segment of the ACA, M1 = M1-segment of the middle cerebral artery (MCA), M2 = M2-segment of the MCA, ICA = internal carotid artery, PcoA = posterior communicating artery, P1 = P1-segment of the posterior cerebral artery, BA = basilar artery, VA = vertebral artery, C7 = C7-segment of the ICA, VBJ = vertebrobasilar junction. Upper right: Classification of CoW anatomical variants according to Lippert's system, figure adapted from (Ophelders et al., 2024). Lower right: Quantification of major CoW artery diameters and bifurcation angles.</figcaption></figure>

## 2. Materials and methods

### 2.1. Challenge organization

The challenge results were presented during a half-day event held at MICCAI 2023, with an open call for submissions beyond the conference. The organizing team consisted of Iris Vos, Ynte Ruigrok, Edwin Bennink, Myrthe Buser, Birgitta Velthuis, and Hugo Kuijf. The training data was released on May 1, 2023 under a Creative Commons Attribution-NonCommercial license (CC-BY-NC).

Participants gained access to the training dataset after registration on the hosting platform (https://crown.isi.uu.nl) and were encouraged to enhance the dataset using publicly available data. The challenge participants had no access to the test set. Developed algorithms had to be submitted for evaluation as Docker container images. Each team was permitted only one submission. Example code for evaluation and building Docker images were provided. Members of the organizers’

institutes could not participate in the challenge. Participants who submitted their results before the deadline for MICCAI 2023 (August 15, 2023), automatically qualified as co-authors. Per team a maximum of three co-authors are included. The CROWN challenge collaborated with the Topology-Aware Anatomical Segmentation of the Circle of Willis for CTA and MRA (TopCoW) challenge (Yang et al., 2023) by sharing a subset (N=20) of the training data.

### 2.2. Mission of the challenge

The objective of the challenge was to benchmark techniques for automated CoW classification and artery quantification in two tasks. Fig. 2 provides an overview for both these tasks. Further details for each task are provided in this section.

**Task 1:** Participants were asked to classify CoW configuration variants on 3D TOF-MRA scans using the scheme introduced by Lippert and Pabst (1985), in which a distinction is made between the anterior and
<table><caption>**Table 1**
Acquisition details of the challenge dataset.</caption><thead><tr><th></th><th>Training</th><th>Test</th></tr></thead><tbody><tr><td>Number of scans</td><td>300</td><td>300</td></tr><tr><td>Field strength (T)</td><td>1.5 (16%)<br>3.0 (84%)</td><td>1.5 (20%)<br>3.0 (80%)</td></tr><tr><td>Field of view</td><td>384 × 384 (&lt;1%)<br>400 × 400 (2%)<br>512 × 512 (4%)<br>560 × 560 (93%)<br>1024 × 1024 (&lt;1%)</td><td>512 × 512 (13%)<br>560 × 560 (83%)<br>576 × 576 (1%)<br>1024 × 1024 (3%)</td></tr><tr><td>Average number of slices</td><td>149 ± 13</td><td>141 ± 4</td></tr><tr><td>In-plane resolution (mm)</td><td>0.20 × 0.20 (3%)<br>0.36 × 0.36 (84%)<br>0.39 × 0.39 (13%)</td><td>0.36 × 0.36 (93%)<br>0.39 × 0.39 (4%)<br>0.50 × 0.50 (3%)</td></tr><tr><td>Slice thickness (mm)</td><td>0.5</td><td>0.5</td></tr><tr><td>Men : women</td><td>49% : 51%</td><td>43% : 57%</td></tr><tr><td>Average ± SD age (years)</td><td>43 ± 19</td><td>40 ± 18</td></tr></tbody></table>

the posterior CoW part with each part containing 10 separate classes (A-J). The desired classification output includes two classes per scan: one for the anterior CoW variant and one for the posterior CoW variant.

**Task 2:** This task was framed as a regression problem aimed at quantifying the diameter of 15 major arteries and the angle of 10 artery bifurcations forming the CoW, using weakly labeled data. The desired regression output contains 25 values per scan corresponding to the diameters and angles.

Considering the aim to improve the current screening process for IAs, a final application would be targeted on individuals with an increased risk for IA development, such as first-degree family members of patients with an unruptured or ruptured IA. The data utilized in the CROWN challenge was also derived from familial screenings cohorts, with the exclusion criterion being individuals diagnosed with IAs. This exclusion was to ensure that the presence of IAs at artery bifurcations did not interfere with the accurate assessment of diameters or angles in those locations.

## 2.3. Data description

### Data source

The dataset for the CROWN challenge was obtained from two cohorts of individuals screened between 1996 and 2021 for IAs at the University Medical Center Utrecht (UMCU), the Netherlands (Vos et al., 2024). From these cohorts, we selected the most recent 3D TOF-MRA scans from 600 individuals with a negative screen (i.e. no diagnosis of IAs). All individuals underwent a 3D T1-weighted gradient echo TOF-MRA acquisition on either a 1.5 T or 3.0 T magnetic resonance imaging (MRI) scan using a 32-channel head coil (Philips Healthcare, Best, the Netherlands) between 2008 and 2021. The study protocol for one of the cohorts was approved by the Medical Ethical Review Committee of the UMCU (approval number 16-777). For the other cohort, we received a waiver from the Medical Ethical Review Committee of the UMCU.

### Training and test cases

The challenge dataset included 300 training and 300 test cases, where each case contained one TOF-MRA scan and its annotations. Acquisition details are summarized in Table 1. To ensure sufficient training instances for each CoW variant, rare variants (<10 occurrences in the training set) were excluded. The images, provided as compressed NIfTI files, underwent full anonymization, including removal of all identifying health records and defacing. No pre-processing was employed. Training data was made available on DataverseNL (DataverseNL, 2023) after signing a confidentiality agreement.

## 2.4. Annotation characteristics

### Annotators

The following annotations were performed: two class labels corresponding to the anterior and posterior CoW variant (Task 1); 15 values corresponding to artery diameters (Task 2); and 10 values corresponding to bifurcation angles (Task 2). Annotations were provided to the participants as JSON files. The distribution of annotations for both training and test data is illustrated in Figs. 3 and 4.

Two annotators classified the CoW variant according to Lippert and Pabst's system as used in a previous study (Ophelders et al., 2024). The first annotator (BV) was an experienced neuroradiologist (> 20 years of experience). The second annotator (MV) was trained specifically for the task by this neuroradiologist.

Another three annotators (IV, MO, BP) underwent training specifically for the task and measured the artery diameters and bifurcation angles for Task 2. These measurements were obtained using semi-automatic methods (further described in the section below), followed by a visual check.

### Annotation protocol

For Task 1, the annotators were asked to classify the anterior and posterior CoW variant using manual diameter measurements.

For Task 2, the annotators were asked to measure diameters at the following locations (both left and right): the A2 segment of the anterior cerebral artery (ACA) at 5 mm beyond the anterior communicating artery, the A1 segment of the ACA at 50% of the total artery length, the M1 segment of the middle cerebral artery (MCA) at 50% of the total artery length, the P1 segment of the posterior cerebral artery (PCA) at 5 mm beyond the basilar artery top, the posterior communicating artery (PcoA) at 50% of the total artery length, the internal carotid artery (ICA) at 5 mm below its top, and the vertebral artery (VA) at 5 mm below the vertebrobasilar junction (VBJ). In addition, the basilar artery (BA) at 5 mm below its top was measured. Bifurcation angles were annotated as follows (again left and right): bifurcation between the A1 and A2 segment of the anterior cerebral artery, bifurcation at the ICA top, bifurcation between the M2 segments of the MCA, bifurcation between the PcoA and the C7 (terminal communicating) segment of the ICA. The VBJ and the BA top were also included. Angles were computed using extracted centerlines and vector directions originating from centerline intersections. In case of artery trifurcations, no annotation was performed. Certain cases may lack annotations because of discarded data and inherent anatomical variability (i.e. pronounced hypoplasia or aplasia of one or more arteries). Predicted outcomes for these arteries or bifurcations are excluded from evaluation. Given the subjective nature and labor-intensive demands of manually measuring diameters and angles on extensive datasets, semi-automatic methods were used to annotate the data for Task 2. These methods were implemented using MeVisLab software (Ritter et al., 2011, version 3.6.1). For the diameter annotations, we employed a method (Vos et al., 2022) that was previously validated on data from the same cohort. Annotators were asked to mark the desired artery location on the TOF-MRA, after which a local region-growing algorithm was employed to derive the artery centerline. Next, a full-width-at-half-maximum (FWHM) analysis was performed on a multi-planar reformation image perpendicular to the artery direction. This method, however, has two notable limitations: (1) due to its dependency on image resolution, the FWHM approach can result in inaccurate measurements for smaller arteries with less defined edges, especially those with diameters below 1.2 mm; and (2) arteries close to each other may result in erroneous measurements. Annotations were visually inspected by the annotator, and any diameters considered inaccurate were discarded.

Angle annotations were obtained using a pre-trained 3D U-Net (de Vos et al., 2021) for artery segmentation, followed by post-processing involving connected component analysis and skeletonization (Selle et al., 2002) to extract centerlines. Annotators were directed to identify
<figure><img src="image_7.png" /><figcaption>**Fig. 3.** Class distribution of training and test cases for Task 1. Anterior part: A, single anterior communicating artery (AcoA); internal carotid artery (ICA) bifurcates into precommunicating anterior cerebral artery segment (A1-ACA) and middle cerebral artery (MCA). B, Two or more AcoAs. C, Medial artery of corpus callosum arises from AcoA (3 A2 segments). D, Fusion of ACAs over short distance. E, ACAs merge into common trunk, then split distally. F, MCA originates as two separate trunks from ICA. G, Hypoplasia or absence of AcoA. H, One A1-ACA hypoplastic or absent. I, Hypoplasia or absence of ICA. J, Hypoplasia or absence of AcoA; MCA arises as two separate trunks. Posterior part: A, Bilateral posterior communicating arteries (PcoAs) present. B, Posterior cerebral artery (PCA) originates mainly from ICA, known as unilateral fetal-type PCA; opposite PcoA is patent. C, Bilateral fetal-type PCAs with both precommunicating segments (P1-PCAs) patent. D, Unilateral PcoA present. E, Hypoplasia or aplasia of both PcoAs. F, Unilateral fetal-type PCA and hypoplasia or absence of the P1-PCA. G, Unilateral fetal-type PCA and hypoplasia or absence of the contralateral PcoA. H, Unilateral fetal-type PCA and hypoplasia or absence of both a P1-PCA and PcoA. I, Bilateral fetal-type PCAs with hypoplasia or absence of both P1-PCAs. J, Bilateral fetal-type PCAs with hypoplasia or absence of one P1-PCA.</figcaption></figure>

the two arteries forming the desired bifurcation angle. Subsequently, angles were computed using the vector directions from the intersection point of the centerlines, with a distance of 5 mm over the centerline from the intersection. Segmenting smaller arteries poses challenges and can yield discontinuous centerlines (Yang et al., 2023). If an artery was partially or entirely invisible, the corresponding angle was not annotated and excluded from the dataset. In cases where the anterior communicating artery (Acom) was not visible, the angle between the A1 and A2 segments of the anterior cerebral artery (ACA) at the point of highest curvature was evaluated instead.

## Annotator variability

Thirty scans underwent duplicate annotation by a single annotator to assess intra-annotator variability for both tasks. Another set of 30

scans was annotated by a second observer to evaluate inter-annotator variability.

For Task 1, we assessed annotator variability using balanced accuracy (BAC). The intra- and inter-annotator BAC scores were 0.89 and 0.77, respectively, for the anterior classification of the CoW variant, and 0.85 and 0.72 for the posterior classification. We observed that most inconsistencies were associated with determining if AcoA or PcoA hypoplasia was present. Fig. 5 illustrates the Bland-Altman plots, suggesting no systematic bias. The limits of agreement for the intra-annotator variability were -0.24 and 0.26 mm for diameter annotations, and -10 and 10 degrees for angle annotations. For the inter-annotator variability, the limits of agreement were -0.24 and 0.24 mm for diameter annotations and -11 and 12 degrees for angle annotations.
<figure><img src="image_8.png" /><figcaption>Fig. 4. Distribution of training and test cases for Task 2. Upper: artery diameters. Lower: bifurcation angles. A2 = A2 segment of the anterior cerebral artery (ACA); A1 = A1 segment of the ACA; M1 = M1 segment of the medial cerebral artery (MCA); M2 = M2 segment of the MCA; ICA = internal carotid artery; C7 = C7 segment of the ICA; PcoA = posterior communicating artery; P1 = P1 segment of the posterior cerebral artery (PCA); BA = basilar artery; VA = vertebral artery; VBJ = vertebrobasilar junction.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 5. Bland-Altman plots illustrating the intra- and inter-annotator variability for the artery diameters (left column) and bifurcation angles (right column). The dotted black lines indicate the mean difference, or bias. The red lines show the 95% limits of agreement (LoA). Upper left: bias 0.01 mm, LoA [-0.24 and 0.26 mm]. Lower left: bias <0.01 mm, LoA [-0.24 and 0.24 mm]. Upper right: bias -0.3 degrees, LoA [-10 and 10 degrees]. Lower right: bias 0.2 degrees, LoA [-11 and 12 degrees].</figcaption></figure>

## 2.5. *Challenge evaluation*

### *Choice of metrics*

We followed the Metrics Reloaded (Maier-Hein et al., 2024) decision guidelines for an imbalanced multi-class classification challenge and used BAC as evaluation metric for Task 1. Correct classification of the anterior and posterior part of the CoW is considered equally important

and thus weighted equally. The results for Task 2 were evaluated using the Mean Absolute Error (MAE) and Pearson correlation coefficient. MAE scores are commonly used for regression tasks and provide an intuitive measure of the average error. The Pearson correlation coefficient helps to quantify the relationship between the predicted and reference values.
## Ranking scheme

We used a similar ranking scheme as in previously organized MICCAI challenges (Timmins et al., 2021; Kuijf et al., 2019). For Task 1, image-level BAC scores were computed for the anterior- and posterior CoW classification separately. The team achieving the best performance was assigned a rank of 0, while the team that performed least well was assigned a rank of 1. All other teams were then ranked relative to those performances. The ultimate ranking was determined by averaging the ranking position for both the anterior- and posterior rankings. Likewise, for Task 2, we obtained MAE and Pearson scores for each diameter or angle. Each team received 25 scores for each metric, and their rankings were relative to their respective performances. The winning team was chosen based on the lowest average of these ranking scores for both metrics.

## 2.6. Further analyses

The aforementioned choice of metrics and ranking strategy were used to establish the winners of the challenge associated with MICCAI 2023. In this section, we further elaborate on the methods and findings, e.g. by including additional metrics or evaluation of data from another institution.

### Evaluation per class, artery diameter, and bifurcation angle

To gain deeper insights into how predictions for specific classes or arteries contribute to overall performance, we assessed the performance of each method per class, artery diameter, and bifurcation angle. For Task 1, we report recall and F1-scores for each class in both anterior and posterior CoW classification. In addition, we examined the confusion matrices for each method. For Task 2, we provide MAE scores and Pearson correlations for each artery diameter and bifurcation angle, as well as a visual analysis of the correlation between predicted and reference values.

### Handling invalid model outputs

For the challenge outcome associated with MICCAI 2023, we addressed invalid model outputs for a single prediction (e.g. NaN or missing values) by substituting them with the largest predicted value corresponding to that artery or bifurcation. According to the guidelines in Metrics Reloaded, missing values are typically handled by assigning them the worst possible value. However, this approach is not straightforward for metrics without upper bounds, such as MAE, or for metrics assessing overall relationships, such as Pearson correlation. For Pearson correlation, assigning the worst value to a single prediction is not feasible, as it evaluates an overall trend. Similarly, adding the maximum MAE to the reference value of a missing value would overestimate the correlation. Therefore, we opted to substitute invalid outputs with the largest predicted value for that artery or bifurcation, specific to each team's predictions.

To provide more insight into the proposed solutions' strengths and limitations, we also analyzed the results with invalid predictions excluded.

### Model performance on data from another institution

We assessed the generalizability of the submitted solutions by evaluating the performance of 30 additional TOF-MRA scans that are part of the TopCoW'23 challenge (Yang et al., 2023). Cases annotated with a class not included in the CROWN training data were excluded from evaluation for anterior or posterior classification.

# 3. Results

## 3.1. Challenge submission

Fifty teams registered for the CROWN challenge and signed the confidentiality agreement between April and August 2023. We received final submissions from six teams representing five different countries, all of whom participated in the challenge session organized at MICCAI 2023.

## 3.2. Participating teams

This section includes the solutions to Task 1 and 2 proposed by each team, ordered based on their final rankings. Detailed descriptions of all methods are published on the Zenodo platform (Vos, 2025).

**Sibets&USTS (Task 1, Geng et al.):** The proposed method by Sibets&USTS employed intensity normalization to extract seed points for region growing segmentation. The resulting segmentations were used to obtain bounding boxes surrounding the CoW arteries. A ResNet50V2 architecture was used to train two separate networks that classified the anterior and posterior part, respectively. Data augmentation was performed using flipping, affine transformation, and gamma changes. The implementation code is publicly accessible (Geng et al., 2025).

**Snaillab (Task 2, Dumais et al.):** This team performed multi-class segmentation of the CoW arteries using previously developed software, eICAB (Dumais et al., 2022). The software is based on convolutional neural networks that were trained on 236 annotated images with a resolution of 0.625 mm isotropic and 18 labels surrounding the CoW. Diameter predictions were obtained by fitting spheres on a binary image mask. These predictions were followed by a harmonization algorithm to standardize diameters according to CROWN reference values. For the computation of the bifurcation angles, segmentations of both the entire brain and the CoW were used to provide skeleton images. Skeleton segments and junction points were extracted and used to compute the vectors and bifurcation angles.

**AlEntropy (Task 1 and 2, Shi et al.):** The proposed method employed graph neural networks (GNNs) for CoW segmentation, building upon their previous work (Shi et al., 2023). A vessel graph was derived from a vascular skeleton to replace the K-nearest neighbour (KNN) graph construction process, enhancing the representation of vascular structures. Additionally, the team included data from the OASIS-3 dataset (LaMontagne et al., 2019), which were labeled in-house using 24 anatomical classes, and employed pseudo-labels for the CROWN dataset during training. Segmentations were used to extract vascular centerlines and boundary distance maps using the distance transform. These components were combined with the multi-class segmentation masks to determine artery coordinates, diameters, and bifurcation angles. Finally, the diameter at key positions was used to classify the CoW variant using logical decision-making. Notably, the team was unable to classify the anterior CoW part due to missing variants in their in-house label classes. Relevant code is available on the Zenodo platform (Shi, 2024).

**Labcom I3M (Task 1 and 2, Coupet et al.):** Labcom I3M proposed an atlas-based approach with registration for CoW classification, that started by registering all images to a chosen reference image. Skeletonization in patient space was performed using segmentations based on entropy thresholding. Each node of interest was located by conducting a spatial search (with a radius of 4 mm) around the registered point to determine its position on the skeleton. Any nodes found beyond this range were disregarded. The identified nodes were subsequently used to address Task 2, by creating spheres that intersect with the skeleton of the artery of interest at a desired distance (5 mm or at 50% of the total artery length). Artery diameters were computed based on image reconstructions in the orthogonal plane at intersection points. Bifurcation angles were derived from two vectors based on the intersection points. Task 1 was addressed by matching the artery diameters with the identified nodes to perform the classification of the CoW variant.

**agaldran_segs (Task 1, Galdran et al.):** The proposed method used a similar approach to "agaldran_ims" in terms of architecture, adding a segmentation step. After obtaining the bounding boxes, a five-fold ensemble model, based on the Swin UNETR segmentation model (Hatamizadeh et al., 2021), was trained using the annotations from the TopCoW challenge. The (cropped) segmentations were subsequently used to train Densenet121 and EfficientNet B0.
<table><caption>**Table 2**
Results and ranking of proposed solutions by each team for Task 1 (classification of the anterior and posterior CoW variant) on the test set.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Anterior classification</th><th colspan="2">Posterior classification</th><th colspan="2">Challenge rank</th></tr><tr><th>BAC<sup>a</sup>↑</th><th>Rank score↓</th><th>BAC<sup>a</sup>↑</th><th>Rank score↓</th><th>Average rank score↓</th><th>Global rank↓</th></tr></thead><tbody><tr><td>Sibets&amp;USTS</td><td><strong>0.26</strong></td><td><strong>0.0</strong></td><td>0.27</td><td>0.2</td><td>0.1</td><td>1</td></tr><tr><td>Alntropy</td><td>0.20</td><td>0.6</td><td><strong>0.40</strong></td><td><strong>0.0</strong></td><td>0.3</td><td>2<sup>a</sup></td></tr><tr><td>Labcom I3M</td><td>0.25</td><td>0.2</td><td>0.20</td><td>0.4</td><td>0.3</td><td>2<sup>a</sup></td></tr><tr><td>agaldran_segs</td><td>0.21</td><td>0.4</td><td>0.13</td><td>0.8</td><td>0.6</td><td>4</td></tr><tr><td>agaldran_ims</td><td>0.20</td><td>0.8</td><td>0.14</td><td>0.6</td><td>0.7</td><td>5</td></tr><tr><td>DCS_CUSAT</td><td>0.20</td><td>1.0</td><td>0.11</td><td>1.0</td><td>1.0</td><td>6</td></tr><tr><td>DCS_CUSAT<sup>b</sup></td><td>0.28</td><td>0.0</td><td>0.20</td><td>0.4</td><td>0.2</td><td>2</td></tr></tbody></table>

The results of the best performing team per category (i.e. anterior and posterior classification) are highlighted in bold. Teams are sorted based on the average rank score. BAC = balanced accuracy. The arrows ↑ and ↓ indicate that higher or lower results correspond to better performance, respectively.

<table><caption>**Table 3**
Results and ranking of proposed solutions by each team for Task 2 (quantification of artery diameters and bifurcation angles) on the test set.</caption><thead><tr><th rowspan="2"></th><th colspan="4">Artery diameters</th></tr><tr><th>Average MAE (mm)↓</th><th>Rank↓</th><th>Average Pearson↑</th><th>Rank↓</th></tr></thead><tbody><tr><td><strong>Snaillab</strong></td><td><strong>0.44 ± 0.21</strong></td><td><strong>0.17</strong></td><td><strong>0.45</strong></td><td><strong>0.23</strong></td></tr><tr><td>Alntropy</td><td>0.50 ± 0.12</td><td>0.37</td><td>0.36</td><td>0.40</td></tr><tr><td>Labcom I3M</td><td>0.87 ± 0.26</td><td>0.97</td><td>0.14</td><td>0.87</td></tr></tbody><thead><tr><th rowspan="2"></th><th colspan="4">Bifurcation angles</th></tr><tr><th>Average MAE (degrees)↓</th><th>Rank↓</th><th>Average Pearson↑</th><th>Rank↓</th></tr></thead><tbody><tr><td>Snaillab</td><td>28 ± 5</td><td>0.75</td><td><strong>0.12</strong></td><td><strong>0.30</strong></td></tr><tr><td>Alntropy</td><td><strong>16 ± 5</strong></td><td><strong>0.05</strong></td><td>0.01</td><td>0.85</td></tr><tr><td>Labcom I3M</td><td>29 ± 6</td><td>0.70</td><td>0.06</td><td>0.35</td></tr></tbody><thead><tr><th></th><th colspan="2">Average rank score↓</th><th colspan="2">Global rank↓</th></tr></thead><tbody><tr><td>Snaillab</td><td>0.36</td><td></td><td>1</td><td></td></tr><tr><td>Alntropy</td><td>0.42</td><td></td><td>2</td><td></td></tr><tr><td>Labcom I3M</td><td>0.73</td><td></td><td>3</td><td></td></tr></tbody></table>

The results of the best performing team per category (i.e. artery diameters and bifurcation angles) and metric are highlighted in bold. Metric values are averaged across all scans and arteries or bifurcations. Teams are sorted based on the average rank score. MAE = mean absolute error. The arrows ↑ and ↓ indicate that higher or lower results correspond to better performance, respectively.

**agaldran_ims (Task 1, Galdran et al.):** The team proposed a method that first trained a model to obtain approximate localizations (bounding boxes) of the CoW. The resulting cropped images were used to train a classifier that employed two architectures: Densenet121 and EfficientNet B0. Both architectures were trained following a five-fold approach, resulting in an ensemble of ten models to predict the class.

**DCS_CUSAT (Task 1, Preena et al.):** The proposed deep learning framework comprised a feature extraction module and a classification module. The feature extraction part included two networks working in parallel: a 3D Residual Network that captured relevant features from the 3D MRA volume, and a 2D Residual Network that extracted spatial information from the 2D Maximum Intensity Projection (MIP) image derived from the input MRA volume. The combined features from these two networks were used as input for anterior and posterior classification of the CoW variant. Their code can be accessed on the Zenodo platform (Preena et al., 2025).

### 3.3. Pre- and post-processing

In addition to diverse model architectures (e.g., CNNs, GNNs, atlas-based), teams adopted various pre- and post-processing techniques that may contribute to performance differences. Common pre-processing steps included intensity normalization and cropping around the CoW (e.g., Sibets&USTS, DCS_CUSAT), as well as registration or resampling

(Labcom I3M, Alntropy). In post-processing, skeletonization and vessel graph construction were used for measurement and classification (Alntropy, Snaillab, Labcom I3M), while other teams applied an ensemble model for robustness (agaldran_segs and agaldran_ims).

### 3.4. Evaluation of metrics and ranking

Tables 2 and 3 summarize the results of the proposed solutions to Tasks 1 and 2, averaged over the 300 scans in the test set.

**Task 1: Classification of CoW anatomical variants**

The top-performing team for Task 1 was Sibets&USTS, achieving an average BAC of 0.26 and 0.27 for anterior and posterior CoW classification, respectively. Following closely, team Alntropy and Labcom I3M shared second place, with BAC scores of 0.20 and 0.40 (Alntropy), and 0.25 and 0.20 (Labcom I3M) for anterior and posterior CoW classification, respectively. These leading three teams employed distinct strategies in tackling the task: Sibets&USTS and Alntropy utilized deep learning approaches, while Labcom I3M adopted an atlas-based approach.

**Task 2: Quantification of artery diameters and bifurcation angles**

For Task 2, the top-performing solution was achieved by team Snaillab, followed by Alntropy (second place) and Labcom I3M (third place). Team Snaillab obtained an average MAE of 0.44 mm and Pearson correlation of 0.45 for artery diameters, and an average MAE of 28 degrees and Pearson correlation of 0.12 for bifurcation angles. All teams utilized voxel-based segmentation for computing artery diameters and bifurcation angles.

### 3.5. Further analyses

**Evaluation per class, artery diameter, and bifurcation angle**

Tables 4 and 5 present the Recall and F1-scores for each class in Task 1, respectively. Across all methods, class “A” (anterior classification) and class “E” (posterior classification) had the best classification performance. These classes constituted the majority of the test cases. Overall, F1-scores were notably lower than the recall, indicating low precision.

The confusion matrices (shown in Figs. A.1 and A.2) provide additional information on the performance of each method per class. For anterior classification, the methods proposed by teams Alntropy, agaldran_segs, and agaldran_ims predominantly predict majority classes, i.e. class “A” or “G”. Minority classes “C”, “D”, and “H” were correctly classified in 8 out of 35 test cases both by Labcom I3M and Sibets&USTS. The results for posterior classification were more widely distributed. Minority classes “C” and “J” were correctly classified in 2/8 and 1/5 cases only by team Sibets&USTS.

<footnote>a Shared 2nd place (alphabetical order).</footnote>

<footnote>b Method submitted after the MICCAI 2023 deadline; rank scores are reported as if the method had competed.</footnote>

<footnote>c Values rounded to two decimal places.</footnote>
<table><caption>**Table 4**
Recall per class for anterior and posterior classification.</caption><thead><tr><th rowspan="2"></th><th colspan="5">Anterior classification</th><th colspan="8">Posterior classification</th></tr><tr><th>A</th><th>C</th><th>D</th><th>G</th><th>H</th><th>A</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>J</th></tr></thead><tbody><tr><td>Sibets&amp;USTS</td><td>0.07</td><td>0.16</td><td>0.38</td><td>0.45</td><td>0.25</td><td>0.02</td><td>0.25</td><td>0.67</td><td>0.33</td><td>0.55</td><td>0.18</td><td>0.00</td><td>0.20</td></tr><tr><td>AIntrony</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.37</td><td>0.00</td><td>0.49</td><td>0.89</td><td>0.45</td><td>0.47</td><td>0.50</td><td>0.00</td></tr><tr><td>Labcom I3M</td><td>0.68</td><td>0.37</td><td>0.00</td><td>0.07</td><td>0.13</td><td>0.26</td><td>0.00</td><td>0.23</td><td>0.88</td><td>0.09</td><td>0.12</td><td>0.00</td><td>0.00</td></tr><tr><td>agaldran_segs</td><td>0.88</td><td>0.00</td><td>0.00</td><td>0.17</td><td>0.00</td><td>0.02</td><td>0.00</td><td>0.04</td><td>0.97</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>agaldran_imms</td><td>0.94</td><td>0.00</td><td>0.00</td><td>0.08</td><td>0.00</td><td>0.02</td><td>0.00</td><td>0.03</td><td>0.98</td><td>0.00</td><td>0.00</td><td>0.06</td><td>0.00</td></tr><tr><td>DCS_CUSAT</td><td>0.53</td><td>0.00</td><td>0.00</td><td>0.46</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.31</td><td>0.31</td><td>0.27</td><td>0.00</td><td>0.00</td><td>0.00</td></tr></tbody></table>

<table><caption>**Table 5**
F1-score per class for anterior and posterior classification.</caption><thead><tr><th rowspan="2"></th><th colspan="5">Anterior classification</th><th colspan="8">Posterior classification</th></tr><tr><th>A</th><th>C</th><th>D</th><th>G</th><th>H</th><th>A</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>J</th></tr></thead><tbody><tr><td>Sibets&amp;USTS</td><td>0.03</td><td>0.05</td><td>0.18</td><td>0.12</td><td>0.10</td><td>0.00</td><td>0.13</td><td>0.13</td><td>0.08</td><td>0.14</td><td>0.06</td><td>0.00</td><td>0.11</td></tr><tr><td>AIntrony</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.00</td><td>0.08</td><td>0.00</td><td>0.11</td><td>0.19</td><td>0.10</td><td>0.21</td><td>0.17</td><td>0.00</td></tr><tr><td>Labcom I3M</td><td>0.16</td><td>0.13</td><td>0.00</td><td>0.03</td><td>0.06</td><td>0.07</td><td>0.00</td><td>0.06</td><td>0.16</td><td>0.06</td><td>0.07</td><td>0.00</td><td>0.00</td></tr><tr><td>agaldran_segs</td><td>0.47</td><td>0.00</td><td>0.00</td><td>0.15</td><td>0.00</td><td>0.02</td><td>0.00</td><td>0.04</td><td>0.49</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>agaldran_imms</td><td>0.48</td><td>0.00</td><td>0.00</td><td>0.07</td><td>0.00</td><td>0.02</td><td>0.00</td><td>0.03</td><td>0.50</td><td>0.00</td><td>0.00</td><td>0.03</td><td>0.00</td></tr><tr><td>DCS_CUSAT</td><td>0.17</td><td>0.00</td><td>0.00</td><td>0.21</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.08</td><td>0.07</td><td>0.09</td><td>0.00</td><td>0.00</td><td>0.00</td></tr></tbody></table>

For Task 2, MAE scores and Pearson correlation coefficients per artery or bifurcation are illustrated in Figs. 6 and 7. The best results for diameter predictions by **Snaillab** were obtained for the A1-, M1-, and P1-segments, ICA, and BA (75th percentile < 0.5 mm), with median MAE scores between 0.17 and 0.39 mm. Best performances for bifurcation angle quantification in terms of MAE scores were obtained by **AIntrony**, with median MAE scores below 20 degrees except for the left M2/M2 bifurcation angle.

### Handling invalid model outputs

We assessed the outcomes of each method submitted to Task 2 by either excluding any invalid model predictions or by assigning the largest value to missing values. The average Mean Absolute Error (MAE) scores were 0.39 ± 0.16 mm for team **Snaillab** when ignoring invalid values (compared to 0.44 ± 0.21 mm when assigning the largest value to missing values), 0.49 ± 0.12 mm for **AIntrony** (compared to 0.50 ± 0.12 mm), and 0.87 ± 0.26 mm for **Labcom I3M** (no missing values). Fig. 8 illustrates the correlation between the reference and predicted values for Task 2 (invalid model predictions ignored).

### Model performance on data from another institution

The results of each proposed solution on 30 TOF-MRA scans from the TopCoW'23 challenge are summarized in Tables 6 and 7. Two cases with posterior class labels not found in the CROWN data were excluded from the posterior classification evaluation (Task 1). Anterior class D was not found in the selection of TopCoW data. Generalizability varied among methods and for anterior/posterior classification, with team **Sibets&USTS** achieving the best overall performance. The results of quantifying artery diameters and bifurcation angles (Task 2) on TopCoW data revealed that the method proposed by team **Snaillab** performed consistently on different data, indicating good generalizability.

## 4. Discussion

This section further discusses the main results, strengths, and limitations of the CROWN challenge 2023.

### 4.1. Challenge design

We developed a standardized approach for evaluating techniques that automatically analyze the morphology of the CoW. This evaluation was conducted against a clinical reference standard using a large dataset collected from familial screening cohorts, which accurately represents the target population for real-world applications. This paper

<table><caption>**Table 6**
Results of proposed solutions by each team for Task 1 (classification of the anterior and posterior CoW variant) on data from another institution (N=30).</caption><thead><tr><th></th><th>Anterior classification<br>BAC↑ (N=30)</th><th>Posterior classification<br>BAC↑ (N=28)</th></tr></thead><tbody><tr><td>Sibets&amp;USTS</td><td>0.42</td><td>0.29</td></tr><tr><td>AIntrony</td><td>0.28</td><td>0.25</td></tr><tr><td>Labcom I3M</td><td>0.12</td><td>0.13</td></tr><tr><td>agaldran_segs</td><td>0.29</td><td>0.16</td></tr><tr><td>agaldran_imms</td><td>0.24</td><td>0.11</td></tr><tr><td>DCS_CUSAT</td><td>0.38</td><td>0.03</td></tr><tr><td>DCS_CUSAT<sup>a</sup></td><td>0.42</td><td>0.20</td></tr></tbody></table>

Reported values are in terms of balanced accuracy (BAC) scores, with higher values indicating better performance.

compares the performance of methods proposed by teams participating in the session hosted at MICCAI 2023.

The intra- and inter-annotator assessment showed some inconsistencies for CoW classification using Lippert's system (Lippert and Pabst, 1985), which can be attributed to the subjectivity involved in manually measuring arteries to identify hypoplasia. The threshold set for hypoplasia at <1 mm approximates the voxel size, meaning minor errors can result in varying classifications. Higher spatial resolutions of the MR acquisitions would yield more accurate measurements, albeit the usage of such acquisitions is still limited in clinical settings. The intra- and inter-annotator variability for diameter measurements aligned with our previous findings (Vos et al., 2024). The computation of bifurcation angles is automated. Thus, any variability can only arise from the annotator selecting a different bifurcation. Most of these inconsistencies were related to the M2-M2 bifurcation, which is challenging to localize due to a variety of MCA variants (Brzegowy et al., 2018). Just a single team (AIntrony), utilized data apart from the provided challenge dataset. AIntrony acquired additional training cases from an internal dataset, potentially accounting for their superior performance in posterior classification. By increasing diversity through the incorporation of data from two institutions, their models may have become more robust and have better generalization.

### 4.2. Automated classification of CoW anatomical variants

In general, the solutions submitted to Task 1 of the CROWN challenge achieved BAC scores between 0.11 and 0.40, underlining the complexity of the problem. While each of the proposed solutions shows

<footnote>a Method submitted after the MICCAI 2023 deadline.</footnote>
<table><caption>**Table 7**
Results of proposed solutions by each team for Task 2 (quantification of artery diameters and bifurcation angles) on data from another institution (N=30).</caption><thead><tr><th rowspan="2"></th><th colspan="2">Artery diameters</th><th colspan="2">Bifurcation angles</th></tr><tr><th>Average MAE (mm)↓</th><th>Average Pearson ↑</th><th>Average MAE (degrees)↓</th><th>Average Pearson ↑</th></tr></thead><tbody><tr><td><b>Snaillab</b></td><td>0.49 ± 0.24</td><td>0.39</td><td>31 ± 10</td><td>0.18</td></tr><tr><td>AIntropy</td><td>0.68 ± 0.36</td><td>0.39</td><td>17 ± 6</td><td>0.01</td></tr><tr><td>Labcom I3M</td><td>1.00 ± 0.43</td><td>0.16</td><td>29 ± 6</td><td>0.06</td></tr></tbody></table>

Metric values are averaged across all scans and arteries or bifurcations. MAE = mean absolute error. The arrows ↑ and ↓ indicate that higher or lower results correspond to better performance, respectively.

<figure><img src="image_10.png" /><figcaption>**Fig. 6.** Results of proposed solutions by each team for Task 2 (quantification of artery diameters and bifurcation angles) on the test set, shown per artery. Lower mean absolute error (MAE) and higher Pearson correlation coefficient are associated with better performance. The method of Labcom I3M yielded MAE scores exceeding 4.0 mm for all arteries except the BA.</figcaption></figure>

promise, the current results fall short of meeting the criteria for clinical applicability compared to the inter- and intra-annotator metrics. Because of the limited number of participating teams, it is difficult to pinpoint key aspects that contribute to strengths or pitfalls in each approach. In addition, there was large diversity in proposed methodologies: the top-3 performing methods employed a ResNet50V2 architecture (Sibets&USTS, # 1), graph neural networks (AIntropy, shared # 2), and an atlas-registration based approach (Labcom I3M, shared # 2). This diversity suggests there is potential for employing multiple strategies to address this task, which may not solely rely on deep learning. This implies that a combination of various methods could yield significant improvements. An ensemble based on the top-3 performing methods using majority voting yielded inferior performance, indicating that majority voting is not a viable strategy.

Team Sibets&USTS correctly classified some cases within each of the minority classes except for one posterior class ("H"). The approach by team AIntropy showed a relatively good performance in posterior classification, but failed to identify the minority anterior classes that were not included as labels in their in-house dataset (e.g. three A2 segments or class "C"). Both team AIntropy and Labcom I3M relied on logical decision making, which offers the advantage of not necessitating training cases. Particularly, Labcom I3M achieved a recall of 0.37 for classification of anterior class "C", demonstrating its potential to identify specific CoW variants that are less prevalent. The results of the methods proposed by team agaldran_ims and agaldran_segs indicate that only the majority classes are predicted. Similarly, team DCS_CUSAT's approach outputs only the majority class for anterior classification. Their improved method, submitted after the
<figure><img src="image_11.png" /><figcaption>**Fig. 7.** Results of proposed solutions by each team for Task 2 (quantification of artery diameters and bifurcation angles) on the test set, shown per bifurcation. Lower mean absolute error (MAE) and higher Pearson correlation coefficient are associated with better performance. The method of Labcom I3M yielded MAE scores exceeding 4.0 mm for all arteries except the BA.</figcaption></figure>

MICCAI 2023 deadline, further builds on their previous approach using graph neural networks. The results indicate that they have effectively addressed the problem of only predicting majority classes.

### 4.3. Automated quantification of artery diameters and bifurcation angles

The proposed methods to Task 2 yielded average MAE scores between 0.44 and 0.87 mm for artery diameters and between 16 and 29 degrees for bifurcation angles. While this performance is inferior to the intra- and inter-annotator variability, it is important to note that the employed manual annotation methods were semi-automatic. For fully manual measurements, this variability is considerably larger (limits of agreement between -0.45 and 0.38 mm as reported in another study Vos et al., 2022).

All three methods used voxel-based segmentation followed by a deterministic approach to compute the artery diameter or bifurcation angle. None of the proposed solutions involved end-to-end training with machine learning, despite encouragement from the challenge organization. The results for artery diameter quantification progress toward clinical relevance, while automated quantification of bifurcation angles proved to be more challenging, and the results have yet to reach clinical significance.

Notably, both methods proposed by Snaillab and by Labcom I3M predict artery diameters below 1.2 mm, which were not included in the reference standard and are considered a limitation of the challenge. This exclusion stems from previous studies suggesting that reliable

measurements require a span of at least two image voxels, and measurements below this threshold are likely to overestimate the true artery size (van Hespen et al., 2019). Team AInropy achieved moderate performance in terms of the MAE for both diameter and angle predictions, although their Pearson correlations were low. Further investigation revealed consistent prediction of discrete values representing the mean artery diameter or bifurcation angle. This indicates that their method successfully localizes and identifies the correct artery or bifurcation, but accurate predictions at those locations were not achieved. The team also competed in the TopCoW'23 challenge on topology-aware anatomical segmentation of the CoW, securing second place (Yang et al., 2023).

### 4.4. Future directions

The current performance achieved for both tasks demonstrates that the problem remains unsolved and does not meet clinical criteria. Errors for both tasks were notably larger than the inter-annotator variability, presenting an ongoing challenge. Interestingly, no end-to-end solutions were proposed, highlighting the complexity of identifying features of only a few voxels in size within a larger image. This raises the question of whether alternative approaches, such as vision transformers, could yield improved results. Since classification of a CoW anatomical variant involves multiple arteries, transformers could offer an advantage by capturing regional or global features, compared to CNNs that focus more on local features. Moreover, as previously
<figure><img src="image_12.png" /><figcaption>**Fig. 8.** Correlation between predicted and references values for each submitted solution to Task 2. The left column depicts plots per method for artery diameter predictions (x-axis) versus reference values (y-axis); the right column shows plots per method for bifurcation angle predictions (x-axis) versus reference values (y-axis). **Alntropy** consistently predicts distinct values that represent the mean for that specific artery or bifurcation. Both **Labcom I3M** and **Snaillab** teams' methods predict diameters below 1.2 mm, indicating underestimations compared to the reference standard. Additionally, an overestimation of smaller bifurcation angles was observed for these two teams.</figcaption></figure>

discussed, integrating various components of different methodologies may prove to be a beneficial strategy worth exploring.

One limitation of this challenge is the relatively small number of final submissions, with six teams participating in Task 1 and three in Task 2. Compared to previous challenges, such as the CrossMoDA challenge (Dorent et al., 2023), this smaller pool of participants may impact the robustness of the conclusions and the comprehensiveness of comparisons among methods. Increased participation could strengthen the benchmarking and provide broader insights in future editions. In addition, while rare variants of the Circle of Willis (CoW) were excluded from the current challenge dataset due to their limited availability, we recognize the importance of including them in the future to ensure a more comprehensive evaluation of methods.

Accurate assessment of the CoW remains challenging and time-consuming, despite an increasing number of studies exploring CoW morphology in relation to neurovascular diseases or across different populations (Kancheva et al., 2022; Vos et al., 2024). Although efforts are increasing to automate this assessment, the process is hindered by the scarcity of publicly available and annotated datasets. Providing more public datasets through scientific challenges would greatly benefit ongoing advancements in this field. Finding the best methodological approach holds significant clinical implications, including the

identification of imaging factors predictive of neurovascular diseases, or enabling genome-wide association studies capable of identifying genetically-regulated components of imaging marker phenotypes. Furthermore, such advancements may extend their benefits to research areas involving other vascular structures, such as the coronary or pulmonary arteries.

## 5. Conclusion

In this study, we compared the performance of several techniques to automatically classify CoW anatomical variants and quantify major CoW artery diameters and bifurcation angles. Although promising, the proposed solutions are not yet suitable for clinical implementation as the current results do not meet the standards set by manual measurements and inter- and intra-annotator agreements. The CROWN challenge remains open for future submissions, providing a benchmark for evaluating new techniques to assess CoW morphology.
<figure><img src="image_13.png" /><figcaption>Fig. A.1. Confusion matrices of the results per team (AIntropy, agaldran_ims, and agaldran_segs) for anterior (left column) and posterior (right column) classification. Rows represent reference classes, while columns represent predicted classes. Values in each cell indicate the count of instances classified accordingly.</figcaption></figure>

CRediT authorship contribution statement

**Iris N. Vos:** Writing – original draft, Visualization, Validation, Software, Resources, Methodology, Investigation, Formal analysis, Conceptualization. **Ynte M. Ruigrok:** Writing – review & editing, Investigation, Funding acquisition, Data curation, Conceptualization. **Edwin Bennink:** Writing – review & editing, Software, Resources. **Mireille R.E. Velthuis:** Data curation. **Barbara Paic:** Data curation. **Maud E.H. Ophelders:** Data curation. **Myrthe A.D. Buser:** Writing – review & editing, Conceptualization. **Bas H.M. van der Velden:** Writing – review & editing, Conceptualization. **Chen Geng:** Software, Methodology. **Matthieu Coupet:** Software, Methodology. **Félix Dumais:** Software, Methodology. **Adrian Galdran:** Software, Methodology. **Junyi Zhang:** Software, Methodology. **Wei Liu:** Software, Methodology. **Ting Ma:** Software, Methodology. **Madhu S. Nair:** Software, Methodology. **Mathieu Naudin:** Software, Methodology. **Preena K.P.:** Software,

**Methodology. Keerthi A.S. Pillai:** Software, Methodology. **Pengcheng Shi:** Software, Methodology. **Thierry Urruty:** Software, Methodology. **Yakang Dai:** Software, Methodology. **Kaiyuan Yang:** Writing – review & editing, Data curation, Conceptualization. **Fabio Musio:** Writing – review & editing, Data curation, Conceptualization. **Bjoern H. Menze:** Writing – review & editing, Data curation, Conceptualization. **Birgitta K. Velthuis:** Writing – review & editing, Investigation, Data curation, Conceptualization. **Hugo J. Kuijf:** Writing – review & editing, Visualization, Validation, Software, Methodology, Investigation, Formal analysis, Conceptualization.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Iris
Vos, Hugo Kuijf, Ynte Ruigrok reports financial support was provided by University Medical Centre Utrecht. Adrian Galdran reports financial support was provided by Pompeu Fabra University. Matthieu Coupet, Mathieu Naudin, Thierry Urruty reports financial support was provided by University of Poitiers. Geng Chen, Dai Yakang reports financial support was provided by Suzhou Institute of Biomedical Engineering and Technology. Zhang Junyi reports financial support was provided by Suzhou University of Science and Technology. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

This study received funding from the European Research Council under the EU's Horizon 2020 research and innovation program

(PRYSM, grant agreement number 852173). YMR received funding from the Dutch Heart Foundation (Dekker grant 03-001-2022-0157). The work of GC, ZJ, and DY was supported by Shandong Natural Science Foundation [ZR2022QF093], Jiangsu Key Technology Research Development Program [BE2022842], and Suzhou Basic Research Pilot Program [SSD2023008]. The work of MC, MN, and TU was supported by the French Nouvelle-Aquitaine region through the ARTERIA project. The work of AG was supported by a Marie Sklodowska-Curie Fellowship, No. 892297. HJK received funding from the Dutch Heart Foundation (Dekker grant project No. 03-004-2021-T043).

## Appendix. Confusion matrices per team for Task 1

See Figs. A.1 and A.2.

<figure><img src="image_14.png" /><figcaption>**Fig. A.2.** Confusion matrices of the results per team (DCS_CUSAT, Labcom I3M, and Sibets&USTS) for anterior (left column) and posterior (right column) classification. Rows represent reference classes, while columns represent predicted classes. Values in each cell indicate the count of instances classified accordingly.</figcaption></figure>
## Data availability

Training data can be found on DataverseNL (DataverseNL, 2023). Our evaluation implementation is publicly available (Vos and Bennink, 2024), and was shared with the challenge participants upon registration to ensure transparency.

## References

Adam, C., Fabre, D., Mougin, J., Zins, M., Azarine, A., Ardon, R., d'Assignies, G., Haulon, S., 2021. Pre-surgical and post-surgical aortic aneurysm maximum diameter measurement: full automation by artificial intelligence. Eur. J. Vasc. Endovasc. Surg. 62 (6), 869-877.

Avadiappan, S., Payabvash, S., Morrison, M.A., Jakary, A., Hess, C.P., Lupo, J.M., 2020. A fully automated method for segmenting arteries and quantifying vessel radii on magnetic resonance angiography images of varying projection thickness. Front. Neurosci. 14, 520810.

Bor, A.S.E., Rinkel, G.J., van Norden, J., Wermer, M.J., 2014. Long-term, serial screening for intracranial aneurysms in individuals with a family history of aneurysmal subarachnoid haemorrhage: a cohort study. Lancet Neurol. 13 (4), 385-392.

Brzegowy, P., Polak, J., Wnuk, J., Łasocha, B., Walocha, J., Popiela, T.J., 2018. Middle cerebral artery anatomical variations and aneurysms: a retrospective study based on computed tomography angiography findings. Folia Morphol. 77 (3), 434-440.

DataverseNL, 2023. Data of the Circle of Willis intracranial artery classification and quantification (CROWN) challenge. http://dx.doi.org/10.3489/R05G1L.

de Vos, V., Timmins, K.M., van der Schaaf, I.C., Ruigrok, Y., Velthuis, B.K., Kuijf, H.J., 2021. Automatic cerebral vessel extraction in TOF-MRA using deep learning. In: Medical Imaging 2021: Image Processing, vol. 11596, SPIE, pp. 651-660.

Dorent, R., Kujawa, A., Ivory, M., Bakas, S., Rieke, N., Joutard, S., Glocker, B., Cardoso, J., Modat, M., Batmanghelich, K., et al., 2023. CrossMoDA 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation. Med. Image Anal. 83, 102628.

Dumais, F., Caceres, M.P., Janelle, F., Seifeldine, K., Arès-Bruneau, N., Gutierrez, J., Bocti, C., Whittingstall, K., 2022. eICAB: A novel deep learning pipeline for Circle of Willis multiclass segmentation and analysis. Neuroimage 260, 119425.

El-Barhoun, E., Gledhill, S., Pitman, A., 2009. Circle of Willis artery diameters on MR angiography: an Australian reference database. J. Med. Imaging Radiat. Oncol. 53 (3), 248-260.

Essadik, I., Nouri, A., Touahni, R., Bourcier, R., Autrusseau, F., 2022. Automatic classification of the cerebral vascular bifurcations using dimensionality reduction and machine learning. Neurosci. Informatics 2 (4), 100108.

Geng, C., Zhang, J., Dai, Y., 2025. 3D-ResNet50V2 and RoI extraction method for task 1 of CROWN challenge 2023. http://dx.doi.org/10.5281/zenodo.14625886.

Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D., 2021. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI Brainlesion Workshop. Springer, pp. 272-284.

Jones, J.D., Castanho, P., Bazira, P., Sanders, K., 2021. Anatomical variations of the circle of Willis and their prevalence, with a focus on the posterior communicating artery: A literature review and meta-analysis. Clin. Anat. 34 (7), 978-990.

Kancheva, A.K., Velthuis, B.K., Ruigrok, Y.M., 2022. Imaging markers of intracranial aneurysm development: A systematic review. J. Neuroradiol. 49 (2), 219-224.

Kuijf, H.J., Biesbroek, J.M., De Bresser, J., Heinen, R., Andermatt, S., Bento, M., Berseth, M., Belyaev, M., Cardoso, M.J., Casamitjana, A., et al., 2019. Standardized assessment of automatic segmentation of white matter hyperintensities and results of the WMH segmentation challenge. IEEE Trans. Med. Imaging 38 (11), 2556-2568.

LaMontagne, P.J., Benzinger, T.L., Morris, J.C., Keefe, S., Hornbeck, R., Xiong, C., Grant, E., Hassenstab, J., Moulder, K., Vlassenko, A.G., et al., 2019. OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease. MedRxiv.

Lazorthes, G., Gouaze, A., Santini, J., Salamon, G., 1979. The arterial circle of the brain circulus arteriosus cerebri. Anat. Clin. 1, 241-257.

Lippert, H., Pabst, R., 1985. Arterial Variations in Man. JF Bergmann-Verlag.

Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M.D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al., 2024. Metrics reloaded: recommendations for image analysis validation. Nature Methods 1-18.

Maier-Hein, L., Reinke, A., Kozubek, M., Martel, A.L., Arbel, T., Eisenmann, M., Hanbury, A., Jannin, P., Müller, H., Onogur, S., et al., 2020. BIAS: Transparent reporting of biomedical image analysis challenges. Med. Image Anal. 66, 101796.

Nader, R., Bourcier, R., Autrusseau, F., 2023. Using deep learning for an automatic detection and classification of the vascular bifurcations along the Circle of Willis. Med. Image Anal. 89, 102919.

Nieuwkamp, D.J., Setz, L.E., Algra, A., Linn, F.H., de Rooij, N.K., Rinkel, G.J., 2009. Changes in case fatality of aneurysmal subarachnoid haemorrhage over time, according to age, sex, and region: a meta-analysis. Lancet Neurol. 8 (7), 635-642.

Noh, S.M., Kang, H.G., 2019. Clinical significance of the internal carotid artery angle in ischemic stroke. Sci. Rep. 9 (1), 4618.

Ophelders, M., van Eldik, M., Vos, I., Beentjes, Y., Velthuis, B., Ruigrok, Y., 2024. Anatomical differences of intracranial arteries according to sex: a systematic review and meta-analysis. J. Neuroradiol. 51 (1), 10-15.

Preena, K.P., A. S. Pillai, K., S. Nair, M., 2025. MICCAI 2023/CROWN_CHALLENGE/DCS_CUSAT_TASK1. Zenodo.

Ritter, F., Boskamp, T., Homeyer, A., Laue, H., Schwier, M., Link, F., Peitgen, H.O., 2011. Medical image analysis. IEEE Pulse 2 (6), 60-70.

Sedghi Gamechi, Z., Bons, L.R., Giordano, M., Bos, D., Budde, R.P., Kofoed, K.F., Pedersen, J.H., Roos-Hesselink, J.W., de Bruijne, M., 2019. Automated 3D segmentation and diameter measurement of the thoracic aorta on non-contrast enhanced CT. Eur. Radiol. 29, 4613-4623.

Selle, D., Preim, B., Schenk, A., Peitgen, H.-O., 2002. Analysis of vasculature for liver surgical planning. IEEE Trans. Med. Imaging 21 (11), 1344-1357.

Shi, P., 2024. CROWN Challenge Submission (Team AlTropy). Zenodo, http://dx.doi.org/10.5281/zenodo.14565066.

Shi, P., Guo, X., Yang, Y., Ye, C., Ma, T., 2023. Nextou: Efficient topology-aware u-net for medical image segmentation. arXiv preprint arXiv:2305.15911.

Timmins, K.M., van der Schaaf, I.C., Bennink, E., Ruigrok, Y.M., An, X., Baumgartner, M., Bourdon, P., De Feo, R., Di Noto, T., Dubost, F., et al., 2021. Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: the ADAM challenge. Neuroimage 238, 118216.

van Hespen, K.M., Zwanenburg, J.J., Harteveld, A.A., Luijten, P.R., Hendrikse, J., Kuijf, H.J., 2019. Intracranial vessel wall magnetic resonance imaging does not allow for accurate and precise wall thickness measurements: an ex vivo study. Stroke 50 (10), e283-e284.

Vos, I., 2025. Solutions to Task 1 and 2 of the CROWN Challenge 2023. Zenodo, http://dx.doi.org/10.5281/zenodo.14878391.

Vos, I., Bennink, E., 2024. irisnadinevos/Crownchallenge-Evaluation-Example: Crownchallenge-Evaluation-Example. Zenodo, http://dx.doi.org/10.5281/zenodo.14568658.

Vos, I., Ophelders, M., Ruigrok, Y., Velthuis, B., Kuijf, H., 2022. Assessment of manual and automated intracranial artery diameter measurements. In: Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment, vol. 12035, SPIE, pp. 10-15.

Vos, I.N., van Tuijl, R.J., Mensing, L.A., Ophelders, M.E., Velthuis, M.R., Zuithoff, N.P., Rinkel, G.J., Kuijf, H.J., Zwanenburg, J.J., van der Schaaf, I.C., et al., 2024. Anatomical markers associated with the presence of intracranial aneurysms in individuals screened for aneurysms. Stroke: Vasc. Interv. Neurol. e001299.

Yang, K., Musio, F., Ma, Y., Juchler, N., Paetzold, J.C., Al-Maskari, R., Höher, L., Li, H.B., Hamamci, I.E., Sekuboyina, A., et al., 2023. Benchmarking the cow with the topcow challenge: Topology-aware anatomical segmentation of the Circle of Willis for cta and mra. ArXiv.

Zhang, X.J., Gao, B.L., Hao, W.L., Wu, S.S., Zhang, D.H., 2018. Presence of anterior communicating artery aneurysm is associated with age, bifurcation angle, and vessel diameter. Stroke 49 (2), 341-347.

Zuurbier, C.C., Greving, J.P., Rinkel, G.J., Ruigrok, Y.M., 2020. Higher risk of intracranial aneurysms and subarachnoid haemorrhage in siblings of families with intracranial aneurysms. Eur. Stroke J. 5 (1), 73-77.