# Inter-Rater Variability of Prostate Lesion Segmentation on Multiparametric Prostate MRI

Thibaut Jeganathan, Emile Salgues, Ulrike Schick, Valentin Tissot, Georges Fournier, Antoine Valéri, Truong-An Nguyen, Vincent Bourbonne

► To cite this version:

Thibaut Jeganathan, Emile Salgues, Ulrike Schick, Valentin Tissot, Georges Fournier, et al.. Inter-Rater Variability of Prostate Lesion Segmentation on Multiparametric Prostate MRI. Biomedicines, 2023, 11 (12), pp.3309. 10.3390/biomedicines11123309. [hal-05247974](https://hal-univ-brest.fr/hal-05247974v1)

HAL Id: hal-05247974

https://hal.univ-brest.fr/hal-05247974v1

Submitted on 10 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

Article

# Inter-Rater Variability of Prostate Lesion Segmentation on Multiparametric Prostate MRI

Thibaut Jeganathan <fnref n="1" />, Emile Salgues <fnref n="1" />, Ulrike Schick <fnref n="2" />,<fnref n="3" />, Valentin Tissot <fnref n="1" />, Georges Fournier <fnref n="3" />,<fnref n="4" />, Antoine Valéri <fnref n="3" />,<fnref n="4" />, Truong-An Nguyen <fnref n="3" />,<fnref n="4" /> and Vincent Bourbonne <fnref n="2" />,<fnref n="3" />,*

<figure><img src="image_3.png" /></figure>

check for
updates

**Citation:** Jeganathan, T.; Salgues, E.; Schick, U.; Tissot, V.; Fournier, G.; Valéri, A.; Nguyen, T.-A.; Bourbonne, V. Inter-Rater Variability of Prostate Lesion Segmentation on Multiparametric Prostate MRI. *Biomedicines* **2023**, *11*, 3309. https://doi.org/10.3390/biomedicines11123309

Academic Editor: Gvantsa Kharaishvili

Received: 25 November 2023

Revised: 10 December 2023

Accepted: 12 December 2023

Published: 14 December 2023

<figure><img src="image_4.png" /></figure>

**Copyright:** © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

**Abstract:** Introduction: External radiotherapy is a major treatment for localized prostate cancer (PCa). Dose escalation to the whole prostate gland increases biochemical relapse-free survival but also acute and late toxicities. Dose escalation to the dominant index lesion (DIL) only is of growing interest. It requires a robust delineation of the DIL. In this context, we aimed to evaluate the inter-observer variability of DIL delineation. Material and Methods: Two junior radiologists and a senior radiation oncologist delineated DILs on 64 mpMRIs of patients with histologically confirmed PCa. For each mpMRI and each reader, eight individual DIL segmentations were delineated. These delineations were blindly performed from one another and resulted from the individual analysis of the T2, apparent diffusion coefficient (ADC), b2000, and dynamic contrast enhanced (DCE) sequences, as well as the analysis of combined sequences (T2ADC, T2ADCb2000, T2ADCDCE, and T2ADCb2000DCE). Delineation variability was assessed using the DICE coefficient, Jaccard index, Hausdorff distance measure, and mean distance to agreement. Results: T2, ADC, T2ADC, b2000, T2 + ADC + b2000, T2 + ADC + DCE, and T2 + ADC + b2000 + DCE sequences obtained DICE coefficients of 0.51, 0.50, 0.54, 0.52, 0.54, 0.55, 0.53, respectively, which are significantly higher than the perfusion sequence alone (0.35, *p* < 0.001). The analysis of other similarity metrics lead to similar results. The tumor volume and PI-RADS classification were positively correlated with the DICE scores. Conclusion: Our study showed that the contours of prostatic lesions were more reproducible on certain sequences but confirmed the great variability of prostatic contours with a maximum DICE coefficient calculated at 0.55 (joint analysis of T2, ADC, and perfusion sequences).

**Keywords:** prostate cancer; multiparametric MRI; segmentation; variability

## 1. Introduction

With 1,400,000 new cases in 2020, prostate cancer (PCa) is the second most common cancer among men in the world, comprising 13.5% of all male cancer diagnoses (after lung cancer, which accounts for 14.5%) [1,2]. External beam radiation therapy (EBRT) is a major curative option for men with localized PCa [3]. Several dose-escalation studies have reported improved biochemical progression-free survival [4,5], with dose escalation being delivered to either the whole prostate gland or the dominant index lesion (DIL) itself.

The ASCENDE-RT trial showed that biochemical disease-free survival (bDFS) was increased in patients who received a brachytherapy boost to the whole prostate gland [6]. Given the rise in acute and late toxicities as a result of dose escalation to the whole prostate gland, there is a growing interest in more focused therapeutic approaches. In the FLAME trial, the addition of a focal boost to the DIL increased bDFS without affecting toxicity

<footnote n="1">¹ Radiology Department, University Hospital, 29200 Brest, France; thibaut.jeganathan@ch-cornouaille.fr (T.J.); emile.salgues@chu-brest.fr (E.S.); valentin.tissot@chu-brest.fr (V.T.)</footnote>

<footnote n="2">² Radiation Oncology Department, University Hospital, 29200 Brest, France; ulrike.schick@chu-brest.fr</footnote>

<footnote n="3">³ INSERM, LaTIM UMR 1101, University of Western Brittany, 29238 Brest, France; georges.fournier@chu-brest.fr (G.F.); antoine.valeri@chu-brest.fr (A.V.); truongan.nguyen@chu-brest.fr (T.-A.N.)</footnote>

<footnote n="4">⁴ Urology Department, University Hospital, 29200 Brest, France</footnote>

<footnote>* Correspondence: vincent.bourbonne@chu-brest.fr; Tel.: +33-298223398</footnote>
or quality of life. Patients with intermediate- or high-risk localized PCa were randomly assigned to the standard treatment arm (77 Gy in 35 fractions of 2,2 Gy) or the focal boost arm (focal boost up to 95 Gy to the macroscopic tumor in addition to the standard treatment). It has shown that the bDFS was significantly higher in the focal boost arm compared with the standard treatment arm (hazard ratio 0.45, 95% CI, 0.28 to 0.71, *p* < 0.001) with an acceptable toxicity profile [7]. Furthermore, in selected individuals with low to intermediate-risk PCa, focused brachytherapy with a high dose rate might obtain a similar relapse-free survival rate as the traditional whole-prostate brachytherapy, with less toxicity [8].

These treatments and clinical settings all have in common the need for a precise delineation of the DIL. Indeed, the delineation of the DIL has been identified as one of the main challenges preventing dose escalation to be broadly used [9].

Because of its high accuracy in lesion localization, multiparametric magnetic resonance imaging (mpMRI) is regarded as the current standard for detection [10–13] and biopsy planning [14,15]. The analysis of prostate mpMRIs requires dedicated training. Studies have previously focused on tumor volume estimation [16], but only a few studies have been conducted on the inter-reader variability of the delineation. With a cohort of 44 participants segmenting 20 prostate cases, the median DICE score was as low as 0.49, with an interquartile range (IQR) of 0.19–0.64 [17]. Furthermore, none of these studies has evaluated the inter-sequence variability. To this day, there is currently no consensus on the optimal magnetic resonance imaging (MRI) series (T2, apparent diffusion coefficient (ADC) cartography, diffusion-weighted imaging (DWI), or dynamic contrast-enhanced (DCE) MRI) to use for the delineation of the DIL. The robustness of the delineation itself was not previously evaluated and is not part of the European Society for Urogenital Radiology (ESUR) guidelines [18]. In this study, we thus aimed to evaluate the inter-reader variability of DIL delineation using mpMRIs.

## 2. Materials and Methods

### 2.1. Population

A retrospective cohort of 64 patients with histologically confirmed localized PCa who did not previously undergo any treatment related to the PCa itself and who were treated at the Brest University Hospital, France, was used for the present study. A multiparametric prostate MRI was mandatory for all patients. No exclusion criteria regarding the prostate or the index lesion volume were used. Antiangiogenic medications, including 5-alpha reductase inhibitors, were not considered as contraindications. This study was approved by the Ethical Committee of the University Hospital Brest and registered on Clinical Trials as NCT05996289. All patients were sent a non-opposition form.

### 2.2. MRI Scan

The mpMRIs were performed following the ESUR and Prostate Imaging Reporting and Data System (PI-RADS) guidelines in use at the time of acquisition [18,19], using the Philips ACHIEVA 3T MR machine of University Hospital Brest (Philips Healthcare©, Amsterdam, The Netherlands) from 19 December 2011 to 25 March 2019.

The mpMRI protocol included the T2w turbo spin echo (TSE) DW-MRI with apparent diffusion coefficient (ADC) maps and dynamic contrast material-enhanced (DCE) sequences. We obtained DWI with calculated b0, b600, and b1000 sequences as well as an independent DWI b2000 sequence. The MRI parameters are summarized in Supplementary Table S1.

The DCE sequence was a T1 weighed sequence performed before, during, and after a bolus injection of a contrast agent (gadolinium) with a 5 s temporal resolution. An endorectal coil was not used.

To ensure comparability, each sequence was registered to the T2 sequence before segmentation using a 6-axis rigid registration. The registration was checked and validated by all readers, focusing on the prostate. The same registration was used for all readers.
## 2.3. Delineation of the Dominant Index Lesion (DIL)

The DILs were delineated by two junior diagnostic radiologists (three and four years of experience, respectively) and a senior radiation oncologist between 1 June 2021 and 10 July 2022. The two junior radiologists each had a 1-year fellowship in abdominal imaging (~250 MRI scans) while the senior radiation oncologist had 5 years of experience in prostate image analysis (~700 MRI scans).

The DIL was defined according to the PI-RADS classification [19]. All mpMRIs and PI-RADS classifications were reviewed according to the PI-RADS v2.1 guidelines [19] by a single radiologist with 15 years of experience in urology imaging.

A delineation workflow was specifically designed using the Workflow tool embedded in MIM Maestro software (MIM v7.1.2, Cleveland, OH, USA). The first step consisted in the registration of all sequences, using the T2 sequence as the reference, as previously described. Before contouring the DIL, each reader first contoured the prostate using the T2 sequence, and then the peripherical (PZ) and transitional zones (TZ).

For each reader, eight DIL delineations were then realized. These delineations were blindly performed from one another and resulted from the individual analysis of the T2, apparent diffusion coefficient (ADC), b2000, and dynamic contrast enhanced (DCE) sequences, as well as the analysis of combined sequences (T2ADC, T2ADCb2000, T2ADCDCE, and the all-combined T2ADCb2000DCE sequence). Delineations were performed in the following order (T2, ADC, T2ADC, b2000, T2ADCb2000, DCE, T2ADCDCE, and the all-combined sequence) as shown in Supplementary Table S2. It has to be noted that each segmentation was masked before displaying the new sequence to be delineated. For instance, after delineating the DIL using the T2 sequence, both the T2 segmentation and T2 sequence were blinded and the ADC sequence was brought up, allowing the reader to delineate the DIL on the ADC sequence, blinded from the other sequences. A total of 4 delineations were added consisting in the union of T2 and ADC delineations (T2 + ADC), T2, ADC, and b2000 delineations (T2 + ADC + b2000), T2, ADC, and DCE delineations (T2 + ADC + DCE) and T2, ADC, b2000, and DCE delineations (T2 + ADC + b2000 + DCE).

It is noteworthy that due to the possible bias of the mis-selection of the DCE phases, the DCE sequence was analyzed using a semi-quantitative descriptive index, the Initial Area Under Gadolinium Curve (IAUCG) [20].

Each reader had the freedom to define and/or modify the viewing parameters, including the color table for the DCE analysis.

## 2.4. Similarity and Statistical Analysis

The spatial correlation between the different prostate contours and prostate tumoral lesion contours were evaluated using the following four geometric metrics: the DICE similarity coefficient (DSC), Jaccard index, Hausdorff distance (HD) measure, and mean distance to agreement (MDA). In order to evaluate the reproducibility of the contours between the different evaluators, an average of these variables was calculated for each sequence.

Similarity metrics were compared using the Student's *t* test. A *p* value < 0.05 was considered as statistically significant.

As an explanatory analysis, the correlation between the similarity metrics and the features of age, prostate and lesion volume, as well as PI-RADS grading, was analyzed using the Spearman correlation coefficient. Based on this univariate analysis, statistically significant features were further processed through a multivariate logistic regression analysis.

Regarding the PI-RADS grading, a Student's *t* test was used to compare the means of the similarity metrics between PI-RADS < 5 and PI-RADS = 5 patients.

## 2.5. Number of Patients

With the assumption that mpMRI would result in a 0.05 DSC gain, reaching 0.55 vs. 0.50 when compared to biparametric MRI (standard deviation of 0.08), a cohort of 64 patients will provide an 80% power with a type-1 error of 0.05.
## 3. Results

### 3.1. Population

Sixty-four patients were randomly selected among the overall cohort. Overall, the median age was 69 years (range 46–90). The PI-RADS classification was distributed as follows: 7 PI-RADS 3 lesions, 12 PI-RADS 4 lesions, and 45 PI-RADS 5 lesions. The characteristics of the patients and tumors are described in Table 1.

<table><caption>**Table 1.** Patients' characteristics.</caption><thead><tr><th>Characteristics of Patients</th><th>Number of Patients = 64</th></tr></thead><tbody><tr><td>Age (median (range))</td><td>69 (46–90)</td></tr><tr><td>PI-RADS Classification</td><td></td></tr><tr><td>PI-RADS 3</td><td>7/64 (10.9)</td></tr><tr><td>PI-RADS 4</td><td>12/64 (18.8)</td></tr><tr><td>PI-RADS 5</td><td>45/64 (70.3)</td></tr><tr><td>Mean PSA (ng/mL)</td><td>9.1</td></tr><tr><td>PSA range (ng/mL)</td><td></td></tr><tr><td>&lt;10</td><td>41/64 (64.1%)</td></tr><tr><td>10–20</td><td>20/64 (31.3%)</td></tr><tr><td>&gt;20</td><td>3/64 (4.7%)</td></tr><tr><td>ISUP Class</td><td></td></tr><tr><td>5</td><td>9/64 (14/1%)</td></tr><tr><td>4</td><td>9/64 (14/1%)</td></tr><tr><td>3</td><td>9/64 (14/1%)</td></tr><tr><td>2</td><td>19/64 (29.7%)</td></tr><tr><td>1</td><td>18/64 (28.1%)</td></tr></tbody></table>

Abbreviations: PSA: Prostate Specific Antigen, ISUP: International Society of Urological Pathology.

### 3.2. Similarity Analysis: Prostate, PZ, and TZ

The prostate volume appeared as similar with respective means of 41.25 mL, 43.77 mL, and 40.07 mL for T.J., E.S., and V.B., respectively (Supplementary Table S3). No significant difference was found between each reader except for the PZ delineation between ES and VB.

A high overlap between each prostate delineation with a mean DSC calculated at 0.89 (TJ/ES), 0.89 (TJ/VB), and 0.90 (ES/VB) was found (Table 2). It is noteworthy that the TZ and PZ delineations were associated with a higher variability with mean DSC values of 0.77 (TJ/ES), 0.79 (TJ/VB), and 0.78 (ES/VB) for the TZ delineations and 0.72 (TJ/ES), 0.75 (TJ/VB), and 0.72 (ES/VB) for the PZ delineations.

<table><caption>**Table 2.** Inter-reader variability of prostate, transitional, and peripheral zone delineations.</caption><thead><tr><th rowspan="2">Pair of Readers</th><th rowspan="2">Variable</th><th colspan="2">Prostate</th><th colspan="2">Transitional Zone</th><th colspan="2">Peripheral Zone</th></tr><tr><th>Mean</th><th>SD</th><th>Mean</th><th>SD</th><th>Mean</th><th>SD</th></tr></thead><tbody><tr><td>TJ/ES</td><td>DICE</td><td>0.89</td><td>0.03</td><td>0.77</td><td>0.13</td><td>0.72</td><td>0.12</td></tr><tr><td>TJ/VB</td><td>DICE</td><td>0.89</td><td>0.03</td><td>0.79</td><td>0.11</td><td>0.75</td><td>0.06</td></tr><tr><td>VB/ES</td><td>DICE</td><td>0.90</td><td>0.04</td><td>0.78</td><td>0.11</td><td>0.72</td><td>0.12</td></tr><tr><td>All</td><td>DICE</td><td>0.90</td><td>0.03</td><td>0.77</td><td>0.13</td><td>0.72</td><td>0.10</td></tr></tbody></table>

Abbreviation: SD: standard deviation.

### 3.3. Similarity Analysis: Index Lesion

After contouring the prostate, PZ and TZ, readers focused on the prostate lesions whose averages volumes are shown in Figure 1 and supplementary Table S4. Displaying all sequences (T2, ADC, b2000 and DCE), the average delineated volume was 3.6 mL. As expected, the combined sequences had higher volumes than single sequences. Without
taking the combined sequences into account, the lowest mean volumes were obtained using the DCE sequence (2.7 mL) and ADC map (2.8 mL), and the highest mean volume was obtained using the T2ADCb2000 (3.6 mL) and T2ADCb2000DCE (3.6 mL) sequences.

<figure><img src="image_5.png" /><figcaption>**Figure 1.** Mean of tumor volume delineated for each reader and each sequence. Abbreviations: ADC: Apparent Diffusion Coefficient, DCE: Dynamic Contrast Enhanced.</figcaption></figure>

Figure 2 as well as Supplementary Table S5 show the mean of the DSC coefficients obtained for each sequence and for each pair of readers. A notable variability between all DILs was found; the mean of the DSC was around 0.50 and 0.54, and even decreased to 0.35 for the DCE sequence. The T2, ADC, T2ADC, b2000, T2ADCb2000, T2ADCDCE and all-combined sequences obtained a significantly higher similarity than the DCE sequence (Table 3, *p* < 0.001).

<figure><img src="image_6.png" /><figcaption>**Figure 2.** Inter-reader variability for the definition of the index lesion for each sequence (DICE). Abbreviations: DSC: DICE Similarity Coefficient, ADC: Apparent Diffusion Coefficient, DCE: Dynamic Contrast Enhanced.</figcaption></figure>
<table><caption>**Table 3.** Difference in mean DICE scores between each sequence (p-values).</caption><thead><tr><th></th><th>T2</th><th>ADC</th><th>T2ADC</th><th>b2000</th><th>T2ADCb2000</th><th>DCE</th><th>T2ADCDCE</th><th>All</th><th>T2 + ADC</th><th>T2 + ADC + b2000</th><th>T2 + ADC + DCE</th><th>T2 + ADC + b2000 + DCE</th></tr></thead><tbody><tr><td>T2</td><td>1.00</td><td>0.27</td><td>0.25</td><td>0.73</td><td>0.25</td><td>&lt;0.0001<sup>†</sup></td><td>0.23</td><td>0.27</td><td>0.15</td><td>0.15</td><td>0.15</td><td>0.02<sup>†</sup></td></tr><tr><td>ADC</td><td>0.27</td><td>1.00</td><td>0.03<sup>†</sup></td><td>0.49</td><td>0.03<sup>†</sup></td><td>0.01<sup>†</sup></td><td>0.02<sup>†</sup></td><td>0.03<sup>†</sup></td><td>0.01<sup>†</sup></td><td>0.01<sup>†</sup></td><td>0.01<sup>†</sup></td><td>0.01<sup>†</sup></td></tr><tr><td>T2ADC</td><td>0.25</td><td>0.03<sup>†</sup></td><td>1.00</td><td>0.16</td><td>1.00</td><td>&lt;0.0001<sup>†</sup></td><td>0.95</td><td>0.90</td><td>0.77</td><td>0.77</td><td>0.80</td><td>0.28</td></tr><tr><td>B2000</td><td>0.73</td><td>0.49</td><td>0.16</td><td>1.00</td><td>0.16</td><td>&lt;0.0001<sup>†</sup></td><td>0.14</td><td>0.19</td><td>0.09</td><td>0.09</td><td>0.10</td><td>0.02<sup>†</sup></td></tr><tr><td>T2ADCb2000</td><td>0.25</td><td>0.03<sup>†</sup></td><td>1.00</td><td>0.16</td><td>1.00</td><td>&lt;0.0001<sup>†</sup></td><td>0.95</td><td>0.90</td><td>0.77</td><td>0.77</td><td>0.80</td><td>0.27</td></tr><tr><td>DCE</td><td>&lt;0.0001<sup>†</sup></td><td>0.00<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>1.00</td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td><td>&lt;0.0001<sup>†</sup></td></tr><tr><td>T2ADCDCE</td><td>0.23</td><td>0.02<sup>†</sup></td><td>0.95</td><td>0.14</td><td>0.95</td><td>&lt;0.0001<sup>†</sup></td><td>1.00</td><td>0.85</td><td>0.81</td><td>0.81</td><td>0.84</td><td>0.30</td></tr><tr><td>All</td><td>0.27</td><td>0.03<sup>†</sup></td><td>0.90</td><td>0.19</td><td>0.90</td><td>&lt;0.0001<sup>†</sup></td><td>0.85</td><td>1.00</td><td>0.67</td><td>0.67</td><td>0.70</td><td>0.22</td></tr><tr><td>T2 + ADC</td><td>0.15</td><td>0.01<sup>†</sup></td><td>0.77</td><td>0.09</td><td>0.77</td><td>&lt;0.0001<sup>†</sup></td><td>0.81</td><td>0.67</td><td>1.00</td><td>1.00</td><td>0.96</td><td>0.43</td></tr><tr><td>T2 + ADC + b2000</td><td>0.15</td><td>0.01<sup>†</sup></td><td>0.77</td><td>0.09</td><td>0.77</td><td>&lt;0.0001<sup>†</sup></td><td>0.81</td><td>0.67</td><td>1.00</td><td>1.00</td><td>0.96</td><td>0.43</td></tr><tr><td>T2 + ADC + DCE</td><td>0.15</td><td>0.01<sup>†</sup></td><td>0.80</td><td>0.10</td><td>0.80</td><td>&lt;0.0001<sup>†</sup></td><td>0.84</td><td>0.70</td><td>0.96</td><td>0.96</td><td>1.00</td><td>0.40</td></tr><tr><td>T2 + ADC + b2000 + DCE</td><td>0.02<sup>†</sup></td><td>0.00<sup>†</sup></td><td>0.28</td><td>0.02<sup>†</sup></td><td>0.27</td><td>&lt;0.0001<sup>†</sup></td><td>0.30</td><td>0.22</td><td>0.43</td><td>0.43</td><td>0.40</td><td>1.00</td></tr></tbody></table>

Abbreviations: SD: standard deviation, ADC: Apparent Diffusion Coefficient, DCE: Dynamic Contrast Enhanced, †: *p* < 0.05.

We also noticed that the combined T2ADC and T2ADCb2000 sequences, with DSCs of 0.54 and 0.54, respectively, obtained higher DSCs than the single T2, ADC, and b2000 sequences (0.51, 0.50, and 0.52, respectively). Finally, the T2ADC and T2ADCb2000 sequences had significantly higher DSCs than ADCs (*p* = 0.0275 and *p* = 0.0261, respectively).

The delineations from the T2 + ADC, T2 + ADC + b2000, T2 + ADC + DCE, T2 + ADC + b2000 + DCE sequences obtained equal or higher DSCs than the single delineations from the T2ADC, T2ADCb2000, T2ADCDCE, and all-combined sequences.

Two patients (high variability and low variability) are presented in Figure 3a and Figure 3b, respectively.

The Jaccard index, HD measure, and MDA were consistent with these findings. The T2ADCDCE analysis obtained higher Jaccard index (0.42), and lower MDA (4.27 mm) values than other sequences (Supplementary Tables S6–S8), while the b2000 sequence scored the lowest mean Hausdorff distance score (10.35 mm).

We also evaluated the similarity of the contours between each sequence for each patient and each reader. The mean DSCs are shown in Supplementary Table S9. Table S9 shows that there was a high correlation between each sequence that contained a T2 sequence. However, a lower similarity was found between sequences that contained a T2 sequence and ADC, b2000, and DCE sequences. This could mean that the ADC, b2000 sequence, and DCE sequence presented different information for the tumor segmentation.

### 3.4. Impact of Tumoral Volume and PI-RADS Classification

A significant correlation between the tumor volume and the reproducibility was found for each sequence, with a positive correlation between the DSC and the lesion volume, as shown in Supplementary Table S10. A significant higher reproducibility was observed between the contours of the PI-RADS 5 lesions than others on each sequence (except the DCE sequence), as shown in Supplementary Table S11. The multivariate logistic regression analysis showed that only the PI-RADS classification remains as a significantly correlated feature.
<figure><img src="image_7.png" /><figcaption>**Figure 3.** Example of tumor delineation on T2 sequence with low and high reproducibility ((a) mean of DICE: 0.31; (b) mean of DICE: 0.75).</figcaption></figure>

**4. Discussion**

The aim of this study was to assess the variability between readers using different MRI sequences. Even if some sequences stood out from the others, we observed a high variability between the contours of the tumor lesions with a maximum mean DSC of 0.55 reached using the combined analysis of the T2, ADC, and DCE sequences. The combined T2 + ADC + DWI + DCE sequence reached the highest DSC (0.58), with the interpretation of these results being limited by the correlation between the DSC and the tumor volume itself. The higher DSC could be due to a larger volume as a result of the union of these contours. Few data are available regarding the inter-reader variability of prostate cancer segmentation. With sixteen participants (four radiologists, four urologists, four urology trainees, and four non-clinicians), a high variance was found in the segmentation of two prostate tumors using single-slice images (mean DSC of 0.81 and 0.58) [21]. Van Schie et al. demonstrated that there were inter-institutional variations in the interpretation of mpMRIs (T2, ADC, and DCE-MRI-Derived Volume Transfer Constant (K trans)) for the prostate cancer delineation of 260 patients in the FLAME study cohort [22]. Similarity indexes were not available.

In a previous study focusing on tumor volume estimation only, DCE sequences achieved the highest result in comparison to the histopathological ground truth. It is noteworthy that, despite reaching the highest correlation, DCE sequences still lead to a substantial underestimation of the DIL volume [16], and suffered from a low reproducibility in our study. It is also noteworthy that, in our study, DCE delineation relied on a parametric map and not the DCE phases themselves. Each DCE sequence is comprised of 30 to 50 phases. Choosing a reference phase for the delineation would have exposed the risk that each reader selects a different phase. The delineation would have been possibly biased by the selection of the phase more than by the sequence itself. We thus used a semi-quantitative descriptive index, the IAUCG [20].
A precise tumor delineation is essential for the development and evaluation of focal treatments in prostate cancer. Our results highlight the high variability between three readers. It would be beneficial to train the observers to improve the consensus of delineations using mpMRIs and/or apply a margin to the delineated tumor to include more tumor tissue. It is noteworthy that in the FLAME trial [7], researchers did not apply a margin to the delineated tumor. With a mean MDA of 4.3 for the T2ADC delineation, a 5 mm margin could be proposed to account for inter-reader variability. Training and/or using additional margins are two solutions. Other efforts have been made to improve delineation reproducibility. The ReIGNITE study evaluated the impact of restriction spectrum imaging restriction scores on inter-reader variability, successfully increasing the median DSC from 0.48 (IQR 0.19–0.64) to 0.66 (IQR 0.55–0.73). Automatic segmentation tools for DIL delineation could possibly outperform inter-reader variability. While automatic prostate segmentation is often highly reproducible and robust [23,24], the automatic segmentation of the DIL performs poorly. Focusing on low-risk prostate cancer patients and with T2, B800, and ADC maps as input data, a model reached AUCs of 0.65, 0.73, and 0.89 for volumes >0.03 cc, >0.1 cc, and >0.5 cc, respectively. Specificity percentages were as low as 43%, 52%, and 74%, depending on the volume cut-off point. DSCs were not available [25]. With the addition of transfer learning and test-time augmentation, a model using a convolutional neural network approach reached a DSC of 0.59 using histopathology as the gold standard [26]. Tsui et al. evaluated the impact of an automatically generated DIL, leading to the conclusion that dosimetry planning was significantly impacted by the accuracy of the DIL delineation [27]. Finally, other imaging modalities, such as PSMA-positron emission tomography (PET), could be used to segment the DIL [28]. The impact of the choice of window levelling should be studied [29].

Despite their interesting character, these results highlight the difficulty in prostate
cancer segmentation, with the only comparison being either to the histopathology data
and its inherent limitation (fusion, cancer retraction after staining), or to a single expert
segmentation. The choice of the similarity metric (DSC) can be challenged. While the DSC
reflects the extent of the overlap [30], other metrics could be adapted to the delineation
task, such as the hit criteria used in the PI-CAI challenge (minimum overlap between the
detection and the ground truth) [31].

Despite our efforts in the design of the present study, several limitations should be
acknowledged. The intra-reader variability was not evaluated. Our cohort focuses on a
single scanner evaluation and on patients who were further treated either with surgery or
EBRT. By definition, this leads to a selection bias, limiting the external validity of our results
in a PI-RADS < 5 cohort. However, it is interesting to note that despite a cohort of >70%
PI-RADS 5 DILs, the highest DSC was as low as 0.55. This low DICE score signifies the
need to exercise caution for margin-free dose-escalation approaches and will be compared
to expert-only results in an additional study. The DCE sequences were analyzed using
the IAUCG and lead to an unexpected high variability and smaller mean volumes in
comparison to T2 and ADC sequences. While the IAUCG eased the delineation workflow, it
is possible that the use of the IAUCG was sub-optimal. It is noteworthy that the use of DCE
sequences is controversial [32–34] and biparametric MRIs (T2 and ADC) are more available
than mpMRIs, considering the long acquisition time of DCE MRIs, and the potential risks
and additional costs of gadolinium injections.

This cohort will be used to develop an automatic segmentation algorithm. Indeed, we
believe that the variability of prostate cancer segmentation should be taken into account
when designing and building automatic segmentation tools. Complementarity between
MRI sequences should also be incorporated. Multiparametric MRIs could thus be used to
its full potential by combining the input on differential segmentations depending on the
sequence. Furthers studies could focus on the factors influencing the variability of prostate
cancer segmentation and evaluate its dosimetric impact.
**5. Conclusions**

A high variability in prostate cancer delineation was observed, with T2, T2ADC, b2000, T2ADCb2000, T2ADCDCE, and all-combined sequences reaching the highest, yet very moderate, similarity scores. This inter-reader variability should be taken into account when planning focal treatments, such as stereotactic radiotherapy or dose-escalated radiotherapy, but also when designing automatic segmentation tools.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/biomedicines11123309/s1, Table S1: MRI parameters; Table S2: Delineation workflow example; Table S3: Prostate, transitional, and peripheral zone delineation (mean); Table S4: Mean of tumor volume contoured for each reader and each sequence; Table S5: Inter-reader variability for the definition of the index lesion for each sequence (DICE); Table S6: Inter-reader variability for the definition of the index lesion for each sequence (Jaccard); Table S7: Inter-reader variability for the definition of the index lesion for each sequence (Max Hausdorff); Table S8: Inter-reader variability for the definition of the index lesion for each sequence (Mean distance to agreement); Table S9: Intersequence contour similarity evaluation (mean DICE coefficients); Table S10: Impact of tumor volume and PI-RADS on inter-reader variability; Table S11: Comparison between mean DICE coefficients for each sequence.

**Author Contributions:** Conceptualization: T.J., E.S., V.T. and V.B.; data curation: T.J., E.S. and V.B.; methodology: T.J., E.S. and V.B.; supervision: U.S. and V.B.; validation, T.J., E.S., U.S., V.T., G.F., A.V., T.-A.N. and V.B.; writing—original draft: T.J. and V.B.; writing—review and editing: T.J., U.S., V.T., G.F., A.V., T.-A.N. and V.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research did not receive any funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of the University Hospital of Brest (B2023CE.33) on 17 July 2023.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data are available upon request to the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

**References**

1. Wang, L.; Lu, B.; He, M.; Wang, Y.; Wang, Z.; Du, L. Prostate Cancer Incidence and Mortality: Global Status and Temporal Trends in 89 Countries from 2000 to 2019. *Front. Public Health* **2022**, *10*, 811044. [CrossRef] [PubMed]

2. Barsouk, A.; Padala, S.A.; Vakiti, A.; Mohammed, A.; Saginala, K.; Thandra, K.C.; Rawla, P.; Barsouk, A. Epidemiology, Staging and Management of Prostate Cancer. *Med. Sci.* **2020**, *8*, 28. [CrossRef] [PubMed]

3. Dearnaley, D.P.; Jovic, G.; Syndikus, I.; Khoo, V.; Cowan, R.A.; Graham, J.D.; Aird, E.G.; Bottomley, D.; Huddart, R.A.; Jose, C.C.; et al. Escalated-Dose versus Control-Dose Conformal Radiotherapy for Prostate Cancer: Long-Term Results from the MRC RT01 Randomised Controlled Trial. *Lancet Oncol.* **2014**, *15*, 464–473. [CrossRef] [PubMed]

4. Kuban, D.A.; Tucker, S.L.; Dong, L.; Starkschall, G.; Huang, E.H.; Cheung, M.R.; Lee, A.K.; Pollack, A. Long-Term Results of the M. D. Anderson Randomized Dose-Escalation Trial for Prostate Cancer. *Int. J. Radiat. Oncol. Biol. Phys.* **2008**, *70*, 67–74. [CrossRef] [PubMed]

5. Beckendorf, V.; Guerif, S.; Le Prisé, E.; Cosset, J.-M.; Bougnoux, A.; Chauvet, B.; Salem, N.; Chapet, O.; Bourdain, S.; Bachaud, J.-M.; et al. 70 Gy versus 80 Gy in Localized Prostate Cancer: 5-Year Results of GETUG 06 Randomized Trial. *Int. J. Radiat. Oncol. Biol. Phys.* **2011**, *80*, 1056–1063. [CrossRef] [PubMed]

6. Morris, W.J.; Tyldesley, S.; Rodda, S.; Halperin, R.; Pai, H.; McKenzie, M.; Duncan, G.; Morton, G.; Hamm, J.; Murray, N. Androgen Suppression Combined with Elective Nodal and Dose Escalated Radiation Therapy (the ASCENDE-RT Trial): An Analysis of Survival Endpoints for a Randomized Trial Comparing a Low-Dose-Rate Brachytherapy Boost to a Dose-Escalated External Beam Boost for High- and Intermediate-Risk Prostate Cancer. *Int. J. Radiat. Oncol. Biol. Phys.* **2017**, *98*, 275–285. [CrossRef] [PubMed]

7. Kerkmeijer, L.G.W.; Groen, V.H.; Pos, F.J.; Haustermans, K.; Monninkhof, E.M.; Smeenk, R.J.; Kunze-Busch, M.; de Boer, J.C.J.; van der Voort van Zijp, J.; van Vulpen, M.; et al. Focal Boost to the Intraprostatic Tumor in External Beam Radiotherapy for Patients With Localized Prostate Cancer: Results From the FLAME Randomized Phase III Trial. *J. Clin. Oncol. Off. J. Am. Soc. Clin. Oncol.* **2021**, *39*, 787–796. [CrossRef]
8. Prada, P.J.; Cardenal, J.; García Blanco, A.; Andreescu, J.; Ferri, M.; Anchuelo, J.; Diaz de Cerio, I.; Sierrasesumaga, N.; Vázquez, A.; Pacheco, M.; et al. Focal High-Dose-Rate Brachytherapy for Localized Prostate Cancer: Toxicity and Preliminary Biochemical Results. *Strahlenther. Onkol. Organ Dtsch. Rontgenes. AI* **2020**, *196*, 222–228. [CrossRef]

9. Zhong, A.Y.; Lui, A.J.; Katz, M.S.; Berlin, A.; Kamran, S.C.; Kishan, A.U.; Murthy, V.; Nagar, H.; Seible, D.; Stish, B.J.; et al. Use of Focal Radiotherapy Boost for Prostate Cancer: Radiation Oncologists' Perspectives and Perceived Barriers to Implementation. *Radiat. Oncol. Lond Engl. 2023*, *18*, 188. [CrossRef]

10. Turkbey, B.; Mani, H.; Aras, O.; Rastinehad, A.R.; Shah, V.; Bernardo, M.; Pohida, T.; Daar, D.; Benjamin, C.; McKinney, Y.L.; et al. Correlation of Magnetic Resonance Imaging Tumor Volume with Histopathology. *J. Urol.* **2012**, *188*, 1157–1163. [CrossRef]

11. Isebaert, S.; Van den Bergh, L.; Haustermans, K.; Joniau, S.; Lerut, E.; De Wever, L.; De Keyzer, F.; Budiharto, T.; Slagmolen, P.; Van Poppel, H.; et al. Multiparametric MRI for Prostate Cancer Localization in Correlation to Whole-Mount Histopathology. *J. Magn. Reson. Imaging JMRI* **2013**, *37*, 1392–1401. [CrossRef] [PubMed]

12. Haider, M.A.; van der Kwast, T.H.; Tanguay, J.; Evans, A.J.; Hashmi, A.-T.; Lockwood, G.; Trachtenberg, J. Combined T2-Weighted and Diffusion-Weighted MRI for Localization of Prostate Cancer. *AJR Am. J. Roentgenol.* **2007**, *189*, 323–328. [CrossRef] [PubMed]

13. Bratan, F.; Melodelima, C.; Souchon, R.; Hoang Dinh, A.; Mège-Lechevallier, F.; Crouzet, S.; Colombel, M.; Gelet, A.; Rouvière, O. How Accurate Is Multiparametric MR Imaging in Evaluation of Prostate Cancer Volume? *Radiology* **2015**, *275*, 144–154. [CrossRef] [PubMed]

14. Kasivisvanathan, V.; Rannikko, A.S.; Borghi, M.; Panebianco, V.; Mynderse, L.A.; Vaarala, M.H.; Briganti, A.; Budäus, L.; Hellawell, G.; Hindley, R.G.; et al. MRI-Targeted or Standard Biopsy for Prostate-Cancer Diagnosis. *N. Engl. J. Med.* **2018**, *378*, 1767–1777. [CrossRef] [PubMed]

15. Eklund, M.; Jäderling, F.; Discacciati, A.; Bergman, M.; Annerstedt, M.; Aly, M.; Glaessgen, A.; Carlsson, S.; Grönberg, H.; Nordström, T.; et al. MRI-Targeted or Standard Biopsy in Prostate Cancer Screening. *N. Engl. J. Med.* **2021**, *385*, 908–920. [CrossRef] [PubMed]

16. Sun, C.; Chatterjee, A.; Yousuf, A.; Antic, T.; Eggener, S.; Karczmar, G.S.; Oto, A. Comparison of T2-Weighted Imaging, DWI, and Dynamic Contrast-Enhanced MRI for Calculation of Prostate Cancer Index Lesion Volume: Correlation with Whole-Mount Pathology. *AJR Am. J. Roentgenol.* **2019**, *212*, 351–356. [CrossRef] [PubMed]

17. Lui, A.J.; Kallis, K.; Zhong, A.Y.; Hussain, T.S.; Conlin, C.; Digma, L.A.; Phan, N.; Mathews, I.T.; Do, D.D.; Domingo, M.R.; et al. ReIGNITE Radiation Therapy Boost: A Prospective, International Study of Radiation Oncologists' Accuracy in Contouring Prostate Tumors for Focal Radiation Therapy Boost on Conventional Magnetic Resonance Imaging Alone or With Assistance of Restriction Spectrum Imaging. *Int. J. Radiat. Oncol. Biol. Phys.* **2023**, *117*, 1145–1152. [CrossRef]

18. Barentsz, J.O.; Richenberg, J.; Clements, R.; Choyke, P.; Verma, S.; Villeirs, G.; Rouviere, O.; Logager, V.; Fütterer, J.J. ESUR Prostate MR Guidelines 2012. *Eur. Radiol.* **2012**, *22*, 746–757. [CrossRef]

19. Weinreb, J.C.; Barentsz, J.O.; Choyke, P.L.; Cornud, F.; Haider, M.A.; Macura, K.; Margolis, D.; Schnall, M.D.; Shtern, F.; Tempany, C.; et al. PI-RADS Prostate Imaging-Reporting and Data System, v2.1. Available online: https://www.acr.org/-/media/ACR/Files/RADS/Pi-RADS/PIRADS-V2-1.pdf?la=en (accessed on 25 September 2023).

20. Mazaheri, Y.; Akin, O.; Hricak, H. Dynamic Contrast-Enhanced Magnetic Resonance Imaging of Prostate Cancer: A Review of Current Methods and Applications. *World J. Radiol.* **2017**, *9*, 416–425. [CrossRef]

21. Chen, M.Y.; Woodruff, M.A.; Dasgupta, P.; Rukin, N.J. Variability in Accuracy of Prostate Cancer Segmentation among Radiologists, Urologists, and Scientists. *Cancer Med.* **2020**, *9*, 7172–7182. [CrossRef]

22. van Schie, M.A.; Dinh, C.V.; van Houdt, P.J.; Pos, F.J.; Heijmink, S.W.T.J.P.; Kerkmeijer, L.G.W.; Kotte, A.N.T.J.; Oyen, R.; Haustermans, K.; van der Heide, U.A. Contouring of Prostate Tumors on Multiparametric MRI: Evaluation of Clinical Delineations in a Multicenter Radiotherapy Trial. *Radiother. Oncol. J. Eur. Soc. Ther. Radiol. Oncol.* **2018**, *128*, 321–326. [CrossRef] [PubMed]

23. Ushinsky, A.; Bardis, M.; Glavis-Bloom, J.; Uchio, E.; Chantaduly, C.; Nguyenat, M.; Chow, D.; Chang, P.D.; Houshyar, R. A. 3D-2D Hybrid U-Net Convolutional Neural Network Approach to Prostate Organ Segmentation of Multiparametric MRI. *AJR Am. J. Roentgenol.* **2021**, *216*, 111–116. [CrossRef] [PubMed]

24. Bardis, M.; Houshyar, R.; Chantaduly, C.; Tran-Harding, K.; Ushinsky, A.; Chahine, C.; Rupasinghe, M.; Chow, D.; Chang, P. Segmentation of the Prostate Transition Zone and Peripheral Zone on MR Images with Deep Learning. *Radiol. Imaging Cancer* **2021**, *3*, e200024. [CrossRef] [PubMed]

25. Arif, M.; Schoots, I.G.; Castillo Tovar, J.; Bangma, C.H.; Krestin, G.P.; Roobol, M.J.; Niessen, W.; Veenland, J.F. Clinically Significant Prostate Cancer Detection and Segmentation in Low-Risk Patients Using a Convolutional Neural Network on Multi-Parametric MRI. *Eur. Radiol.* **2020**, *30*, 6582–6592. [CrossRef] [PubMed]

26. Hoar, D.; Lee, P.Q.; Guida, A.; Patterson, S.; Bowen, C.V.; Merrimen, J.; Wang, C.; Rendon, R.; Beyea, S.D.; Clarke, S.E. Combined Transfer Learning and Test-Time Augmentation Improves Convolutional Neural Network-Based Semantic Segmentation of Prostate Cancer from Multi-Parametric MR Images. *Comput. Methods Programs Biomed.* **2021**, *210*, 106375. [CrossRef] [PubMed]

27. Tsui, J.M.G.; Kehayias, C.E.; Leeman, J.E.; Nguyen, P.L.; Peng, L.; Yang, D.D.; Moningi, S.; Martin, N.; Orio, P.F.; D'Amico, A.V.; et al. Assessing the Feasibility of Using Artificial Intelligence-Segmented Dominant Intraprostatic Lesion for Focal Intraprostatic Boost With External Beam Radiation Therapy. *Int. J. Radiat. Oncol. Biol. Phys.* **2024**, *118*, 74–84. [CrossRef] [PubMed]

28. Gaudreault, M.; Chang, D.; Hardcastle, N.; Jackson, P.; Kron, T.; Hofman, M.S.; Siva, S. Feasibility of Biology-Guided Radiotherapy Using PSMA-PET to Boost to Dominant Intraprostatic Tumour. *Clin. Transl. Radiat. Oncol.* **2022**, *35*, 84–89. [CrossRef]
29. Draulans, C.; De Roover, R.; van der Heide, U.A.; Kerkmeijer, L.; Smeenk, R.J.; Pos, F.; Vogel, W.V.; Nagarajah, J.; Janssen, M.; Isebaert, S.; et al. Optimal 68Ga-PSMA and 18F-PSMA PET Window Levelling for Gross Tumour Volume Delineation in Primary Prostate Cancer. *Eur. J. Nucl. Med. Mol. Imaging* **2021**, *48*, 1211–1218. [CrossRef]

30. Guzene, L.; Beddok, A.; Nioche, C.; Modzelewski, R.; Loiseau, C.; Salleron, J.; Thariat, J. Assessing Interobserver Variability in the Delineation of Structures in Radiation Oncology: A Systematic Review. *Int. J. Radiat. Oncol. Biol. Phys.* **2023**, *115*, 1047–1060. [CrossRef]

31. Saha, A.; Twilt, J.J.; Bosma, J.S.; van Ginneken, B.; Yakar, D.; Elschot, M.; Veltman, J.; Fütterer, J.; de Rooij, M.; Huisman, H. Artificial Intelligence and Radiologists at Prostate Cancer Detection in MRI: The PI-CAI Challenge (Study Protocol) (1.1). *Zenodo* **2022**. [CrossRef]

32. Stanzione, A.; Imbriaco, M.; Cocozza, S.; Fusco, F.; Rusconi, G.; Nappi, C.; Mirone, V.; Mangiapia, F.; Brunetti, A.; Ragozzino, A.; et al. Biparametric 3T Magnetic Resonance Imaging for Prostatic Cancer Detection in a Biopsy-Naïve Patient Population: A Further Improvement of PI-RADS V2? *Eur. J. Radiol.* **2016**, *85*, 2269–2274. [CrossRef]

33. Hansford, B.G.; Peng, Y.; Jiang, Y.; Vannier, M.W.; Antic, T.; Thomas, S.; McCann, S.; Oto, A. Dynamic Contrast-Enhanced MR Imaging Curve-Type Analysis: Is It Helpful in the Differentiation of Prostate Cancer from Healthy Peripheral Zone? *Radiology* **2015**, *275*, 448–457. [CrossRef]

34. Scialpi, M.; Prosperi, E.; D'Andrea, A.; Martorana, E.; Malaspina, C.; Palumbo, B.; Orlandi, A.; Falcone, G.; Milizia, M.; Mearini, L.; et al. Biparametric versus Multiparametric MRI with Non-Endorectal Coil at 3T in the Detection and Localization of Prostate Cancer. *Anticancer Res.* **2017**, *37*, 1263–1271. [CrossRef]

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.