# Confidence intervals for performance estimates in 3D medical image segmentation

Rosana El Jurdi, Gaël Varoquaux, Olivier Colliot

<figure><img src="image_1.png" /></figure>

## To cite this version:

Rosana El Jurdi, Gaël Varoquaux, Olivier Colliot. Confidence intervals for performance estimates in 3D medical image segmentation. 2023. hal-04166803

HAL Id: hal-04166803

https://hal.science/hal-04166803

Preprint submitted on 20 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Confidence intervals for performance estimates in brain MRI segmentation

Rosana El Jurdi, Gaël Varoquaux, Olivier Colliot

► To cite this version:

Rosana El Jurdi, Gaël Varoquaux, Olivier Colliot. Confidence intervals for performance estimates in brain MRI segmentation. Medical Image Analysis, 2025, 103, pp.103565. 10.1016/j.media.2025.103565. hal-04166803v2

HAL Id: hal-04166803

https://hal.science/hal-04166803v2

Submitted on 23 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Confidence intervals for performance estimates in 3D medical image segmentation

Rosana El Jurdi, Gaël Varoquaux and Olivier Colliot

*Abstract*—Medical segmentation models are evaluated empirically. As such an evaluation is based on a limited set of example images, it is unavoidably noisy. Beyond a mean performance measure, reporting confidence intervals is thus crucial. However, this is rarely done in medical image segmentation. The width of the confidence interval depends on the test set size and on the spread of the performance measure (its standard-deviation across of the test set). For classification, many test images are needed to avoid wide confidence intervals. Segmentation, however, has not been studied, and it differs by the amount of information brought by a given test image. In this paper, we study the typical confidence intervals in medical image segmentation. We carry experiments on 3D image segmentation using the standard nnU-net framework, two datasets from the Medical Decathlon challenge and two performance measures: the Dice accuracy and the Hausdorff distance. We show that the parametric confidence intervals are reasonable approximations of the bootstrap estimates for varying test set sizes and spread of the performance metric. Importantly, we show that the test size needed to achieve a given precision is often much lower than for classification tasks. Typically, a 1% wide confidence interval requires about 100-200 test samples when the spread is low (standard-deviation around 3%). More difficult segmentation tasks may lead to higher spreads and require over 1000 samples.

*Index Terms*—Segmentation, Performance measure, Validation, Statistical analysis, Confidence interval, Standard error.

## I. INTRODUCTION

Solid evaluation is crucial for automatic tools used to process medical images, as medical professionals must trust the automated system's output. Most modern segmentation tools rely on deep learning [1], and their performance depends on details of the training run. Trained models are evaluated by using unseen data to estimate their expected performance. Evaluation is harder than it seems. First, it requires an adequate choice of metric [2], [3], [4]. Second, it is inevitably noisy, if only because it uses a finite number of samples. It is thus crucial to quantify the uncertainty on an estimated performance, for instance using confidence intervals¹ (CIs). Indeed, model developers can easily fall into the trap of apparent model

improvements due to evaluation noise, that do not generalize to new data, as in some medical-imaging competitions [5], [6].

And yet, evaluation uncertainty or CIs are seldom reported in the medical image segmentation literature (including in our own previous papers, e.g. [7], [8], [9]). To illustrate how prevalent the problem is, we conducted a survey of all papers on 3D medical image segmentation published in 2022 in the following three journals: IEEE Transactions on Medical Imaging, Medical Image Analysis and Journal of Medical Imaging. As reported in Table I, only 11 of the corresponding 133 paper reported CIs. Such quick survey, in no way exhaustive, does suggest that, across three representative journals of the community, the vast majority (> 90%) of recent papers do not report confidence intervals for their trained models.

Two main factors affect the precision of model evaluation: the size of the test set, and how much the performance metric varies across test-set samples. Given a high spread of the metric across the test set, the precision will be low and the confidence interval will be wider. Alternatively, increasing the sample size of the test set leads to tighter confidence intervals. In 3D medical image segmentation, the size of the set used to evaluate the performance is often of small to moderate size, typically in the order of dozens of subjects, at best hundreds, as obtaining the ground truth requires voxel-wise annotation by trained raters. One fear is thus that most segmentation performances assessments will be associated to large error bars. For image classification, studies have shown that large sample sizes are needed for a precise estimation of the prediction accuracy (typically 10,000 samples to achieve a 1%-wide confidence interval) [10], [11]. Surprisingly, this question has not yet been addressed for medical image segmentation.

This paper addresses the following questions. What precision can be expected in 3D medical image segmentation for typical test set sizes? How trustworthy are the average performance estimates (for instance Dice coefficients or Hausdorff distance) reported in medical image segmentation papers? The main objectives are: i) to raise awareness on the importance of reporting confidence interval on independent test sets; ii) to provide the community with typical values of CI that can be expected for various sample sizes and performance variability.

To this end, we present in this paper a series of experiments that deploy nnU-net [12] –a standard framework for medical image segmentation– on two classical segmentation tasks from the Segmentation Decathlon Challenge [13] in order to estimate confidence intervals that are obtained for test sets of variable size. For each task, we study the following performance measures: the Dice accuracy and the Hausdorff distance. We then report CIs using both bootstrap (which is

The research leading to these results has received funding from the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute) and reference ANR-10-IAHU-0006 (Agence Nationale de la Recherche-10-IA Institut Hospitalo-Universitaire-6)".

Rosana El Jurdi and Olivier Colliot are with Sorbonne Université, Institut du Cerveau – Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié-Salpêtrière, F-75013, Paris, France.

Gaël Varoquaux is with the Soda team, Inria, Saclay, France.

Corresponding author: Rosana El Jurdi (eljurdi.rosana@gmail.com).

¹Here we use confidence interval in its broader meaning; in terms of strict statistical definitions, this encompasses the notion of credible intervals.
# Confidence Intervals for Performance Estimates in Brain MRI Segmentation

Rosana El Jurdi<sup>a,*</sup>, Gaël Varoquaux<sup>b</sup>, Olivier Colliot<sup>a</sup>

<sup>a</sup>Sorbonne Université, Institut du Cerveau – Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié-Salpêtrière, F-75013, Paris, France.

<sup>b</sup>Soda team, Inria, Saclay, France

## Abstract

Medical segmentation models are evaluated empirically. As such an evaluation is based on a limited set of example images, it is unavoidably noisy. Beyond a mean performance measure, reporting confidence intervals is thus crucial. However, this is rarely done in medical image segmentation. The width of the confidence interval depends on the test set size and on the spread of the performance measure (its standard-deviation across the test set). For classification, many test images are needed to avoid wide confidence intervals. Segmentation, however, has not been studied, and it differs by the amount of information brought by a given test image. In this paper, we study the typical confidence intervals in the context of segmentation in 3D brain magnetic resonance imaging (MRI). We carry experiments on using the standard nnU-net framework, two datasets from the Medical Decathlon challenge that concern brain MRI (hippocampus and brain tumor segmentation) and two performance measures: the Dice Similarity Coefficient and the Hausdorff distance. We show that the parametric confidence intervals are reasonable approximations of the bootstrap estimates for varying test set sizes and spread of the performance metric. Importantly, we show that the test size needed to achieve a given precision is often much lower than for classification tasks. Typically, a 1% wide confidence interval requires about 100-200 test samples when the spread is low (standard-deviation around 3%). More difficult segmentation tasks may lead to higher spreads and require over 1000 samples. The corresponding code and notebooks are available on GitHub at https://github.com/rosanajurdi/SegVal_Repo

**Keywords:** Segmentation, Performance measure, Validation, Statistical analysis, Confidence interval, Standard error.

## I. Introduction

Solid evaluation is crucial for automatic tools used to process medical images, as medical professionals must trust the automated system's output. Most modern segmentation tools rely on deep learning Hesamian et al. (2019), and their performance depends on details of the training run. Trained models are evaluated by using unseen data to estimate their expected performance. Evaluation is harder than it seems. First, it requires an adequate choice of metric Reinke et al. (2022); Maier-Hein et al. (2024); Reinke et al. (2024). Second, it is inevitably noisy, if only because it uses a finite number of samples. It is thus crucial to quantify the uncertainty on an estimated

performance, for instance using confidence intervals<fnref n="1" /> (CIs). Indeed, model developers can easily fall into the trap of apparent model improvements due to evaluation noise, that do not generalize to new data, as in some medical imaging competitions Maier-Hein et al. (2018); Varoquaux and Cheplygina (2022).

And yet, evaluation uncertainty or CIs are seldom reported in the medical image segmentation literature. To illustrate how prevalent the problem is, we conducted a survey of all papers on 3D medical image segmentation published in 2022 in the following three journals: IEEE Transactions on Medical Imaging, Medical Image Analysis and Journal of Medical

<footnote n="1">¹Here we use confidence interval in its broader meaning; in terms of strict statistical definitions, this encompasses the notion of credible intervals.</footnote>

<footnote>*Corresponding author: eljurdi.rosana@gmail.com</footnote>
<table><caption>TABLE I: Papers reporting precision of segmentation-performance estimates. Summary of the survey conducted on papers tackling 3D medical image segmentation via machine/deep learning published in 2022 in the journals IEEE Transactions on Medical Imaging (IEEE TMI), Medical Image Analysis (MedIA) and Journal of Medical Imaging (JMI). We report the number of papers and experiments and the number and proportion of papers that reported the precision of their estimates (we counted both papers reporting confidence intervals (CI) and those reporting standard error of the mean (SEM), since it is very often possible to derive a reasonable estimate of CI from SEM).</caption><thead><tr><th>Journal</th><th>Nb papers</th><th>Nb exp.</th><th>Nb (%) papers reporting CI (or SEM)</th></tr></thead><tbody><tr><td>TMI</td><td>51</td><td>107</td><td>3 (6%)</td></tr><tr><td>MedIA</td><td>70</td><td>171</td><td>7 (10%)</td></tr><tr><td>JMI</td><td>12</td><td>22</td><td>1 (8%)</td></tr><tr><td>Total</td><td>133</td><td>300</td><td>11 (8%)</td></tr></tbody></table>

the most general approach) and a parametric estimation, and varying the test set size. We find that the parametric estimation is in general reasonable, even for small test sets and for metrics which distribution is far from being Gaussian. Building on this, we perform simulations for other sizes and spreads. We demonstrate that the test size needed to achieve a given precision is lower for segmentation than for classification. This is due to the continuous nature of the evaluation metrics [14]. Indeed, segmentation algorithms are evaluated with continuous metrics that aggregate multiple observations at the level of a single image. As each image brings more information to characterize the errors of the algorithm, tight confidence intervals can often be obtained with a moderate number of images.

The remaining sections of this paper are organized as follows. In section II, we describe the statistical tools that we use in this paper to conduct our analysis. In section III, we present the datasets examined and the experimental setup. Section IV reports results on the whole test sets. In Section V, we then experimentally study how the precision varies when varying the test set size. In Section VI, we provide a table of confidence intervals when varying the test set sizes and the performance spread (as measured by $\sigma$). Finally, we discuss our findings and conclude in sections VII and VIII

## II. COMPUTING CONFIDENCE INTERVALS

In this paper, we are interested in computing confidence intervals for the mean of a given performance metric (e.g. Dice, Hausdorff). We will consider 95% confidence intervals, which are the most common, but everything could be done similarly for other confidence levels.

A general way to compute confidence intervals is through the bootstrap [15], [16]. It has the advantage that it does not make any assumption on the distribution of the metric nor on the sampling distribution of the mean. Let us briefly remind the corresponding procedure. Given a test set of size $n$, $M$ bootstrap samples of size $n$ are drawn with replacement from the test set. Each bootstrap sample is denoted as $S_m^*$ where $m \in \{1, \dots, M\}$. In our experiments,

we use $M = 15000$. We denote $\mu_m^* \approx 2$ the mean of $S_m^*$. The 95% confidence interval $[a^*, b^*]$ is the set of values between the 2.5% and 97.5% percentiles of the sorted bootstrap means $\{\mu_1^*, \mu_2^*, \dots, \mu_m^*, \dots, \mu_M^*\}$. The standard error of the mean $\mu^*$ obtained via bootstrapping (SEM$^*$) is the standard deviation of the means of all bootstrap samples: $SEM^* = \sqrt{\frac{1}{M} \sum_{m=1}^M (\mu_m^* - \mu^*)^2}$ where $\mu^*$ is the mean of the bootstrap sample means $\mu_m^*$.

While the bootstrap is easy to perform, it must be performed by the researcher evaluating the model, since it requires to have the performance value for each subject of the test set. Knowing typical confidence intervals to expect given a test size can help evaluating the precision of a result reported without bootstrap. To build such charts, we perform simulations to gauge the following classical approximation:

$$ 
\begin{gathered}
SEM = \frac{\sigma}{\sqrt{n}} \qquad (1) \\
[a, b] = [\mu - 1.96 \times SEM, \mu + 1.96 \times SEM]
\end{gathered}
$$

where $\mu$ and $\sigma$ respectively denote the mean and standard deviation of the metric over the test set, $n$ the test set size, SEM the standard error of the mean and $a$ (resp. $b$) are the lower (resp. upper) bounds of the 95% confidence interval. This approximation relies on the assumption that the sampling distribution of the mean follows a normal distribution. It holds asymptotically with only weak assumptions on the metric distribution (finite expected value and variance) and thus should be valid for sufficiently large samples. However, in practice, it is unknown under which test size this approximation is reasonable. This will be assessed through the experiments in the paper. If the metric distribution is close to normal, the approximation is more likely to be correct even for relatively small samples (the sampling distribution of the mean would thus follow a Student's distribution which becomes quickly close to a normal distribution when $n$ increases). In the following, we will refer to estimations derived from Equations 1 as parametric estimations to distinguish them from bootstrap estimations.

Finally, we derive the following quantities from the confidence intervals. First, we are not interested in the absolute values of the boundaries of the confidence interval. Thus, we report confidence intervals independently of the mean, as follows:

$$ CI = [a - \mu, b - \mu] \qquad (2) $$

and we also define the width of CI as $w = b - a$. When the parametric estimation is used, $w = 2 \times 1.96 \times SEM$. Moreover, different performance measures have different ranges of values (for instance Dice is between 0% and 100% while the Hausdorff distance can be arbitrarily large). For the reader to have a better intuition of the precision of the estimate, we thus normalize the width by the mean value of the metric. To this end, we define the normalized width:

$$ \text{Normalized width} \qquad \nu = \frac{w}{\mu} \qquad (3) $$

<footnote n="2">²Throughout the paper, the bootstrap estimate of a given $x$ is always denoted as $x^*$</footnote>
Imaging. As reported in Table 1, only 11 of the correspond-
ing 133 paper reported CIs. Such quick survey, in no way ex-
haustive, does suggest that, across three representative jour-
nals of the community, the vast majority (> 90%) of recent
papers do not report confidence intervals for their trained
models. Since the initial writing of the present paper, a survey
of segmentation papers published at MICCAI (Medical Im-
age Computing and Computer-Assisted Intervention) 2023
has been reported in Christodoulou et al. (2024). They found
that more than 50% of papers do not assess performance vari-
ability at all and that only 0.5% of papers reported confidence
intervals (CIs).

Two main factors affect the precision of model evaluation: the size of the test set, and how much the performance metric varies across test-set samples. Given a high spread of the metric across the test set, the precision will be low and the confidence interval will be wider. Alternatively, increasing the sample size of the test set leads to tighter confidence intervals. In 3D medical image segmentation, the size of the set used to evaluate the performance is often of small to moderate size, typically in the order of dozens of subjects, at best hundreds, as obtaining the ground truth requires voxel-wise annotation by trained raters. One fear is thus that most segmentation performance assessments will be associated to large error bars. For image classification, studies have shown that large sample sizes are needed for a precise estimation of the prediction accuracy (typically 10,000 samples to achieve a 1%-wide confidence interval) Varoquaux (2018); Varoquaux and Colliot (2023). Surprisingly, this question has not yet been addressed for medical image segmentation.

The scope of this paper is that of segmentation of 3D brain
magnetic resonance images (MRI). Within this scope, the pa-
per addresses the following questions. What precision can
be expected in 3D brain MRI segmentation for typical test
set sizes? How trustworthy are the average performance es-
timates (for instance Dice coefficients or Hausdorff distance)
reported in brain MRI segmentation papers? The main ob-
jectives are: i) to raise awareness on the importance of report-
ing confidence interval on independent test sets; ii) to provide
the community with typical values of CI that can be expected
for various sample sizes and performance variability.

To this end, we present in this paper a series of experi-
ments that deploy nnU-net Isensee et al. (2021) –a standard
framework for medical image segmentation– on two classical
segmentation tasks from the Segmentation Decathlon Chal-
lenge Antonelli et al. (2022) in order to estimate confidence

intervals that are obtained for test sets of variable size. The
choice of nnUnet was based on its victory in the 2018 chal-
lenge, and since then, it has consistently performed compa-
rably to top-ranked models in the live challenge, with little
to no difference in performance Antonelli et al. (2022). For
each task, we study the following performance measures: the
Dice Similarity Coefficient and the Hausdorff distance. We
then report CIs using both bootstrap (which is the most gen-
eral approach) and a parametric estimation, and varying the
test set size. We find that the parametric estimation is in gen-
eral reasonable, even for small test sets and for metrics which
distribution is far from being Gaussian. Building on this, we
perform simulations for other sizes and spreads. We demon-
strate that the test size needed to achieve a given precision is
lower for segmentation than for classification. This is due to
the continuous nature of the evaluation metrics Altman and
Royston (2006). Indeed, segmentation algorithms are evalu-
ated with continuous metrics that aggregate multiple observa-
tions at the level of a single image. As each image brings more
information to characterize the errors of the algorithm, tight
confidence intervals can often be obtained with a moderate
number of images.

We previously published preliminary work as a conference
paper assessing confidence intervals for segmentation El Jurdi
and Colliot (2023). The present work differs on many aspects:
i) we use a state-of-the-art nnU-net instead of a standard U-
Net; ii) we study the Brain Tumor task in addition to the hip-
pocampus; iii) we study both Dice Similarity Coefficient and
Hausdorff distance as metrics; iv) we provide an extensive dis-
cussion.

The remaining sections of this paper are organized as fol-
lows. In section 2, we describe the statistical tools that we use
in this paper to conduct our analysis. In section 3, we present
the datasets examined and the experimental setup. Section 4
reports results on the whole test sets. In Section 5, we then
experimentally study how the precision varies when varying
the test set size. In Section 6, we provide a table of confidence
intervals when varying the test set sizes and the performance
spread (as measured by σ). Finally, we discuss our findings
and conclude in sections 7 and 8.

## 2. Computing Confidence Intervals

In this paper, we are interested in computing confidence
intervals for the mean of a given performance metric (e.g.
Dice, Hausdorff). We will consider 95% confidence intervals,
where $\mu$ is the average performance metric mean across the test set. In the case of bootstrap estimation, the above quantities are denoted as CI*, $w^*$ and $v^*$.

# III. DATASETS AND SEGMENTATION METHOD

## A. Datasets

We used the Hippocampus and Brain Tumor datasets from the Medical Decathlon challenge [13]. The Hippocampus dataset is mono-modal and composed of 260 3D T1-weighted (T1-w) MPRAGE brain images. The task is to segment the anterior and posterior parts of the hippocampus (denoted respectively as L1 and L2 according to the challenge annotations). The Brain Tumor dataset is a multi-modal dataset comprising T1w preconstrast, T1w postcontrast, fluid-attenuated inversion recovery (FLAIR), and T2-weighted images. It is composed of 484 samples and dedicated to glioma segmentation. The regions to segment are the edema (L1), the non-enhancing tumor (L2), and the enhancing tumor (L3), according to the Decathlon annotations. The data contains a subset of the data used in the 2016 and 2017 Brain Tumor Segmentation (BraTS) challenges [17]. The brain tumor task is considered difficult (best Dice is around 70%) while the Hippocampus is easier in comparison (best Dice around 90%).

From the 260 and 484 samples of the Hippocampus and Brain Tumor dataset respectively, 100 patients were randomly selected for training, 50 for validation and the remaining samples constituted the test set. The size of the training and validation subsets were the same for both datasets to ensure that the number of training samples has no impact on the model performances.

## B. Segmentation Methods

We used the Decathlon Challenge winner's nnU-net, as a framework to perform our experiments. nnU-net [12] is a fully automated segmentation framework that makes use of typical 2D and 3D U-Net architectures by dynamically and automatically adapting the hyper-parameters to a particular dataset. Given an arbitrary dataset, the nnU-net pipeline first extracts data fingerprints (e.g. voxel spacing, intensity distribution, modalities, class ratios etc.). Pre-processing of the datasets is conducted automatically via cropping to non-zero volumes to reduce the computational burden, resampling relative to voxel spacing to allow spatial semantics, and z-score normalization, if need be. We allowed nnU-Net to automatically determine the best configuration of parameters. In nnU-net, one can either predetermine which architecture to train on the given dataset or allow nnU-net to choose the best configuration. In the present paper, we conducted our experiments using the 3D full-resolution U-Net and the 2D U-Net. It is possible that even better performances could have been obtained using the cascade approach or letting nnU-net choose the best approach. However, the aim of the present paper is not achieve the highest possible performance but to obtain a set of results which are representative of the state-of-the-art. All networks are trained via a combined Dice and cross-entropy loss and the Adam optimizer. Further information concerning model parameters and the training scheme can be found in [12].

## C. Performance Metrics and Setup

We have conducted our study for two common performance metrics: the Dice similarity coefficient computed for each class separately and the 95% Hausdorff distance<fnref n="1" />. For sake of conciseness, we only reported results for the first region of each task (L1). In the Hippocampus dataset, it is the anterior part. In the Brain Tumor dataset, it corresponds to the edema. Note nevertheless that the training was done simultaneously on all regions. The full code in Python and Jupyter notebooks to reproduce our experiments are available on GitHub<fnref n="2" />

# IV. EXPERIMENTAL STUDY ON THE WHOLE TEST SETS

In this section, we study the precision of the performance evaluation using the full test sets (test set sizes are 110 for the Hippocampus and 334 for the Brain Tumor, respectively).

The distribution of Dice accuracy and Hausdorff distance values over the test set are shown in Figure 1. One can observe that the Dice accuracy's distribution is not very far from Gaussian despite outliers and skewness in the histogram plots. It is not the case for the Hausdorff distance (Figure 2). First, this measure is lower-bounded by one and a lot of values lie to one, which makes the distribution highly skewed. Also,

<figure><img src="image_3.png" /><figcaption>(a) Hippocampus dataset, 3D U-net</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(c) Brain Tumor dataset, 3D U-net</figcaption></figure>

<caption>(d) Brain Tumor dataset, 2D U-net</caption>

Fig. 1: **Histogram of Dice accuracy over the entire test set**, shown together with a kernel density estimation (KDE) which smoothes the observations with a Gaussian kernel. (a) Hippocampus dataset, 3D U-Net. (b) Hippocampus dataset, 2D U-Net. (c) Brain Tumor dataset, 3D U-Net. (d) Brain Tumor dataset, 2D U-Net.

<footnote n="1">¹Performance metrics were computed using this code: https://github.com/deepmind/surface-distance</footnote>

<footnote n="2">²https://github.com/rosanajurdi/SegVal.git</footnote>
<table><caption>Table 1: Papers reporting precision of segmentation-performance estimates. Summary of the survey conducted on papers tackling 3D medical image segmentation via machine/deep learning published in 2022 in the journals IEEE Transactions on Medical Imaging (IEEE TMI), Medical Image Analysis (MedIA) and Journal of Medical Imaging (JMI). We report the number of papers and experiments and the number and proportion of papers that reported the precision of their estimates (we counted both papers reporting confidence intervals (CI) and those reporting standard error of the mean (SEM), since it is very often possible to derive a reasonable estimate of CI from SEM).</caption><thead><tr><th>Journal</th><th>Nb papers</th><th>Nb exp.</th><th>Nb (%) papers reporting CI (or SEM)</th></tr></thead><tbody><tr><td>TMI</td><td>51</td><td>107</td><td>3 (6%)</td></tr><tr><td>MedIA</td><td>70</td><td>171</td><td>7 (10%)</td></tr><tr><td>JMI</td><td>12</td><td>22</td><td>1 (8%)</td></tr><tr><td>Total</td><td>133</td><td>300</td><td>11 (8%)</td></tr></tbody></table>

which are the most common, but everything could be done similarly for other confidence levels.

A general way to compute confidence intervals is through the bootstrap Efron and Tibshirani (1993); Platt et al. (2000). It has the advantage that it does not make any assumption on the distribution of the metric nor on the sampling distribution of the mean. Let us briefly remind the corresponding procedure. Given a test set of size $n$, $M$ bootstrap samples of size $n$ are drawn with replacement from the test set. Each bootstrap sample is denoted as $S_m^*$ where $m \in \{1, \dots, M\}$. In our experiments, we use $M = 15000$. We denote $\mu_m^*$ the mean of $S_m^*$. The 95% confidence interval $[a^*, b^*]$ is the set of values between the 2.5% and 97.5% percentiles of the sorted bootstrap means $\{\mu_1^*, \mu_2^*, \dots, \mu_m^*, \dots, \mu_M^*\}$. The standard error of the mean $\mu^*$ obtained via bootstrapping (SEM$^*$) is the standard deviation of the means of all bootstrap samples: $SEM^* = \sqrt{\frac{1}{M} \sum_{m=1}^{M} (\mu_m^* - \mu^*)^2}$ where $\mu^*$ is the mean of the bootstrap sample means $\mu_m^*$.

While the bootstrap is easy to perform, it must be performed by the researcher evaluating the model, since it requires to have the performance value for each subject of the test set. Knowing typical confidence intervals to expect given a test size can help evaluating the precision of a result reported without bootstrap. To build such charts, we perform simulations

to gauge the following classical approximation:

$$ SEM = \frac{\sigma}{\sqrt{n}} \qquad (1) \\ [a, b] = [\mu - 1.96 \times SEM, \mu + 1.96 \times SEM] $$

where $\mu$ and $\sigma$ respectively denote the mean and standard deviation of the metric over the test set, $n$ the test set size, SEM the standard error of the mean and $a$ (resp. $b$) are the lower (resp. upper) bounds of the 95% confidence interval. This approximation relies on the assumption that the sampling distribution of the mean follows a normal distribution. It holds asymptotically with only weak assumptions on the metric distribution (finite expected value and variance) and thus should be valid for sufficiently large samples. However, in practice, it is unknown under which test size this approximation is reasonable. This will be assessed through the experiments in the paper. If the metric distribution is close to normal, the approximation is more likely to be correct even for relatively small samples (the sampling distribution of the mean would thus follow a Student's distribution which becomes quickly close to a normal distribution when $n$ increases). In the following, we will refer to estimations derived from Equations 1 as parametric estimations to distinguish them from bootstrap estimations.

Finally, we derive the following quantities from the confidence intervals. First, we are not interested in the absolute values of the boundaries of the confidence interval. Thus, we report confidence intervals independently of the mean, as follows:

$$ CI = [a - \mu, b - \mu] \qquad (2) $$

and we also define the width of CI as $w = b - a$. When the parametric estimation is used, $w = 2 \times 1.96 \times SEM$. Moreover, different performance measures have different ranges of values (for instance Dice is between 0% and 100% while the Hausdorff distance can be arbitrarily large). For the reader to have a better intuition of the precision of the estimate, we thus normalize the width by the mean value of the metric. To this end, we define the normalized width:

$$ \text{Normalized width} \qquad v = \frac{w}{\mu} \qquad (3) $$

where $\mu$ is the average performance metric mean across the test set. In the case of bootstrap estimation, the above quantities are denoted as $CI^*$, $w^*$ and $v^*$.

<footnote n="2">²Throughout the paper, the bootstrap estimate of a given $x$ is always denoted as $x^*$</footnote>
<table><caption>TABLE II: Results on the full test set (n = 110 for Hippocampus and n = 334 for Brain Tumor) for region L₁. μ and σ are the empirical mean and standard deviation of the Dice accuracy (Dice) and 95% volumetric Hausdorff distance (HD) across all patients in the test set. SEM is the standard error of the mean, CI is the 95% confidence interval independent of the mean calculated using the parametric estimation, ν is the normalized CI. μ*, SEM*, CI* and ν* are the bootstrap estimates.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">Parametric</th><th colspan="5">Bootstrap</th></tr><tr><th>μ</th><th>σ</th><th>SEM</th><th>CI</th><th>ν</th><th>μ*</th><th>SEM*</th><th>CI*</th><th>ν*</th></tr></thead><tbody><tr><td rowspan="2">Hippocampus<br>(n = 110)</td><td rowspan="2">3D U-Net</td><td>Dice<br>95% HD</td><td>89.714<br>1.205</td><td>2.797<br>0.472</td><td>0.267<br>0.045</td><td>[-0.52, 0.52]<br>[-0.08, 0.08]</td><td>0.012<br>0.149</td><td>89.715<br>1.205</td><td>0.263<br>0.045</td><td>[-0.53, 0.51]<br>[-0.08, 0.09]</td><td>0.012<br>0.149</td></tr><tr><td>Dice<br>95% HD</td><td>88.197<br>1.311</td><td>3.267<br>0.806</td><td>0.311<br>0.077</td><td>[-0.61, 0.61]<br>[-0.15, 0.15]</td><td>0.014<br>0.229</td><td>88.199<br>1.31</td><td>0.313<br>0.077</td><td>[-0.64, 0.59]<br>[-0.13, 0.17]</td><td>0.014<br>0.229</td></tr><tr><td rowspan="2">Brain Tumor<br>(n = 334)</td><td rowspan="2">3D U-Net</td><td>Dice<br>95% HD</td><td>80.265<br>7.726</td><td>11.947<br>10.634</td><td>0.654<br>0.582</td><td>[-1.28, 1.28]<br>[-1.14, 1.14]</td><td>0.032<br>0.294</td><td>80.268<br>7.73</td><td>0.659<br>0.581</td><td>[-1.31, 1.24]<br>[-1.08, 1.18]</td><td>0.032<br>0.294</td></tr><tr><td>Dice<br>95% HD</td><td>77.489<br>8.855</td><td>13.115<br>11.262</td><td>0.718<br>0.616</td><td>[-1.41, 1.41]<br>[-1.21, 1.21]</td><td>0.036<br>0.272</td><td>77.488<br>8.862</td><td>0.717<br>0.62</td><td>[-1.43, 1.38]<br>[-1.22, 1.22]</td><td>0.036<br>0.272</td></tr></tbody></table>

V. EXPERIMENTAL STUDY ON SUB-SAMPLES OF THE TEST SETS

In this section, we study experimentally the relationship between the test set size and the precision of the estimation of the segmentation performance. To that end, we draw subsamples of variable size

$$k \in K = \{10, 20, 30, 50, 100, \dots, n\}$$

where n is the whole test set size. For the Hippocampus dataset n = 110 and K = {10, 20, 30, 50, 100, 110} and for the Brain Tumor dataset, n = 334 and K = {10, 20, 30, 50, 100, 200, 300, 334}.

For small subsamples, the estimates can vary substantially from one drawing to another. In order not to depend on a particular drawing, which may be lucky or unlucky, we repeat the procedure 100 times for each value of k. We denote the subsamples as (S<sub>k,j</sub>) where k is the subsample size and j ∈ {1, ..., 100} is the index of a particular drawing. We then compute the precision measures (SEM, CI, ν and bootstrap counterparts) for the subsamples of different sizes. The exact formulas for the estimation of the different measures are detailed in appendix A.

A. Dice Accuracy Performance Measure

Results for the Dice accuracy for different sample sizes k are reported in Figure 3 (details in Tables IV, V, VI and VII). As expected, as the sample size decreases, the estimates become less precise (the standard error increases, the confidence interval widens). Moreover, the confidence intervals obtained via parametric estimation are very close, though slightly below, to the ones obtained via bootstrapping. Overall, since the CI obtained via the parametric estimates are slightly narrower than that of the bootstrap values, one can assume that the parametric estimates could serve as a lower bound to the test set size needed to insure a particular precision. Finally, one can observe that bootstrap confidence intervals are slightly skewed. However, this skewness rapidly fades away as the test set sample size k decreases.

B. Hausdorff Distance Performance Measure

Results for the Hausdorff distance metric are reported in Figure 3 (details in Tables VIII, IX, XI and X). Even though

<figure><img src="image_5.png" /><figcaption>Fig. 2: Histogram of 95% Hausdorff distance over the entire test set, shown together with a kernel density estimation (KDE) which smoothes the observations with a Gaussian kernel. (a) Hippocampus dataset, 3D U-Net. (b) Hippocampus dataset, 2D U-Net. (c) Brain Tumor dataset, 3D U-Net. (d) Brain Tumor dataset, 2D U-Net.</figcaption></figure>

due to the discrete nature of the images, some values will never occur hence histograms are sparse.

We then compute, for both metrics and for both 2D and 3D networks, the test sample mean μ, the test sample standard deviation σ, SEM, CI and ν. The computations are done as described in Section II, using both parametric and bootstrap estimations. Results are reported in Table II. One can see that, under such test set sizes, the computations using parametric estimation are close to those obtained with the bootstrap, even for the Hausdorff distance. This indicates that it is a reasonable approximation under this size regime. In the next section, we will study if this still holds for smaller test set sizes.
# 3. Datasets and Segmentation Method

## 3.1. *Datasets*

We used the Hippocampus and Brain Tumor datasets from the Medical Decathlon challenge Antonelli et al. (2022). The Hippocampus dataset is mono-modal and composed of 260 3D T1-weighted (T1-w) MPRAGE brain images. The task is to segment the anterior and posterior parts of the hippocampus (denoted respectively as L1 and L2 according to the challenge annotations). The Brain Tumor dataset is a multi-modal dataset comprising T1w preconstrast, T1w postconstrast, fluid-attenuated inversion recovery (FLAIR), and T2-weighted images. It is composed of 484 samples and dedicated to glioma segmentation. The regions to segment are the edema (L1), the non-enhancing tumor (L2), and the enhancing tumor (L3), according to the Decathlon annotations. The data contains a subset of the data used in the 2016 and 2017 Brain Tumor Segmentation (BraTS) challenges Menze et al. (2015). The brain tumor task is considered difficult (best Dice is around 70%) while the Hippocampus is easier in comparison (best Dice around 90%).

From the 260 and 484 samples of the Hippocampus and Brain Tumor dataset respectively, 100 patients were randomly selected for training, 50 for validation and the remaining samples constituted the test set. The size of the training and validation subsets were the same for both datasets to ensure that the number of training samples has no impact on the model performances.

## 3.2. *Segmentation Methods*

We used the Decathlon Challenge winner's nnU-net, as a framework to perform our experiments. nnU-net Isensee et al. (2021) is a fully automated segmentation framework that makes use of typical 2D and 3D U-Net architectures by dynamically and automatically adapting the hyper-parameters to a particular dataset. Given an arbitrary dataset, the nnU-net pipeline first extracts data fingerprints (e.g. voxel spacing, intensity distribution, modalities, class ratios etc.). Pre-processing of the datasets is conducted automatically via cropping to non-zero volumes to reduce the computational burden, resampling relative to voxel spacing to allow spatial semantics, and z-score normalization, if need be. We allowed nnU-Net to automatically determine the best configuration of parameters. In nnU-net, one can either predetermine which architecture to train on the given dataset or allow nnU-net to choose the best configuration. In the present paper, we

conducted our experiments using the 3D full-resolution U-Net and the 2D U-Net. It is possible that even better performances could have been obtained using the cascade approach or letting nnU-net choose the best approach. However, the aim of the present paper is not achieve the highest possible performance but to obtain a set of results which are representative of the state-of-the-art. All networks are trained via a combined Dice and cross-entropy loss and the Adam optimizer. Further information concerning model parameters and the training scheme can be found in Isensee et al. (2021).

## 3.3. *Performance Metrics and Setup*

We have conducted our study for two common performance metrics: the Dice Similarity Coefficient computed for each class separately and the 95% Hausdorff distance<fnref n="1" />. For sake of conciseness, we only reported results for the first region of each task (L1). In the Hippocampus dataset, it is the anterior part. In the Brain Tumor dataset, it corresponds to the edema. Note nevertheless that the training was done simultaneously on all regions. The full code in Python and Jupyter notebooks to reproduce our experiments are available on GitHub²

# 4. Experimental Study on the Whole Test Sets

In this section, we study the precision of the performance evaluation using the full test sets (test set sizes are 110 for the Hippocampus and 334 for the Brain Tumor, respectively).

The distribution of Dice Similarity Coefficient and Hausdorff distance values over the test set are shown in Figure 1. One can observe that the Dice Similarity Coefficient's distribution is not very far from Gaussian despite outliers and skewness in the histogram plots. It is not the case for the Hausdorff distance (Figure 2). First, this measure is lower-bounded by one and a lot of values lie to one, which makes the distribution highly skewed. Also, due to the discrete nature of the images, some values will never occur hence histograms are sparse.

We then compute, for both metrics and for both 2D and 3D networks, the test sample mean $\mu$, the test sample standard deviation $\sigma$, SEM, CI and $v$. The computations are done

<footnote n="1">¹Performance metrics were computed using this code: https://github.com/deepmind/surface-distance
²https://github.com/rosanajurdi/SegVal.Repo</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 3: Parametric and bootstrap CIs: Dice accuracy (detailed results in Tables IV, V, VI, VII)</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 4: Parametric and bootstrap CIs: Hausdorff Distance (detailed results in Tables VIII, IX, X, XI)</figcaption></figure>

we saw that the Hausdorff distance is far from being Gaussian distributed, it is interesting to note that obtained parametric estimates are actually close to bootstrap ones. It thus seems reasonable to assume that parametric estimates can, as in the case of the Dice accuracy, be used to derive a lower bound on the minimal test set size needed to achieve a given precision.

Inspecting the normalized width $v$ (reported in appendix B) shows that it is much larger for the Hausdorff distance than for the Dice accuracy for the same model and the same dataset. This indicates that more test samples are required to achieve a particular precision. Possible explanations include the discontinuity of the Hausdorff distance and the fact that it corresponds to a maximum rather than an average.

## VI. A TABLE OF GAUSSIAN CONFIDENCE INTERVALS

The previous section shows that parametric estimates match well bootstrap confidence intervals. Confidence intervals can be useful even before running an experiment, to give a lower bound for the dataset size needed to achieve a given precision. To facilitate the use of confidence intervals, Table III list them for different values of test set size $k$ and $\sigma$ (which reflects how variable is the performance across test samples), computed via Equations 1. Some values of $\sigma$ match those experimentally measured in our experiences in Table II (0.47, 0.81, 2.79, 3.26, 10.63, 11.26, 13.12). We recall that the value of $\mu$, in itself, has no impact on the width of the confidence interval nor on the SEM –even though it is usually observed that lower performing models, thus associated with a lower value of $\mu$, also have a more variable performance and thus a larger value of $\sigma$.

## VII. DISCUSSION

We have studied the precision with which segmentation performance can be estimated in typical medical imaging studies. We provided typical values for confidence intervals of the performance of trained models. Such values can be useful to authors and reviewers, e.g. to roughly estimate confidence intervals that can be expected for a given study. Finally, we have shown that, under typical performance spreads, the sample size needed to achieve a given confidence interval is

<table><caption>TABLE III: Table of SEM and confidence intervals CI for different sizes $k$ of the test set and for different values of $\sigma$ including the experimental values obtained in Table II.</caption><thead><tr><th>σ</th><th>k →</th><th>10</th><th>20</th><th>30</th><th>50</th><th>100</th><th>200</th><th>300</th><th>500</th><th>1000</th><th>1500</th><th>2000</th><th>2500</th><th>3000</th></tr></thead><tbody><tr><td>0.47</td><td>SEM CI</td><td>0.15<br>[-0.29, 0.29]</td><td>0.11<br>[-0.21, 0.21]</td><td>0.09<br>[-0.17, 0.17]</td><td>0.07<br>[-0.13, 0.13]</td><td>0.05<br>[-0.09, 0.09]</td><td>0.03<br>[-0.07, 0.07]</td><td>0.03<br>[-0.05, 0.05]</td><td>0.02<br>[-0.04, 0.04]</td><td>0.01<br>[-0.03, 0.03]</td><td>0.01<br>[-0.02, 0.02]</td><td>0.01<br>[-0.02, 0.02]</td><td>0.01<br>[-0.02, 0.02]</td></tr><tr><td>0.81</td><td>SEM CI</td><td>0.26<br>[-0.5, 0.5]</td><td>0.18<br>[-0.35, 0.35]</td><td>0.15<br>[-0.29, 0.29]</td><td>0.26<br>[-0.22, 0.22]</td><td>0.08<br>[-0.16, 0.16]</td><td>0.05<br>[-0.11, 0.11]</td><td>0.05<br>[-0.09, 0.09]</td><td>0.04<br>[-0.07, 0.07]</td><td>0.03<br>[-0.05, 0.05]</td><td>0.02<br>[-0.04, 0.04]</td><td>0.02<br>[-0.04, 0.04]</td><td>0.02<br>[-0.04, 0.04]</td></tr><tr><td>1</td><td>SEM CI</td><td>0.32<br>[-0.62, 0.62]</td><td>0.22<br>[-0.44, 0.44]</td><td>0.18<br>[-0.36, 0.36]</td><td>0.14<br>[-0.28, 0.28]</td><td>0.1<br>[-0.2, 0.2]</td><td>0.07<br>[-0.14, 0.14]</td><td>0.06<br>[-0.11, 0.11]</td><td>0.04<br>[-0.09, 0.09]</td><td>0.03<br>[-0.06, 0.06]</td><td>0.02<br>[-0.04, 0.04]</td><td>0.02<br>[-0.04, 0.04]</td><td>0.02<br>[-0.04, 0.04]</td></tr><tr><td>2.79</td><td>SEM CI</td><td>0.88<br>[-1.73, 1.73]</td><td>0.62<br>[-1.22, 1.22]</td><td>0.51<br>[-1.0, 1.0]</td><td>0.39<br>[-0.77, 0.77]</td><td>0.28<br>[-0.55, 0.55]</td><td>0.23<br>[-0.39, 0.39]</td><td>0.16<br>[-0.32, 0.32]</td><td>0.12<br>[-0.24, 0.24]</td><td>0.09<br>[-0.17, 0.17]</td><td>0.07<br>[-0.14, 0.14]</td><td>0.06<br>[-0.12, 0.12]</td><td>0.05<br>[-0.11, 0.11]</td></tr><tr><td>3.26</td><td>SEM CI</td><td>1.03<br>[-2.02, 2.02]</td><td>0.73<br>[-1.43, 1.43]</td><td>0.6<br>[-1.17, 1.17]</td><td>0.46<br>[-0.9, 0.9]</td><td>0.33<br>[-0.64, 0.64]</td><td>0.23<br>[-0.45, 0.45]</td><td>0.19<br>[-0.37, 0.37]</td><td>0.15<br>[-0.29, 0.29]</td><td>0.1<br>[-0.2, 0.2]</td><td>0.08<br>[-0.16, 0.16]</td><td>0.07<br>[-0.14, 0.14]</td><td>0.06<br>[-0.12, 0.12]</td></tr><tr><td>5</td><td>SEM CI</td><td>1.58<br>[-3.1, 3.1]</td><td>1.12<br>[-2.19, 2.19]</td><td>0.91<br>[-1.79, 1.79]</td><td>0.71<br>[-1.39, 1.39]</td><td>0.5<br>[-0.98, 0.98]</td><td>0.35<br>[-0.69, 0.69]</td><td>0.29<br>[-0.57, 0.57]</td><td>0.22<br>[-0.44, 0.44]</td><td>0.16<br>[-0.31, 0.31]</td><td>0.13<br>[-0.25, 0.25]</td><td>0.11<br>[-0.22, 0.22]</td><td>0.09<br>[-0.18, 0.18]</td></tr><tr><td>10.63</td><td>SEM CI</td><td>3.36<br>[-6.59, 6.59]</td><td>2.38<br>[-4.66, 4.66]</td><td>1.94<br>[-3.8, 3.8]</td><td>1.5<br>[-2.95, 2.95]</td><td>1.06<br>[-2.08, 2.08]</td><td>0.75<br>[-1.47, 1.47]</td><td>0.61<br>[-1.2, 1.2]</td><td>0.48<br>[-0.93, 0.93]</td><td>0.34<br>[-0.66, 0.66]</td><td>0.27<br>[-0.54, 0.54]</td><td>0.24<br>[-0.47, 0.47]</td><td>0.21<br>[-0.42, 0.42]</td><td>0.19<br>[-0.38, 0.38]</td></tr><tr><td>11.26</td><td>SEM CI</td><td>3.56<br>[-6.98, 6.98]</td><td>2.52<br>[-4.93, 4.93]</td><td>2.06<br>[-4.03, 4.03]</td><td>1.59<br>[-3.12, 3.12]</td><td>1.13<br>[-2.21, 2.21]</td><td>0.8<br>[-1.56, 1.56]</td><td>0.65<br>[-1.27, 1.27]</td><td>0.5<br>[-0.99, 0.99]</td><td>0.36<br>[-0.7, 0.7]</td><td>0.29<br>[-0.57, 0.57]</td><td>0.25<br>[-0.49, 0.49]</td><td>0.23<br>[-0.44, 0.44]</td><td>0.21<br>[-0.4, 0.4]</td></tr><tr><td>12</td><td>SEM CI</td><td>3.79<br>[-7.44, 7.44]</td><td>2.68<br>[-5.26, 5.26]</td><td>2.19<br>[-4.29, 4.29]</td><td>1.7<br>[-3.33, 3.33]</td><td>1.2<br>[-2.35, 2.35]</td><td>0.85<br>[-1.66, 1.66]</td><td>0.69<br>[-1.36, 1.36]</td><td>0.54<br>[-1.05, 1.05]</td><td>0.38<br>[-0.74, 0.74]</td><td>0.31<br>[-0.61, 0.61]</td><td>0.27<br>[-0.53, 0.53]</td><td>0.24<br>[-0.47, 0.47]</td><td>0.22<br>[-0.43, 0.43]</td></tr><tr><td>13.12</td><td>SEM CI</td><td>4.15<br>[-8.13, 8.13]</td><td>2.94<br>[-5.75, 5.75]</td><td>2.4<br>[-4.69, 4.69]</td><td>1.86<br>[-3.64, 3.64]</td><td>1.31<br>[-2.57, 2.57]</td><td>0.93<br>[-1.82, 1.82]</td><td>0.76<br>[-1.48, 1.48]</td><td>0.59<br>[-1.15, 1.15]</td><td>0.41<br>[-0.81, 0.81]</td><td>0.34<br>[-0.66, 0.66]</td><td>0.29<br>[-0.58, 0.58]</td><td>0.26<br>[-0.51, 0.51]</td><td>0.24<br>[-0.47, 0.47]</td></tr><tr><td>20</td><td>SEM CI</td><td>6.32<br>[-12.4, 12.4]</td><td>4.47<br>[-8.77, 8.77]</td><td>3.65<br>[-7.16, 7.16]</td><td>2.83<br>[-5.54, 5.54]</td><td>2.0<br>[-3.92, 3.92]</td><td>1.41<br>[-2.77, 2.77]</td><td>1.15<br>[-2.26, 2.26]</td><td>0.89<br>[-1.75, 1.75]</td><td>0.63<br>[-1.24, 1.24]</td><td>0.52<br>[-1.01, 1.01]</td><td>0.45<br>[-0.88, 0.88]</td><td>0.4<br>[-0.78, 0.78]</td><td>0.37<br>[-0.72, 0.72]</td></tr><tr><td>30</td><td>SEM CI</td><td>9.49<br>[-18.59, 18.59]</td><td>6.71<br>[-13.15, 13.15]</td><td>5.48<br>[-10.74, 10.74]</td><td>4.24<br>[-8.32, 8.32]</td><td>3.0<br>[-5.88, 5.88]</td><td>2.12<br>[-4.16, 4.16]</td><td>1.73<br>[-3.39, 3.39]</td><td>1.34<br>[-2.63, 2.63]</td><td>0.95<br>[-1.86, 1.86]</td><td>0.77<br>[-1.52, 1.52]</td><td>0.67<br>[-1.31, 1.31]</td><td>0.6<br>[-1.18, 1.18]</td><td>0.91<br>[-1.07, 1.07]</td></tr><tr><td>50</td><td>SEM CI</td><td>15.81<br>[-30.99, 30.99]</td><td>11.18<br>[-21.91, 21.91]</td><td>9.13<br>[-17.89, 17.89]</td><td>7.07<br>[-13.86, 13.86]</td><td>5.0<br>[-9.8, 9.8]</td><td>3.54<br>[-6.93, 6.93]</td><td>2.89<br>[-5.66, 5.66]</td><td>2.24<br>[-4.38, 4.38]</td><td>1.58<br>[-3.1, 3.1]</td><td>1.29<br>[-2.53, 2.53]</td><td>1.12<br>[-2.19, 2.19]</td><td>1.0<br>[-1.96, 1.96]</td><td>0.91<br>[-1.79, 1.79]</td></tr></tbody></table>
<table><caption>Table 2: **Results on the full test set** (n = 110 for Hippocampus and n = 334 for Brain Tumor) for region L₁. μ and σ are the empirical mean and standard deviation of the Dice Similarity Coefficient (Dice) and 95% volumetric Hausdorff distance (HD) across all patients in the test set. SEM is the standard error of the mean, CI is the 95% confidence interval independent of the mean calculated using the parametric estimation, ν is the normalized CI. μ*, SEM*, CI* and ν* are the bootstrap estimates.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">Parametric</th><th colspan="5">Bootstrap</th></tr><tr><th>μ</th><th>σ</th><th>SEM</th><th>CI</th><th>ν</th><th>μ*</th><th>SEM*</th><th>CI*</th><th>ν*</th></tr></thead><tbody><tr><td rowspan="2">Hippocampus<br>(n = 110)</td><td rowspan="2">3D U-Net<br>Dice<br>95% HD</td><td>89.714</td><td>2.797</td><td>0.267</td><td>[-0.52, 0.52]</td><td>0.012</td><td>89.715</td><td>0.263</td><td>[-0.53, 0.51]</td><td>0.012</td></tr><tr><td>1.205</td><td>0.472</td><td>0.045</td><td>[-0.08, 0.08]</td><td>0.149</td><td>1.205</td><td>0.045</td><td>[-0.08, 0.09]</td><td>0.149</td></tr><tr><td rowspan="2">Brain Tumor<br>(n = 334)</td><td rowspan="2">2D U-Net<br>Dice<br>95% HD</td><td>88.197</td><td>3.267</td><td>0.311</td><td>[-0.61, 0.61]</td><td>0.014</td><td>88.199</td><td>0.313</td><td>[-0.64, 0.59]</td><td>0.014</td></tr><tr><td>1.311</td><td>0.806</td><td>0.077</td><td>[-0.15, 0.15]</td><td>0.229</td><td>1.31</td><td>0.077</td><td>[-0.13, 0.17]</td><td>0.229</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td>80.265</td><td>11.947</td><td>0.654</td><td>[-1.28, 1.28]</td><td>0.032</td><td>80.268</td><td>0.659</td><td>[-1.31, 1.24]</td><td>0.032</td></tr><tr><td>7.726</td><td>10.634</td><td>0.582</td><td>[-1.14, 1.14]</td><td>0.294</td><td>7.73</td><td>0.581</td><td>[-1.08, 1.18]</td><td>0.294</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td>77.489</td><td>13.115</td><td>0.718</td><td>[-1.41, 1.41]</td><td>0.036</td><td>77.488</td><td>0.717</td><td>[-1.43, 1.38]</td><td>0.036</td></tr><tr><td>8.855</td><td>11.262</td><td>0.616</td><td>[-1.21, 1.21]</td><td>0.272</td><td>8.862</td><td>0.62</td><td>[-1.22, 1.22]</td><td>0.272</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Figure 1: **Histogram of Dice Similarity Coefficient over the entire test set**, shown together with a kernel density estimation (KDE) which smoothes the observations with a Gaussian kernel. (a) Hippocampus dataset, 3D U-Net. (b) Hippocampus dataset, 2D U-Net. (c) Brain Tumor dataset, 3D U-Net. (d) Brain Tumor dataset, 2D U-Net.</figcaption></figure>

<caption>Figure 2: **Histogram of 95% Hausdorff distance over the entire test set**, shown together with a kernel density estimation (KDE) which smoothes the observations with a Gaussian kernel. (a) Hippocampus dataset, 3D U-Net. (b) Hippocampus dataset, 2D U-Net. (c) Brain Tumor dataset, 3D U-Net. (d) Brain Tumor dataset, 2D U-Net.</caption>
smaller than for classification tasks. To that purpose, we have
conducted an extensive set of experiments using both para-
metric and bootstrap estimates. Results show that parametric
estimates give reasonable results even when the performance
metric substantially deviates from a Gaussian distribution.

*Confidence intervals cannot be computed post-hoc:* The width of the confidence interval depends on two factors: the size of the test set (n) and the variability of the performance (σ). In classification, a binomial distribution gives parametric confidence intervals –though slightly conservative [10], [11]. Confidence intervals can be computed post-hoc as a function of the accuracy, e.g. when reading a manuscript. This is not the case in image segmentation: for a given metric, σ is highly task dependent and not directly linked to the performance (in practice, lower performing models are often associated to higher standard deviations). Hence, σ must be computed during model evaluation, and studies should report confidence intervals and potentially σ.

**A table for confidence intervals:** As parametric estimates give reasonable results, one can compute typical confidence intervals for given *n* and *σ* and we provide this information in Table III. These values can be useful to researchers preparing a segmentation study, to help them choose the size of the test set. Investigators must then make an hypothesis on the expected value of *σ* (as when doing a power analysis), for example using previously published studies on similar segmentation tasks (given that these report *σ*). The values can also be useful to reviewers who can use these to gauge plausible confidence intervals, should these not be reported in the paper. Again, if *σ* is not reported in the paper, one will need to make an hypothesis on its expected value.

*Test-set size for statistical control:* Classification tasks need large test-set sizes for tight confidence intervals (though higher-performing models lead to less uncertainty). Typically, for classification a 1%-wide confidence interval requires about 10 000 samples for models with over 90% accuracy [11].

Our results show that the typical test size needed for a
given confidence interval is smaller for segmentation than for
classification. This is good news as it is arguably more difficult
to obtain a large test set in the case of segmentation as it
requires voxel-wise annotation by a trained rater. However,
smaller does not mean small. When dispersion is low (σ ≤ 3
for Dice accuracy), hundreds of samples may suffice for tight
confidence intervals (e.g. 1%-wide). When the dispersion is
larger (for example σ ≈ 15), as often for more difficult tasks,
a 1%-wide confidence interval may require more than 1000
test samples. More relaxed confidence intervals (e.g. 4%-wide)
may still require about 200 test samples.

**A call for larger test sets:** Are the practices of medical-imaging studies in line with such test set sizes? As mentioned in the introduction, we surveyed papers published in 2022 in three medical imaging journals: IEEE Transactions on Medical Imaging (IEEE TMI), Medical Image Analysis (MedIA), and SPIE's Journal of Medical Imaging (JMI) dealing with segmentation of 3D images. This survey, though not exhaustive, sheds light on current practices in recent medical-imaging papers. Figure 5 shows that the test set size is in general small and highly variable across studies (median, 25; minimum: 1;

<figure><img src="image_8.png" /><figcaption>Fig. 5: Histogram of the test-set size of different experiments from segmentation papers published in JMI, TMI, and MedIA in 2022. The red line represents the median test size (n = 25).</figcaption></figure>

maximum: 1543). Thus, with the exception of a few studies,
there is a need to increase the size of test sets to obtain
reasonably precise estimates of the performance.

*Reporting confidence intervals is important:* Trustworthy evidence on segmentation tools is important for their clinical adoption. Confidence intervals and bootstrapping are classic statistical practice; and yet medical image segmentation papers seldom report confidence intervals: we found them in only 10% of papers of our survey (Table I). Beyond immediate clinical adoption, controlled statistical evidence is a component of reproducibility, needed for scientific progress [18], [19]. *Statistical reproducibility* is suited to reason on generalization to unseen data: results need to be statistically compatible rather than identical. Reporting confidence intervals is central to asserting statistical reproducibility of studies. Awareness about this issue has risen, as can been seen for example in reproducibility checklists (e.g. MICCAI 2021³). However, the fact that confidence intervals are so rarely reported in recent papers indicates yet more awareness is needed.

Reporting confidence intervals enables progress at the level of the community. To disseminate a trained model (for use by clinical researchers, neuroscientists ...), the users should understand expected variances in the performance of the model. Confidence intervals also enable comparing performance to inter-rater (or intra-rater) variability. When computing inter-rater variability on the test set used to evaluate the trained model, one can obtain confidence intervals for the trained model and for the inter-rater and check if they overlap.

*Recommendations:* We recommend that authors systematically report confidence intervals for trained models. Even though we showed that parametric approximations can yield reasonable estimates, we recommend to use the bootstrap method as it makes much fewer assumptions and is easy to perform since it only requires to resample performance metrics obtained on the test set (without recomputing them). Finally, confidence intervals should be complemented by displaying

<footnote n="3">³https://miccai2021.org/files/downloads/MICCAI2021-Reproducibility-Checklist.pdf</footnote>
as described in Section 2, using both parametric and bootstrap estimations. Bootstrapping on the entire dataset took around 0.53 second for the hippocampus and 0.62 second for the Brain Tumor dataset on a standard laptop. Results are reported in Table 2. One can see that, under such test set sizes, the computations using parametric estimation are close to those obtained with the bootstrap, even for the Hausdorff distance. This indicates that it is a reasonable approximation under this size regime. In the next section, we will study if this still holds for smaller test set sizes.

## 5. Experimental Study on Sub-samples of the Test Sets

In this section, we study experimentally the relationship between the test set size and the precision of the estimation of the segmentation performance. To that end, we draw sub-samples of variable size

$$k \in K = \{10, 20, 30, 50, 100, \dots, n\}$$

where n is the whole test set size. For the Hippocampus dataset n = 110 and K = {10, 20, 30, 50, 100, 110} and for the Brain Tumor dataset, n = 334 and K = {10, 20, 30, 50, 100, 200, 300, 334].

For small subsamples, the estimates can vary substantially from one drawing to another. In order not to depend on a particular drawing, which may be lucky or unlucky, we repeat the procedure 100 times for each value of k. We denote the subsamples as (S$_{k,j}$) where k is the subsample size and j ∈ {1, ..., 100} is the index of a particular drawing. We then compute the precision measures (SEM, CI, v and bootstrap counterparts) for the subsamples of different sizes. The exact formulas for the estimation of the different measures are detailed in appendix Appendix .1.

### 5.1. Dice Similarity Coefficient Performance Measure

Results for the Dice Similarity Coefficient for different sample sizes k are reported in Figure 3 (details in Tables .4, .5, .6 and .7). As expected, as the sample size decreases, the estimates become less precise (the standard error increases, the confidence interval widens). Moreover, the confidence intervals obtained via parametric estimation are very close, though slightly below, to the ones obtained via bootstrapping. Overall, since the CI obtained via the parametric estimates are slightly narrower than that of the bootstrap values, one can assume that the parametric estimates could serve as a lower

<figure><img src="image_3.png" /><figcaption>Figure 3: Parametric and bootstrap CIs: Dice Similarity Coefficient (detailed results in Tables .4, .5, .6, .7)</figcaption></figure>

bound to the test set size needed to insure a particular precision. Finally, one can observe that bootstrap confidence intervals are slightly skewed. However, this skewness rapidly fades away as the test set sample size k decreases.

### 5.2. Hausdorff Distance Performance Measure

Results for the Hausdorff distance metric are reported in Figure 3 (details in Tables .8, .9, .11 and .10). Even though we saw that the Hausdorff distance is far from being Gaussian distributed, it is interesting to note that obtained parametric estimates are actually close to bootstrap ones. It thus seems reasonable to assume that parametric estimates can, as in the case of the Dice Similarity Coefficient, be used to derive a lower bound on the minimal test set size needed to achieve a given precision.

Inspecting the normalized width v (reported in appendix Appendix .2) shows that it is much larger for the Hausdorff distance than for the Dice Similarity Coefficient for the same model and the same dataset. This indicates that more test samples are required to achieve a particular precision. Possible explanations include the discontinuity of the Hausdorff distance and the fact that it corresponds to a maximum rather than an average.

## 6. A table of Gaussian confidence intervals

The previous section shows that parametric estimates match well bootstrap confidence intervals. Confidence inter-
the distribution of the performance metrics (e.g. as a violin plot or a box plot) on the test set and reporting its standard deviation.

*More work to benchmark learning procedures:* We investigated the sampling noise in the test set, the only source of variance when evaluating a *trained model*. However evaluating a *learning procedure* –as when comparing two methods– comes with other sources of variance such as hyperparameters or random seeds [20], [11]. Methods to build confidence intervals for *learning procedures* typically involve running multiple trainings with different data splits (to account for sampling noise in training and testing sets) but also different hyperparameters or random seeds [20], [11]. They require much more computing power and lead to wider confidence intervals.

*Limitations:* While we studied the two most widely used performance metrics, there are many other metrics, some being more appropriate for certain settings [3]. Also, due to space constraints, we only studied two datasets. We have chosen them to include both a relatively easy and a more difficult segmentation task. However, conducting similar experiments on other datasets would be ground for more general conclusions.

## VIII. CONCLUSION

Confidence intervals are needed to characterize the uncertainty on the performance of image-segmentation tools. We have shown that simple parametric estimates can serve as a reasonable reference to guide researchers on the test set size needed for a target precision. Importantly, results show that the test set size needed to achieve a given precision is lower for segmentation than for classification tasks. Confidence intervals can be computed easily; we hope that they will be used more in image-segmentation studies.

## REFERENCES

[1] M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep learning techniques for medical image segmentation: achievements and challenges,” *Journal of digital imaging*, vol. 32, pp. 582–596, 2019.

[2] A. Reinke, E. Christodoulou, B. Glocker, P. Scholz, F. Isensee, and et al, “Metrics reloaded - a new recommendation framework for biomedical image analysis validation,” in *Medical Imaging with Deep Learning*, 2022. [Online]. Available: https://openreview.net/forum?id=24kBqy8rcB_

[3] L. Maier-Hein, A. Reinke, P. Godau, M. D. Tizabi, E. Christodoulou, B. Glocker, and et al, “Metrics reloaded: Pitfalls and recommendations for image analysis validation,” *arXiv preprint*, vol. arXiv:2206.01653, 2022. [Online]. Available: https://arxiv.org/abs/2206.01653

[4] A. Reinke, M. D. Tizabi, M. Baumgartner, M. Eisenmann, D. Heckmann-Nötzel, A. E. Kavu, T. Rädsch, C. H. Sudre, L. Acion, M. Antonelli et al., “Understanding metric-related pitfalls in image analysis validation,” *arXiv preprint arXiv:2302.01790*, 2023. [Online]. Available: https://arxiv.org/abs/2302.01790

[5] L. Maier-Hein, M. Eisenmann, A. Reinke, S. Onogur, M. Stankovic, P. Scholz, T. Arbel, H. Bogunovic, A. P. Bradley, A. Carass et al., “Why rankings of biomedical image analysis competitions should be interpreted with care,” *Nature communications*, vol. 9, no. 1, p. 5217, 2018.

[6] G. Varoquaux and V. Cheplygina, “Machine learning for medical imaging: methodological failures and recommendations for the future,” *NPJ digital medicine*, vol. 5, no. 1, p. 48, 2022.

[7] M. Chupin, E. Gérardin, R. Cuingnet, C. Boutet, L. Lemieux, S. Lehéricy, H. Benali, L. Garnero, and O. Colliot, “Fully automatic hippocampus segmentation and classification in Alzheimer’s disease and mild cognitive impairment applied on data from ADNI,” *Hippocampus*, vol. 19, no. 6, pp. 579–587, 2009.

[8] T. Samaille, L. Fillon, R. Cuingnet, E. Jouvent, H. Chabriat, D. Dormont, O. Colliot, and M. Chupin, “Contrast-based fully automatic segmentation of white matter hyperintensities: method and validation,” *PLoS one*, vol. 7, no. 11, p. e48953, 2012.

[9] R. El Jurdí, C. Petitjean, P. Honeine, V. Cheplygina, and F. Abdallah, “A surprisingly effective perimeter-based loss for medical image segmentation,” in *Medical Imaging with Deep Learning*, 2021, pp. 158–167.

[10] G. Varoquaux, “Cross-validation failure: Small sample sizes lead to large error bars,” *NeuroImage*, vol. 180, pp. 68–77, 2018.

[11] G. Varoquaux and O. Colliot, “Evaluating machine learning models and their diagnostic value,” *HAL preprint*, vol. hal-03682454, 2022. [Online]. Available: https://hal.archives-ouvertes.fr/hal-03682454

[12] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” *Nature Methods*, vol. 18, no. 2, pp. 203–211, Feb 2021.

[13] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger et al., “The medical segmentation decathlon,” *Nature Communications*, vol. 13, no. 1, p. 4128, 2022.

[14] D. G. Altman and P. Royston, “The cost of dichotomising continuous variables,” *BMJ*, vol. 332, no. 7549, p. 1080, May 2006.

[15] B. Efron and R. J. Tibshirani, *An introduction to the bootstrap*. CRC press, 1993.

[16] R. W. Platt, J. A. Hanley, and H. Yang, “Bootstrap confidence intervals for the sensitivity of a quantitative diagnostic test,” *Statistics in medicine*, vol. 19, no. 3, pp. 313–322, 2000.

[17] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, and et al, “The multimodal Brain Tumor Image Segmentation benchmark (BRATS),” *IEEE Transactions on Medical Imaging*, vol. 34, no. 10, pp. 1993–2024, 2015.

[18] M. McDermott, S. Wang, N. Marinsek, R. Ranganath, M. Ghassemi, and L. Foschini, “Reproducibility in machine learning for health,” *arXiv preprint arXiv:1907.01463*, 2019.

[19] O. Colliot, E. Thibeau-Sutre, and N. Burgos, “Reproducibility in machine learning for medical imaging,” *arXiv preprint arXiv:2209.05097*, 2022.

[20] X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V. Voleti et al., “Accounting for variance in machine learning benchmarks,” *Proceedings of Machine Learning and Systems*, vol. 3, pp. 747–769, 2021.

## APPENDIX

### A. Details on the estimated measures on subsamples of the test sets

a) *Parametric estimates:* For the parametric estimation, we denote as

$$ \mu_{k,j} = \frac{1}{k} \sum_{l=1}^{k} P_{j,k,l}, \quad \sigma_{k,j} = \sqrt{\frac{1}{k} \sum_{l=1}^{k} (P_{j,k,l} - \mu_{k,j})^2} $$

the empirical mean and standard deviation for the subsample $S_{k,j}$ where $P_{j,k,l}$ is the performance metric (in our case either Dice accuracy or Hausdorff distance) of a given subject in the sub-sample $S_{k,j}$. Similarly, we use the notations

$$ \text{SEM}_{k,j} = \frac{\sigma_{k,j}}{\sqrt{k}}, \quad \nu_{k,j} = \frac{2 * 1.96 * \text{SEM}_{k,j}}{\mu_{k,j}}. $$

We then compute the average of $\mu_{k,j}$, $\sigma_{k,j}$, $\text{SEM}_{k,j}$, $\nu_{k,j}$ across the different subsamples $S_{k,j}$ for $k$ fixed and $j \in \{1, \dots, 100\}$. This provides the following estimates $\mu_k$, $\sigma_k$, $\text{SEM}_k$ and $\nu_k$. Similarly, we define $\text{CI}_k = [-1.96 \times \text{SEM}_k, 1.96 \times \text{SEM}_k]$.
<figure><img src="image_4.png" /><figcaption>Figure 4: Parametric and bootstrap CIs: Hausdorff Distance (detailed results in Tables .8, .9, .10, .11)</figcaption></figure>

vals can be useful even before running an experiment, to give
a lower bound for the dataset size needed to achieve a given
precision. To facilitate the use of confidence intervals, Table 3
list them for different values of test set size *k* and *σ* (which
reflects how variable is the performance across test samples),
computed via Equations 1. Some values of *σ* match those ex-
perimentally measured in our experiences in Table 2 (0.47,
0.81, 2.79, 3.26, 10.63, 11.26, 13.12). We recall that the value of *μ*,
in itself, has no impact on the width of the confidence inter-
val nor on the SEM –even though it is usually observed that
lower performing models, thus associated with a lower value
of *μ*, also have a more variable performance and thus a larger
value of *σ*.

7. Discussion

We have studied the precision with which segmentation
performance can be estimated in typical 3D brain imaging
studies. We provided typical values for confidence intervals of
the performance of trained models. Such values can be useful
to authors and reviewers, e.g. to roughly estimate confidence
intervals that can be expected for a given study. Finally, we
have shown that, under typical performance spreads, the sam-
ple size needed to achieve a given confidence interval is smaller
than for classification tasks. To that purpose, we have con-
ducted an extensive set of experiments using both parametric
and bootstrap estimates. Results show that parametric esti-

mates give reasonable results even when the performance met-
ric substantially deviates from a Gaussian distribution.

*Test-set size for statistical control.* Classification tasks need large test-set sizes for tight confidence intervals (though higher-performing models lead to less uncertainty). Typically, for classification a 1%-wide confidence interval requires about 10 000 samples for models with over 90% accuracyVARQUAUX and COLLIOT (2023).

It is intuitive that the typical test size needed for a given con-
fidence interval is smaller for segmentation than for classifica-
tion. This is due to the continuous nature of segmentation
metrics which aggregate information across multiple pixels or
voxels. On the contrary, in classification, the metric associ-
ated to each image is binary (correct or wrong) and these are
then aggregated at the sample level to form a continuous met-
ric (e.g. accuracy). In other words, in classification, one image
provides only one bit of information to the estimation of the
performance. In contrast, in segmentation, each pixel/voxel
provides one bit of information, and thus each image con-
tributes many bits. This intuition is indeed verified by our ex-
perimental results which show that indeed, to achieve a given
CI width, fewer test samples are usually needed for segmenta-
tion than for classification. This is good news, as it is arguably
more difficult to obtain a large test set in the case of segmen-
tation, as it requires voxel-wise annotation by a trained rater.

However, smaller does not mean small. When dispersion is low ($\sigma \le 3$ for Dice Similarity Coefficient), hundreds of samples may suffice for tight confidence intervals (e.g. 1%-wide). When the dispersion is larger (for example $\sigma \approx 15$), as often for more difficult tasks, a 1%-wide confidence interval may require more than 1000 test samples. More relaxed confidence intervals (e.g. 4%-wide) may still require about 200 test samples.

*A call for larger test sets.* Are the practices of medical-imaging
studies in line with such test set sizes? As mentioned in the
introduction, we surveyed papers published in 2022 in three
medical imaging journals: IEEE Transactions on Medical
Imaging (IEEE TMI), Medical Image Analysis (MedIA), and
SPIE's Journal of Medical Imaging (JMI) dealing with seg-
mentation of 3D images. This survey, though not exhaustive,
sheds light on current practices in recent medical-imaging pa-
pers. Figure 5 shows that the test set size is in general small and
highly variable across studies (median, 25; minimum: 1; maxi-
mum: 1543). Thus, with the exception of a few studies, there
b) *Bootstrap estimates*: Bootstrap estimations are performed as follows. For a given subsample $S_{k,j}$ of size $k$ and index $j$, $M = 15000$ bootstrap samples of size $k$ are drawn with replacement. We denote a given bootstrap sample as $S_{k,j,m}^*$ and its mean as $\mu_{k,j,m}^*$ where $m \in \{1, \dots, M\}$ is the index of the $m^{th}$ bootstrap sample of subsample $S_{k,j}$. The bootstrap mean $\mu_{k,j}^*$ of $S_{k,j}$ is the mean of the bootstrap sample means $\mu_{k,j,m}^*$:

$$ \mu_{k,j}^* = \frac{1}{M} \sum_{m=1}^{M} \mu_{k,j,m}^* $$

The standard error of the mean $\mu_{k,j}^*$ (denoted as $SEM_{k,j}^*$) obtained via bootstrapping is the standard deviation of the means of all bootstrap samples of subsample $S_{k,j}$:

$$ SEM_{k,j}^* = \sqrt{\frac{1}{M} \sum_{m=1}^{M} (\mu_{k,j,m}^* - \mu_{k,j}^*)^2} $$

The 95% confidence interval of the sample $S_{k,j}$ is denoted as $[a_{k,j}^*, b_{k,j}^*]$ and is the set of values between the 2.5% and 97.5% percentile of the sorted bootstrap means of subsample $S_{k,j}$. We then compute the average of $\mu_{k,j}^*$, and $SEM_{k,j}^*$ across the different subsamples $S_{k,j}$ for $k$ fixed and $j \in \{1, \dots, 100\}$ and denote the results as $\mu_k^*$ and $SEM_k^*$. For the confidence intervals, we compute the average values $a_k^* = \frac{1}{100} \sum_{j=1}^{100} a_{k,j}^*$ and $b_k^* = \frac{1}{100} \sum_{j=1}^{100} b_{k,j}^*$ to obtain the average confidence interval independent of the mean $CI_k^* = [a_k^* - \mu_k^*, b_k^* - \mu_k^*]$. We finally compute the normalized width of the confidence interval as: $\nu_k^* = \frac{b_k^* - a_k^*}{\mu_k^*}$.

## B. Detailed results on subsamples of the test sets

Detailed results on subsamples of the test set, which were summarized in Figures 3 and 4, are presented in Tables IV, V, VI, VII, VIII, IX, X and XI.

<table><caption>TABLE IV: Hippocampus dataset, 3D U-net, Dice accuracy. Results on sub-samples of size $k \le 110$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>89.751</td><td>2.578</td><td>0.815</td><td>[-1.6, 1.6]</td><td>0.036</td><td>89.752</td><td>0.815</td><td>[-1.647, 1.525]</td><td>0.035</td></tr><tr><td>20</td><td>89.723</td><td>2.666</td><td>0.596</td><td>[-1.17, 1.17]</td><td>0.026</td><td>89.724</td><td>0.597</td><td>[-1.204, 1.128]</td><td>0.026</td></tr><tr><td>30</td><td>89.681</td><td>2.785</td><td>0.508</td><td>[-0.995, 0.995]</td><td>0.022</td><td>89.682</td><td>0.508</td><td>[-1.023, 0.968]</td><td>0.022</td></tr><tr><td>50</td><td>89.768</td><td>2.707</td><td>0.383</td><td>[-0.75, 0.75]</td><td>0.017</td><td>89.768</td><td>0.383</td><td>[-0.766, 0.734]</td><td>0.017</td></tr><tr><td>100</td><td>89.721</td><td>2.788</td><td>0.279</td><td>[-0.545, 0.545]</td><td>0.012</td><td>89.721</td><td>0.279</td><td>[-0.557, 0.536]</td><td>0.012</td></tr><tr><td>110</td><td>89.714</td><td>2.784</td><td>0.265</td><td>[-0.52, 0.52]</td><td>0.012</td><td>89.714</td><td>0.266</td><td>[-0.529, 0.512]</td><td>0.012</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Figure 5: Histogram of the test-set size of different experiments from segmentation papers published in JMI, TMI, and MedIA in 2022. The red line represents the median test size (n = 25).</figcaption></figure>

is a need to increase the size of test sets to obtain reasonably precise estimates of the performance.

*Reporting confidence intervals is important.* Trustworthy evidence on segmentation tools is important for their clinical adoption. Confidence intervals and bootstrapping are classic statistical practice; and yet medical image segmentation papers seldom report confidence intervals: we found them in only 10% of papers of our survey (Table 1). Beyond immediate clinical adoption, controlled statistical evidence is a component of reproducibility, needed for scientific progress McDermott et al. (2019); Colliot et al. (2022). *Statistical reproducibility* is suited to reason on generalization to unseen data: results need to be statistically compatible rather than identical. Reporting confidence intervals is central to asserting statistical reproducibility of studies. Awareness about this issue has risen, as can been seen for example in reproducibility checklists (e.g. MICCAI 2021³). However, the fact that confidence intervals are so rarely reported in recent papers indicates yet more awareness is needed.

Reporting confidence intervals enables progress at the level of the community. To disseminate a trained model (for use by clinical researchers, neuroscientists ...), the users should un-

derstand expected variances in the performance of the model. Confidence intervals also enable comparing performance to inter-rater (or intra-rater) variability. When computing inter-rater variability on the test set used to evaluate the trained model, one can obtain confidence intervals for the trained model and for the inter-rater and check if they overlap.

The absence of confidence intervals in model evaluations can have serious implications for the interpretation of published results. When performance metrics such as Dice coefficients or Hausdorff distances are presented without associated confidence intervals, it becomes challenging for the community to assess the reliability of these metrics, especially when comparing different models or methods. For example, small differences in performance could be due to random fluctuations in the evaluation process rather than genuine improvements. This issue is particularly critical in medical imaging, where decisions based on model outputs can directly impact patient care. Thus, failing to report confidence intervals can lead to overconfidence in model performance and may prevent the detection of models that, while performing well on a given test set, fail to generalize to other datasets or clinical settings.

Confidence intervals (and standard errors) are tools from inferential statistics, specifically from statistical estimation. They indicate how *precise* an *estimate* (e.g. the mean Dice) is. They are dependent on sample size: when the sample size goes to infinity the standard errors tend towards zero and the CI tends towards a single point. Standard-deviation and other quantities such as inter-quartile ranges) are tools from descriptive statistics. They describe how *variable* a *measure-ment* (e.g. the Dice Similarity Coefficient across the test set) is. They don't shrink with increasing sample size. Of course, the estimate of the standard-deviation is more *precise* for a larger sample but its order of magnitude does not change. Therefore, inferential and descriptive statistics are complementary and both should be reported.

*A table for confidence intervals.* The width of the confidence interval depends on two factors: the size of the test set (n) and the variability of the performance (σ). As parametric estimates give reasonable results, one can compute typical confidence intervals for given n and σ and we provide this information in Table 3. These values can be useful to researchers preparing a segmentation study, to help them choose the size of the test set. Investigators must then make an hypothesis on the expected value of σ (as when doing a power analysis),

<footnote n="3">³https://miccai2021.org/files/downloads/MICCAI2021-Reproducibility-Checklist.pdf</footnote>
<table><caption>TABLE V: Hippocampus dataset, 2D U-net, Dice Accuracy. Results on subsamples of size $k \le 110$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="5">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>88.334</td><td>2.759</td><td>0.872</td><td>[-1.71, 1.71]</td><td>0.039</td><td>88.335</td><td>0.872</td><td>[-1.78, 1.608]</td><td>0.038</td></tr><tr><td>20</td><td>88.189</td><td>3.082</td><td>0.689</td><td>[-1.35, 1.35]</td><td>0.031</td><td>88.19</td><td>0.689</td><td>[-1.425, 1.263]</td><td>0.030</td></tr><tr><td>30</td><td>88.126</td><td>3.281</td><td>0.599</td><td>[-1.175, 1.175]</td><td>0.027</td><td>88.126</td><td>0.599</td><td>[-1.24, 1.102]</td><td>0.027</td></tr><tr><td>50</td><td>88.254</td><td>3.12</td><td>0.441</td><td>[-0.865, 0.865]</td><td>0.020</td><td>88.255</td><td>0.441</td><td>[-0.905, 0.822]</td><td>0.020</td></tr><tr><td>100</td><td>88.212</td><td>3.261</td><td>0.326</td><td>[-0.64, 0.64]</td><td>0.015</td><td>88.212</td><td>0.326</td><td>[-0.664, 0.612]</td><td>0.014</td></tr><tr><td>110</td><td>88.197</td><td>3.252</td><td>0.31</td><td>[-0.61, 0.61]</td><td>0.014</td><td>88.197</td><td>0.31</td><td>[-0.631, 0.584]</td><td>0.014</td></tr></tbody></table>

<table><caption>TABLE VI: Brain Tumor dataset, 3D U-net, Dice Accuracy. Results on subsamples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="5">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>80.457</td><td>10.612</td><td>3.356</td><td>[-6.58, 6.58]</td><td>0.164</td><td>80.461</td><td>3.356</td><td>[-7.051, 5.853]</td><td>0.160</td></tr><tr><td>20</td><td>80.188</td><td>11.281</td><td>2.522</td><td>[-4.945, 4.945]</td><td>0.123</td><td>80.189</td><td>2.523</td><td>[-5.28, 4.541]</td><td>0.122</td></tr><tr><td>30</td><td>79.925</td><td>11.839</td><td>2.162</td><td>[-4.235, 4.235]</td><td>0.106</td><td>79.924</td><td>2.162</td><td>[-4.512, 3.925]</td><td>0.106</td></tr><tr><td>50</td><td>80.375</td><td>11.593</td><td>1.64</td><td>[-3.215, 3.215]</td><td>0.080</td><td>80.374</td><td>1.639</td><td>[-3.384, 3.015]</td><td>0.080</td></tr><tr><td>100</td><td>80.388</td><td>11.389</td><td>1.139</td><td>[-2.23, 2.23]</td><td>0.055</td><td>80.388</td><td>1.139</td><td>[-2.321, 2.137]</td><td>0.055</td></tr><tr><td>150</td><td>80.329</td><td>11.788</td><td>0.962</td><td>[-1.885, 1.885]</td><td>0.047</td><td>80.328</td><td>0.963</td><td>[-1.954, 1.817]</td><td>0.047</td></tr><tr><td>200</td><td>80.253</td><td>11.936</td><td>0.844</td><td>[-1.655, 1.655]</td><td>0.041</td><td>80.252</td><td>0.844</td><td>[-1.709, 1.601]</td><td>0.041</td></tr><tr><td>250</td><td>80.295</td><td>11.919</td><td>0.754</td><td>[-1.475, 1.475]</td><td>0.037</td><td>80.296</td><td>0.754</td><td>[-1.521, 1.431]</td><td>0.037</td></tr><tr><td>300</td><td>80.28</td><td>11.924</td><td>0.688</td><td>[-1.35, 1.35]</td><td>0.034</td><td>80.281</td><td>0.688</td><td>[-1.386, 1.311]</td><td>0.034</td></tr><tr><td>334</td><td>80.265</td><td>11.929</td><td>0.653</td><td>[-1.28, 1.28]</td><td>0.032</td><td>80.264</td><td>0.653</td><td>[-1.313, 1.245]</td><td>0.032</td></tr></tbody></table>

<table><caption>TABLE VII: Brain Tumor dataset, 2D U-net, Dice Accuracy. Results on subsamples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="5">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>77.811</td><td>11.799</td><td>3.731</td><td>[-7.315, 7.315]</td><td>0.188</td><td>77.815</td><td>3.731</td><td>[-7.856, 6.566]</td><td>0.185</td></tr><tr><td>20</td><td>77.334</td><td>12.631</td><td>2.824</td><td>[-5.535, 5.535]</td><td>0.143</td><td>77.334</td><td>2.825</td><td>[-5.879, 5.137]</td><td>0.142</td></tr><tr><td>30</td><td>77.101</td><td>13.084</td><td>2.389</td><td>[-4.68, 4.68]</td><td>0.121</td><td>77.101</td><td>2.389</td><td>[-4.94, 4.389]</td><td>0.121</td></tr><tr><td>50</td><td>77.607</td><td>12.897</td><td>1.824</td><td>[-3.575, 3.575]</td><td>0.092</td><td>77.606</td><td>1.823</td><td>[-3.736, 3.39]</td><td>0.092</td></tr><tr><td>100</td><td>77.564</td><td>12.797</td><td>1.28</td><td>[-2.51, 2.51]</td><td>0.065</td><td>77.564</td><td>1.279</td><td>[-2.588, 2.42]</td><td>0.065</td></tr><tr><td>150</td><td>77.583</td><td>13.01</td><td>1.062</td><td>[-2.08, 2.08]</td><td>0.054</td><td>77.582</td><td>1.063</td><td>[-2.139, 2.023]</td><td>0.054</td></tr><tr><td>200</td><td>77.468</td><td>13.182</td><td>0.932</td><td>[-1.825, 1.825]</td><td>0.047</td><td>77.467</td><td>0.932</td><td>[-1.871, 1.778]</td><td>0.047</td></tr><tr><td>250</td><td>77.549</td><td>13.056</td><td>0.826</td><td>[-1.62, 1.62]</td><td>0.042</td><td>77.549</td><td>0.826</td><td>[-1.654, 1.582]</td><td>0.042</td></tr><tr><td>300</td><td>77.501</td><td>13.104</td><td>0.757</td><td>[-1.485, 1.485]</td><td>0.038</td><td>77.502</td><td>0.756</td><td>[-1.511, 1.448]</td><td>0.038</td></tr><tr><td>334</td><td>77.489</td><td>13.115</td><td>0.718</td><td>[-1.405, 1.405]</td><td>0.036</td><td>77.488</td><td>0.717</td><td>[-1.43, 1.376]</td><td>0.036</td></tr></tbody></table>

<table><caption>TABLE VIII: Hippocampus dataset, 3D U-net, Hausdorff Distance. Results on sub-samples of size $k \le 110$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="5">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>1.214</td><td>0.416</td><td>0.132</td><td>[-0.26, 0.26]</td><td>0.428</td><td>1.214</td><td>0.131</td><td>[-0.205, 0.286]</td><td>0.404</td></tr><tr><td>20</td><td>1.206</td><td>0.435</td><td>0.097</td><td>[-0.19, 0.19]</td><td>0.315</td><td>1.206</td><td>0.097</td><td>[-0.165, 0.21]</td><td>0.310</td></tr><tr><td>30</td><td>1.221</td><td>0.468</td><td>0.086</td><td>[-0.17, 0.17]</td><td>0.278</td><td>1.22</td><td>0.086</td><td>[-0.149, 0.183]</td><td>0.271</td></tr><tr><td>50</td><td>1.203</td><td>0.46</td><td>0.065</td><td>[-0.125, 0.125]</td><td>0.208</td><td>1.203</td><td>0.065</td><td>[-0.116, 0.137]</td><td>0.211</td></tr><tr><td>100</td><td>1.205</td><td>0.47</td><td>0.047</td><td>[-0.09, 0.09]</td><td>0.149</td><td>1.205</td><td>0.047</td><td>[-0.086, 0.097]</td><td>0.152</td></tr><tr><td>110</td><td>1.205</td><td>0.47</td><td>0.045</td><td>[-0.09, 0.09]</td><td>0.149</td><td>1.205</td><td>0.045</td><td>[-0.08, 0.09]</td><td>0.149</td></tr></tbody></table>
for example using previously published studies on similar seg-
mentation tasks (given that these report $\sigma$). The values can
also be useful to reviewers who can use these to gauge plausi-
ble confidence intervals, should these not be reported in the
paper.

*When standard deviation is not reported.* The width of the confidence interval depends on two factors: the size of the test set (n) and the variability of the performance (σ). In classification problems, for some metrics such as accuracy, there is a closed-form relationship between SD and the mean since the metric follows a binomial proportion distribution (the variance is p × (1 − p) where p is the accuracy). In such a case, we have an analytical formula which shows that the variability decreases as performance approaches its extremes. As a consequence, for classification, using a binomial distribution gives parametric confidence intervals Varoquaux (2018); Varoquaux and Colliot (2023). Confidence intervals can thus easily be computed post-hoc as a function of the accuracy, e.g. when reading a manuscript.

However, for segmentation metrics like the Dice Similarity Coefficient (DSC), no such closed-form relationship exists. Nevertheless, in practice, Christodoulou et al. (2024) have shown that, empirically, one observes the same pattern of relationship between mean and SD, although not in an exact manner. Specifically, they showcase that SD can be approximated by a second-order polynomial function of the mean DSC. Such an approximation can offer a way to have a rough estimate of SD when it is not reported. This in turns provides a rough confidence interval. This approximation, while useful, does not eliminate the necessity of directly reporting standard-deviation and confidence intervals in future work.

***Preliminary recommendations.*** Providing definite recom-
mendations for reporting is clearly beyond the scope of the
present paper. Indeed, such recommendations would require
further experiments with more tasks, models and metrics as
well as involving more members of the community to reach a
consensus.

However, we believe that we can still provide preliminary
recommendations which would anyway constitute an im-
provement over current typical practices. We recommend
that authors systematically report confidence intervals for
trained models. Even though we showed that parametric ap-
proximations can yield reasonable estimates, we tend to rec-
ommend to use the bootstrap method as it does not make as-

sumptions on the distribution of the metric. Moreover, it is
easy to perform since it only requires to resample performance
metrics obtained on the test set (without recomputing them).
Furthermore, unlike what is commonly believed, bootstrap-
ping on the test set is actually very fast: less than a second on
a standard laptop in our experiments (specifically 0.53 second
for the hippocampus and 0.62 second for the Brain Tumor
dataset). This is negligible compared to the cost of training a
deep learning model. Note nevertheless, that both the para-
metric method and the standard bootstrap assume that the
test observations are independent. If this does not hold, CIs
will be invalid. The case of dependent observations is be-
yond the scope of the present paper. Finally, confidence inter-
vals should be complemented by descriptive statistics, both in
graphical (e.g. displaying the distribution of the performance
metrics with a box plot) and numerical form (reporting stan-
dard deviation, median and inter-quartile range for instance).

***More work to benchmark learning procedures.*** We investigated the sampling noise in the test set, the only source of variance when evaluating a *trained model*. However evaluating a *learning procedure* –as when comparing two methods– comes with other sources of variance such as hyperparameters or random seeds Bouthillier et al. (2021); Varoquaux and Colliot (2023). Methods to build confidence intervals for *learning procedures* typically involve running multiple trainings with different data splits (to account for sampling noise in training and testing sets) but also different hyperparameters or random seeds Bouthillier et al. (2021); Varoquaux and Colliot (2023). They require much more computing power and lead to wider confidence intervals.

***Limitations.*** While we studied the two most widely used performance metrics, there are many other metrics, some being more appropriate for certain settings Maier-Hein et al. (2024). Also, our scope is limited to brain MRI tasks, specifically hippocampus and tumor segmentation. We have included both a relatively easy and a more difficult segmentation task which allows to see the behavior of CIs across different levels of performance. However, other other modalities, such as abdominal CT or cardiac ultrasound, introduce additional variability and annotation challenges. It remains to be seen if our observations hold for other organs and modalities. Also, we only studied one method, the nnU-net, and future work would need to study other architectures.
<table><caption>TABLE IX: Hippocampus dataset, 2D U-net, Hausdorff Distance. Results on subsamples of size $k \le 110$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>1.296</td><td>0.538</td><td>0.17</td><td>[-0.335, 0.335]</td><td>0.517</td><td>1.296</td><td>0.17</td><td>[-0.267, 0.367]</td><td>0.490</td></tr><tr><td>20</td><td>1.302</td><td>0.599</td><td>0.134</td><td>[-0.265, 0.265]</td><td>0.407</td><td>1.302</td><td>0.134</td><td>[-0.219, 0.289]</td><td>0.391</td></tr><tr><td>30</td><td>1.331</td><td>0.718</td><td>0.131</td><td>[-0.255, 0.255]</td><td>0.383</td><td>1.331</td><td>0.131</td><td>[-0.211, 0.288]</td><td>0.375</td></tr><tr><td>50</td><td>1.31</td><td>0.729</td><td>0.103</td><td>[-0.2, 0.2]</td><td>0.305</td><td>1.31</td><td>0.103</td><td>[-0.168, 0.228]</td><td>0.303</td></tr><tr><td>100</td><td>1.31</td><td>0.793</td><td>0.079</td><td>[-0.155, 0.155]</td><td>0.237</td><td>1.31</td><td>0.079</td><td>[-0.133, 0.175]</td><td>0.234</td></tr><tr><td>110</td><td>1.311</td><td>0.803</td><td>0.077</td><td>[-0.15, 0.15]</td><td>0.229</td><td>1.311</td><td>0.077</td><td>[-0.13, 0.17]</td><td>0.229</td></tr></tbody></table>

<table><caption>TABLE X: Brain Tumor dataset, 3D U-net, Hausdorff Distance. Results on subsamples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>8.171</td><td>8.806</td><td>2.785</td><td>[-5.46, 5.46]</td><td>1.336</td><td>8.168</td><td>2.784</td><td>-4.278, 6.046</td><td>1.264</td></tr><tr><td>20</td><td>7.635</td><td>9.098</td><td>2.034</td><td>[-3.985, 3.985]</td><td>1.044</td><td>7.634</td><td>2.034</td><td>-3.268, 4.459</td><td>1.012</td></tr><tr><td>30</td><td>7.95</td><td>9.782</td><td>1.786</td><td>[-3.5, 3.5]</td><td>0.881</td><td>7.95</td><td>1.787</td><td>-2.98, 3.909</td><td>0.866</td></tr><tr><td>50</td><td>7.563</td><td>9.791</td><td>1.385</td><td>[-2.715, 2.715]</td><td>0.718</td><td>7.563</td><td>1.383</td><td>-2.36, 3.002</td><td>0.709</td></tr><tr><td>100</td><td>7.702</td><td>10.468</td><td>1.047</td><td>[-2.05, 2.05]</td><td>0.532</td><td>7.702</td><td>1.046</td><td>-1.848, 2.228</td><td>0.529</td></tr><tr><td>150</td><td>7.691</td><td>10.339</td><td>0.844</td><td>[-1.655, 1.655]</td><td>0.430</td><td>7.693</td><td>0.844</td><td>-1.522, 1.775</td><td>0.428</td></tr><tr><td>200</td><td>7.698</td><td>10.413</td><td>0.736</td><td>[-1.445, 1.445]</td><td>0.375</td><td>7.697</td><td>0.736</td><td>-1.341, 1.535</td><td>0.374</td></tr><tr><td>250</td><td>7.758</td><td>10.663</td><td>0.674</td><td>[-1.32, 1.32]</td><td>0.340</td><td>7.757</td><td>0.674</td><td>-1.238, 1.397</td><td>0.340</td></tr><tr><td>300</td><td>7.695</td><td>10.513</td><td>0.607</td><td>[-1.19, 1.19]</td><td>0.309</td><td>7.694</td><td>0.607</td><td>-1.12, 1.257</td><td>0.309</td></tr><tr><td>334</td><td>7.726</td><td>10.618</td><td>0.581</td><td>[-1.14, 1.14]</td><td>0.295</td><td>7.726</td><td>0.581</td><td>-1.076, 1.196</td><td>0.294</td></tr></tbody></table>

<table><caption>TABLE XI: Brain Tumor dataset, 2D U-net, Hausdorff Distance. Results on sub-samples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>9.544</td><td>10.409</td><td>3.292</td><td>[-6.45, 6.45]</td><td>1.352</td><td>9.544 ± 3.441</td><td>3.294</td><td>-5.099, 7.152</td><td>1.284</td></tr><tr><td>20</td><td>8.454</td><td>9.178</td><td>2.052</td><td>[-4.02, 4.02]</td><td>0.951</td><td>8.453 ± 2.302</td><td>2.05</td><td>-3.454, 4.441</td><td>0.934</td></tr><tr><td>30</td><td>9.204</td><td>10.794</td><td>1.971</td><td>[-3.86, 3.86]</td><td>0.839</td><td>9.205 ± 1.786</td><td>1.973</td><td>-3.374, 4.269</td><td>0.830</td></tr><tr><td>50</td><td>8.836</td><td>10.86</td><td>1.536</td><td>[-3.01, 3.01]</td><td>0.681</td><td>8.837 ± 1.359</td><td>1.537</td><td>-2.696, 3.282</td><td>0.676</td></tr><tr><td>100</td><td>8.758</td><td>10.966</td><td>1.097</td><td>[-2.15, 2.15]</td><td>0.491</td><td>8.756 ± 0.938</td><td>1.096</td><td>-1.974, 2.307</td><td>0.489</td></tr><tr><td>150</td><td>8.886</td><td>11.158</td><td>0.911</td><td>[-1.785, 1.785]</td><td>0.402</td><td>8.885 ± 0.673</td><td>0.912</td><td>-1.671, 1.894</td><td>0.401</td></tr><tr><td>200</td><td>8.806</td><td>11.068</td><td>0.783</td><td>[-1.535, 1.535]</td><td>0.349</td><td>8.806 ± 0.471</td><td>0.783</td><td>-1.445, 1.617</td><td>0.348</td></tr><tr><td>250</td><td>8.873</td><td>11.32</td><td>0.716</td><td>[-1.405, 1.405]</td><td>0.317</td><td>8.873 ± 0.357</td><td>0.715</td><td>-1.33, 1.469</td><td>0.315</td></tr><tr><td>300</td><td>8.83</td><td>11.173</td><td>0.645</td><td>[-1.265, 1.265]</td><td>0.287</td><td>8.831 ± 0.214</td><td>0.645</td><td>-1.204, 1.32</td><td>0.286</td></tr><tr><td>334</td><td>8.855</td><td>11.245</td><td>0.615</td><td>[-1.205, 1.205]</td><td>0.272</td><td>8.856</td><td>0.615</td><td>-1.154, 1.257</td><td>0.272</td></tr></tbody></table>
Furthermore, many factors contribute to the variability of performance, including lesion size and image quality. Larger lesions generally result in higher DSC scores, while smaller lesions often lead to lower DSC scores. Similarly, image quality, which can vary due to factors such as motion artifacts, noise, or differences in resolution, introduces additional variability, making it harder for models to consistently segment lesions accurately across different scans. Assessing the influence of these factors is another important future work.

*Distribution shifts during deployment.* It is crucial to acknowledge the potential for distribution shifts during deployment, such as when applying the method to healthy scans or cases with lesions of significantly different sizes. Confidence intervals provide a measure of variability under random sampling from a given test distribution, but they do not account for distribution shifts that may arise during deployment. Distribution shifts are a separate and important topic, as explored in studies Koch et al. (2024).

## 8. Conclusion

Confidence intervals are needed to characterize the uncertainty on the performance of image segmentation tools. In the context of 3D brain MRI segmentation, we have shown that simple parametric estimates can serve as a reasonable reference to guide researchers on the test set size needed for a target precision. Importantly, results show that the test set size needed to achieve a given precision is lower for segmentation than for classification tasks. Confidence intervals can be computed easily; we hope that they will be used more in image segmentation studies.

## Disclosure of interests

**Competing financial interests related to the present article:** none to disclose for all authors.

**Competing financial interests unrelated to the present article:** OC reports having received consulting fees from AskBio and Therapanacea. OC reports that his laboratory has received grants (paid to the institution) from Qynapse and that members from his laboratory have co-supervised a PhD thesis with Qynapse. OC's spouse was an employee myBrainTechnologies and is now an employee of DiamPark. OC holds a patent registered at the International

Bureau of the World Intellectual Property Organization (PCT/IB2016/0526993, Schiratti J-B, Allassonniere S, Colliot O, Durrleman S, A method for determining the temporal progression of a biological phenomenon and associated methods and devices) (2017).

## Acknowledgements

The research leading to these results has received funding from the French government under management of Agence Nationale de la Recherche as part of the "France 2030" program (reference ANR-23-IACL-0008, project PRAIRIE-PSAI), as part of the "Investissements d'avenir" program (reference ANR-19-P3IA-0001, project PRAIRIE 3IA Institute and reference ANR-10-IAIHU-06, project Agence Nationale de la Recherche-10-IA Institut Hospitalo-Universitaire-6), from Agence Nationale de la Recherche (reference ANR-23-CE17-0054, project LYMP-PD), from the European Union's Horizon Europe Framework Programme (grant number 101136607, project CLARA), was supported by a grant from Inserm and the French Ministry of Health in the context of the MESSIDORE 2023 call operated by IReSP (reference AAP-2023-MSDR-341011), and by the Fonds Recherche Neurosciences under the project "NEIMO - Monitoring Neuroinflammation: a Paris Brain Institute-Yale Neurology Collaborative program

*Appendix 1. Details on the estimated measures on subsamples of the test sets*

*Parametric estimates.* For the parametric estimation, we denote as

$$ \mu_{k,j} = \frac{1}{k} \sum_{l=1}^{k} P_{j,k,l}, \quad \sigma_{k,j} = \sqrt{\frac{1}{k} \sum_{l=1}^{k} (P_{j,k,l} - \mu_{k,j})^2} $$

the empirical mean and standard deviation for the subsample $S_{k,j}$ where $P_{j,k,l}$ is the performance metric (in our case either Dice Similarity Coefficient or Hausdorff distance) of a given subject in the sub-sample $S_{k,j}$. Similarly, we use the notations

$$ SEM_{k,j} = \frac{\sigma_{k,j}}{\sqrt{k}}, \quad \nu_{k,j} = \frac{2 * 1.96 * SEM_{k,j}}{\mu_{k,j}} $$

We then compute the average of $\mu_{k,j}$, $\sigma_{k,j}$, $SEM_{k,j}$, $\nu_{k,j}$ across the different subsamples $S_{k,j}$ for $k$ fixed and $j \in \{1, \dots, 100\}$. This provides the following estimates $\mu_k$, $\sigma_k$, $SEM_k$ and $\nu_k$. Similarly, we define $CI_k = [-1.96 \times SEM_k, 1.96 \times SEM_k]$.
*Bootstrap estimates.* Bootstrap estimations are performed as follows. For a given subsample $S_{k,j}$ of size $k$ and index $j$, $M = 15000$ bootstrap samples of size $k$ are drawn with replacement. We denote a given bootstrap sample as $S_{k,j,m}^*$ and its mean as $\mu_{k,j,m}^*$ where $m \in \{1, \dots, M\}$ is the index of the $m^{th}$ bootstrap sample of subsample $S_{k,j}$. The bootstrap mean $\mu_{k,j}^*$ of $S_{k,j}$ is the mean of the bootstrap sample means $\mu_{k,j,m}^*$:

$$ \mu_{k,j}^* = \frac{1}{M} \sum_{m=1} \mu_{k,j,m}^* $$

The standard error of the mean $\mu_{k,j}^*$ (denoted as $SEM_{k,j}^*$) obtained via bootstrapping is the standard deviation of the means of all bootstrap samples of subsample $S_{k,j}$:

$$ SEM_{k,j}^* = \sqrt{\frac{1}{M} \sum_{m=1}^{M} (\mu_{k,j,m}^* - \mu_{k,j}^*)^2} $$

The 95% confidence interval of the sample $S_{k,j}$ is denoted as $[a_{k,j}^*, b_{k,j}^*]$ and is the set of values between the 2.5% and 97.5% percentile of the sorted bootstrap means of subsample $S_{k,j}$. We then compute the average of $\mu_{k,j}^*$, and $SEM_{k,j}^*$ across the different subsamples $S_{k,j}$ for $k$ fixed and $j \in \{1, \dots, 100\}$ and denote the results as $\mu_k^*$ and $SEM_k^*$. For the confidence intervals, we compute the average values $a_k^* = \frac{1}{100} \sum_{j=1}^{100} a_{k,j}^*$ and $b_k^* = \frac{1}{100} \sum_{j=1}^{100} b_{k,j}^*$ to obtain the average confidence interval independent of the mean $CI_k^* = [a_k^* - \mu_k^*, b_k^* - \mu_k^*]$. We finally compute the normalized width of the confidence interval as: $\gamma_k^* = \frac{b_k^* - a_k^*}{\mu_k^*}$.

## Appendix 2. Detailed results on subsamples of the test sets

Detailed results on subsamples of the test set, which were summarized in Figures 3 and 4, are presented in Tables .4, .5, .6, .7, .8, .9, .10 and .11.
<table><caption>Table 3: Table of SEM and confidence intervals CI for different sizes k of the test set and for different values of σ including the experimental values obtained in Table 2.</caption><thead><tr><th>k →</th><th>10</th><th>20</th><th>30</th><th>50</th><th>100</th><th>200</th><th>300</th><th>500</th><th>1000</th><th>1500</th><th>2000</th><th>2500</th><th>3000</th></tr></thead><tbody><tr><td>0.47</td><td>CI</td><td>0.15<br>(-0.29, 0.29)</td><td>0.11<br>(-0.21, 0.21)</td><td>0.09<br>(-0.17, 0.17)</td><td>0.07<br>(-0.13, 0.13)</td><td>0.05<br>(-0.09, 0.09)</td><td>0.03<br>(-0.07, 0.07)</td><td>0.02<br>(-0.05, 0.05)</td><td>0.01<br>(-0.05, 0.05)</td><td>0.01<br>(-0.02, 0.02)</td><td>0.01<br>(-0.02, 0.02)</td><td>0.01<br>(-0.02, 0.02)</td><td>0.01<br>(-0.02, 0.02)</td></tr><tr><td>0.81</td><td>CI</td><td>0.26<br>(-0.5, 0.5)</td><td>0.18<br>(-0.35, 0.35)</td><td>0.15<br>(-0.29, 0.29)</td><td>0.11<br>(-0.22, 0.22)</td><td>0.08<br>(-0.16, 0.16)</td><td>0.06<br>(-0.11, 0.11)</td><td>0.05<br>(-0.09, 0.09)</td><td>0.04<br>(-0.07, 0.07)</td><td>0.03<br>(-0.05, 0.05)</td><td>0.02<br>(-0.04, 0.04)</td><td>0.02<br>(-0.04, 0.04)</td><td>0.01<br>(-0.05, 0.03)</td></tr><tr><td>1</td><td>CI</td><td>0.32<br>(-0.62, 0.62)</td><td>0.22<br>(-0.44, 0.44)</td><td>0.18<br>(-0.36, 0.36)</td><td>0.14<br>(-0.28, 0.28)</td><td>0.1<br>(-0.2, 0.2)</td><td>0.07<br>(-0.14, 0.14)</td><td>0.06<br>(-0.11, 0.11)</td><td>0.04<br>(-0.09, 0.09)</td><td>0.03<br>(-0.06, 0.06)</td><td>0.03<br>(-0.05, 0.03)</td><td>0.02<br>(-0.04, 0.04)</td><td>0.02<br>(-0.04, 0.04)</td></tr><tr><td>2.79</td><td>CI</td><td>0.88<br>(-1.75, 1.75)</td><td>0.62<br>(-1.22, 1.22)</td><td>0.51<br>(-1.0, 1.0)</td><td>0.39<br>(-0.77, 0.77)</td><td>0.28<br>(-0.55, 0.55)</td><td>0.2<br>(-0.39, 0.39)</td><td>0.16<br>(-0.32, 0.32)</td><td>0.12<br>(-0.24, 0.24)</td><td>0.09<br>(-0.17, 0.17)</td><td>0.07<br>(-0.14, 0.14)</td><td>0.06<br>(-0.12, 0.12)</td><td>0.05<br>(-0.11, 0.11)</td></tr><tr><td>3.26</td><td>CI</td><td>1.03<br>(-2.02, 2.02)</td><td>0.73<br>(-1.43, 1.43)</td><td>0.6<br>(-1.17, 1.17)</td><td>0.46<br>(-0.9, 0.9)</td><td>0.33<br>(-0.64, 0.64)</td><td>0.23<br>(-0.45, 0.45)</td><td>0.19<br>(-0.37, 0.37)</td><td>0.15<br>(-0.29, 0.29)</td><td>0.07<br>(-0.2, 0.2)</td><td>0.08<br>(-0.16, 0.16)</td><td>0.07<br>(-0.14, 0.14)</td><td>0.06<br>(-0.12, 0.12)</td></tr><tr><td>5</td><td>CI</td><td>1.58<br>(-3.1, 3.1)</td><td>1.12<br>(-2.19, 2.19)</td><td>0.91<br>(-1.79, 1.79)</td><td>0.71<br>(-1.39, 1.39)</td><td>0.5<br>(-0.98, 0.98)</td><td>0.35<br>(-0.69, 0.69)</td><td>0.29<br>(-0.57, 0.57)</td><td>0.22<br>(-0.44, 0.44)</td><td>0.16<br>(-0.31, 0.31)</td><td>0.13<br>(-0.25, 0.25)</td><td>0.11<br>(-0.22, 0.22)</td><td>0.09<br>(-0.2, 0.2)</td></tr><tr><td>10.65</td><td>CI</td><td>3.36<br>(-6.59, 6.59)</td><td>2.38<br>(-4.66, 4.66)</td><td>1.94<br>(-3.8, 1.8)</td><td>1.5<br>(-2.95, 2.95)</td><td>1.06<br>(-2.08, 2.08)</td><td>0.75<br>(-1.47, 1.47)</td><td>0.61<br>(-1.2, 1.2)</td><td>0.48<br>(-0.95, 0.93)</td><td>0.34<br>(-0.66, 0.66)</td><td>0.27<br>(-0.54, 0.54)</td><td>0.24<br>(-0.47, 0.47)</td><td>0.19<br>(-0.38, 0.18)</td></tr><tr><td>11.26</td><td>CI</td><td>3.56<br>(-6.98, 6.98)</td><td>2.52<br>(-4.95, 4.93)</td><td>2.06<br>(-4.03, 4.03)</td><td>1.59<br>(-3.12, 3.12)</td><td>1.13<br>(-2.21, 2.21)</td><td>0.8<br>(-1.56, 1.56)</td><td>0.5<br>(-1.27, 1.27)</td><td>0.5<br>(-0.99, 0.99)</td><td>0.36<br>(-0.7, 0.7)</td><td>0.29<br>(-0.57, 0.57)</td><td>0.25<br>(-0.49, 0.49)</td><td>0.21<br>(-0.44, 0.44)</td></tr><tr><td>12</td><td>CI</td><td>3.79<br>(-7.44, 7.44)</td><td>2.68<br>(-5.26, 5.26)</td><td>2.19<br>(-4.29, 4.29)</td><td>1.7<br>(-3.33, 3.33)</td><td>1.2<br>(-2.35, 2.35)</td><td>0.85<br>(-1.66, 1.66)</td><td>0.69<br>(-1.36, 1.36)</td><td>0.34<br>(-1.05, 1.05)</td><td>0.38<br>(-0.74, 0.74)</td><td>0.31<br>(-0.61, 0.61)</td><td>0.27<br>(-0.53, 0.53)</td><td>0.24<br>(-0.47, 0.47)</td></tr><tr><td>13.12</td><td>CI</td><td>4.15<br>(-8.13, 8.13)</td><td>2.94<br>(-5.75, 5.75)</td><td>2.4<br>(-4.69, 4.69)</td><td>1.86<br>(-3.64, 3.64)</td><td>1.34<br>(-2.57, 2.57)</td><td>0.93<br>(-1.82, 1.82)</td><td>0.76<br>(-1.48, 1.48)</td><td>0.59<br>(-1.35, 1.15)</td><td>0.41<br>(-0.81, 0.81)</td><td>0.34<br>(-0.66, 0.66)</td><td>0.29<br>(-0.38, 0.38)</td><td>0.24<br>(-0.51, 0.51)</td></tr><tr><td>20</td><td>CI</td><td>6.32<br>(-12.4, 12.4)</td><td>4.47<br>(-8.77, 8.77)</td><td>3.65<br>(-7.16, 7.16)</td><td>2.83<br>(-5.54, 5.54)</td><td>2.0<br>(-3.92, 3.92)</td><td>1.41<br>(-2.77, 2.77)</td><td>1.15<br>(-2.26, 2.26)</td><td>0.89<br>(-1.75, 1.75)</td><td>0.63<br>(-1.24, 1.24)</td><td>0.52<br>(-1.01, 1.01)</td><td>0.45<br>(-0.88, 0.88)</td><td>0.4<br>(-0.78, 0.78)</td></tr><tr><td>30</td><td>CI</td><td>9.49<br>(-18.59, 18.59)</td><td>6.71<br>(-13.15, 13.15)</td><td>5.48<br>(-10.74, 10.74)</td><td>4.24<br>(-8.32, 8.32)</td><td>3.0<br>(-5.88, 5.88)</td><td>2.12<br>(-4.16, 4.16)</td><td>1.73<br>(-3.39, 3.39)</td><td>1.34<br>(-2.61, 2.61)</td><td>0.95<br>(-1.31, 1.31)</td><td>0.77<br>(-1.18, 1.18)</td><td>0.67<br>(-1.18, 1.18)</td><td>0.55<br>(-1.07, 1.07)</td></tr><tr><td>50</td><td>CI</td><td>15.81<br>(-30.99, 30.99)</td><td>11.18<br>(-21.91, 21.91)</td><td>9.13<br>(-17.89, 17.89)</td><td>7.07<br>(-13.86, 13.86)</td><td>5.0<br>(-9.8, 9.8)</td><td>3.54<br>(-6.93, 6.93)</td><td>2.89<br>(-5.66, 5.66)</td><td>2.24<br>(-4.38, 4.38)</td><td>1.58<br>(-3.14, 3.14)</td><td>1.29<br>(-2.19, 2.19)</td><td>1.12<br>(-1.96, 1.96)</td><td>1.01<br>(-1.79, 1.79)</td></tr></tbody></table>
<table><caption>Table 4: Hippocampus dataset, 3D U-net, Dice accuracy. Results on sub-samples of size $k \le 110$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>89.751</td><td>2.578</td><td>0.815</td><td>[-1.6, 1.6]</td><td>0.036</td><td>89.752</td><td>0.815</td><td>[-1.647, 1.525]</td><td>0.035</td></tr><tr><td>20</td><td>89.723</td><td>2.666</td><td>0.596</td><td>[-1.17, 1.17]</td><td>0.026</td><td>89.724</td><td>0.597</td><td>[-1.204, 1.128]</td><td>0.026</td></tr><tr><td>30</td><td>89.681</td><td>2.785</td><td>0.508</td><td>[-0.995, 0.995]</td><td>0.022</td><td>89.682</td><td>0.508</td><td>[-1.023, 0.968]</td><td>0.022</td></tr><tr><td>50</td><td>89.768</td><td>2.707</td><td>0.383</td><td>[-0.75, 0.75]</td><td>0.017</td><td>89.768</td><td>0.383</td><td>[-0.766, 0.734]</td><td>0.017</td></tr><tr><td>100</td><td>89.721</td><td>2.788</td><td>0.279</td><td>[-0.545, 0.545]</td><td>0.012</td><td>89.721</td><td>0.279</td><td>[-0.557, 0.536]</td><td>0.012</td></tr><tr><td>110</td><td>89.714</td><td>2.784</td><td>0.265</td><td>[-0.52, 0.52]</td><td>0.012</td><td>89.714</td><td>0.266</td><td>[-0.529, 0.512]</td><td>0.012</td></tr></tbody></table>

<table><caption>Table 5: Hippocampus dataset, 2D U-net, Dice Accuracy. Results on subsamples of size $k \le 110$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>88.334</td><td>2.759</td><td>0.872</td><td>[-1.71, 1.71]</td><td>0.039</td><td>88.335</td><td>0.872</td><td>[-1.78, 1.608]</td><td>0.038</td></tr><tr><td>20</td><td>88.189</td><td>3.082</td><td>0.689</td><td>[-1.35, 1.35]</td><td>0.031</td><td>88.19</td><td>0.689</td><td>[-1.425, 1.263]</td><td>0.030</td></tr><tr><td>30</td><td>88.126</td><td>3.281</td><td>0.599</td><td>[-1.175, 1.175]</td><td>0.027</td><td>88.126</td><td>0.599</td><td>[-1.24, 1.102]</td><td>0.027</td></tr><tr><td>50</td><td>88.254</td><td>3.12</td><td>0.441</td><td>[-0.865, 0.865]</td><td>0.020</td><td>88.255</td><td>0.441</td><td>[-0.905, 0.822]</td><td>0.020</td></tr><tr><td>100</td><td>88.212</td><td>3.261</td><td>0.326</td><td>[-0.64, 0.64]</td><td>0.015</td><td>88.212</td><td>0.326</td><td>[-0.664, 0.612]</td><td>0.014</td></tr><tr><td>110</td><td>88.197</td><td>3.252</td><td>0.31</td><td>[-0.61, 0.61]</td><td>0.014</td><td>88.197</td><td>0.31</td><td>[-0.631, 0.584]</td><td>0.014</td></tr></tbody></table>
<table><caption>Table .6: Brain Tumor dataset, 3D U-net, Dice Accuracy. Results on subsamples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>80.457</td><td>10.612</td><td>3.356</td><td>[-6.58, 6.58]</td><td>0.164</td><td>80.461</td><td>3.356</td><td>[-7.051, 5.853]</td><td>0.160</td></tr><tr><td>20</td><td>80.188</td><td>11.281</td><td>2.522</td><td>[-4.945, 4.945]</td><td>0.123</td><td>80.189</td><td>2.523</td><td>[-5.28, 4.541]</td><td>0.122</td></tr><tr><td>30</td><td>79.925</td><td>11.839</td><td>2.162</td><td>[-4.235, 4.235]</td><td>0.106</td><td>79.924</td><td>2.162</td><td>[-4.512, 3.925]</td><td>0.106</td></tr><tr><td>50</td><td>80.375</td><td>11.593</td><td>1.64</td><td>[-3.215, 3.215]</td><td>0.080</td><td>80.374</td><td>1.639</td><td>[-3.384, 3.015]</td><td>0.080</td></tr><tr><td>100</td><td>80.388</td><td>11.389</td><td>1.139</td><td>[-2.23, 2.23]</td><td>0.055</td><td>80.388</td><td>1.139</td><td>[-2.321, 2.137]</td><td>0.055</td></tr><tr><td>150</td><td>80.329</td><td>11.788</td><td>0.962</td><td>[-1.885, 1.885]</td><td>0.047</td><td>80.328</td><td>0.963</td><td>[-1.954, 1.817]</td><td>0.047</td></tr><tr><td>200</td><td>80.253</td><td>11.936</td><td>0.844</td><td>[-1.655, 1.655]</td><td>0.041</td><td>80.252</td><td>0.844</td><td>[-1.709, 1.601]</td><td>0.041</td></tr><tr><td>250</td><td>80.295</td><td>11.919</td><td>0.754</td><td>[-1.475, 1.475]</td><td>0.037</td><td>80.296</td><td>0.754</td><td>[-1.521, 1.431]</td><td>0.037</td></tr><tr><td>300</td><td>80.28</td><td>11.924</td><td>0.688</td><td>[-1.35, 1.35]</td><td>0.034</td><td>80.281</td><td>0.688</td><td>[-1.386, 1.311]</td><td>0.034</td></tr><tr><td>334</td><td>80.265</td><td>11.929</td><td>0.653</td><td>[-1.28, 1.28]</td><td>0.032</td><td>80.264</td><td>0.653</td><td>[-1.313, 1.245]</td><td>0.032</td></tr></tbody></table>

<table><caption>Table .7: Brain Tumor dataset, 2D U-net, Dice Accuracy. Results on subsamples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>77.811</td><td>11.799</td><td>3.731</td><td>[-7.315, 7.315]</td><td>0.188</td><td>77.815</td><td>3.731</td><td>[-7.856, 6.566]</td><td>0.185</td></tr><tr><td>20</td><td>77.334</td><td>12.631</td><td>2.824</td><td>[-5.535, 5.535]</td><td>0.143</td><td>77.334</td><td>2.825</td><td>[-5.879, 5.137]</td><td>0.142</td></tr><tr><td>30</td><td>77.101</td><td>13.084</td><td>2.389</td><td>[-4.68, 4.68]</td><td>0.121</td><td>77.101</td><td>2.389</td><td>[-4.94, 4.389]</td><td>0.121</td></tr><tr><td>50</td><td>77.607</td><td>12.897</td><td>1.824</td><td>[-3.575, 3.575]</td><td>0.092</td><td>77.606</td><td>1.823</td><td>[-3.736, 3.39]</td><td>0.092</td></tr><tr><td>100</td><td>77.564</td><td>12.797</td><td>1.28</td><td>[-2.51, 2.51]</td><td>0.065</td><td>77.564</td><td>1.279</td><td>[-2.588, 2.42]</td><td>0.065</td></tr><tr><td>150</td><td>77.583</td><td>13.01</td><td>1.062</td><td>[-2.08, 2.08]</td><td>0.054</td><td>77.582</td><td>1.063</td><td>[-2.139, 2.023]</td><td>0.054</td></tr><tr><td>200</td><td>77.468</td><td>13.182</td><td>0.932</td><td>[-1.825, 1.825]</td><td>0.047</td><td>77.467</td><td>0.932</td><td>[-1.871, 1.778]</td><td>0.047</td></tr><tr><td>250</td><td>77.549</td><td>13.056</td><td>0.826</td><td>[-1.62, 1.62]</td><td>0.042</td><td>77.549</td><td>0.826</td><td>[-1.654, 1.582]</td><td>0.042</td></tr><tr><td>300</td><td>77.501</td><td>13.104</td><td>0.757</td><td>[-1.485, 1.485]</td><td>0.038</td><td>77.502</td><td>0.756</td><td>[-1.511, 1.448]</td><td>0.038</td></tr><tr><td>334</td><td>77.489</td><td>13.115</td><td>0.718</td><td>[-1.405, 1.405]</td><td>0.036</td><td>77.488</td><td>0.717</td><td>[-1.43, 1.376]</td><td>0.036</td></tr></tbody></table>
<table><caption>Table .8: Hippocampus dataset, 3D U-net, Hausdorff Distance. Results on sub-samples of size *k* ≤ 110. We show the mean over all the sub-samples *S*<sub>*k*,*j*</sub> of a given size *k* (*k* is fixed and *j* ∈ {1, ..., 100}).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>1.214</td><td>0.416</td><td>0.132</td><td>[-0.26, 0.26]</td><td>0.428</td><td>1.214</td><td>0.131</td><td>[-0.205, 0.286]</td><td>0.404</td></tr><tr><td>20</td><td>1.206</td><td>0.435</td><td>0.097</td><td>[-0.19, 0.19]</td><td>0.315</td><td>1.206</td><td>0.097</td><td>[-0.165, 0.21]</td><td>0.310</td></tr><tr><td>30</td><td>1.221</td><td>0.468</td><td>0.086</td><td>[-0.17, 0.17]</td><td>0.278</td><td>1.22</td><td>0.086</td><td>[-0.149, 0.183]</td><td>0.271</td></tr><tr><td>50</td><td>1.203</td><td>0.46</td><td>0.065</td><td>[-0.125, 0.125]</td><td>0.208</td><td>1.203</td><td>0.065</td><td>[-0.116, 0.137]</td><td>0.211</td></tr><tr><td>100</td><td>1.205</td><td>0.47</td><td>0.047</td><td>[-0.09, 0.09]</td><td>0.149</td><td>1.205</td><td>0.047</td><td>[-0.086, 0.097]</td><td>0.152</td></tr><tr><td>110</td><td>1.205</td><td>0.47</td><td>0.045</td><td>[-0.09, 0.09]</td><td>0.149</td><td>1.205</td><td>0.045</td><td>[-0.08, 0.09]</td><td>0.149</td></tr></tbody></table>

<table><caption>Table .9: Hippocampus dataset, 2D U-net, Hausdorff Distance. Results on subsamples of size *k* ≤ 110. We show the mean over all the sub-samples *S*<sub>*k*,*j*</sub> of a given size *k* (*k* is fixed and *j* ∈ {1, ..., 100}).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>1.296</td><td>0.538</td><td>0.17</td><td>[-0.335, 0.335]</td><td>0.517</td><td>1.296</td><td>0.17</td><td>[-0.267, 0.367]</td><td>0.490</td></tr><tr><td>20</td><td>1.302</td><td>0.599</td><td>0.134</td><td>[-0.265, 0.265]</td><td>0.407</td><td>1.302</td><td>0.134</td><td>[-0.219, 0.289]</td><td>0.391</td></tr><tr><td>30</td><td>1.331</td><td>0.718</td><td>0.131</td><td>[-0.255, 0.255]</td><td>0.383</td><td>1.331</td><td>0.131</td><td>[-0.211, 0.288]</td><td>0.375</td></tr><tr><td>50</td><td>1.31</td><td>0.729</td><td>0.103</td><td>[-0.2, 0.2]</td><td>0.305</td><td>1.31</td><td>0.103</td><td>[-0.168, 0.228]</td><td>0.303</td></tr><tr><td>100</td><td>1.31</td><td>0.793</td><td>0.079</td><td>[-0.155, 0.155]</td><td>0.237</td><td>1.31</td><td>0.079</td><td>[-0.133, 0.175]</td><td>0.234</td></tr><tr><td>110</td><td>1.311</td><td>0.803</td><td>0.077</td><td>[-0.15, 0.15]</td><td>0.229</td><td>1.311</td><td>0.077</td><td>[-0.13, 0.17]</td><td>0.229</td></tr></tbody></table>

<table><caption>Table .10: Brain Tumor dataset, 3D U-net, Hausdorff Distance. Results on subsamples of size *k* ≤ 334. We show the mean over all the sub-samples *S*<sub>*k*,*j*</sub> of a given size *k* (*k* is fixed and *j* ∈ {1, ..., 100}).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>8.171</td><td>8.806</td><td>2.785</td><td>[-5.46, 5.46]</td><td>1.336</td><td>8.168</td><td>2.784</td><td>[-4.278, 6.046]</td><td>1.264</td></tr><tr><td>20</td><td>7.635</td><td>9.098</td><td>2.034</td><td>[-3.985, 3.985]</td><td>1.044</td><td>7.634</td><td>2.034</td><td>[-3.268, 4.459]</td><td>1.012</td></tr><tr><td>30</td><td>7.95</td><td>9.782</td><td>1.786</td><td>[-3.5, 3.5]</td><td>0.881</td><td>7.95</td><td>1.787</td><td>[-2.98, 3.909]</td><td>0.866</td></tr><tr><td>50</td><td>7.563</td><td>9.791</td><td>1.385</td><td>[-2.715, 2.715]</td><td>0.718</td><td>7.563</td><td>1.383</td><td>[-2.36, 3.002]</td><td>0.709</td></tr><tr><td>100</td><td>7.702</td><td>10.468</td><td>1.047</td><td>[-2.05, 2.05]</td><td>0.532</td><td>7.702</td><td>1.046</td><td>[-1.848, 2.228]</td><td>0.529</td></tr><tr><td>150</td><td>7.691</td><td>10.339</td><td>0.844</td><td>[-1.655, 1.655]</td><td>0.430</td><td>7.693</td><td>0.844</td><td>[-1.522, 1.775]</td><td>0.428</td></tr><tr><td>200</td><td>7.698</td><td>10.413</td><td>0.736</td><td>[-1.445, 1.445]</td><td>0.375</td><td>7.697</td><td>0.736</td><td>[-1.341, 1.535]</td><td>0.374</td></tr><tr><td>250</td><td>7.758</td><td>10.663</td><td>0.674</td><td>[-1.32, 1.32]</td><td>0.340</td><td>7.757</td><td>0.674</td><td>[-1.238, 1.397]</td><td>0.340</td></tr><tr><td>300</td><td>7.695</td><td>10.513</td><td>0.607</td><td>[-1.19, 1.19]</td><td>0.309</td><td>7.694</td><td>0.607</td><td>[-1.12, 1.257]</td><td>0.309</td></tr><tr><td>334</td><td>7.726</td><td>10.618</td><td>0.581</td><td>[-1.14, 1.14]</td><td>0.295</td><td>7.726</td><td>0.581</td><td>[-1.076, 1.196]</td><td>0.294</td></tr></tbody></table>
<table><caption>Table .11: Brain Tumor dataset, 2D U-net, Hausdorff Distance. Results on sub-samples of size $k \le 334$. We show the mean over all the sub-samples $S_{k,j}$ of a given size $k$ ($k$ is fixed and $j \in \{1, \dots, 100\}$).</caption><thead><tr><th rowspan="2">Subsample size k</th><th colspan="5">Parametric</th><th colspan="4">Bootstrap</th></tr><tr><th>μ<sub>k</sub></th><th>σ<sub>k</sub></th><th>SEM<sub>k</sub></th><th>CI<sub>k</sub></th><th>ν<sub>k</sub></th><th>μ<sup>*</sup><sub>k</sub></th><th>SEM<sup>*</sup><sub>k</sub></th><th>CI<sup>*</sup><sub>k</sub></th><th>ν<sup>*</sup><sub>k</sub></th></tr></thead><tbody><tr><td>10</td><td>9.544</td><td>10.409</td><td>3.292</td><td>[-6.45, 6.45]</td><td>1.352</td><td>9.544 ± 3.441</td><td>3.294</td><td>[-5.099, 7.152]</td><td>1.284</td></tr><tr><td>20</td><td>8.454</td><td>9.178</td><td>2.052</td><td>[-4.02, 4.02]</td><td>0.951</td><td>8.453 ± 2.302</td><td>2.05</td><td>[-3.454, 4.441]</td><td>0.934</td></tr><tr><td>30</td><td>9.204</td><td>10.794</td><td>1.971</td><td>[-3.86, 3.86]</td><td>0.839</td><td>9.205 ± 1.786</td><td>1.973</td><td>[-3.374, 4.269]</td><td>0.830</td></tr><tr><td>50</td><td>8.836</td><td>10.86</td><td>1.536</td><td>[-3.01, 3.01]</td><td>0.681</td><td>8.837 ± 1.359</td><td>1.537</td><td>[-2.696, 3.282]</td><td>0.676</td></tr><tr><td>100</td><td>8.758</td><td>10.966</td><td>1.097</td><td>[-2.15, 2.15]</td><td>0.491</td><td>8.756 ± 0.938</td><td>1.096</td><td>[-1.974, 2.307]</td><td>0.489</td></tr><tr><td>150</td><td>8.886</td><td>11.158</td><td>0.911</td><td>[-1.785, 1.785]</td><td>0.402</td><td>8.885 ± 0.673</td><td>0.912</td><td>[-1.671, 1.894]</td><td>0.401</td></tr><tr><td>200</td><td>8.806</td><td>11.068</td><td>0.783</td><td>[-1.535, 1.535]</td><td>0.349</td><td>8.806 ± 0.471</td><td>0.783</td><td>[-1.445, 1.617]</td><td>0.348</td></tr><tr><td>250</td><td>8.873</td><td>11.32</td><td>0.716</td><td>[-1.405, 1.405]</td><td>0.317</td><td>8.873 ± 0.357</td><td>0.715</td><td>[-1.33, 1.469]</td><td>0.315</td></tr><tr><td>300</td><td>8.83</td><td>11.173</td><td>0.645</td><td>[-1.265, 1.265]</td><td>0.287</td><td>8.831 ± 0.214</td><td>0.645</td><td>[-1.204, 1.32]</td><td>0.286</td></tr><tr><td>334</td><td>8.855</td><td>11.245</td><td>0.615</td><td>[-1.205, 1.205]</td><td>0.272</td><td>8.856</td><td>0.615</td><td>[-1.154, 1.257]</td><td>0.272</td></tr></tbody></table>
References

Altman, D.G., Royston, P., 2006. The cost of dichotomising continuous variables. BMJ 332, 1080.

Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., et al., 2022. The medical segmentation decathlon. Nature Communications 13, 4128.

Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., Nichyporuk, B., Szeto, J., Mohammadi Sepahvand, N., Raff, E., Madan, K., Voleti, V., et al., 2021. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems 3, 747-769.

Christodoulou, E., Reinke, A., Houhou, R., Kalinowski, P., Erkan, S., Sudre, C.H., Burgos, N., Boutaj, S., Loizillon, S., Solal, M., Rieke, N., Cheplygina, V., Antonelli, M., Mayer, L.D., Tizabi, M.D., Cardoso, M.J., Simpson, A., Jäger, P.F., Kopp-Schneider, A., Varoquaux, G., Colliot, O., Maier-Hein, L., 2024. Confidence intervals uncov- ered: Are we ready for real-world medical imaging AI?, in: proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, Springer Nature Switzerland.

Colliot, O., Thibeau-Sutre, E., Burgos, N., 2022. Repro- ducibility in machine learning for medical imaging. arXiv preprint arXiv:2209.05097.

Efron, B., Tibshirani, R.J., 1993. An introduction to the boot- strap. CRC press.

El Jurdi, R., Colliot, O., 2023. How precise are performance estimates for typical medical image segmentation tasks?, in: Proc. IEEE International Symposium on Biomedical Imaging (ISBI 2023).

Hesamian, M.H., Jia, W., He, X., Kennedy, P., 2019. Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging 32, 582-596.

Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H., 2021. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Na-ture Methods 18, 203-211.

Koch, L.M., Baumgartner, C.F., Berens, P., 2024. Distribu-tion shift detection for the postmarket surveillance of med-ical ai algorithms: a retrospective simulation study. npj Digital Medicine 7, 120.

Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A.P., Carass, A., et al., 2018. Why rankings of biomedical image analysis competitions should be inter- preted with care. Nature communications 9, 5217.

Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M.D., Buet- tner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al., 2024. Metrics reloaded: recommendations for image analysis validation. Nature methods 21, 195-212.

McDermott, M., Wang, S., Marinsek, N., Ranganath, R., Ghassemi, M., Foschini, L., 2019. Reproducibility in ma-chine learning for health. arXiv preprint arXiv:1907.01463 .

Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Fara-hani, K., et al, 2015. The multimodal Brain Tumor Image Segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging 34, 1993-2024. doi:10.1109/TMI. 2014.2377694.

Platt, R.W., Hanley, J.A., Yang, H., 2000. Bootstrap confi-dence intervals for the sensitivity of a quantitative diagnos-tic test. Statistics in medicine 19, 313-322.

Reinke, A., Christodoulou, E., Glocker, B., Scholz, P., Isensee, F., et al, 2022. Metrics reloaded - a new recommen-dation framework for biomedical image analysis validation, in: Medical Imaging with Deep Learning. URL: https://openreview.net/forum?id=24kBqy8rcB_.

Reinke, A., Tizabi, M.D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A.E., Rädsch, T., Sudre, C.H., Acion, L., Antonelli, M., et al., 2024. Understand-ing metric-related pitfalls in image analysis validation. Na-ture methods 21, 182-194.

Varoquaux, G., 2018. Cross-validation failure: Small sample sizes lead to large error bars. NeuroImage 180, 68-77. doi:https://doi.org/10.1016/j.neuroimage.2017.06.061.
Varoquaux, G., Cheplygina, V., 2022. Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ digital medicine 5, 48.

Varoquaux, G., Colliot, O., 2023. Evaluating machine learning models and their diagnostic value, in: Machine Learning for Brain Disorders. Springer. URL: https://hal.archives-ouvertes.fr/hal-03682454.