On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice
Authors: Marcus Nordstrom
Henrik Hult
Fredrik Löfman
Jonas Söderberg
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image Segmentation
Computer Vision

Abstract: We study two of the most popular performance metrics in medical image segmentation, Accuracy and Dice, when the target labels are noisy. For both metrics, several statements related to characterization and volume properties of the set of optimal segmentations are proved, and associated experiments are provided. Our main insights are: (i) the volume of the solutions to both metrics may deviate significantly from the expected volume of the target, (ii) the volume of a solution to Accuracy is always less than or equal to the volume of a solution to Dice and (iii) the optimal solutions to both of these metrics coincide when the set of feasible segmentations is constrained to the set of segmentations with the volume equal to the expected volume of the target.

On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice
Marcus Nordström
Henrik Hult Department of Mathematics
Department of Mathematics
KTH Royal Institute of Technology Stockholm, Sweden mar barcno@kth.es
KTT Royal Institute of Technology Stockholm, Sweden hult@kth.
Jonas Söderberg Department Machine Learning
Fredrik Löfman
Department of Machine Learning RaySearch Laboratories
RaySearch Laboratories
Stockholm, Sweden
Stockholm, Sweden fredrik. hww..aaaaaaarayyaraabbbb.ooo
jonas www.sssdddrbrrrggggssrrcaaaa.a.a
Abstract
We study two of the most popular performance metrics in medical image segmen- tation, Accuracy and Dice, when the target labels are noisy. For both metrics, several statements related characterization and volume properties of the of optimal segmentations are proved, and associated experiments are provided. Our main insights are: (i) the volume of the solutions
both metries may deviate significantly from the expected volume of the target, (ii) the volume of a solution Accuracy is always less than equal to the volume of solution Dice and (iii) the optimal solutions to both of these metrics coincide when the set of feasible segmentations is constrained to the set of segmentations with the volume equal the expected volume of the target.
Introduction
One of the most central problems in medical image analysis is to identify the region of an image associated with a certain target structure. This problem, referred to as image segmentation or delineation, is often very time consuming to solve manually. Consequently, there is great interest in the development of methods that can assist in automation of the procedure. Since 2015, it has become increasingly popular to address the segmentation problem using machine learning based methods, and in particular, fully convolutions neural networks with U-net architecture. Such methods commonly dominate the winning submissions to segmentation contests and are backed by a large
base of supporting literature 66, 13
19
28
32).
Despite the success, performance of these methods will. like any machine learning method, depend on the quality of the available data 4 37 Since it is well known that the data commonly used in practice produced by medical practitioners that delineate structures an inconsistent manner (25]. it is important to understand the impact of label noise. One way to study the influence of label noise is to consider how the noise impacts segmentations that are theoretically optimal with respect to the metric used for measuring performance. Even if theoretically optimal solutions may not be
Author also affiliated with RaySearch Laboratories.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

attainable in practice, studying them gives important insights into what the effect label noise has on the particular metric and any training method that is designed maximize it.
the fraction of the image that is correctly delineated. This metric is most commonly targeted by taking a 1 1/2-threshold of predictions from a model trained with the cross-entropy loss [24]. However, since Accuracy may not reflect the desired behaviour when the data is unbalanced, that is, when the target structure is much smaller than the background, alternative metrics are often preferred. The most popular such alternative is the Sørensen- Dice coefficient, or Dice for short and is related to the Fi-metric used in binary classification. This metric is most commonly targeted by taking a 1/ /22tthreshold of predictions from a model trained with the soft-Dice loss, a smoothed version of the Dice metric [24). Other examples considered in the literature include the Jaccard index and variations of the Haussdorff metric
Arguably, the most simple and classical choice of metric is Accuracy:
In this work, we conduct a theoretical investigation of the effect label noise has on optimal segmenta- tions with respect to the performance metries Accuracy and Dice. Because the volume of proposed segmentation may be used for important properties such as estimating the size of a tumor [9]. we pay special attention the effect noise may have on the volume of the optimal segmentations.
Contributions: A characterization of all optimal solutions to Accuracy and Dice when the target is noisy is provided. This characterization is used to analyze the volume of the optimal solutions and we prove: (i) sharp upper and lower bounds on the volume of optimal segmentations with respect to Accuracy and Dice, (ii) that the volume of the optimal solutions Accuracy always is less or equal to the volume of the optimal solutions to Dice and (iii) that the optimal solutions to both metrics coincide when the volume is held fix. We also show the relevance of the problem in a practical setting by including experiments on data from the Gold Atlas project (25) and The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI)
2 Related work
Deep learning methods are playing an increasingly vital part in the development of medical image analysis. However, deep learning models require large annotated data sets for successful training that rarely are available in the clinics. Even if some data sets are being curated for training deep learning models, it is generally difficult and expensive to accurately annotate large collections of medical images. Moreover, training data may include corrupted or noisy labels. This particularly the case in image segmentation where different annotators may have different views on the correct delineation of a region of interest, leading to uncertainty about the true label, see e.g. 23]. Noisy labels may also appear due to automated systems or non-expert systems being used to annotate large volumes of data, see 55
There is large body of literature on the impact of label noise in image segmentation, see 34 and [4] for recent reviews. The proposed solutions limit the loss of performance when the labels are noisy include label cleaning and pre-processing, e.g. [T0], modification of network architechtures, c.g. robustification of loss functions, e.g. 211, reweighting of training data, see 22 39,, and many others. These approaches are of practical nature and generally address methodology that improve the
36
performance on some chosen noisy data set. On the contrary, the literature that address the effect of label noise from a theoretical point of view in the context of image segmentation is rather limited. It was shown that the loss function soft-Dice, in contrast to the loss function cross-entropy, does not yield optimal predictions that coincide with the pixel-wise marginals and that the associated volume is biased [2]. This motivated methods for post-calibrating uncalibrated marginal estimates [2] and more general investigation of the relationship between volume and marginal calibration 226. Finally, an alternative volume preserving segmentation method based on optimal transport theory
was proposed and investigated in 220).
Another domain of related work can be found in the binary classification literature. The connection between the study of solutions to Accuracy and Dice in segmentation and the study of Accuracy and F, in binary classification was investigated classifiers to Accuracy and F,. That is, classifiers that are obtained by processing the posterior class probabilities or estimates thereof using a threshold. For Accuracy, this relates to the classical Bayes classifier which has been studied since the origin of the field [35]. Early work on the existence of
[24]. Of importance to us are optimal plug-in

---

such a classifier for F1, and the fact that this threshold is lower than the threshold for Accuracy can be traced 38] Further work showed this threshold to be equal to half of the maximal attainable Fi-score 18 Lots of extensions of these works have been proposed, but to the best of our knowledge no such extension is the direction of our work.
Preliminaries
In work we find that it is convenient do the theoretical analysis over continuous domain and where all encountered continuous spaces are equipped with their associated Borel a-fields. Formally, let Ω = [0, 1] C R" be the unit cube of dimension n 2 and À be the associated standard normalized Lebesgue measure such that X(92) = 1. The space of segmentations is denoted by 8 and is formally given by the space of measurable functions from S? the binary numbers {0, 1). For any segmentation 8 E S, s(w) = 1 implies that the object of interest occupies the site w E Q. In numerical setting, discretization of the domain often represented by a three dimensional voxel grid of an approximate order of 512 x 512 x 100. In such situations the space of segmentations are given by all possible binary functions defined on this voxel grid. Note that any segmentation on discretized domain can be incorporated in our continuous framework by using appropriate step functions. Details on this can found in the end of Section 4.
commonly used. For instance, 3D CT scans are
Beyond the space of segmentations, several other technical constructions are introduced. This includes the space of measurable functions from Q to [0, 1] which we denote by M and refer to the marginal functions. We also let || = Sg [Xdw)) and f 1 f. where f is any measurable function defined Ω. Throughout are equal if they are equal X-a.e. We will use f(-} to denote the identity function, and when F is cumulative distribution function, we will denote the left limit F(t-) lim Fs)). Finally, for given volume E [0, 1]. we let S. = {s S ||s||| = 2 be the set of segmentations with volume v.
adopt the convention that two X-measurable functions f.g
Classically, metrics in medical image segmentation deterministic segmentations 33 When noise is present, the label becomes a random variable and the metries need to be extended to functional over one deterministic segmentation and one random label segmentation. In this work, the soft labeling convention for this extension is adopted 31, Also, since do our analysis with respect measurable functions on continuous domain instead of functions on a finite index set, we replace the sums usually used with integrals. This, however, does not change the intuition of the metries, and the definition using sums can be seen a special case.
defined per image as functionals over two
I1 15 16
Definition 1. For any m E M, Accuracy is given by
Am As)= = [s(W))(w) (W))(()))ddw)),, s sES
Definition 2. For any m M. Dice is given by
(s( = S s(w)) (ww)m(())ddw)
S
(2)
| + ml/
For noisy segmentation L, that is, a random variable taking values in S, can be taken to be the exact marginal success probability m(w) = B[[(w),, w Q. Such marginal functions are important in theory but can rarcly be obtained in practice. Alternative choices of marginal functions include finite sample approximations, that is, point-wise averages over finite observations of L, 16 BI] These choices of m important because they are sometimes used for training machine learning models. Finally, note that (s) (s)] and (s)) is common alternative way of specifying the metric. For Dice, this sort of relationship does not hold in general (EL (s) + E(Dr(ss). However, it does hold that (s) = (+)) when the volume of the noisy labels is constant Var(I] [[[1]] O, and it holds approximately (s) Rs [s)r when the variance of the volume of the noisy labels is small Var ~ 0, which is often the case in medical image segmentation applications. Examples of observations of particular L for couple of different target structures are depicted in Figure T
and estimates of E[L] according to single annotator [T]] 15
Because of the fact that [[m]]a = [[][][[]] when m taken to the exact marginal success probability, ||m]] plays an important role in the medical segmentation context, either theoretically as
3

---

the expected volume of the target as approximation thereof. Understanding how ||ma]] relates |||| i. where s an optimal segmentation to Accuracy or Dice will be central our work. To the best of our knowledge, this has not been studied in prior work.
Figure 1: To the left is Urinary bladder and to the right is Neurovascular bundles for one patient in the Gold Atlas data [25]. Each line is associated with the boundary of a segmentation produced by particular annotator. The screenshots are taken with RayStation 12A (RaySearch Laboratories AB, Stockholm Sweden).
Main results
The objective of our analysis is to characterize the optimizers to A, and Dm and give detailed description of the volume of the optimal segmentations. That is, for given m E M, identify properties (e.g. volume) of the optimal segmentation s E S that maximimize Accuracy or Dice. To
this end, consider the probability measure on [0, 1] given by the push-forward measure À m (() and let Fm. denote its cumulative distribution function,
(t) Ao m -(0, t)) =
stAA(ww), E [0, 1].
(3)
The function (1) may be interpreted the volume of the of sites with non-success probability less than or equal to t. In other words, the volume of the sub-level set of m at level t.
Since F.. is the cumulative distribution function of probability distribution [o, 1]. it has several well-known properties making it casy to work with, c.g., Fm is non-decreasing and right-continuone with F., (1) 1. Of particular interest to is that it has generalized inverse given by
(v) = inf(l: (!) 2 v). E 10,
which can be interpreted as the minimum level at which the volume of the corresponding sub-level set of m, is at least This function is often referred to as the quantile function and also has several well known properties; is non-decreasing and left-continuous. Moreover, it allows us to define the following important class of segmentations for given E M.
((w))( < Fm (v)) =0, 0,
Sm =
s E S.
E [0. 1].
(5)
[ ((w)[[((w) > -- Fm (U) = O,
The described class Sm,y v. is informally the set of segmentations with volume that assigns to sites where m(w) is large. Ift F,,.! (v) is continuity point of F i.e. X(w mm(w t) 0. then S m, only consist of the elements that are X-a.c. equal the segmentation s(w) (m)(w) > 1 - F, (v)).

---

mi
0.5
112
0.5
m2
0.5
0
0
0
0.5
0.5
0.5
W
W
1
1
Fm it)
Fim (t)
F (t)
0.5
0.5
0.5
0
0
0
0.5
0.5
0
0.5
1
(2)
Fm (1)
0.5
ma
0.5
0.5
0
0
0
0.5
0.5
0.5
V
v
Figure 2: Three examples of marginal functions mi, m2, my on a one-dimensional domain w E
Ω [0, 1] together with the associated Fm, Fm, and F= F.m. The mi occurs when there is no noise. The case m2 can for instance occur when the segmentation is given by 7(0.2 S v 0.6) with probability 0.5 and / (0.44 < 0.8) with probability 0.5. The ms can occur when the edges of the segmentation are randomly perturbed by some continuous random variable.
A lot of our analysis can be simplified if Fn is assumed to be invertible almost everywhere. However, this would require m to not have any non-neglible constant regions, which for instance excludes any m that is given by an empirical approximations using finite number of samples. Consequently, in the sequel, we treat general F..- Our first result contains the essential ingredients for characterizing the optimizers Accuracy and Dice.
Lemma 1, For any m E .M and E [0,1]
sup / sES.
s(w)m(w))((w) (1-FFF (u))
(6)
and the elements where the supremum is attained is given by Sm,y-
A complete proof is given in the Supplementary Document and outlined as follows. The first part shows that the class Sm,v is the class of optimal solutions by showing that for any Sm, and E S, Sm,,. (a (w) s(w))m(() (((w))(dw)) > 0. The second part proves the equality (6) using an application of the quantile transform. That is, if U has uniform distribution [0, 1] then F (U) has edf given by Fm, and f/ t.m,, (dt) (0(+,,;(()))
So F (u)du.
Lemma I allows us to reduce the constrained optimization problem over the rather complicated space one-dimensional integral with respect the quantile function. It is the starting point for our analysis.
of segmentations,
In the remaining section our main theoretical results are presented. In Theorem we provide characterization of all of the optimal solutions Accuracy based on volume. addition, sharp upper and lower bounds the volume of the associated optimal segmentations are provided.
5

---

Theorem 1. For any m E M. the class of maximizers to Am is given by UsEVam Sm, where
VAm (1m22/))), (m)
Moreover, Vio satisfies the following bounds
Van S [[max(2]]m]]]] 1, 03, min (2|mmll 1}},
where the bounds are sharp in the sense that there for any [0, 1] exist mo. mi M such that [mol]] = || 1/mil11-0 = v and
inf yAmo max (2|lmoll 1,03, sup VAm, min (2) mm [l, 1}.
The complete proof is given in the Supplementary Document and outlined as follows. First, the function
Om (v) = u || 1
-2/ |
(m)(u)du, VE [0, !],
(10)
is introduced and then Lemma is used to show that sup.es (m(s) = suP am (v). Conse- quently, the class of optimal solutions Am is given by UNeVAm Sm... where VA is the set of optimizers to Om- The rest of the proof consists of detailed analysis of am and is composed of three parts. The first part is show (7) by finding one optimal solution and then identifying all volumes that yield the same optimal value. The second part is to provide the lower and upper bounds on the elements of yA in terms of ||m]|] given by The third part to provide examples of situations where the extreme cases occur In Figure upper sense illustrated.
a case that is extreme both in the lower and in the
0.5
0.5
0.5
0.5
1
Figure 3: To the left is a particular quantile function F= and to the right is the associated function (t)) E [0, 1] with ||>=|] = 0.4
am given by (TO). Here (1) = 1 (0.2]] (v) +
0), min min(2llm (2][m]]]. 13] and consequently is an extreme case to in both the lower sense and the upper sense.
that satisfies AAm. 10, 2||mm/|/] [max m2]
In Theorem 2 we provide a characterization of all of the optimal solutions to Dice based on volume. In addition, sharp upper and lower bounds on the volume of the associated optimal segmentations provided.
Theorem 2. For any m E M. the class of maximizers Dm given by Uneyom where
yDm = (F sup D. ()) (m. sup Dm (s)22) sES
(11)
sES
Moreover, DD satisfies the following bounds
Dm C [[]m 1],
(12)
where the bounds are sharp in the sense that there for any E (0, 1] exist mo, E M such that [mollla || [l
and
inf D mm] sup Din = 1.
(13)
The complete proof is given in the Supplementary Document and outlined as follows. First, the function
2 Fr-1(()) (w)) dm (v)
10.
(14)
|m||| ||||| +

---

is introduced and then Lemma is used to show that supses Dm (s) = (0 dm)()). Conse- quently, the class of optimal solutions to Dm is given by UvE Sm... where y:Dm. is the set of optimizers to dm- The remaining proof consists of detailed analysis of dm and composed of three parts. The first part is to show
(TT) which is derived by careful investigation of the properties of the
function ((v) = [I]ml] ++)* (,dm(w)) which has the same sign d. (v) and therefore can be used identify optimal values of dm. The second part is to provide the lower and upper bounds on the elements of yD in terms of |m]]| given by (12). The third part is to provide examples of situations
where the extreme values occur In Figure case that is extreme both in the lower and the
upper sense is illustrated.
0.5
0.5
0.5
0
0.5
1
V
V
Figure 4: To the left is a particular quantile function F and to the right is the associated function
dm given by (14). Here F (v) (1 []m]]])(1 [ - m11.. (v) with ||/m]] = 0.4 that
satisfies DD [|]m|]] 1] and consequently is an extreme case to (13) in both the lower sense and
the upper sense.
In Theorem 3] we relate the volume of the optimal segmentations of Accuracy and the optimal segmentations of Dice for a given marginal probability m ... Theorem 3. For any m E M, VÅ given by and Dm given by satisfy
sup Am inf D...
(15)
The complete proof is given in the Supplementary Document and outlined follows. First note that (s) I for any E S and then consider separately the cases when (s) < for all E S and when there exist some that supses (s) < 1/2 which implies that Fm (1/2) I Fm - supses D (m, (s)s2)-) For the second case, we show that the volume of the optimizers are uniquely given by D ydm (1/2)) In either case, (I5) holds.
S such that (s) 1. For the first case, it is obvious
In Theorem 4 the set of optimal solutions to Accuracy and Dice when constrained specific volume is shown to coincide.
Theorem 4. For any m E M and v E [0, 1] the maximizers to the problems,
Am(s) sES,
sup
and sup Dm sES,
(16)
coincide and are given by Sm,
The complete proof is given in the Supplementary Document and a straightforward application of Lemma] [ Of particular interest is the when ||m]| since this correspond to the situation when the metrics are maximized under the constraint that there should be no volume bias.
It follows from Theorem and Theorem 2 that the optimizers to both Accuracy and Dice are of the form UveySm,rs where V = [F (t-), Fi (t) for some E [0. 1]. This type of charecterization is practical for proving properties on volume, but inconvenient for other tasks. In Theorem 5 we provide an alternative charecterization using threshold segmentations of the form s(w) = / (mi(w) > a. or s(w) = 7(m(w) 2 a., for some Even if there exist optimal segmentations that are not necessarily of threshold type, they can always bounded, above and below, by optimal segmentations of threshold type.
7

---

Theorem 5. For any m E M and E (0. 1, let $1 (w) [m)(w) 2 1--1 and su(w) = [mm(w)
>
1 t}. Then, ||sol| F.. (t-), || Fm (t)
and
s E (F. (t-), (+) Sm.e
(w( S s(w) S $1 (ww), -- a
(17)
The complete proof is given in the Supplementary Document and outlined as follows. Note that ||sul]a = So / (m((w) < [dw) = (t-) and ||s1|| = Sn I [m((w)= I {))(dw) = Fm (t). Now, take each direction of (17) separately. For the = part, we first show the upper bound s(w) - 81 (w)), a.e. and then show the lower bound so(w) I s(w), X-a.e. For the upper bound, with A = {w
s(w) > $1 [w)). we first observe that 1 [(w E A} = s(w) (w) and then, using the definition of Sm,e we prove that
X(A) 1 s(w)s - J ((w)I(m(w>> > (o))((())) O.
(18)
The lower bound is similar, but slightly more involved. For the = part, we first note that the (t-) ||so||a S ||>||| S ||sı|| (t), and then show that for = || E Sm,.-
In numerical applications, the continuum 9 is usually partitioned into finite collection of voxels [St)aez. Marginal functions are then constrained to the subset of .M that is compatible with the voxelization in the sense that m is measurable with respect to the a-field generated by the partition. t} for some E (0, 1]. then I(m(w) 2 1 - t) and su(w) = [m((w) [Theorem 2 and Theorem [S] the also Su and 81 are compatible with the voxelization. By Theorem [ segmentations with least and greatest volume that are optimal with respect to Accuracy (Dice) are compatible with the voxelization. For the metrics respectively, we denote the segmentations with the greatest volume by:
Note that if s1 (w)
8 Am (w) = [mm(w) 2 1/2], E Ω
(19)
D (w) = [m((w) sup Dm (5)/2}, WEΩ. seS
(20)
Note that Am is analogous to the Bayes classifier and s Dm is analogous to the threshold classifier described in [18]. Both of these are trivial to compute from given marginal function m compatible to some voxelization and code for doing so is available in the Supplementary Material.
Experiments
The sharp bounds on volume in Thereom D and Theorem 2 iimplies that there exist marginal functions for which the volume of the optimal segmentations to Accuracy and Dice deviate significantly from the expected target volume. In this section we conduct experiments on marginal functions formed from real world data to compare the volume of optimal segmentations to the expected volume in practice.
For our experiments we investigate two data sets. The first data set (G) contains segmentations in the pelvic area and is part of the Gold Atlas project (25). The data is in 3D with a resolution of 512 x 512 pixels per slice and consist of 19 patients with different ROI's (region of interest), each of which have been delineated by 5 experts (see Figure [ for an illustration of the segmentations associated with two different ROl's for one patient). The second data set (L) contains segmentations in the thorax area and is part of The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) ① and is hosted by TCIA 7 The data in 3D with a resolution of 512 x 512 pixels per slice and contains 1018 cases with lung nodules delineated by 4 experts. For each data set, ROI and patient, a marginal function m is formed by taking the fraction of which each pixel has been selected by the annotators, that is, finite sample approximation is considered. The resulting marginal functions are then used to compute the segmentations AA... (19) and g P 20 For (G) we make use of the software Plastimatch (30] and for (L) we make use of the python package pylidc [12]. Details on the experiments can be found in the Supplementary Document. Code and instructions how to reproduce the experiments
be found at https
//github. gggithub.coommmacccu nwwdstrooddooooo http://ttmmmllllee solutioos -tre--tt-tccccrgooo
accuracy and- dice
From our experiments we report the quantities ||اÅm ||1 /|mm|| and P || //]m||| i, which in a relative sense describe how much the volume of the computed optimal segmentations with respect to Accuracy and Dice deviate from the expected target volume. In Figure S and Figure 0 these quantities are
8

---

Urinary bladder
Rectum
Anal canal
1.6
1.6
1.6
[|sD ||2/// ||||||
[|raa]
| Ila [mrlll
1.3
|| /11/ ////////ma
1.3
1.3
0.7
0.7
0.7
0.4
0.4
0.4
0.4
0.7
1.3
1.6
0.4
0.7
1
1.3
1.6
0.4
0.7
1
1.3
1.6
|| |1 ///ll
||
||
|| /
Penile bulb
Neurovascular bundles
Femoral head R
1.6
1.6
1.6
DD |/ //mmml
[| |a 11/////a
1.3
1.3
1.3
0.7
0.7
0.7
0.4
0.4
0.4
0.4
0.7
1.3
1.6
0.4
0.7
1.3
1.6
0.4
0.7
1
1.3
1.6
|| || [|ml|
|| A
||//||m||
||5Am
Femoral head L
Prostate
Seminal vesicles
1.6
1.6
1.6
|| |I1 ////|
D| ||r/ ||m]]
IieDa || mell
1.3
1.3
1.3
1
0.7
0.7
0.7
0.4
0.4
0.4
0.4
0.7
1.3
1.6
0.4
0.7
1
1.3
1.6
0.4
0.7
1
1.3 1.6
||sA" || /|/l]]
Am ||: /|///
||
/|//|//]|
Figure 5: Results of our experiments the pelvic data in the Gold Atlas project 25 For each ROI and patient, the associated m is formed by finite sample approximation and used to compute s ^ as defined by (19) and _.... as defined by (20)
Lung nodules
[|1 //|m||
2
2
3
4
A ||
|m/]
Figure 6: Results of experiments on the lung nodules in the LIDC-IDRI data set M For each patient, the associated m is formed by finite sample approximation and used to compute s Am as defined by (19) and P... as defined by 20

---

Table Results of our experiments on the pelvic data the Gold Atlas project (G) 225 and the lung
nodule data in LIDC-IDRI (L) [ For each ROI and patient, the associated m is formed by finite sample approximation and used to compute AAm as defined by (19) and S' as defined by (20p.
||s |I ||m|]
P| 11 /I
ROI
Mean
Std
Min
Max
Mean
Std
Min
Max
(G) Urinary bladder
1.004
0.009
0.991
1.035
1.004
0.009
0.991
1.035
(G) Rectum
0.994
0.047
0.912
1.094
0.994
0.047
0,912
1.094
(G) Anal canal
0.916
0.068
0.753
1.067
1.075
0.182
0,877
1,560
(G) Penile bulb
0.929
0.072
0.696
1.022
1.065
0.157
0.863
1.365
G) Neurovascular b.
0.778
0.110
0,461
0.928
1.267
0.115
1.070
1.481
GG) Femoral head R
0.988
0.011
0.963
1.006
0.988
0.011
0,963
1.006
G)) Femoral head L
0,994
0,022
0.958
1.070
0.994
0.022
0,958
1.070
G) Prostate
0.978
0.027
0.894
1.011
0.978
0.027
0.894
1.011
G) Seminal vesicles
0.903
0.085
0.669
1.028
1.029
0.142
0.855
1.325
(L.) Lung nodules
1.002
0.362
0.000
2.000
1.432
0.893
0.451
4.000
illustrated for each patient and ROI in seatter plots. In Table aggregated statistics of these quantities with respect to all patients are shown. By simple inspection it is clear that the volume of optimal segmentations Accuracy and Dice often deviate significantly from the expected target volume. For (G) the extreme cases are given by some marginal function m for which ||s^m /I/// s 0.5 and
some marginal function for which | m//lm 11 1.5. For (L) the extreme cases are given by
some marginal function m for which | ||/ ~ 0 and some marginal function m for which "m
||1/ [m]]] s= 4.
6 Conclusion
In this work, we have theoretically investigated the optimal segmentations with respect to the performance metrics Accuracy and Dice. We have given a detailed rigorous characterization of the optimizers and upper and lower bounds on the volume of optimal segmentations. Finally, we have shown the relevance of our theoretical observations practice by comparing the volume optimal segmentations with respect to the performance metrics to the expected volume, on two real world data sets. We conclude that noise may cause optimal segmentations to have a volume that deviates significantly from the expected target volume and that the reason for this may be what metric is chosen, or implicitly, what metric a chosen training method targets.
Broader impacts: Formalizing the evaluation process automated segmentation methods can be done in many ways, each with its pros and cons. Even if this work can be interpreted describing the problems with using Dice for this formalization, it can still paradoxically contribute to an unhealthy fixation of Dice the gold standard for segmentation evaluation in medical image analysis. This in turn can lead to that medical practitioners put too much faith in models that have been shown to perform well with respect to the metric on some test data. One solution this is make sure that clinical practitioners using such models are educated in the problems associated with the metric.
Limitations: In order for the volume bounds to be sharp in Theorem and Theorem 2 we construct extreme cases of m. These extreme cases might only be representable approximately with step functions for particular choice of voxelization. Consequently, the most extreme cases we can construct in numerical setting may not be as extreme as those we have constructed in the continuous setting. However, in medical image analysis it is common to deal with voxelizations of the order of 512 x 512 x 100 voxels which means that the approximation error would be negligible. Our work is also limited by the amount of experiments included. Additional numerical experiments wider range of data sets would give a more comprehensive picture on the impact of different noise distributions.
Acknowledgement:
The authors were supported by RaySearch Laboratories AB.
10

---

References
[1] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman, et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI); A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics, 38(2)915-931, 2011.
[2] J. Bertels, D. Robben, D. Vandermeulen, and P. Suetens. Theoretical Analysis and Experimental Validation of Volume Bias of Soft Dice Optimized Segmentation Maps the Context of Inherent Uncertainty. Medical Image Analysis, 67:101833, 2021.
[3] Bridge, A. Fielding, Rowntree, and A. Pullar. Intraobserver Variability: Should We Worry? Journal of Medical Imaging and Radiation Sciences, 47(3):217-220, 2016.
[4] W. Chi, L. Ma, J. Wu, M. Chen, W. Lu, and X. Gu. Deep Learning-Based Medical Image Segmentation With Limited Labels. Physics in Medicine & Biology, 65(23):235001, 2020.
[5] Chiaroni, M.-C. Rahal, Hueber, and F, Dufaux. Hallucinating a Cleanly Labeled Aug- mented Dataset From Noisy Labeled Dataset Using GAN. In 2019 IEEE International Conference on Image Processing (ICIP), pages 3616-3620. IEEE, 2019.
[6] Ó. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger. 3D U-Net: Learning Dense Volumetric Segmentation From Sparse Annotation. In International Conference on Medical Image Computing and Compuler-Asisted Intervention, pages 424 432. Springer, 2016.
[7] K. Clark, B. Vendt, K. Smith, J. Freymann, Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, et al. The Cancer Imaging Archive (TCIA): Maintaining and Operating Public Information Repository. Journal of Digital Imaging, (6(6))1045--055 2013.
[8] M. Drozdzal, E. Vorontsov, G. Chartrand, S. Kadoury, and C. Pal. The Importance of Skip Connections in Biomedical Image Segmentation. In Deep Learning and Data Labeling for Medical Applications, pages 179-187. Springer, 2016.
[9] H.-H. Dubben, Thames, and H.-P. Bock-Bornholdt. Tumor Volume: A Basic and Specific Response Predictor in Radiotherapy. Radiotherapy and Oncology, 47(2):167-177, 1998.
[10] B.-B. Gao, C. Xing, C.-W. Xie, J. Wu, and X. Geng. Deep Label Distribution Learning With Label Ambiguity. IEEE Transactions Image Processing, 2666):2825-28888 2017.
[11] C. Gros, A. Lemay, and J. Cohen- SoftSeg: Advantages of Soft Versus Binary Training for Image Segmentation. Medical Image Analysis, 71:102038, 2021.
[12] M. C. Hancock and J. F. Magnan. Lung nodule malignancy classification using only radiologist- quantified image features inputs to statistical learning algorithms: probing the lung image database consortium dataset with two statistical learning methods. Journal of Medical Imaging, (4):0445 2016.
[13] Z. Jiang, C. Ding, M. Liu, and D. Tao. Two-Stage Cascaded U-Net: 1st Place Solution to BraTS Challenge 2019 Segmentation Task. In International MICCAI Brainlesion Workshop, pages 231-241. Springer, 2019.
[14] D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour. Deep Learning With Noisy Labels: Exploring Techniques and Remedies in Medical Image Analysis. Medical Image Analysis, 65:101759, 2020.
[15] E. Kats, J. Goldberger, and H. Greenspan. Soft Labeling by Distilling Anatomical Knowledge for Improved MS Lesion Segmentation. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 1563-1566. IEFF, 2019.
[16] A. Lemay, C. Gros, and Cohen-Adad. Label Fusion and Training Methods for Reliable Representation of Inter- Rater Uncertainty, arXiv preprint arXiv: 2022.
11

---

[17] H. Li, D. Wei, S. Cao. K. Ma, L. Wang, and Y. Zheng. Superpixel- Guided Label Softening for Medical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 227-237. Springer, 2020.
[18] Z. C. Lipton, C. Elkan, and B. Naryanaswamy. Optimal Thresholding of Classifiers to Maximize Fl Measure. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 225-239. Springer, 2014.
[19] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez. A Survey Deep Learning in Medical Image Analysis. Medical Image Analysis, 42:60-88, 2017.
[20] J. Liu, X. Wang, and X.-c Tai. Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-Shape Priors for Image Segmentation. Journal of Mathematical Imaging and Vision, pages 1-21, 2022.
[21] D. J. Matuszewski and L.M. Sintorn. Minimal Annotation Training for Segmentation of Microscopy Images. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 387-390. IEEE, 2018.
[22] 7. Mirikharaji, Y. Yan, and G. Hamarneh. Learning to Segment Skin Lesions From Noisy Annotations. In Domain adaptation and representation transfer and medical image learning with less labels and imperfect data, pages 207-215. Springer, 2019.
[23] G. Nir, S. Hor, D. Karimi, L. Fazli, B. F. Skinnider, P. Tavassoli, D. Turbin, C. F. Villamil, G. Wang, R. S. Wilson, al. Automatic Grading of Prostate Cancer in Digitized Histopathology Images: Learning From Multiple Experts. Medical Image Analysis, 50:167-180, 2018.
[24] M. Nordström, H. Bao, F. Löfman, H. Hult, A. Maki, and M. Sugiyama. Calibrated Surrogate Maximization of Dice. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 269-278. Springer, 2020.
[25] T. Nyholm, S. Svensson, S. Andersson, J. Jonsson, M. Sohlin, C. Gustafssonn, E. Kjellén, K. Söderström, P. Albertsson, L. Blomqvist, et al. MR and CT Data With Multiobserver Delineations of Organs in the Pelvic Area-Part of the Gold Atlas Project. Medical Physics, 45(3):1295- 2018.
[26] T. Popordanoska, J. Bertels, D. Vandermeulen, F. Maes, and M. B. Blaschko. On the Relationship Between Calibrated Predictors and Unbiased Volume Estimation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 678-688. Springer, 2021.
[27] A. J. Ratner, C. M. De Sa, S. Wu, D. Selsam, and C. Ré. Data Programming: Creating Large Training Sets, Quickly. Advances in neural information processing systems, 29, 2016.
(28) O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 234-241. Springer, 2015.
[29] A.-J. Rousseau, T. Becker, J. Bertels, M. B. Blaschko, and D. Valkenborg. Post Training Uncertainty Calibration of Deep Networks for Medical Image Segmentation. In 2021 IEEE 18th International Symposium Biomedical Imaging (ISBI), pages 1052-1056. IEEE, 2021.
[30] G. C. Sharp, R. Li, J. Wolfgang, G. Chen, M. Peroni, M. F. Spadea, S. Mori, J. Zhang. J. Shack- leford, and N. Kandasamy. Plastimatch: an open source software suite for radiotherapy image processing. In Proceedings of the XVI'th th International Conference on the use of Computers in Radiotherapy (ICCR), Amsterdam, Netherlands, 2010.
[31] J. L.. Silva and A. L. Oliveira. Using Soft Labels to Model Uncertainty in Medical Image Segmentation. arXiv preprint arXiv:2 /2622, 2021.
[32] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso. Generalised Dice Overlap as Deep Learning Loss Function for Highly Unbalanced Segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 240-248. Springer, 2017.
12

---

[33] A. A. Taha and A. Hanbury. Metrics for Evaluating 3D Medical Image Segmentation: Analysis, Selection, and Tool. BMC Medical Imaging, 15(1):1-28, 2015.
[34] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. N. Chiang, Z. Wu, and X. Ding. Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation. Medical Image Analysis, 63:101693, 2020.
[35] V. Vapnik. The Nature of Statistical Learning Theory. Springer science & business media, 1999.
[36] J. Yao, J. Wang, I. W. Tsang, Y. Zhang, J. Sun, C. Zhang, and R. Zhang. Deep Learning From Noisy Image Labels With Quality Embedding. IEEE Transactions on Image Processing. 18(4):1909-1922, 2018.
[37] S. Yu, M. Chen, E. Zhang, J. Wu, H. Yu, Z. Yang. L. Ma, X. Gu, and W. Lu. Robustness Study of Noisy Annotation in Deep Learning Based Medical Image Segmentation. Physics in Medicine & Biology, 65(17):155007,2222
[38] M.-. Zhao. N. Edakunni, A. Pocock, and G. Brown. Beyond Fano's Inequality: Bounds on the Optimal F-score, BER, and Cost-Sensitive Risk and Their Implications. The Journal of Machine Learning Research, 14(1):1033-1000, 2013.
[39] H. Zhu, J. Shi, and J. Wu. Pick-and-Learn. Automatic Quality Evaluation for Noisy-Labeled Image Segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 576-584. Springer, 2019.
13


Reviews:
[{"review_id": "tOni8NH9RQz", "reviewer": "Reviewer_RPBQ", "summary": "The authors provided a theoretical analysis of how the optimal segmentation to the metrics Accuracy and Dice can deviate regarding the segmentation volume. In addition, the upper and lower bounds on the volume of optimal segmentation are provided. Unfortunately, although the authors attempt to show the relevance of the theoretical observation in a real dataset, the experiments are not well-introduced and contain many flaws (listed in the weakness below), which makes the experimental results less convincing.   ", "questions": "See the weakness part.", "limitations": "Limitations are discussed except for the aspects listed in the weakness.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ A theoretical analysis of how the volume of optimal segmentation with respect to the performance of Accuracy and Dice could deviate is presented.\n\nWeakness:\n- The experiments are not well-introduced, and the results are not well-illustrated. For example, how exactly the marginal function m is computed by \"taking the pixel-wise average of the different segmentations\" is not clearly introduced? In addition, it is unknown what's the optimal segmentations in the provided dataset (with labels from multiple annotators).\n- The selected dataset is small, with only 18 patients, and the ROIs in each case are annotated with unknown medical practitioners. \n- It remains unclear how the level of noise in the annotation will affect the evaluation capability of metrics. A conclusion of \" the noise has a significant effect on the volume of optimal segmentations\" is not practically valuable for detailed applications.  \n- The authors also mentioned that the shape of the target is related to the effect of noise. Still, only a vague clue is given that spherical structures will be less affected. \n- FCN and U-Net-based segmentation methods have been around and popular since 2015, not \"during the last years,\" as stated at the beginning of the Introduction section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rW5zPABkoUb", "reviewer": "Reviewer_esZf", "summary": "This paper tackled the noisy labels of image segmentation in a theoretical manner. The study is performed on accuracy and dice metrics in medical image segmentation. ", "questions": "No questions.", "limitations": "Yes", "rating": 7, "confidence": 1, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "From a couple of years ago, dataset curation has been addressed in a wide range. However, most of the findings have not been shared with the community due to the private corporation policies. Thanks to the inconsistency of the  labels detected in the regular datasets used in the wide range of computer vision tasks, efficient solutions have been discovered and proposed. Therefore, this is one of the papers that can address not only segmentation but also edge detection, to name an example. Hence, the paper presents the first discoveries and solutions for the medical image segmentation; It also includes experiments on data from the Gold Atlas project.\n\nMinor errors:\n* Page 3 line 123 typo.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "_t6TFJEamNY", "reviewer": "Reviewer_5JSU", "summary": "In this paper, the authors present an analysis of the often used performance metrics for image segmentation (Accuracy, Dice) for noisy target labels. They performed a theoretical analysis of the problem and evaluation on a real-world data set. In their results, they provide a characterization of upper and lower bounds on the volume of optimal segmentation.", "questions": "None", "limitations": "Yes, the authors have addressed the limitations of their work.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written and very detailed. The paper provides a detailed investigation of how noise labels influence the volume of optimal segmentation. This provides interesting insides on how the quality of the manual segmentations influences the final segmentation performance in relation to the state-of-the-art evaluation metrics Dice and accuracy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZnEf_OYfNEI", "reviewer": "Reviewer_wPye", "summary": "It is a theoretical paper that studies two popular performance metrics in medical image segmentation, i.e., Accuracy and Dice while considering the target labels are noisy/random. The noisy labels scenario would be common in segmentation, for example, when different annotators give different boundaries to differ two classes. The authors theoretically prove several theorems to provide a form of optimal solutions to Accuracy and Dice metrics, and derive multiple properties of the optimal solutions, including 1) bounds on the volume of optimal segmentation, 2) relation between the volume of optimal solutions to two metrics. Further, the authors use an experiment to show that the optimal segmentations with respect to Accuracy and Dice deviate from the expected volume of the target.", "questions": "Suggestions:\n1. As mentioned in Strengths And Weaknesses, the reviewer suggests adding more \"informal\" descriptions to the notations defined and giving some concrete examples when introducing the theory.\n2.  As mentioned in Strengths And Weaknesses, the reviewer suggests adding more information about the significance of the work.\n\nQuestions:\n1. The theory developed in the paper seems to focus more on the performance evaluation step when using these metrics. Could the current theory help with improving the training steps in some ways?\n2. How do we make use of these properties to do better image segmentation? Maybe some examples could be provided.\n\n\n", "limitations": "1. I agree with the limitations mentioned by the authors, i.e., the amount of experiments included is limited in the paper, because to demonstrate the usefulness and generality of the theory, results from different medical image segmentation applications may be needed to draw a conclusion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n* Strengths: The theoretical analysis on Accuracy and Dice looks original to the reviewer.\n\nQuality:\n* Strengths: The main contributions of the paper, i.e., theorems are well-stated and written in a good format, and lemmas and theorems are rigorously proved.\n\nClarity:\n* Strengths: The main ideas of the paper are clearly presented.\n* Weaknesses: As the measure theory is widely used in the paper for strictness and generality, sometimes the readers would not be able to get the true meaning/insight of some notations corresponding to their use in the medical segmentation. It is highly recommended that examples (even in an informally speaking way) are given to describe the notations when they first appear in the paper. For example, add some description that $\\Omega$ is informally the continuous space for voxels from the data when $n=3$, and give some concrete examples for noisy segmentation $L$. Also, adding a non-measure-theory-style description for Accuracy and Dice would also help understand the paper. \n\nSignificance:\n* Weaknesses: Authors may need to highlight how their work could help with current medical image segmentation work, and how their insights from the theorems can help the people/scholars in this area, for example, selecting metrics etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept